Inference of root mean square value in terms of house prediction

The RMSE is a measure for how "wrong" a regression model's predictions are on average, and is mostly useful as a relative metric for determining which of a set of models is best. If you want an absolute baseline against which to compare any given model you can always compute the standard deviation of your target, which is the RMSE of a model which predicts the average value of $y$ for every observation.

I would also argue that for something like housing prices a more meaningful error measure is root mean square logarithmic error, which is the RMSE after taking the log of $y$ (or $y + 1$ if $y$ contains zeros) and the predicted values. The reason for this is that we're interested in errors as proportions of our target (an error of \$50,000 is much more serious for a \$100,000 house compared to a \$1,000,000 house), and taking logs is the way to adjust for this. Also be sure to focus on test or cross validation error, else you will always choose your most overfit model.


Mathematics: Can the result of a derivative for the Gradient Descent consist of only one value?

Gradient, g(x) = 0.4*x

At [x1 = 1], Gradient,
g(1) = 0.4
x2 = x1- step*gradient
=>x2 = 1 - 0.1*0.4
=>x2 = 0.96
At [x2 = 0.96], Gradient,
g(0.96) = 0.4*0.96 = 0.384
=> x3 = 0.96 - 0.1*0.384 = 0.9216

Continue following the same steps and will reach near the minima.


How to best visualize data when outliers lead to lack of contrasting colors for the rest of the plot?

With Wolfram Language you may use GeoRegionValuePlot with the ColorFunction option to customise the colour scaling.

With some "AdministrativeDivision" "Population" data for the Philippines.

popPH = EntityValue[
   Entity["Country", "Philippines"]["AdministrativeDivisions"],
   "Population",
   "EntityAssociation"];

First@Normal@popPH

Then GeoRegionValuePlot with and without a custom ColorFunction.

linear = GeoRegionValuePlot[popPH,
  ColorFunction -> ColorData["SolarColors"],
  PlotLabel -> "Linear ColorFunction",
  PlotLegends -> Placed[Automatic, Below]
  ];

log = GeoRegionValuePlot[popPH,
  ColorFunctionScaling -> False,
  ColorFunction -> (ColorData["SolarColors"][
      Rescale[Log@#, Log@QuantityMagnitude@MinMax[popPH]]] &),
  PlotLabel -> "Log ColorFunction",
  PlotLegends -> Placed[Automatic, Below]
  ];

Row[{linear, log}]

Hope this helps.


Find the time between two events by customer id

You can split it into two dataframe containing only one of P1 and P2 first.

df1 = df[df.index[df['Product'] == 'P1'].tolist()]
df2 = df[df.index[df['Product'] == 'P2'].tolist()]

And then marge df1 and df2 on Customer ID and Product

df1.merge(df2, 'inner', left_on=['Customer ID', 'Product'], right_on=['Customer ID', 'Product'], copy=False)


Python - Create many dummy variables from one text variable?

On the kaggle page, there is a kernel available which focuses on exactly your problem. This is the code from user JAGADEESHWARA VARA PRASAD:

# load your dataset into df
df = pd.read_csv("../input/train.csv")

# trimm { } symbols and split at ',' and trimm " from each word
l = [[word.strip('[" ]') for word in row[1:-1].split(',')]
     for row in list(df['amenities'])]

# form a set of distinct text
cols = set(word for row in l for word in row)
cols.remove('')

# create and fill new data frame
new_df = pd.DataFrame(columns=cols)
for row_idx in range(len(l)):
    for col in cols:
        new_df.loc[row_idx,col]=int(col in l[row_idx])

print(new_df)


Where can I find free spatio-temporal dataset for download?

If you have R and the spacetime package then you are only data(package="spacetime") away from a list of space-time data sets bundled with the package:

Data sets in package ‘spacetime’:

DE_NUTS1 (air)          Air quality data, rural background PM10 in
                        Germany, daily averages 1998-2009
fires                   Northern Los Angeles County Fires
rural (air)             Air quality data, rural background PM10 in
                        Germany, daily averages 1998-2009

then for example:

> data(fires)
> str(fires)
'data.frame':   313 obs. of  3 variables:
 $ Time: int  5863 5870 6017 6018 6034 6060 6176 6364 6366 6372 ...
 $ X   : num  63.9 64.3 64.1 64 64.4 ...
 $ Y   : num  19.4 20.1 19.7 19.8 20.3 ...


What are practical differences between kernel k-means and spectral clustering?

The differences are indeed not too large. There is a paper called
Kernel k-means, Spectral Clustering and Normalized Cuts by
Inderjit S. Dhillon, Yuqiang Guan, Brian Kulis from KDD 2004
that is discussing that relationship.
The authors show that spectral clustering of normalized cuts is a special case of weighted kernel k-means. The reason is that the "graph cut problem and weighted kernel k-means problem can both be written as trace maximization problems".
They write that

this has important implications: a) eigenvector-based algorithms, which can be computationally prohibitive, are not essential for minimizing normalized cuts, b) various techniques, such as local search and acceleration schemes, maybe used to improve the quality as well as speed of kernel k-means.

Further they combine ideas from both to improve the total results and

show that using eigenvectors to initialize kernel k-means gives better initial and final objective function values and better clustering results.

Regarding robustness in spectral clustering you might want to look at Robust Spectral Clustering for Noisy Data by Aleksandar Bojchevski, Yves Matkovic, Stephan Günnemann, published at the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2017: https://www.kdd.org/kdd2017/papers/view/robust-spectral-clustering-for-noisy-data

Inderjit S. Dhillon, Yuqiang Guan, Brian Kulis, Kernel k-means: spectral clustering and normalized cuts, KDD '04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 2004, Pages 551–556, https://doi.org/10.1145/1014052.1014118


Trying to understand the Fowlkes-Mallows Score

Yes, its a mistake.

FN: they should be together but you said they are not, you false negatively ordered them to a different class.
FP: they should not be together, but you said they are you false positevly ordered them together


Does it generate the set of the same image classes in the same order on each iteration?

No, a basic generator in a GAN is typically fed a small random vector as input; e.g. a column of 100 gaussian samples, with mean 0, standard deviation 1. It then must use this random "embedding" in the feed forward network to produce output that is in the target distribution of real data that the discriminator is trying to assess it against.

So a GAN typically learns a mapping from an arbitrary relatively low dimensional space onto a higher-dimensional target space. DCGANs do this with image data.

The class of an image can be used as conditioning data, if it is fed as additional input to the the generator and discriminator. The sequence of image classes can be anything you like, and the GAN is not required to work in a particular sequence (unless perhaps you are working with a RNN-based GAN and are trying to learn to produce sequences).

  What exactly does generator produce in DCGANs?

It produces an output image depending on the input. The output is drawn from a distribution based on the training data, that the discriminator would struggle to distinguish from real data. If fed a random input vector, generated in same way as training stages, then the idealised output should appear to have been drawn equally randomly from the imagined population of all possible training data.


Welcome to SO! It looks like you have a time series problem. Typically the first step when dealing with time series is to consider the difference. Let us define $f(t)$ as the fuel level at time $t$. You would want to calculate $diff_{\text{fuel}}(t) = f(t) - f(t - 1)$.

After this step you will likely see that the spikes that you identify as bad data are outliers. You could detect these by for example looking at all the data below and above your 2.5 percentile or 5 percentile. Typically this requires some careful analysis work to ensure that you do not delete too much. Once you have identified what the best workable percentile is you can could Winsorize your data.

Lastly, you would look at the data points in the bottom of your resulting distribution. These will likely be the points you identify as theft.

I hope this helps.


Validation loss and validation accuracy stay the same in NN model

You say it's a regression task, predicting valence and arousal values, although you use accuracy as a performance metric. This does not make much sense, so your accuracy graph doesn't really say much. MSE is a valid performance metric for regression tasks in general, so your loss graph is more descriptive of what is going on. The loss-graph most definitely displays the characteristic of over-fitting, so I would recommend you to add regularization to your model.
This can be done by for example incorporating dropout and L1/L2-regularisation.


The definition of a Language Model (LM) is a probability distribution over sequences of words.

The simple illustration of an LM is predicting the next word given the previous word(s).

For example, if I have a language model and some initial word(s):

I set my initial word to My
My model predicts there is a high probability that name appears after My.
By setting the initial words to  My name, my model predicts there is a high probability that is appears after My name.
So it's like: My -> My name -> My name is -> My name is Tom, and so on.

You can think of the autocompletion on your smartphone keyboard. In fact, LM is the heart of autocompletions.

So, LSTM-LM is simply using an LSTM (and softmax function) to predict the next word given your previous words.

By the way, Language Model is not limited to LSTM, other RNNs (GRU), or other structured models. In fact, you can also use feedforward networks with context/sliding/rolling window to predict the next word given your initial words.


How interpret keras training loss without compare with validation loss?

In Keras model it defines its own bias and weight..and eventually they converge closer to Y as loss decreases..the output graphs would change based on learning rate, batch size and epochs


How to calculate precision and recall in an unsupervised ranking method in order to be compared with a supervised one?

You first need to decide what your "label" is, and this label will apply to both your supervised and unsupervised models.  Obviously, in a supervised world, knowing your label make training the model and evaluating it easy, so I won't go into that because it seems like you have a good handle.

In the unsupervised world, you'll have to think really hard about the mapping of the unsupervised outputs to the label you've decided to use for evaluation.  A lot of this is going to come down to how you structure that unsupervised problem.  Like, if you're going to be using unsupervised clustering, then maybe you run the model, see what the clusters look like, and see if there's a way to map those clusters the label you're evaluating.

It sounds like you're using unsupervised ranking and extracting top key phrases after the unsupervised ranking.  In this case, you'd probably want to map the possible top key phrases to the labels you want to evaluate.  Then after ranking and key phrase extraction, you can get a mapped "prediction label" based on the key phrases for each prediction.  Of course, you'll likely have a messy collection of "prediction labels" because it's unlikely that each test prediction will output N-key phrases that perfectly match 1 label which means you'll have to come up with a way to round those maps (maybe you take the mode, maybe you round the average, it all depends on what you value most) to a singular prediction label.  From there, doing recall and precision and f1 is straight forward.

I will say though, that trying to bend an unsupervised model to evaluation like a supervised model is only useful if recall, precision, and f1 are the best way to evaluate your model's usefulness in its problem space.  This is very different from evaluating its effectiveness.

This question is: Does this models output fit the use case of the problem best?  When I implement this output to an end user, would they appreciate this model more than another model?  I'd encourage you to, on top of using classic evaluation scores to measure model effectiveness, also find a way to measure or understand model usefulness in the application that you have.


How can I parallelize GloVe reverse lookups in PyTorch?

It's usually not very efficient to approach these types of problems in pythonic ways, with list comprehensions and such. This whole process can be done with some matrix math, which will be substantially faster (and able to be computed on the GPU using PyTorch). Using torch.dist with default p=2 will compute the Euclidean distance between two tensors, which is defined as

$$
\text{euclidean distance} = d(\mathbf{v}, \mathbf{w}) = \sqrt{\sum_d \left(\mathbf{v}_d - \mathbf{w}_d\right)^2}
$$

You can do this efficiently in PyTorch for every word in your vocab by broadcasting your query word over the whole matrix of word vectors:

def closest(vec):
    dists = torch.sqrt(((glove.vectors - vec) ** 2).sum(dim=1))
    return dists.argmin() # or glove.itos[dists.argmin()] if you want a string output

However, usually people use the cosine similarity to find the closest word vectors rather than the Euclidean distance. This is the cosine of the angle between the vectors, and can be calculated as

$$
\text{Cosine similarity} = \frac{\mathbf{v}\cdot \mathbf{w}}{\|\mathbf{v}\| \|\mathbf{w}\|}
$$

which can similarly be implemented in PyTorch as the following:

glove_lengths = torch.sqrt((glove.vectors ** 2).sum(dim=1))

def closest_cosine(vec):
    numerator = (glove.vectors * vec).sum(dim=1)
    denominator = glove_lengths * torch.sqrt((vec ** 2).sum())
    similarities = numerator / denominator
    return glove.itos[similarities.argmax()]

Note that we moved the $\|\mathbf{v}\|$ part of the equation outside the function here, so we don't have to recompute the lengths of each glove vector every time we want to run it.


The problem was that outputs and y_train_torch had different shapes.


Since I cannot comment because I don't have enough reputation, I will post this as an answer.

If your goal is to "find a point where the increase of promotion don't increase profit or new_users", I won't do a simple regression, since the regression will tell you that if you do more promotions, you will always inscrease the profits. I would say that, in reality, the relationship between promotions and profits or new users is not linear. Because the number of new users is limited and the promotions are not.

A better model is to say that there is a optimal promotion that will give you the best increase of profits and new users.

(if you have a real business to optimise, I would introduce the Customer Lifetime Value of new users. Because generally, the new users you get when doing huge promotions will not come back...)


What pre-processing of the image is needed before feeding it into the convolutional neural network?

If you have a trained network that accepts input of 100 * 100 you can either scale your image accordingly or slice it into smaller parts.

Scaling the image

Scaling your image down is really straight forward. It will lower the information and might distort it too if your aspect ratio doesn't match. In your case it is a little off, but most networks are trained with a little distortion.

You will have to ask yourself if the details you are interested in are still discernable, and whether they (still) match the examples the network is trained on.

Slicing smaller parts

If you are interested in features rather than the larger scene, you could slice your image into bits of 100 * 100. I recommend you use a sliding window, wherein you tile the subsamples as if they were shingles. That way you avoid that a feature is only availble in bit's and doenst get recognized.


Semantic analysis score as input to LSTM model for improving stock price prediction accuracy

It is possible. Your problem will then become multivariate, where on top of the opening price for a particular day, you would also add a sentiment score for that day.
LSTM can handle multiple inputs natively, so there shouldn't be much for you to change.


Accuracy differs between MATLAB and scikit-learn for a decision tree

It is hard to make a direct comparison between a white box implementation (scikit-learn) and a black box implementation (MATLAB).

One guess they are using different algorithms. scikit-learn uses an optimized version of the CART algorithm. Maybe MATLAB uses  ID3, C4.5, or something else. Another guess two implementations are using different hyperparameters (e.g., different splitting criteria, max depth, minimum node size, ...).

Since decision trees are white-box models, you can examine their internal structure. Plot both trained trees. See how they each are making the splits and how many splits are being made.


Use a single Gaussian to represent a mixture of Gaussians

One option is to re-estimate a single Gaussian distribution directly from the empirical data.


These are calculated based on millions of images of ImageNet.

Ref - SO
Ref - MachinelearningMastery


There are many other options instead of doing one-hot encoding. You can do target encoding, count encoding and many others. Here is a nice library for encoding categorical variables. Explore all the operations in here. Here's the link.
Target encoding would be my bet.


How do I fix space allocation error on a machine with plenty of space?

It doesn't matter that your instance has more than 57.8GB. R is asking for another 57.8GB on top of whatever it is already using. Not to mention any operating system overhead.

Shrink your dataset and see if it what you are doing works at small scale before trying to do it at big scale. Maybe it will work on 9,917 rows and that's good enough. Or it might fail on 500 rows in which case you need to rethink what you are doing.


How can I create a VM instance with GPUs on Google Cloud Platform?


What is the feedforward network in a transformer trained on?

Let's take the common translation task which transformers can be used for as an example: If you would like to translate English to German one example of your training data could be

("the cat is black", "die Katze ist schwarz").

In this case your target is simply the German sentence "die Katze ist schwarz" (which is of course not processed as a string but using embeddings incl. positional information). This is what you calculate your loss on, run backprop on, and derive the gradients as well as weight updates from.

Accordingly, you can think of the light blue feed forward layers of a transformer

as a hidden layer in regular feed forward network. Just as for a regular hidden layer its parameters are updated by running backprop based on transformer $loss(output,target)$ with target being the translated sentence.


Keras Multiple “Softmax” in last layer possible?

I would use the functional interface.

Something like this:

from keras.layers import Activation, Input, Dense
from keras.models import Model
from keras.layers.merge import Concatenate

input_ = Input(shape=input_shape)

x = input_
x1 = Dense(4, x)
x2 = Dense(4, x)
x3 = Dense(4, x)
x1 = Activation('softmax')(x1)
x2 = Activation('softmax')(x2)
x3 = Activation('softmax')(x3)
x = Concatenate([x1, x2, x3])

model = Model(inputs=input_, outputs=x)


Regression and Machine learning models for panel data

Sales data is Time-series data. You should use the method of Time-series forecasting.

Below is a list of useful links to start with -

Analytics Vidhya
Yashuseth blog
Kaggle blog

This is a good book to grasp the concepts -
Forecasting: Principles and Practice

Also, do check this similar question.
Stack Exchange


Pros and Cons of Positive Unlabeled learning?

I don't think it's possible to know for sure if PU learning would work in your setting or not. It's certainly relevant to cases like the one you describe, so it would be worth trying. But there are other valid options, and even within PU learning there are different approaches to choose from (you might be interested in this question).
In my opinion the alternative you propose with regression makes some sense and it might work, but it's not very "clean" in terms of design: first the choice of 0.1 is arbitrary (why not 0.2 or 0.05 or ...?). Second, it means that you're telling the regression algorithm that "this instance should have probability 0.1" for many negative instances and also for a few negative instances: this is different than saying "I don't know the target value for this instance".
Note that you could also consider one class classification in this kind of setting, (as part of PU learning or not).


About training and cross validation on a time series problem

You might want to check out my answer to a related question here.

If you want to cross-validate time series data, I would suggest creating some sort of sliding window on which you train your model and predict the next month following that window.

For example, you could train your model on February data and then predict the data of the first week of March. Then slide your window to include data from the second week of February through the first week of March and then predict the data of the second week in March. The length of the windows that you use to train and test on are parameters that you will need to play around with yourself to see what gets you the best results.

Once you feel like you are getting good results from your cross-validation, I would try training a new model on all of the data you have and see how it performs on the new data coming in. Depending on the data and model you are using, this may provide better results having trained on more data. Of course, this depends on the problem you are trying to solve, the nature of your data and your choice of model.


ValueError:invalid literal for int() with base 10: 'No''

It seems that your 'Result' column is of type int and you are trying to set some rows to str. Stick to int or category (docs).


Perform bootstrapping of an ordinary linear regression model, using B=100 bootstrap resamples of my dataset, and getting RMSE

The following code will run Ordinary linear regression and ridge regression on B=100 different samples of Boston data set and calculate the RMSE on the test set for all 100 different test sets and store the RMSE for ordinary linear regression in olr_rmse_all  and for ridge in ridge_rmse_all.

You can do any type of analysis on RMSE vectors.

I calculated mean and standard deviation of the RMSE of B=100 for Ordinary Linear regression and Ridge regression

library(MASS)
library (ridge)

data1 = Boston

RMSE = function(actual, predicted){
  sqrt(mean((actual - predicted)^2))
}

test_perc = 0.3
B = 100

olr_rmse_all = c()
ridge_rmse_all = c()

for (i in c(1:B)){

  cat("running for sample = ", i, '\n')
  train_rows = sample(nrow(data), (1-test_perc)*nrow(data))
  test_rows = setdiff(c(1:nrow(data)) , train_row)

  Boston.train <- Boston[train_rows, ]
  Boston.test <- Boston[test_rows, ]

  olr_model  <- lm(medv ~ ., data = Boston.train)
  ridge_model <- linearRidge(medv ~ ., data = Boston.train)

  test_predicted_olr = predict(olr_model, newdata = Boston.test)
  test_predicted_ridge = predict(ridge_model, newdata = Boston.test)

  test_actual = Boston.test$medv

  rmse_test_olr = RMSE(test_actual, test_predicted_olr)
  rmse_test_ridge = RMSE(test_actual, test_predicted_ridge)

  cat(paste("olr_rmse: " , rmse_test_olr,  "ridge_rmse: ", rmse_test_ridge, sep = " "), '\n')

  olr_rmse_all = c(olr_rmse_all, rmse_test_olr)
  ridge_rmse_all = c(ridge_rmse_all, rmse_test_ridge)

}

#-- Mean of RMSE----

olr_rmse_all_mean = mean(olr_rmse_all)
ridge_rmse_all_mean = mean(ridge_rmse_all)

#-- Standard Deviation of RMSE----

olr_rmse_all_sd = sd(olr_rmse_all)
ridge_rmse_all_sd = sd(ridge_rmse_all)

Hope, this answer your question.

There is package for bootstrapping in R, name "boot", you can also use that. I do it without package for more understanding.

reference: https://www.statmethods.net/advstats/bootstrapping.html


Is there room to improve the model? if the train data accuracy is 99.8% but test data accuracy is 90%?

Your model is overfit: it learns patterns on the training data which are specific to the training data, hence the big difference in performance on the test data. This is bad, because it means that your model wrongly assumes that some patterns are good indication for the class even though it's just noise.

It's possible that by solving the overfitting problem the performance would improve. However nobody can give you any advice without knowing anything about your actual problem and data. Btw you don't explain why you use accuracy as evaluation measure, it might or might not be optimal for your problem.


How to apply a different Loss function to one specific Label?

Using mean absolute error might damage your output. You must do some regularization. One thing you can do would be to assign custom weights for each class metric.

Though in general, I would strongly advise against doing that. If this class is so important do separate model in one-vs-all fashion. Be aware you'll be overfitted.


"R" write/export dataframe preserving the class information

The best way to achieve this would be to save the data as an R data file using either save() or saveRDS():
# option 1
save(df, file="data.Rdata")
load("data.Rdata")

# option 2
saveRDS(df, file="data2.Rds")
df <- readRDS("data2.Rds")


Keras, now fully merged with the new TensorFlow 2.0, allows you to call a long list of pre-trained models. If you want to create an Inception V3, you do:

from tensorflow.keras.applications import InceptionV3

That InceptionV3 you just imported is not a model itself, it's a class. You now need to instantiate an InceptionV3 object, with:

my_model = InceptionV3()

at this point, my_model is a Keras Sequential() model with the architecture and trained weights of Inception V3, that you can re-train, freeze, save, and load as you need.

Check also the full list of models available in the module, it's great.


when speed is not a major concern, what is the best ML algorithm for high dimensional data?

There is no one-size-fits-all

Also, data dimensionality is not the same everywhere. Text data, which is inherently sparse, has a very different intrinsic dimensionality than e.g. random Gaussians.

For text data, linear SVMs are known to work very well.

RBF kernels do not work well with high-dimensional data, because they are distance-based at the core, and choosing the sigma parameter becomes next to impossible.

If you can "fold" dimensions, you also get very different behavior. Im image recognition, you usually have thousands of pixels. However, you never look at all of them at once. Instead, you use convolutional kernels that move over the data space, and they may have only say 32x32 pixels. That is still 1024 dimensions, but not millions anymore.


How to cope with new/unseen targets classes in incremental learning algorithms

Scikit-multiflow is a package designed for incremental learning for multi-output/multi-label problems.


Do smaller neural nets always converge faster than larger ones?

For most cases, probably. For all cases, no. Especially if you are training on small data with very aggressive regularization in place, you may need a very long time until the desired performance level is achieved.
For instance, for some popular text generation networks called Transformers trained on small datasets, it is necessary to use very aggressive regularization techniques and train during a very very large number of iterations (see this Twitter thread where they describe how to train a Transformer model on the PTB and Wikitext-103 datasets).


Feature Engineering of mixed data type column

You have several choices. Among them try these:

Discretize your target and use chi-square association measure or Cramer's V to see the association between variables.
The other way around i.e. convert availability labels to numerics and simply try different association analysis methods like Correlation (Pearson, Spearman, Kendall) or Mutual Information.

PS: Please note that Spearman or Kendall Rank Correlations can also be applied to your categorical variables as they are ordered (dates).


How to avoid covariate shift in python and distribute classes in each train and test phase?

You need to call train_test_split(X, y, test_size=0.33, stratify=y) assuming that y contains your class labels. Alternatively, you can call StratifiedShuffleSplit directly.


Is web analytics similar to data science?

We can see Web Analytics like an application of Data Science concepts. Almost a field of it. Usually it's very close to Business Intelligence for marketing aims and the main goal is to analyse users information.
So it's about marketing.


Clustering will be much too expensive for your purpose (most are O(n^2), and the good ones like HAC may even be O(n^3) - you won't be able to run them on 300k instances). Also beware of the prerequisites of the various algorithms - they may not be applicable to your data, or the results may be as bad as random! Also, clustering algorithms may behave rather unpredictable, and will profuce unbalanced data sets. It would not be unusual if 99% of your data ends up in the same cluster, while other clusters are almost empty (or even empty!)

I would consider first to remove all duplicates and maybe near-duplicate records (ignoring identifiers, of course). These algorithms are much more scalable and tolerate more data types. That may already be enough to achieve your goal of data reduction.


How to mathematically define the architecture of neural network model? And the function space associated with it?

You can use directed graphs. Each neuron is a node in the graph. Each neuron has list of its neighbours and activation function. You can add remove nodes to this graph.

If you just want to search for simple architectures you can use two lists. One for number of neurons in layer and one for activation function of neurons in that layer.

So this is a neural network with 3 layers. First layer has 5 neurons and echo of them have Relu activation function.

  [5,3,2] ['Relu', 'sigmoid', 'Relu']

Hope that helps.


can I use public pretrained word2vec, and continue train it for domain specific text?

fastText pretrained models should give you a boost to classification task.

gensim on the other hand has possibility to load model and train it with new texts but if you need to account for new words, you need to use

  build_vocab(update=True...)

So you can take with fastText pretrained embeddings to gensim and update with your texts.


What are "VGG54" and "VGG22" derived from the VGG19 CNN?

Reading the article, it seems like they define VGG54 as the loss calculated from the euclidean distance between the $\phi_{5,4}$ feature maps derived from both the high and low resolution images using the VGG19 network. Where $\phi_{i,j}$ is defined as "the feature map obtained by the j-th convolution (after activation) and before the i-th max-pooling layer within the VGG19 network".


receipt-parser is receipt parser package in Python built on tesseract.


The theory of backpropagation should work with complex numbers, it doesn't make any assumptions about all numbers being real. However, most NN libraries would not be able to calculate with them. Your weights and biases would also need to be complex. I have no idea if you could reasonably expect convergence using gradient descent - many activation functions have periodic output and numerically unstable regions when calculating with complex numbers, which makes me think it would be risky.

The simplest thing to do would be to split each complex number up into its components at the input, representing as a 2-dimensional vector of real numbers - either using cartesian or polar approach. Neural networks should cope just fine with splitting features up like this.


How to train a classification algorithm with normalized data set using scikit-learn python

I would suggest you to scale your data using standard scaler and do it before you split it into X and Y here in your case. Why ? Please check this answer on stats sc.
Also, keep the target variable (Front in your case) as it is. So, according to me, the right choice looks like this, however you can try and experiment with min max scaler too:

from sklearn.preprocessing import StandardScaler
import pandas as pd
df = pd.read_csv(r"D:\path\data.csv", sep=",")
X = df[["Title words", "Hour posted", "Author age", "Is Text", "Subreddit age"]]
Y = df['Front']
scaler = StandardScaler()
scaled_X = scaler.fit_transform(X)
scaled_X = pd.DataFrame(scaled_X, columns = X.columns)
X_train, X_test, y_train, y_test = train_test_split(scaled_X, Y, test_size=0.2, random_state=42)

To start, you can try with logistic regression and move on to svm, or maybe neural networks, if your data is big enough.

from sklearn.linear_model import LogisticRegression
lr = LogisticRegression(class_weight='balanced', C = 0).fit(X_train, y_train)
y_pred = lr.predict(X_test)
precision, recall, fscore, support = score(y_test, y_pred, pos_label= 1, average='binary')

Now, you can choose a good performance metric for the same, looking at the distribution of the class label, whether they are balanced/imbalanced etc.


What is a manifold for Unsupervised Learning?

Depends who you ask but generally mannifold is just some  structure in a (high) dimensional space having finite dimension : a line, a curve, a plane, a rock, a sphere, a ball, a cylinder, a torus, a "blob"... something like this :

If you ask a mathematician
they would say that it is a general term describing "a curve" (dimension 1) or "surface" (dimension 2), or a 3D object (dimension 3)... for any possible finite dimension $n$. A one dimensional manifold is simply a curve (line, circle...). A two dimensional manifold is simply a surface (plane, sphere, torus, cylinder...). A three dimensional manifold is a "full object" (ball, full cube, the 3D space around us...).

TO ANSWER YOUR QUESTION what he is referring is that information in this high-dimensional space can be compressed and perserved in some smaller space. (remember images when quantified live in some space that contains information about them)


Accuracy changes very little when modifying parameters in a Keras model

When you can't improve the model any more, improve the data.

are there any unisex names in your dataframe? Those will obviously hurt your accuracy. Either toss them out for now or do multi label binary classification.
Visualize your data: are the labels clearly separated? Which are the top losses?
I just saw that the data has the popularity of a name. I think it would make sense to try adding this popularity as a weight of each datapoint when training the model. Intuitively, you want your model to understand what does it mean for a name to be feminine. And "Anna" (score=2604) is much more representative of this meaning than "Jewel" (score=5), don't you think?

PS: I just saw that "Joe" is included in the list of feminine names.


Loading a Keyword and Evaluating the Information

You ask for quite a bit. The webpage is dynamic, so you would need to run an automated browser, navigate to the search page, enter the service number, press "search" and see if there is any item indicating a digital copy.

One way to do this is in Python, a free programming language.
Find Python here and a "howto install" tutorial here.

Make sure you install Python with pip, since it lets you install additional free modules very easily. To install additional packages you simply need to type cmd in the search field in windows. This shows you the command line interface (a black window). When you have Python with pip installed, you can simply type pip install XYZ (press enter), to install the package XYZ (where XYZ is the name of the package).

You also need to download the Selenium Chrome driver (the version must match the version number of you current Chrome browser). This free software allows Python to automatically use Chrome to browser pages. The Chrome driver is a single file, which must be copied to the place where you have Python on your computer.

The script below does exactly what you want. It searches a list of "numbers" from an excel file one by one on the NAA page. For each request, the script looks for digital documents and "notes" the number of found digital documents. Finally, the results are saved to an excel file. My dummy result would look like:

     servnr  digicopy
0      1234         2
1      6516        15
2     51651         2
3  51651651         0

Maybe you have a friend or someone in your family who can help you install Python. Usually this is a little daunting for people with no programming experiance. If you elaborate my script below, you could even download the digital documents or extract information (e.g. by using OCR software like tesseract).

Basic code for the webcrawler:

# Import required additional packages (to be installed via pip first)
from bs4 import BeautifulSoup
from urllib.request import urlopen
import re, time, datetime, json, requests
import pandas as pd
# Chrome Webdriver https://chromedriver.chromium.org/downloads
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

# When you see something like "ModuleNotFoundError: No module named 'openpyxl'"
# You need to install the pip package, e.g. pip install openpyxl
# Install pip packages in the console window (type cmd in the search field in Windows bottom left, open it, type pip...)

# Load Excel containing service numbers
# Change the path to the file is needed
df = pd.read_excel('D:/service_numbers.xlsx')

# Start chrome driver https://chromedriver.chromium.org/downloads
options = webdriver.ChromeOptions()
#options.add_argument('headless') # in background option
options.add_argument('log-level=3')
browser = webdriver.Chrome(chrome_options=options)

# Set up empty data frame to store results
results = pd.DataFrame([])

# Loop over each number (aka row) in the excel file (column name 'snr')
for s in df['snr']:
    browser.get("https://www.naa.gov.au/")
    time.sleep(0.5)
    # Go to record search
    browser.find_element_by_xpath("//*[@id=\"block-naa-mainpagecontent\"]/div/div[4]/div[2]/div/div/div/div/a").click()
    time.sleep(0.5)

    # Find searchfield and send number "s", press enter
    print("Loading number %s" %s)
    searchfield = browser.find_element_by_xpath("//*[@id=\"ContentPlaceHolderSNR_tbxKeyword\"]") # find field
    searchfield.send_keys(s) # enter searchterm (the service number)
    searchfield.send_keys('\n') # press enter
    time.sleep(0.5)

    # Load the content of the current page (I do not iterate over all resultrs but only look at the first page here)
    soup = BeautifulSoup(browser.page_source, "html.parser")
    table = soup.find("table", id="ContentPlaceHolderSNR_tblItemDetails")
    # You can also load/print the entire table, you can even go directly to the digital contant (if present)
    #print(table.text)

    # See how many "images" indicating digital content are there
    imgs = soup.find_all('img', {'class':'digital_copy'})
    try:
        numberofdigiitems = len(imgs)
    except:
        numberofdigiitems = 0
    print("There are %s digital copys" %numberofdigiitems)

    # Append result to data frame
    results = results.append(pd.DataFrame({'servnr': s, 'digicopy': numberofdigiitems}, index=[0]), ignore_index=True)

print(results.head())

# Save to excel
results.to_excel("D:/myresults.xlsx", index=False)


Difference between segmentation and effect of explanatory variables

If you choose to build a model where one of the categorical features such as a gender plays a big role in the outcome, then the overall model results will give a large weight to the gender, and other features will have a smaller weight. While your model will be applicable for the unsegmented population, the predictive power might not be very good.

Now, suppose that you wanted to identify if the weights of the other features differed for males vs. females and if the segmented models had better predictive power, then by segmenting the population and creating models for each gender, you could explore changes in the magnitude or direction of the weights of each feature & evaluate the goodness of fit of your models to see if there is an improvement in the segmented population models vs. the overall model.

If performing logistic regression, you could scale the coefficients or Z-values returned in the model parameters for each model and check if the relative magnitude & direction of the overall vs. the segmented models gave you similar or different results. You could also look at the confusion matrix and use various accuracy measures to assess if the segmented models performed better, worse or the same as the overall model. In this way, you could determine whether for the outcome independent gender-based models are more appropriate for your question than a single overall model.

But, this leads to the question of if I have "n" features, should I create multiple models by segmenting the "x" categories in each of the "n" features?  This really depends on the data question being asked. In some cases, it is valuable to run multiple models where each feature is dropped one-by-one to assess feature importance. In some cases, just evaluating the weight & relevance of each feature from the overall model is sufficient. Going beyond, one could look at optimizing models by choosing which of all "n" features should be segmented to yield the model with the best predictive power. This paper from FICO describes one methodology. This article details an approach and comparison of various models.


How to handle large number of features in machine learning?

There are 4 ways I know in Python. In the following I copied the code I wrote for regression purposes. Classification would be very similar :

First: SelectKBest:

from sklearn.feature_selection import SelectKBest, f_regression
train_data = train_data.apply(pd.to_numeric).astype('float32')

kb = SelectKBest(score_func=f_regression, k=70)
kb.fit(train_data.loc[:, train_data.columns != 'SalePriceLog'], train_data.SalePriceLog)
indices = np.argsort(kb.scores_)[::-1]
selected_features = []
for i in range(5):
  selected_features.append(train_data.columns[indices[i]])
plt.figure()
plt.bar(selected_features, kb.scores_[indices[range(5)]], color='r', align='center')
plt.xticks(rotation=45)

results:

Second: RFE

 from sklearn.linear_model import LogisticRegression, LinearRegression
    from sklearn.feature_selection import RFE
    model = LinearRegression()
    rfe = RFE(model, 10)
    fit_rfe = rfe.fit(train_data.loc[:, train_data.columns != 'SalePriceLog'], train_data.SalePriceLog)
indices_rfe = np.argsort(fit_rfe.ranking_)
selected_features_rfe = []
for i in range(10):
    selected_features_rfe.append(train_data.columns[indices_rfe[i]])
selected_features_rfe
plt.figure()
plt.bar(selected_features_rfe, fit_rfe.ranking_[indices[range(10)]], color='r', align='center')
plt.xticks(rotation=45)

results:

Third: PCA

from sklearn.decomposition import PCA
# pca = PCA(n_components=5)
pca = PCA(0.999)
fit = pca.fit(train_data.loc[:, train_data.columns != 'SalePriceLog'])

Fourth: ExtraTrees

from sklearn.ensemble import ExtraTreesRegressor

model_extra_tree = ExtraTreesRegressor()
model_extra_tree.fit(train_data.loc[:, train_data.columns != 'SalePriceLog'], train_data.SalePriceLog)
indices_extra_tree = np.argsort(model_extra_tree.feature_importances_)[::-1]
selected_feature_extra_tree = []
for i in range(10):
    selected_feature_extra_tree.append(train_data.columns[indices_extra_tree[i]])
plt.figure
plt.bar(selected_feature_extra_tree, model_extra_tree.feature_importances_[indices_extra_tree[range(10)]])
plt.xticks(rotation=45)

results:


'RandomForestClassifier' object has no attribute 'oob_score_ in python

This kaggle guide explains Random Forest.

Setting warm_start to True might give you a solution to your problem. The default value is False.


Distance measure calculation addresses for record linking

Is the problem that you can resolve a new address B to a location reliably, but, don't know if it's the same address as an existing address A? then it seems straightforward to perform the expensive check for near-matches on only addresses that also resolve to a nearby location, and consider them in order of distance. The simple distance calculation shouldn't be expensive.

Or are you saying that you can't resolve B to a location by itself, and have to resolve it by finding a fuzzy match with some A, but you don't know where A might be? Then you need some cheap heuristic match to compute over all records. What about simply the distribution of characters/digits in every string? easy to compute, small, and easy to compare. Presumably near-matches have nearly the same letters and digits, so might correlate well with actual fuzzy matches.

Edited to add:

You could try to build a classifier that can tell when two different strings represent the same address. That's not your ultimate goal, I know. But along the way you might be able to learn a useful intermediate vector representation of the strings that is relevant for determining same/not same. And then use that intermediate representation.

In theory that's what a neural network could do. Inputs are two strings; they share weights/layers that transform to an intermediate rep; then the top of the network learns same/not same.

"Simple as that" but in practice you might find that learning this way takes such a huge network to do well that it's infeasible.

Maybe someone knows an equivalent idea that doesn't need the brute force of deep learning.


Finding out why your model is doing better?

First, it makes sense that adding features will improve your performance, just make sure you do evaluation carefully and not overuse the same validation dataset (and if yes try to re evaluate it on an unseen independent different test set) to ensure you are not overfitted.
After that, you can use Shapley values in its aggregate look to see which features (or specific values in your encoding) impacted the model decision making


How to label overlapping objects for deep learning model training

There is no common practice in labeling the bounding boxes. It is always problem dependent. For example, if you want to count the chickens then you should also label the whole chicken as one instance of a chicken.
If you simply what to detect if there is a chicken in the picture you should label the unoccluded part.

You have to think about your problem. What is the goal of the algorithm? Could a human do the task without imagining where the rest of the object is? You should also consider the pixel imbalance for your problem. In general, the first method is a harder task than the second method because even humans have problems in labeling the bounding box for occluded objects. Hence, you will have a lot of variance due to this factor. If you label only what you see the bounding box labeling will be more reliable. As far as I know, the PASCAL Visual Object Classes data set which was used in the YOLO publication did only label what you can see and not what is occluded.

BTW I hope your task aims to improve the live quality of the chickens. It would be a shame if machine learning would be used to harm them.


What is a better solution for text classification than use of perplexity

You can evaluate pre-built language models. Advantage is :

These models are usually trained on very large text corpora (Such a a snapshot of  text from billions of Web pages).
It requires very little development effort for testing models for a given classification problem.

Examples :

https://spacy.io/usage/examples

https://uber.github.io/ludwig/examples/


Euclidean distance between time series in Python

This only makes sense if you look at the time series as vectors with same length. In that case, there is no need for a package. Just use, for example,
numpy.linalg.norm(a-b).


Can we use sentence transformers to embed sentences without labels?

The link you provided of Siamese Bert is an instance of a Bert or Roberta finetuned on STS or NLI data. Which can have the format sentence 1 is similar 3 out of 5 to sentence 2 (STS). Hence, is supervised, it does not fit your purpose.
Nonetheless, do not despair, there are some that do not require training, although may not perform as good as the supervised one. The below use word embeddings which you can train on your data corpora to generate sentence embeddings:

Word Mower's Distance
Sentence Embedding S3E

Or by feeding just sentences line by line:

DeCLUTR
Sent2Vec

P.S. I have not tried all of the solutions, to my knowledge I suggest these, cause either they are quite known or are quite recent.


How to make a dataframe with lists or vectors as its elements


How to use the finite difference to compute gradient for very complex loss function

You may consider using a score function estimator (also known as REINFORCE), which defines an estimator of the gradient of a scoring function that does not need to be differentiable. This can be achieved thanks to the log derivative trick.

You may find its mathematical foundations in the original REINFORCE article by Williams or in Ian Goodfellow's seminal book on Deep Learning (section 20.9.1 Back-Propagating through Discrete Stochastic Operations).


Pretrained features return worse results for sklearn classifier/pipeline

Normalization

Long story short normalization was the problem in my case. Because I haven't shared the transformer pipelines code in FeatureUnion, I've missed to tell that each one of them ends with normalization. So that's what the biggest difference.

In case with pretrained -> normalization is done over the whole dataset and nothing more
In case of pipeline -> normalization is done 10 times (twice per each fold -> once for train data, once for test data)

And that's how I do get different values => such difference in results.


What is the best file format to store an uncompressed 2D matrix?

I would go with .csv as it's universally accepted and can be read in different programming languages easily. Moreover you can simply open it with an office software. If you are using your matrix only in Python I also recommend Pickle library which writes your matrix in a .p format and can be easily read in Python with a simple load function.


Is the size of the output vector of all machine learning algorithms the same?

No, the output vector depends on the architecture that you define, which should itself be dependent on the data that you have.

Can't an ML algorithm predict only one value as output?

Absolutely it can. Think for instance of regression problems. I cannot think of any algorithms that don't allow you to have an arbitrary sized output vector.

Given the LSTM is many-to-one type how does it produce two values as output(x,y)?

I don't think you are using "many-to-one" correctly in this case. It seems that you have a single input which happens to be a time-series and you want to predict a vector containing two values, so it is in fact a "one-to-one" problem.

A network can produce as many outputs as you want. It all depends on the architecture.

Could an MLP also do that?

Yes, absolutely. RNNs are essentially a specific way to use MLPs to achieve a specific purpose (in this case, use the positional information). SO anything an LSTM can do could be replicated using MLPs.


Replace a value in a column if that value appears only once

It is easier if you store the value count separately to avoid redoing it inside the apply loop. You can do it like this:

food_count = data["food_group"].value_counts()
data["food_group"] = data["food_group"].apply(lambda x: "other" if food_count[x]==1 else x)


Any case studies using Bayesian Networks for system design trades?

As I can see you're searching for a case study. Learning from data is useful because you can learn both the conditional probability distributions (CPDs) and the structure of the network. The best resource I've read is a freely-available Microsoft tech report.

Have a look at the paper, maybe it can help you upto some extend. I'll suggest you to start with Chapter 7 of it.

Hope it helps. Cheers!


handling missing values for LightGBM model

The default behavior allows the missing values to be sent down either branch of a split. Replacing with a negative value that is less than all your data forces the (originally) missing values to take the left branch, and so your model has (slightly) less capacity.  That may be a good or a bad thing, depending on where you land on the bias-variance curve.
So, I think the best answer here is "it depends on your data."  If your missing values actually behave like lesser values, then encoding them as large negative numbers enforces that, reducing capacity in a probably-beneficial way.  But without digging into exploratory data analysis, or complex imputation methods (MICE), I'd personally stick to the default behavior.


Neither, there is not enough discriminatory information in data (yet)

Dont squeeze the data until it tells you the truth. You can change the metric (malahobian distance for example) and the algo but you cant expect it to show miracles.

Using elbow method, as you increase the number of clusters it will always become more homogenous. You dont have a "kink" indicating optimal clustering number.

And with Silhouete (be careful those are averaged Silhoute score per cluster, in other words for k=4 you have 4 scores indicating wether points are lying inside/border/should- be-other-cluster that are averaged) you get that all of the points (in average) lie on the border of the clusters, without clear distinction of the clusters (thats what 0 means).

Advice Find better quantitative representation of data, new feature, reduce noise etc...


sklearn NearestNeighbors returning misaligned results

You can refactor your code to make this issue easier to investigate.

Something like this:

nbrs = NearestNeighbors(n_neighbors=1, n_jobs=-1).fit(tfidf)
orgs = list(set(names['VariationName'].values))

queryTFIDF_ = vectorizer.transform(orgs)
distances, indices = nbrs.kneighbors(queryTFIDF_)

matches = pd.DataFrame(columns=['Match confidence (lower is better)', 'Matched name', 'Original name'])

for distance, index, org in zip(distances, indices, orgs):
    match_confidence = round(distance, 2)
    matched_name = clean_org_names.values[index][0][0]
    matches.append(match_confidence, matched_name, org)


Alternative Hunspell dictionary for stemming

I can't comment on your question (my reputation isn't high enough), but I've got a few clarifying questions to ask: Why do you want to stem words?  Text classification?  Sentiment analysis?  Something else?

I've used both the 'tm' package and the 'RTextTools' package for stemming when doing text classification.  Both have some built-in functions for stemming.

Whether they would be useful to you or not depends on why you want to stem words ...

FWIW, here are some links related to those packages:

tm: https://cran.r-project.org/web/packages/tm/index.html
RTextTools: http://www.rtexttools.com/documentation.html


Intent Classification in Question Answering

I don't know about the granularity of the intents. If they are just Person/Loc/Official, why not use question words(where,who,what etc.) to identify the intent? For example, "where" corresponds to "Loc", "Who" corresponds to "Person" etc. You can also use this as a feature in you classifier.
Hope it helps!!


EfficientNet function composition or Hadamard

It probably represents repeated function composition.
Key reasons:

The authors' wording before this equation is: "a list of composed layers".
A Hadamard product could not be applied to layers of different sizes, whereas function composition of course can.
Yes, EfficientNet has skip connections, but I think they try to validate function composition via a distinction between "layers" and "stages". They say that "$F_i^{L_i}$ denotes layer $F_i$ is repeated $L_i$ times in stage $i$". They also explain that "all layers in each stage share the same architecture. Therefore, I think they put skip-connections in the form of function composition, but it is confusing. I think it's their way of compactly expressing the (often) repeated architecture of skip-connected layers.


Magnification factor in image classification

This question greatly depends on the pixel-space scale of the objects, but if the scale of the objects is similar and enough data are provided to curb overfitting, then yes, it should work both ways.

The Caveats:
A Neural Network is merely a glorified (hierarchical) Template Matcher. If the visual appearance of multiple objects in an image is strongly correlated with the learned template, those objects will each get a high response from that convolutional kernel. Convolutional templates can achieve some degree of scale-invariance through stacking multiple layers and using a downsampling strategy such as Max Pooling, but virtually all convolutional neural networks are practically quite limited as to the scale of features that they will respond to. As a simple example, a neural network consisting of two convolutional layers of 3x3 spatial kernels each with padding of 1 separated by a 2x2 Max Pooling downsampling layer would have a receptive field of at most 8x8 pixels per output location (3x2+2), where the last 3 is the last layer's kernel width, the middle 2 is the downsampling factor, and the last 2 is the input padding reachable from the first kernel's output. Trying to classify any objects larger than this would result in a failure to match the learned template, so in fact CNNs are sensitive to the scale of their inputs, but due to weight sharing across spatial locations, a strong response trained from a single item should activate for multiple items of similar scale and (to some extent) vice-versa.


Disproving or proving claim that if VCdim is "n" then it is possible that a set of smaller size is not shattered

I agree, the claim as written is incorrect.  If $C^*$ is shattered by $\mathcal{H}$, and $C\subseteq C^*$, then $C$ is also shattered by $\mathcal{H}$; to be possibly over-pedantic:
For each $B\subseteq C$, because then also $B\subseteq C^*$, we have that there is an $H\in\mathcal{H}$ such that $C^*\cap H=B$.  Note that this implies $B\subseteq H$.  Now, $C\subseteq C^*$ implies that $C\cap H\subseteq C^*\cap H$, so we have $C\cap H\subseteq B$.  And having both $B\subseteq C$ and $B\subseteq H$ implies that $B\subseteq C\cap H$.  So $B=C\cap H$.


General strategy for imbalanced, semi-supervised, sparse problem

It's a binary semi-supervised classification problem. First, establish a base-line for the supervised case. Then try if the unlabeled data helps

Supervised

From your labeled data: create a training, validation and test set. Don't touch the test set until the very end.
Try something simple, e.g. a multilayer Perceptron (MLP) with 350 input nodes and 1 output node (giving the probability of "true").
Try more stuff (e.g. https://github.com/MartinThoma/algorithms/blob/master/ML/mnist/many-classifiers/python.py#L92).
Try to combine classifiers in ensembles (see examples).
A (naive) bayes classifier might be worth being investigated.

Semi-supervised

You could train an auto-encoder with a bottleneck on the unlabeled data. Then remove everything after the bottleneck. Use this network as a preprocessing step. The idea is that this network finds a more meaningful abstraction of the relevant data. However, similar as PCA can have a very low projection error and still destroy the possibility to distinguish the classes.
SVMs can also be used in a semi-supervised setting.
You could try to find clusters in the unlabeled data + labeled data. Then you can get an a priori probability of "true" for each cluster (ignoring the unlabeled data). You could train very small models (overfitting!) on the data in the clusters.

Usually, you can interpret features. This might help to develop new features.


I finally used this website, since I found it easier to found any ontology.
For example I found this one, very similar to what I was looking for.


Implementing a model for a language to another

In case you want some state-of-the-art technique, you can implement a neural machine translation model with attention, as fully described in this course with Google members libraries like Trax.
In case of short texts, you can still use sequence-to-sequence models without attention, but the attention mechanism prevents you from suffering the vanishing gradient problem on longer sentences.
As a very brief schema of the type of model, find the image below (from the same given source):


Trained Tensorflow model performs poorly on inference

I believe the samples that you are using for inference belongs to a different distribution of data from training and test set. So, I would suggest you to verify if the above case is true and if it is then you should fine-tune your model for the examples that you are trying to carry out inference.


Visualizing 28 different variables with 28 different colors?

This is the best solution I have come up with, I simply found a larger color pallet. The largest I could find was:

http://bl.ocks.org/aaizemberg/78bd3dade9593896a59d

So I merged d3 catagory 20 with the first few from d3 category20b which was inspired from this heatmap tools heatmap tool's source code:


Learn the evaluation matrix for hire a job candidate

its a simple DNN problem when you have x features and y labels you can have some data for NN training and then NN can be used to classify other candidates. I have a program already made for this plz comment if you need one .


How to store complex tables and structures?

In third normal form

IndvidualID
CatID
CaseID
Value

Primary Key:  IndvidualID, CatID, CaseID

If you need to constrain CatID and/or CaseID then have a FK

From that structure you can easily loop and create long format or multidimensional array.
It is also space efficient as a null is simply no row.

In sql an avg

select CatID, avg(value)
  from table
 where CaseID = x
 group by CatID

Or you can just loop on those same constraints
Or a tool like LINQ


Checking Correlation of Categorical variables in SPSS

For testing the correlation between categorical variables, you can use:

binomial test: A one sample binomial test allows us to test whether the proportion of successes on a two-level categorical dependent variable significantly differs from a hypothesized value.  For example, using the hsb2 data file, say we wish to test whether the proportion of females (female) differs significantly from 50%, i.e., from .5.  We can do this as shown below.
chi-square test: A chi-square goodness of fit test allows us to test whether the observed proportions for a categorical variable differ from hypothesized proportions.  For example, let's suppose that we believe that the general population consists of 10% Hispanic, 10% Asian, 10% African American and 70% White folks.  We want to test whether the observed proportions from our sample differ significantly from these hypothesized proportions.

You can find the code for doing above analysis from the link below:

Source: http://www.ats.ucla.edu/stat/spss/whatstat/whatstat.htm

I'm not sure about comparing numerical and categorical variables but one thing you can do is to transform the numerical variable into categories. For example, if it is age, then, you can transform it to: [0,10), [10,20), ... , [90,100), so that you can compare the correlation of age with other categorical variables.

Hope it helps.


Supervised Learning could be biased if we use obsolete data

One approach to using knowledge from the example you gave above is to use that information as a prior for your current model. While it is unlikely that the exact trends observed a while ago will predict future trends, some general observations/correlations will likely still be relevant for the present day.

Concerning your example of criminal profiling. Making predictive models in such cases can be highly biased and is controversial. I invite you to read the Outlook section XVIII.D of this ML review on the social implications of machine learning and the references mentioned.


Can feedback loops occur in machine learning that cause the model to become less precise?

Yes, this is a real problem that manifests once system is used by real users.

Most prominent example is News Echo Chamber (accentuated by ML based recommendation systems)

ML algo sees that you like news / videos related to certain point of view, you watch more of such videos and model becomes more convinced of your choice. So it suggests even more content with similar views.

https://en.wikipedia.org/wiki/Echo_chamber_(media)

http://theconversation.com/explainer-how-facebook-has-become-the-worlds-largest-echo-chamber-91024

https://www.theguardian.com/science/blog/2017/dec/04/echo-chambers-are-dangerous-we-must-try-to-break-free-of-our-online-bubbles

https://www.quora.com/Would-you-say-that-Quoras-generated-news-feed-suffers-from-an-echo-chamber-dilemma


How to fill an missing values in a column based on another column

import pandas as pd
import numpy as np

shoes = pd.DataFrame({'Brand':['Ugg', 'Prada', 'Clark', 'Ugg', 'Clark'],
                     'Comment':[np.NaN, np.NaN, np.NaN, np.NaN, np.NaN]})

df_shoes = pd.DataFrame({('Ugg','Made from sheep'),
                     ('Prada', 'Made from pig leather'),
                     ('Clark', 'Made from Cow leather')}, columns=['Brand', 'Comment'])

shoes.merge(df_shoes, on=['Brand'], how='left', suffixes=('_x', '_y'))

The result will show like:

You can then drop the null columns.

EDIT:
As discussed in the comments, in case for the edited question, you can do that:

shoes[shoes.Comment.isnull()].merge(df_shoes,on=['Brand'], how='left',suffixes=('', '_notnull'))

shoes.Comment.fillna(value=temp.Comment_notnull)


Prediction interval for very small dataset

In the case of quantile modeling, it makes sense as long as the quantile model makes sense. If you are able to build a good quantile model with 60 observations, then your estimates of intervals between quantiles should also be good. Keep in mind how errors will propagate when building intervals:

The bias in the width of the interval will be the sum of the biases in the quantile estimates; they might balance out, or they might make things worse.
The variance of the width of the interval will also be the sum of the variances of the quantile estimates. This could be a pretty big number, knowing that with 60 observations the "true" model variance will be large and hard to estimate.

Outside of that, it all depends on the model. Common statistical models tend to have explicit expressions for prediction intervals. In MCMC-based inference they can be inferred from the sampled values. In these cases, 60 is as good as 6,000 as long as the model works. As above, keep in mind that error propagation is not always easy to control. If these intervals are needed for serious applications, it's a good idea to generate fake data and run a sensitivity analysis.

The most generic solution is to obtaining a prediction interval is resampling. With just 60 observations, you can run full Leave-One-Out cross-validation (LOOCV). Or you can do bootstrap resampling. It might be a good idea, or the variance of the procedure will be through the roof. Again, simulation-based sensitivity analysis will help you decide.


Clustering with Only Categorical Features

You can one-hot encode all your features, first. Then, you will face with a sparse feature space. To resolve this issue, you can use an auto-encoder to encode all these values to a low-dimensional and more dense space. Then run one of your clustering methods such as k-means.


Is NLP suitable for my legal contract parsing problem?

Here's what I would have done.

extract raw data from the the pdfs.
use rule based NER from spacy(it's quite fast), but it will require some manual pattern making.
Also apply multiprocessing if you can.
https://spacy.io/usage/rule-based-matching
You can also write a document classification to segregate the templates and send the templates to appropriate code,running parallely.
Also checkout blackstone library for legal texts, it might be of some use https://spacy.io/universe/project/blackstone


How to get data back into two separate audio files after successfully applying kmeans clustering on an audio file?

What you are trying to do is Blind Source Separation, and unfortunately it does not works as easy as that. When you are extracting MFCC from an audio file with a mixture of voices, MFCC clusters won't represent two people. For the separation, you should end up with two sound signals with the same number of samples. The MFCC are used as the feature's space for speakaer recognition tasks, for example, but not for separation in general. For more mathematical background about BSS you can visit: https://arxiv.org/pdf/1603.03089.pdf

The most simple case is the so-called Cocktail Party Problem, in where you have at least as many microphones as persons and the mixture has to be instantaneous (no delay between signals, only attenuations due to path-losses). This very simple case can be solved applying Independent Component Analysis, with the FastICA algorithm: http://scikit-learn.org/stable/auto_examples/decomposition/plot_ica_blind_source_separation.html

For delayed mixtures you can try the DUET algorithm: https://pdfs.semanticscholar.org/1413/746141f2871e0f45056a7696e019b8f8a100.pdf

For convolved mixtures and undetermined problems (more speakers than microphones) this PARAFAC-based algorithm performs quite well: http://people.ece.umn.edu/~nikos/05233821.pdf


Need machine learning algorithm to fill in time-series data

You can approximate a time series with a polynomial of degree n_degree by using ridge regression. You can try different degree numbers (e.g. [2,3,4,5,6]) and choose the best one. Keep in mind that higher degree models, always get lower error values. So you should somehow penalize higher degrees.

from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

for count, degree in enumerate([2, 3, 4, 5, 6]):
    model = make_pipeline(PolynomialFeatures(degree), Ridge())
    model.fit(Time, y)

More details here.


So, you can build a pretty good starter set of airlines from this wikipedia page. If you scraped the page, extracted the Airline column, and lowercased the text, you would be able to have some seed data to work with.

This is also a shot in the dark, but you could scrape the model names from this toy/model planes website, also generating some base set of strings you can use to automatically validate the first two fields in your output.

The trick to reducing your time here is to figure out the 80/20 (get 80% of the work done with 20% of the effort). I have no answers for how you would be able to differentiate the last field. But these are the steps I would take to reduce the effort altogether


Method to filter out objective statements (or say facts)

There isn't really an "objective" statement.

In your example: what if he wasn't born that day? Same statement, but is it still objective?

You may assume that "1 + 1 = 2" is an objective statement. But what if I'm doing binary math, and 1+1=0 then? So even that is subjective.

How would a machine tell apart these things, where philosophers will disagree with each other?

All you can do is provide training data examples of "your" (subjective) idea of objectiveness.


ATM I know of TSimulus and TimeSynth to generate data programatically in a controlled manner (instead of generating random data).
TSimulus allows to generate data via various generators.
TimeSynth is capable of generating signal types
Harmonic functions(sin, cos or custom functions)
Gaussian processes with different kernels
    Constant
    Squared exponential
    Exponential
    Rational quadratic
    Linear
    Matern
    Periodic
Pseudoperiodic signals
Autoregressive(p) process
Continuous autoregressive process (CAR)
Nonlinear Autoregressive Moving Average model (NARMA)

and noise types
White noise
Red noise

If you are looking for a graphical way to generate data TimeSeriesMaker is the only tool able to do this.


Training and test split for time series analysis

Your code looks incomplete but you can definitely try the following to split your dataset-

X_train, X_test, y_train, y_test = train_test_split(dataset,
                                                    y,
                                                    test_size=0.3,
                                                    shuffle=False)

Note: y will be a series object for your dependent variable.

Because of this reason, using train_test_split with shuffle=True is not a good practice and should strictly be avoided.

Alternately, for time series you can use TimeSeriesSplit() and look for the implementation here.


Which metric to use for classify a binary logistic regression

The performance measure should be related to the actual real world problem you face. (Imagine, you are a football trainer. You are seeking for young talents to introduce them to your very expensive training camp. It might be more important to you to prevent expenses on kids, that proof not to be as talented as hoped for, than reaching all hidden talents with the program. So you might be interested in strong precision of beforehand assessement. But if you dont have to care about funding, your priorities might change. You could be more interested in finding any last talented kid. Then you would be more interested in recall.)

If you want good probabilty predictions, use logloss. If you want the prediction score to separate very well among classes, rather use area under RO curve. Your choice should depend on your real world cost for missclassification or bad predictions.

The averages are merely an aggregation of the class specific statistics. This is mostly useful for multi class problems. For two class problems I recommend to look at the class specific statistics instead. Anywasy, how averageing is conducted, you can read here: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html


Answering the first question:
Is it possible to train a LIME explainer for a binary classfier on a dataset without labels?
From https://christophm.github.io/interpretable-ml-book/lime.html
Local surrogate models are interpretable models that are used to explain individual predictions of black box machine learning models. - So, no, you cannot use Lime on binary dataset without labels, because you cannot train classification model in the first place.
So, you need labels - one approach would be to cluster your data and if the structure in your data will be found, assign clusters to the dataset, train model, use Lime => interpret results.
Second part:
In general, does it make sense to keep a trained LIME explainer around to generate explanations during production or is it better to train the LIME explainer on production data whenever is needed?
As far as I know, there is no clear prove on if you should use LIME trained on train model and then use it on validation dataset or if you can use LIME on the trained row themselves. I would train the LIME model on train dataset and corresponding model and would then use the same LIME in production to explain particular data.
Generally, this book covers all topics (or most of them) regarding interpretable ml. (Im pretty sure that this book has a paragpaph talking about use of trained explainer on test/train dataset, but I dont remember where exactly)
https://christophm.github.io/interpretable-ml-book/


How do I implement the sigmoid function in Octave?

This will compute the sigmoid of a scalar, vector or matrix.

function g = sigmoid(z)
%   SIGMOID Compute sigmoid function
%   g = SIGMOID(z) computes the sigmoid of z.

% Compute the sigmoid of each value of z (z can be a matrix,
% vector or scalar).

SIGMOID = @(z) 1./(1 + exp(-z));

g = SIGMOID(z);

end


Data Science oriented dataset/research question for Statistics MSc thesis

Just head to kaggle.com; it'll keep you busy for a long time. For open data there's the UC Irvine Machine Learning Repository. In fact, there's a whole Stackexchange site devoted to this; look there.


Correct me if I'm wrong, but it seems you have a binary classification problem - the image either contains a cat or a dog.

If that's the case, then:

output = Dense(1, activation='sigmoid')(flatten)

and

xception_model.compile(Adam(lr=.0001, decay=1e-6), loss='binary_crossentropy', metrics=['binary_accuracy'])

This thread is a great resource on the difference between the two questions.

Let me know if it works.

EDIT: Further explanation

Although we normally talk about “binary classification”, the way the outcome is usually modeled is as a Bernoulli random variable, conditioned on the input data. So:

$$P(y = 1|\mathbf{x}) = p, \ 0\leq p\leq1$$

A Bernoulli random variable takes on values between 0 and 1. So that’s what our network should produce. One idea might be to just clip all values of $wth+b$ outside that interval. But if we do this, the gradient in these regions will be 0: The network cannot learn.

A better way is to squish the complete incoming interval into the range (0,1), using the sigmoid function:

$$\sigma(x) = \frac{1}{1 + e^{(-x)}}$$

But the sigmoid function saturates when its input gets very large, or very small. Is this problematic? It depends. In the end, what we care about is if the cost function saturates. If we follow the general principle of maximum likelihood/cross entropy, the loss will be:

$$ - log P (y|\mathbf{x}) $$

where the $log$ undoes the $exp$ in the sigmoid.

In Keras, the corresponding loss function is binary_crossentropy. For a single item, the loss will be:

$−log(p)$ when the ground truth is 1
$−log(1−p)$ when the ground truth is 0

Here, you can see that when for an individual example, the network predicts the wrong class and is highly confident about it, this example will contribute very strongly to the loss. Therefore, you only need to have one output node, because binary classification is "symmetrical". By that, I mean that if we say that an item is of class 0 with probability p, we are also referring to the fact that this same item is of class 1 with probability 1 - p.


derivation for expected value for variance

The expected value of a random variable is defined as (without entering into probability/measure theory):

For discrete distribution $F$, we have $E(X)=\sum_x P_F(X=x) x$
For (absolutely) continuous distribution $F=\int f(x) dx$, we have $E(X) = \int f(x) dx$

Therefore, the first equality is just the definition of expected value for the discrete case.
And finally, the law of the unconscious statistician (LOTUS) states that
$$E(g(X)) = \int g(x) f(x) dx$$ (for continuous F), or
$$E(g(X)) = \sum g(x) P(X=x)$$ (for discrete F).


Most clustering algorithms work best with continuous values (in particular k-means). Many cannot use categorial attributes and won't work very well with one-hot encodings either.


how to remove unwanted characters from data

Should have been posted in SO as others rightly pointed out.

Simple solution would be to:

Split on '*' and select the first part of the split.
Apply (1) over list items and return a vector.

x <- c("21,34,99*", "56,90*", "45*")               #test case

data_clean <- function(x) sapply (strsplit(x , '[*]' ), `[` , 1)
data_clean(x)


How to retrieve images from a url in a pandas dataframe and store them as PIL object in a new column

I just changed the User-agent in the for loop so that now the request line in the loop is: r = requests.get(df.iloc[i,1], headers=headers) with headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/XXX.XX (KHTML, like Gecko) Chrome/XX.X.XXXX.XXX Safari/XXX.XX'} and this solved the error. I also added a r.raise_for_status() to check the status before using the r.content

Final code:

import pandas as pd
from PIL import Image
import requests
from io import BytesIO

df = pd.DataFrame({'project_id':["1", "2"],
                    'image_url':['http://www.personal.psu.edu/dqc5255/gl-29.jpg',
                                'https://www.iprotego.com/wp-content/uploads/google.jpg']})

pictures = [None] * 2

df.insert(2, "pictures", pictures, True)

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/xxx.xx (KHTML, like Gecko) Chrome/xx.x.xxxx.xxx Safari/xxx.xx'}

for i in range(2):
    r = requests.get(df.iloc[i,1], headers=headers)
    r.raise_for_status()
    df.iloc[i,2] = Image.open(BytesIO(r.content))

df
````


Map similar like clusters in two different cluster sets

I would investigate a soft clustering approach using HDBSCAN clustering.
Some reading on text clustering here, which includes discussion on clustering with DBSCAN, HDBSCAN's non-hierarchical precursor


Padding sequences for neural sequence models (RNNs)

Padded values are noise when they are regarded as actual values. For example, a padded temperature sequence [20, 21, 23, 0, 0] is the same as a noisy sequence where sensor has failed to report the correct temperature for the last two readings. Therefore, padded values better be cleaned (ignored) if possible.

Best practice is to use a Mask layer before other layers such as LSTM, RNN,.. to mask (ignore) the padded values. This way, it does not matter if we place them first or last.

Check out this post (my answer) that shows how to pad and mask the sequences with different length (with a sample code). You can experiment with the code to see the effect of removing the mask (treating padded values as actual values) on the deterioration of model performance.

This is the python snippet for quick reference:

model.add(Masking(mask_value=special_value, input_shape=(max_seq_len, dimension)))
model.add(LSTM(lstm_units))

where special_value is the padded value that should not have overlap with actual values.


How to convert categorical data to numerical data in Pyspark

This can be done using StringIndexer in PySpark and the reverse using IndexToString for reference please check this:

from pyspark.ml.feature import StringIndexer

df = sqlContext.createDataFrame(
    [(0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c")],
    ["id", "category"])
indexer = StringIndexer(inputCol="category", outputCol="categoryIndex")
indexed = indexer.fit(df).transform(df)
indexed.show()

For more details, please check the spark documentation


Performance metric in recommender systems with implicit feedback

I think your post is missing a clear question.

I'll answer your concerns anyway:

For a implicit feedback problem like the one described RMSE, indeed, doesn't make much sense. First of all it is not clear what the difference between the real value (number of interactions, e.g. watched movie 4 times) and the estimation would mean. Second, in recommender systems you usuallly want to give the user 10-20 relevant recommendations and you don't care for the rest. Hence, it makes sense to optimize your algorithm to get these top recommendations right while disregarding your performance further down in the ranking.
See my first answer. Also, the authors of your cited paper argue for a rank-based metric.


Why DDPG's policy target is Q-value itself?

In DDPG, the actor network is used to calculate the action that maximises the expected reward in a given state, i.e. the action that maximises the $Q$ function. Critic network is used to calculate the action value in a given state. So the critic network is updated just like the way in DQN. While updating the actor however, we don't have a supervised target action (actual action that maximises the Q function) and our aim is to generate the action that maximises $Q(s,a)$, So we generate an action using actor, apply it to the critic function, then modify the actor params to maximise the $Q(s,a)$, i.e. the gradients flow all the way back from Q network to the actor and the update is done using gradient ascent. So, the policy network is not trained independently, it is done with the help of critic network using the $Q$ value as the target and I'd say the loss of the policy is $-E[Q(s, \mu(s))]$.


How to train a machine learning model if there is a relationship between two different data points?

This is a very broad question and really depends on the type of data you have and how it is distributed.

There are many types of classifiers that you can use.
This link, from scikit-learn shows you a comparison between many algorithms that could help you choose which one to pick.

I understand you want to write the algorithm yourself, but I would recommend you looking at scikit-learn.org and trying out different algorithms. Once you have tried and wanted to implement yourself, you can have a look at blogs, like this one, which explains in details the intrinsic works of models, like k-nearest-neighbours for example.


How can decision trees be tuned for non-symmetrical loss?

Theoretically, decision tree algorithms specify the feature as well as the threshold that maximize the separation between classes at each node. This can be done by solving optimization problem related to the entropy at the children nodes.
You can modify this optimization problem by including the miss- classification costs you have in the optimization problem so the algorithm will be biased toward the most expensive class


You can use groupby and then sum
Take a look at https://stackoverflow.com/questions/39922986/pandas-group-by-and-sum


How to save and test CNN model on test set after training

I don't know exactly where you have the problem but according to comments, take a look at the following line.

_, c = sess.run([optimizer, cost], feed_dict={x_img: X, y_label: Y})

feed_dict is used for passing data to your network. As you can see, X is the training data. You can replace it with the test data. You should also change the y_labels to the labels of the test data.


How to select variables based on the mean correlation in a correlation matrix?

There is no universal definition of "strongly correlated" and thus there is no "correct" answer to your question. One approach would be to perform a significance test to determine whether there is strong evidence that the correlations are nonzero. The mathematical details of the test are here, and the relevant Scipy API documentation is here.

However, "probably nonzero" may not be what you're looking for when you say "strongly correlated." If that's the case, you might want to consider why you're interested in finding the strongly correlated variables in the first place. Are you interested in identifying the strongly correlated features for a predictive problem? If so, you'll likely be better off skipping this exercise and starting to experiment directly with learning algorithms. Are you interested in determining what predictor variables cause changes in a certain response variable? In this case, you'll probably want to look into some kind of statistical regression model (probably starting with linear regression) and be very careful about all of the assumptions involved when drawing causal conclusions.


If I were you I would take a MOOC or two (e.g., Algorithms, Part I, Algorithms, Part II, Functional Programming Principles in Scala), a good book on data structures and algorithms, then just code as much as possible. You could implement some statistics or ML algorithms, for example; that would be good practice for you and useful to the community.

For a PhD program, however, I would also make sure I were familiar with the type of maths they use. If you want to see what it's like at the deep end, browse the papers at the JMLR. That will let you calibrate yourself in regards to theory; can you sort of follow the maths?

Oh, and you don't need a PhD to work at top companies, unless you want to join research departments like his. But then you'll spend more time doing development, and you'll need good coding skills...


Switching Keras backend Tensorflow to GPU

I suggest reinstalling the GPU version of Tensorflow, although you can install both version of Tensorflow via virtualenv. GPU version of Tensorflow supports CPU computation, you can switch to CPU easily:

with device('/cpu:0'):
    # your code here

I have been using GPU version of Tensorflow on my Tesla K80 for a few months, it works like a charm. Feel free to have a try!


What is the difference between these 2 Python Pandas queries

Depends on what you are trying to do. What is the mean that you are trying to find. Your query return the mean of the number of correct matching rows divided by all the rows in the dataset.

Query 10: It first filters the data out on year > 2000 and then check the total number of rows in this filtered dataset which have ['avg_temp'] < 20. So if your whole dataset had 100 rows, your filtered dataset now will have lets say 10 rows. After the avg_temp condition, only 5 rows match that criterion. So the mean will be 0.5

Query 11 and 12:

It filters the original dataframe on both the conditions at the same time. SO from the sample example as above, your total number of rows in the dataset is 100 and the number of rows that match the criterion are still 5 so the mean will be 0.05.

So it depends on what you are trying to do here.


Is there any python framework which take a dataframe and give all important relation?

There is a one-liner solution using a Python package called pandas-profiling that gives you a quick way into most crucial statistical explanatory analysis including various correlations and many more. The documentation provides a demo that is worth checking.


Which cross-validation type best suits to binary classification problem

You will have best results if you care to build the folds so that each variable (and most importantly the target variable) is approximately identically distributed in each fold. This is called, when applied to the target variable, stratified k-fold. One approach is to cluster the inputs and make sure each fold contains the same number of instances from each cluster proportional to their size.


Implementing the Trapezoid rule without the formula for the curve

You're assuming that the points are equally spaced along the fpr axis, which is generally not true.  See e.g. the "Uniform grid" vs "Nonuniform grid" sections of the wikipedia article.  You need something like
delta_xs = np.diff(fpr)
left_endpoints_y = tpr[:-1]
right_endpoints_y = tpr[1:]
trap_areas = 0.5 * (left_endpoints_y + right_endpoints_y) * delta_xs
area = trap_areas.sum()

(That's more verbose than it needs to be, probably not the most efficient, and I don't know what order your fpr/tpr lists are, so it'll need some finagling.)
Using thresholds at 0.01 spacing is a little unusual too: the ROC curve is represented by using every predicted probability as a threshold (together with $\pm\infty$, or a convention that $(0,0)$ and $(1,1)$ are on the curve).


How to train the machine so that it can give 'out of bound/classes' as an output for neural network

I think the best way would be to augment some data and have an additional output class "unknown". However, if that is not possible or the neural net can not be retrained I would compare the distribution of the outputs of a hidden layer.

For the CNN architecture below, calculate the empirical distribution for the outputs of a hidden layer after the flatten layer for the training data (e.g.$n1$). Save this distribution and during deployment, calculate the output of $n1$ for the test instance. If the output of $n1$ corresponds to a highly unlikely value with respect to the empirical distribution of training examples, return "unknown". Else, predict the digit for this instance.

(image source)


Sequential Modelling: Multiple Sequence to One or Sequence to Sequence

As you are talking about RNN, I'm going to change the parameters of your question a little bit.

The multi sequence to one will be:

$[x_1] , [y_1]$

$[x_1, x_2, y_1] , [y_2]$

...

$[x_1, x_2, ..., x_n, y_{(n-1)}], [y_n]$

as the previous state will always be used.

This is typically a Markov process with a state containing a n input vector. It has known properties and that's what is used by RNN to be able to create a sequence one element by one element.

On the case of one sequence to one sequence, there is no such state, and we don't have the concept of one word following the other. It's basically n values in the input and n value as the output. But there is no constraint about "one after the other" in such a pseudo sequence anymore, so the outputs could be scrambled and it would still be a sequence to sequence output.

Now, for NLP, the fact that you don't have a state before time 0 is not a problem. You can discard the first n-1 elements even and keep the Markov property of the system.

This is exactly what I'm doing in chapter 8 in Building Machine Learning with Python and it works well if you start retrieving words when the system is in a proper state.


Why does it not need to set test group when using 'rank:pairwise' in xgboost?

The output is a score that can be used to rank the samples, and the point in this sort of ranking problem is that you'll only care about ranking samples within the same group (which you think of as being results from a given query).

But that can be safely left to you on the testing set.  (Indeed, you might as well only run the prediction for each group separately.  You might think about the output in your case as assuming that the test set is all from a single query.)  For scoring on the test set, it might matter what the specified groups are, but not for just making predictions.

For training, the group data is needed so the algorithm knows not to calibrate the rankings for intergroup comparisons.

See also:
How fit pairwise ranking models in xgBoost?
https://github.com/dmlc/xgboost/blob/master/doc/tutorials/input_format.rst#group-input-format


What are bias and variance in machine learning?

What are Bias and Variance?
Let's start with some basic definitions:

Bias: it's the difference between average predictions and true values.
Variance: it's the variability of our predictions, i.e. how spread out your model predictions are.

They can be understood from this image:

(source)
What to do about bias and variance?
If your model suffers from a bias problem you should increase its power. For example, if the prediction of your neural network is not good enough, add more parameters, add a new layer making it deeper, etc.
If your model suffers from a variance problem instead, the best possible solution is coming from ensembling. Ensembles of Machine Learning models can significantly reduce the variance in your predictions.
The Bias-Variance tradeoff
If your model is underfitting, you have a bias problem, and you should make it more powerful. Once you made it more powerful though, it will likely start overfitting, a phenomenon associated with high variance. For that reason, you must always find the right tradeoff between fighting the bias and the variance of your Machine Learning models.

(source)
Learning how to do that is more an art than a science!


Using LSTM's on Multivariate Input AND Multivariate Output

You can simply change the number of units at the last layer if you want to predict multiple outputs (or just said differently, one output with multiple features). For example from the link you shared, you can change the units of the Dense layer:

model = Sequential()
model.add(LSTM(50, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(2))
model.compile(loss='mae', optimizer='adam')

The above model would now predict the next step of an output with 2 "features".
Note that you output should be of shape num_samplesx2 now.


Relation between amount of training samples and model depth?

Yes, this is common knowledge. Every time you add a parameter to a model, you will need to give it more data in order for it to learn as well as the simpler model. Every individual weight in a neural network is a parameter. The more weights, the more parameters, and the more data is needed.

One of the fundamental tasks in model-building is finding a good tradeoff between having enough parameters to learn fine details and having enough data to train all those parameters. Too many parameters lead to overfitting in part because if you don't have enough data you end up memorizing individual cases, as opposed to learning about and averaging over a spread across several cases.

This is one reason why class imbalance is a problem. If you don't have enough data on one of the classes, then that class will be poorly understood by the model.


Using model.fit() will always train on the whole dataset for n epochs. The batch_size argument denotes how many samples are used to calculate the gradient and updates the parameters. If you want to train the model on just a batch of data (i.e. a subset of your total dataset) simply pass the subset of data to model.fit().