Variational auto-encoders (VAE): why the random sample?

To have a common mental image of AE and VAE please take a look at this answer first.

Lets go through this "why not?" thought process step by step:

Why not deterministic? lets directly encode the latent vector $z$ inside a layer of neural network. But this way, $z$ would be deterministic, meaning a fix input $x$ always produces a fix latent vector $z$, thus, $z$ would not have distribution $q(z|x)$. This is the ordinary auto-encoder. Sampling $z$ from  $q(z|x)$ means if we try the same input $x$ twice, we should get two different values for $z$. Deterministic computations (layers) cannot achieve this. Therefore, we need to inject a random element in the calculation of $z$, otherwise the same $x$ always gives the same $z$. Note that getting a different $z$ for a different training point $x$ implies the existence of distribution $p(z)$, which exists for ordinary auto-encoder too, not the existence of conditional distribution $q(z|x)$,
Why not only mean? OK, lets add these random elements to a layer to get our beloved probabilistic $z$, lets call this layer $\mu$. So, suppose $\mu$ is calculated from previous layer $y$ as $\mu=\mbox{tanh}(y)$, then we select the random elements $\epsilon_d \sim  N(0,1)$ per dimension of $\mu$, and then calculate $z= \mu + \epsilon$ which is the same as sampling $z_d \sim  N(\mu_d, 1)$, now our $z$ is probabilistic and we have got rid of standard deviation $\sigma$ and its regularization all together, we now only need to regularize mean $\mu$ to $0$,
Why not only random elements? Lets go one step further, lets throw $\mu$ away too and set $z_d = \epsilon_d \sim  N(0,1)$, and get rid of all the regularizations. But wait a minute! now $z$ is completely disconnected from previous layers, so no information is delivered from input $x$ to latent variable $z$.

Going from VAE with two parameters $(\mu, \sigma)$ to (2) with one parameter $\mu$ and then (3) with zero parameter is the same as saying instead of using parameter $w$ and regularizing it via $\parallel w \parallel^2$, lets set $w=0$ and get rid of the regularization. Even though we want parameters to be close to zero but still we want them not to be zero and carry information. This is the same as $(\mu, \sigma)$ in VAE, we want them to carry information, although being closer to $(0, 1)$ is favorable. $(\mu, \sigma)$ are two channels through which information is distilled from $x$ to $z$. We may even add a third variable (channel) to our distribution if complexity-performance trade-off is favorable. It is safe to say that researchers have experimented with only $\mu$ and observed that adding another channel for variance would be beneficial, so here we are with two channels $(\mu, \sigma)$ and a crucial random vector $\epsilon$.

Why would VAE work better than AE?

The main difference that VAE introduces compared to AE is that now $z$ is loosely (probabilistically) related to $x$ compared to AE, this creates an additional regularization effect that throws some information of $x$ out by not obeying the exact network computations for $z$, thus this extra regularization could improve the performance in practice. Of course, this is not a complete dominance. In some tasks this extra regularization may work against VAE.

In my opinion, the why's could be answered this far as they reach a similar level to "Full Bayesian vs regularized least squares regression".


Once is all it takes; you're done. The point of dealing with outliers is that they are rare. Forcing your model to fit to those outliers can make its performance against inliers (which it will encounter far more often) worse, and that's A Bad Thing. Although you now have values that look like outliers when you plot them, this is an artifact of your earlier transformation - those values should be within the "normal" expected range of values for your data.


Yes, this basically means that no sensible rules can be built with the constrains of support and confidence you impose. Note that your lift is always trivially one. The solution to this is given in the documentation of the package:

Note: The default value in APparameter for minlen is 1.
This means that rules with only one item (i.e., an empty
antecedent/LHS) like {} => {beer} will be created. These
rules mean that no matter what other items are involved
the item in the RHS will appear with the probability given
by the rule's confidence (which equals the support). If
you want to avoid these rules then use the argument
parameter=list(minlen=2).


KNN Regression: Distance function and/or vector representation for datetime features

I like your idea of converting to 2d (the unit circle), 2d numpy array would be the way to go here Specifically, try puting the days_x and days_y into separate columns if you take the unit circle approach.
An alternative idea - it looks like there's a 'precomputed' option for distance, which will let you use the distance you "really" want, and should not be slow, since there's no computation to be done.


How do i get to make colab read my images from my google drive?

There are some commands you must run, designed specifically for Colab:

from google.colab import drive
drive.mount('/content/drive')

With this, an authentication page will open. They want you to explicitly allow an access to your Drive. Click on the link and get a key code, that you can insert into an input line. At that point, you Colab Notebook is ready to access the data within you Drive account.

Each file path commonly starts with:

"/content/drive/My Drive/ ... your folders.../"

Within Colab Notebook, linux shell commands are also working, and can be accessd by putting ! before them (as in Jupyter).

NB: google.colab commands do not work outside Google Colab Notebooks, therefore, if you download your .ipybn file, remember to change those line or it'll throw error.


Classification with millions of records, thousands of categories - keep memory use efficient?

I assume you are using the sklearn implementation of random forests. If you are not bound to use sklearn, you could try another implementation, like the one from h2o, that supports enum categorical variables.

Alternatively, you could first determine if all of the 12000 levels are relevant: for each of the 12000 levels, you could compute something like information gain, which will tell you if that level provides any information relevant to predicting the class. You may then use a threshold on the information gain to discard all levels that provide no information, reducing the number of levels and, perhaps, allowing you to use one-hot encoding.

Basically, if $H$ is the entropy (of the class label) for a set of examples, given a dataset $D$, you could discard any level $v$ of an attribute $a$ for which the following is below a certain threshold (to be determined given how many levels you want to keep):

$$
H(D) - \frac{|D_v|}{|D|} \cdot H(D_v)
$$

where $D_v = \{x \in D|value(x, a) = v\}$


Fine tuning means changing the weights such that the VGGNet can perform the task you want in your dataset. The reason why fine-tuning is not called training (which is what you are doing) is because it implies that you already use a network that has been trained on a dataset. However, the concept is the same as training, but you just happen to it do with a convenient set of initial weights.


How to Avoid rarely used discrete feature values in a dataset

Working with Categorical Features can raise a few challenges. For instance, you might encounter features with high cardinality in certain values or the other case (your case), features with rare categories.

The first thing you can consider is, what you will lose if you drop those rows. What extra info do you get with keeping the rare categories. Here you probably need the domain knowledge of the project to identify the impact of the specific data. If your dataset is small or you don't feel safe to remove those rows, then you should look for alternatives.

The second approach is to group all the rare categories in one new called "Rare" or "Other". So, whenever you have a record with that value, you replace it with the new one.

Finally, you can try either a Dimentionality Reduction such as PCA or a Feature Hashing where you will map multiple categories in one new value.


How can I compare the contribution of two predictors in two different sorts for machine learning algorithms?

No you can not.
You are forgeting about variable interactions (C-A,D-A,E-A, etc...)  that could favor A.

You could answer following qestion: If I were to measure information in my variables via variability how would I proceede? For example PCA

The more information you have in the variable(predictor) the higher the chance that it will contribute more to accuracy


Evaluating Logistic Regression Model in Tensorflow

I think you are correct. The line should be

loss_batch, logits_batch = sess.run([loss, logits], feed_dict={X: X_batch, Y:Y_batch})


From $$y^{(n)}(w^T\cdot x^{(n)} + b)=1,$$

since $y^{(n)}$ is binary, we have $$w^T\cdot x^{(n)} + b = y^{(n)}$$

That is $$b=y^{(n)}-w^T\cdot x^{(n)}.\tag{2}$$

Now, let's examine

\begin{align}&\sum_{m \in S}  \alpha_m y^{(m)}\langle x^{(n)}, x^{(m)}\rangle\\&=\sum_{m \in S}  \alpha_m y^{(m)}\langle x^{(n)}, x^{(m)}\rangle + \sum_{m \notin S}  0\cdot y^{(m)}\langle x^{(n)}, x^{(m)}\rangle \\
&=\sum_{m \in S}  \alpha_m y^{(m)}\langle x^{(n)}, x^{(m)}\rangle + \sum_{m \notin S}  \alpha_m\cdot y^{(m)}\langle x^{(n)}, x^{(m)}\rangle\\
&=\sum_{m=1}^N \alpha_m y^{(m)}\langle x^{(n)}, x^{(m)}\rangle \\
&=\langle \sum_{m=1}^N \alpha_my^{(m)}x^{(m)}, x^{(n)}\rangle\\
&= w^Tx^{(n)} \tag{3}\end{align}

Using $(2)$ and $(3)$, you should be able to obtain the conclusion.


There are a number of approaches to try and improve the model but more importantly to understand what is happening.
First of all, you've displayed a plot with over 200 'features' and despite scaling the input the coefficients are in the order of 1E17!!! Edit: house prices (in dollars are in the range 1E5-E7, ie less than $10M), yet the coefficients are at least 7 orders of magnitude greater or smaller.
Therefore, two immediate steps to take either together or in combination is conduct:

dimensionality reduction either using a formal method e.g. PCA or just dropping terms that have coefficients close to zero backed with some plots to help you decide.

regularisation - at at the moment you have some extreme and opposite coefficients around +7.5E17 and -7.5E17.  See models here https://scikit-learn.org/stable/modules/linear_model.html#linear-model that implement ways to suppress extreme coefficients. Edit: If your input data is between 0-1 ask yourself why for some parameters are at opposite ends, in effect cancelling each other out.

Use some personal domain knowledge and evaluate the individual columns. Is there anything that has no bearing on the sale price? Some silly examples might the name of first person who bought it, which has no relevance to house price. Or a telephone number etc.

As we're now entering the world of 'hyper-parameter' tuning don't forget to split your data into 3 parts instead of 2.


Algorithms for Sentiment Analysis on Entity

A roll-your-own solution using Mathematica is described at https://pdfs.semanticscholar.org/0245/c38fd1f3c7859c5e22382426178090faa395.pdf

I've not found such capability built into Mathematica's core, although that research paper is from 2013.

IBM Cloud's Text Analysis API also includes this capability: https://console.bluemix.net/apidocs/natural-language-understanding#entities

See also the responses to sentiment analysis for multiple entry in one text

There may be others...


How to visualize segmentation result with CrossEntropyLoss

In either case (with or without using nn.LogSoftMax()) you should do some argmax operation to get the class from probabilities.

The reason for that is that order of probabilities will always stay the same after the nn.LogSoftmax() since that operation squeezes the numbers in the [0, 1] interval, but keeps the ordering (biggest one before this operation will also remain the biggest in the [0, 1] interval, and so on ...).

Hence, for the purpose of determinating the class, it doesn't matter whether you do nn.LogSoftmax() or not.

nn.LogSoftmax() is needed only during the training process in order to calculate the loss and backpropagate that information to update the parameters of the network.


What loss function should I use if I have been working on a classification problem which involves both multi-label and multi-class labels?

The labels of data are not mutually exclusive so you can't say this is a one vs. all problem, because more than one entry may be one in the output vector. Moreover, if in the seen there should be an apple or pear this can be considered as an exhaustive problem which means one of them should happen for each input.

My opinion is that for this problem you don't have to make a new cost function. For the mutually exclusive part, as you have truly stated that, and for the second part of your vector, the well known cross-entropy cost function will perform fine. I guess the problem is something else. For problems with mutually exclusive classes, we use soft max layer as the last layer for neural nets while for cases which classes are not mutually exclusive, you can use sigmoid as the activation function. In your case that you have combination of them I suggest you an alternative approach:

Change your mutually exclusive part to a binary output, means if the corresponding entry is less than half, you can understand that it is e.g. an apple otherwise, it is the other class. and for the rest, just keep the output vector as it is. Finally use sigmoid activation function as the last layer if you are using neural nets.


Here is solution if somebody will need it someday

def groupreg(df):
    a=df[['EUR']].values
    # train isolation forest
    model = IsolationForest()
    model.fit(a)
    return pd.Series(model.predict(a))

result=df.groupby('id').apply(groupreg)
#print(result)

#print(isolationforest(data2))
df['outlier1'] = result.tolist()
print(df)


Not only the naming has changed, also the individual widget settings and layouts are completely different. Some Orange 2 widgets are not even available in Orange 3 yet, or have been consolidated into other ones. I'm afraid Orange 2 workflows are just not compatible with Orange 3.


Extracting patterns from the spatiotemporal movement of objects

You could take the difference between coordinates occuring consecutively. This gives your displacement vector. Then create two plots one indicating the magnitude of displacement( $((\delta x_{t})^{2}+(\delta y_{t})^{2})^{1/2}$) vs time and another plot of the direction indicated by the angle of the vector from the x axis; given by $\theta_{t} = tan^{-1}(\delta y/\delta x)$.
Hence you could visualize the magnitude and direction with respect to time.


How to train a model using a daunting huge training dataset

Quite often with massive datasets the model doesn't actually need the whole data. So I think the first step is to check whether using the whole data is useful: run an ablation study where you use say 1%, then 2%, 3%, .., up to say 10% of the data (adapt the levels to your case of course). Each run consists in training on the x % subset and evaluating on a validation set (be sure to separate your real final test set before anything, this study should only use a validation set).
The goal is to estimate how much gain in performance is obtained by adding training data. Plotting the performance as a function of the amount of data should give a decent idea of the trend, even if it doesn't reach the point of maximal performance, i.e. when more data doesn't improve performance anymore. With this information you can make a better decision about how to proceed with the real training.


Url string processing: what is the best way?

I don't think you need any ML, in the best case it's going to be very slow compared to direct processing and it's very likely to cause errors.

For each address you can parse the url string and extract the domain substring (in your example they are all prefixes, if it's always the case it's very easy). Then you just group them by domain in a map... done.


Are you looking for something like this?

"Introduction to Tensor Decompositions and their
Applications in Machine Learning"
https://arxiv.org/pdf/1711.10781.pdf

I'm not sure if there are better papers for it, but from what I can tell tensor decomposition seems like what you're describing.


I just processed a bunch of files from a random corpus and the results seem to make sense generally speaking, the entities list (at the bottom) does not contain any suspicious words -there may be an issue with false negatives, but there seems to be no obvious problem with false positives.

Any chance you mistook the keys of the variable dictionary for the contents of nnp? Check this out:

# print(dictionary.keys())
dict_keys(['thi', 'excel', 'start', 'film', 'career',
'.', 'hi', 'talent', 'show', 'long', 'ahead', 'the', 'car',
 'truck', 'chase', 'excit', '1937', 'era', 'seri', 'american',
 'treasur', 'book', 'spring', 'perform', 'usual', 'owner',
'right', ',', 'take', 'chanc', 'get', 'produc', 'I', 'think',
'would', 'winner', 'recent', 'air', 'episod', "'s", 'It',
#
# ... (skipping many items) ...
#
'bang-up', 'horror/act', 'hybrid'])

# print(nnp)
{'Better', 'Rackham', 'Unicorn', 'Rooney', 'Alucarda',
'Mejia', 'Alejandra', 'Red', 'Soviets', 'Marisol', 'Remi',
'Captain', 'Santacruz', 'Digest', 'Mickey', 'de', 'Infernal',
'PBS', 'Andy', 'Recently', 'England', 'Director/co-writer',
'yummy', 'Anda', 'DVD', 'Mauri', "I'ts", 'Hergé.', 'Zero',
'Alberto', 'Herge.', 'Baron', 'Byington', 'Sadoul', 'Galindo',
 'Mauricio', 'Tono', 'Free', 'Mexican', 'P.O.V', 'Boys',
'Author', 'Swedish', 'Snowy', 'Breaking', 'Vietnam', 'TinTin',
 'Georges', 'Adriana', 'Antonio', 'Anyway', 'Remy', 'Charly',
'Daniels', 'Mr', 'J.', 'Chan', 'A', 'Mr.', 'Plascencia',
'Brilliantly', 'Congo', 'Pedro', '>', '<', 'Hardy', 'Hergé',
'Awesome', 'Valentino', 'Blue', 'Please', 'Jesse', 'Gonzalez',
 'Fanny', 'Children', 'Tintin', 'Numa', 'Sea', 'Vega', 'Panic',
 '/', 'Herge', 'Edith', 'III', 'Catholic', 'Nacho', 'Lotus',
 'Sharks', 'CD', 'Fernandez'}


Why Self-Organizing Map (SOM) units are called neurons?

Neurons do not have to be only logistic regression units. Even neurons in a simple feed-forward network may not all be logistic. For example, neurons in the output layer of a network for a regression problem are usually linear.

An artificial neuron is any computing unit that takes inputs and performs some computation. It can be any computation, linear, non-linear, logistic, weird, whatever. The goal of connecting multiple neurons into a network is achieving the ability to perform a complicated computation, even though each neuron can do only a very simple task.

A neuron in a self-organizing map does perform computations. When an input vector is presented to the network, every neuron in a SOM computes the distance between its weight vector and the input vector. This is a computation. Then it compares this distance with the computations of other neurons. This is also a computation which is possible only because the neurons are connected in the network.

The subsequent behavior of the neuron depends on the result of these computations. If it is the winner, it does another computation - modifies its weights, and one more computation - pulls the neighboring units (also a networking thing).

If it is not the winner, it performs a very important action: it does nothing, which is often the best thing that one can do.

The decision for a neuron to update its weights or not to update them is a non-linear operation: if an input vector changes slightly, the response of the neuron may change sharply or not to change at all.

The result is that the network performs a complicated computation: it builds a discrete, topology-preserving approximation of the distribution of training samples. This is useful in many applications.

When a SOM is trained, then every neuron also does a non-linear computation. It either fires or not depending on whether the input vector is closer to that neuron than to any other neuron.


What is the better to pre-process data, MS-Excel or Python/R

I mean most of the time it's very much about preference really.

If you want any indication on when to use what:

MS-Excel can only handle so much data. Good for quick analysis of small data but neither good for production or middle to large amounts of data
R great tool for calculating stuff, running models, etc. Pre-Processing is good, lots of libraries to use. Some people don't like using R for production pipelines but I know companies where this is working just fine
Python is a great overall language for pre-processing and creating production worthy data pipelines. So if you want to have a scalable, reusable pre prcessing pipeline this is certainly one of the ways to go

Of course Excel has the lowest learning curve compared to R and Python. The latter ones are similar in that.


How many images per class are sufficient for training a CNN

From  How few training examples is too few when training a neural network? on CV:

It really depends on your dataset, and network architecture. One rule of thumb I have read (2) was a few thousand samples per class for the neural network to start to perform very well. In practice, people try and see.

A good way to roughly assess to what extent it could be beneficial to have more training samples is to plot the performance of the neural network based against the size of the training set, e.g. from (1):

(1) Dernoncourt, Franck, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. "De-identification of Patient Notes with Recurrent Neural Networks" arXiv preprint arXiv:1606.03475 (2016).
(2) Cireşan, Dan C., Ueli Meier, and Jürgen Schmidhuber. "Transfer learning for Latin and Chinese characters with deep neural networks." In The 2012 International Joint Conference on Neural Networks (IJCNN), pp. 1-6. IEEE, 2012. https://scholar.google.com/scholar?cluster=7452424507909578812&hl=en&as_sdt=0,22 ; http://people.idsia.ch/~ciresan/data/ijcnn2012_v9.pdf:

  For classification tasks with a few thousand samples per
  class, the benefit of (unsupervised or supervised) pretraining is not easy to demonstrate.


r2 for regression models it is a score or error?

R2 of 0.83 means that approx 83% of the variance can be explained by the model, 17% of the variance can't be explained by the model.

This is one of the measures to evaluate a model, but you shouldn't try and maximize any number, just for the sake of maximizing the number. An high R2 could be a sign of over fitting, and a model with low R2 could still be useful.

But in a simplistic way, yes they could be seen as "score" and "error". But the implementations of the number can vary.


Layer shape computation in convolutional neural net (pyTorch)

Well, with conv layers in pyTorch, you don't need to specify the input size except the number of channels/depth. However, you need to specify it for fully connected layers.
So, when defining the input dimension of the first linear layer, you have to know what is the size of the images you feed.

You can find information on the output size calculation of conv layers and pooling layers here and here or here

If you feed images of size 32x32, the outputs layer by layer of this model are :

conv1 : $6$ feature maps of size $\left \lfloor{\frac{32 + 2\times0 -1\times(3-1)-1}{1}+1}\right \rfloor   = 30$
max_pool2d: $6$ feature maps of size $\left \lfloor{\frac{30 + 2\times0 -1\times(2-1)-1}{2}+1}\right \rfloor = 15$
conv2 : $16$ feature maps of size $\left \lfloor{\frac{15+ 2\times0 -1\times(3-1)-1}{1}+1}\right \rfloor = 13$
max_pool2d: $16$ feature maps of size $\left \lfloor{\frac{13+ 2\times0 -1\times(2-1)-1}{2}+1}\right \rfloor = 6$

Therefore, for the flattened size of the output before the first linear layer is $16\times6\times6$ :

 self.fc1 = nn.Linear(16 * 6 * 6, 120)

By doing all sizes calculations in reverse, you could have find that the input size must be $32\times32$.


Tool vs Python Script for Transforming Data in Mongo

Many ETL tools support MongoDB and are commonly used for normalizing and standardizing data. For example:

http://www.pentaho.com/analytics-for-mongodb#tab-2


sklearn - SimpleImputer in an empty Pipeline

All(?) the sklearn transformers do a check on input data (check_X_y), which includes a check for an empty dataframe.  You could probably monkey-patch out that check, but that seems like overkill.

Instead, ColumnTransformer seems the way to go.  Its main purpose fits your situation.  It deals with an empty columns selector gracefully, by just not calling fit on that transformer:

  transformers_ : list
  The collection of fitted transformers... In case there were no columns selected, this will be the unfitted transformer.

Unless you're removing columns earlier in the pipeline?  In that case, please provide that additional context.


Doing Supervised Learning where output should be a probability that adds to 1

IF YOU HAVE SMALL NUMBER OF INPU_ID
Start with simples approach which is multiclass classification. Take the combination of INPUT_ID and IS_TRUE_INPUT as labels. The rest are your features. Convert categorical features to numericals (e.g. one-hot). Let the model decide.
I don't know why you need the probability distribution of predictions per INPUT_ID. You enter a new input and model gives you a probability of being member of a class. If I did not understand your use-case please comment and I will update.
Example for lines 77 and 78:
line 77 --> input = feature_set, output = {is_i171:1, is_i120:0, ...}
line 78 --> input = feature_set, output = {is_i171:0, is_i120:0, ...}

Depending on the model you use, probabilistic output will be a probability distribution over is_i171, is_i120, etc. e.g. a softmax decision function in NNs.
IF THERE ARE SO MANY INPUT_ID
Embedd all data into a n-dimensional vector space. Calculate the mean vector of TRUE instances for each INPUT_ID. Foe a new query (input), calculate distances from each INPUT_ID mean vector and divide distances by sum of distances. Then you get a probability distribution of a new input being true for any of your input classes.


How to use "sann" function in R to solve graph problems?

First, some clarification on terminology.

A package in R is a collection of R functions, data, and compiled code in a well-defined format.

SANN (sann) is not a package. Depending on which package you're using, sann is either a function or, more often, a method used within an optimization function.

Packages containing sann include optim, trustOptim, consPlan, and constrOptim.

In the package optim, the sann method is implemented as:

> func <- function(x){
+   out <- (x[1]-2)^2 + (x[2]-1)^2
+   return <- out
+   }>
> optim(par=c(0,0), fn=func, gr = NULL,
+       method = c("SANN"),
+       lower = -Inf, upper = Inf,
+       control = list(), hessian = T)

As you said, for the "SANN" (sann) method gr is used to generate a new candidate point. If it is NULL a default Gaussian Markov kernel is used.

Now in your use case -- the case of a graph -- what you probably want to do is to use par and value to pass values to fn and gr. This is a nice feature of this implementation of SANN in optim which is covered a little more than half way  through this documentation page.

par is the best set of parameters found and value is the value of fn corresponding to par.


Spark 1.6.1 - Determining the number of clusters in a data set

You can use the Elbow method. https://en.wikipedia.org/wiki/Elbow_method_(clustering)


Visitor's probability to purchase on eCommerce site, based on aggregate historic data

This is a big topic known as Recommender Systems or Recommendation Engines.

The most common method is use what is known as collaborative filtering. Most methods on the internet try to guess what the user will rank an item, but I think this can be extended to if the user will make a purchase or not, given that if a user were to rank something highly they would also be more likely to buy that item.

The idea is that each item has a n-dimensional vector, and each user has an n-dimensional vector. You can predict a user's predicted rating by some operation between the two vectors. In this example lets use the dot product between the two vectors as a means to predict the user's rating for an item they haven't yet seen.

While taking the 2 vectors and getting a prediction is not a machine learning problem, finding suitable values for the vectors is most definitely a machine learning problem. I won't get too deep into how you can get these vectors because many different algorithms apply and it seems that you are asking more of a system design question.

Once you have trained 1 vector for each user and 1 vector for each item, you have 2 more problems to solve:
1. Estimate the top N candidate items to recommend
2. Take the dot product between each of these N items and recommend the few with the highest score.

Number 1 will likely involve some use of spatial indexing in your database (example PostGIS for PSQL) so you can find some vectors that would give reasonably good dot-products. You may want to consider using a ball tree data structure to search for a good set of candidate items.

Number 2 will be pretty simple by just dotting the vectors and picking the best few.

It will be a big challenge for you to undertake but I'm certain you'll learn a lot. Since it was such a big system you want to create I wasn't able to go into detail but hopefully, I pointed you in the right direction.


Recommendations and Missing Data in Deep Learning

This depends on what you mean by 'unseen products'. If your definition is like in the paper

[...] new  feature  combinations  that have  never  or  rarely
occurred  in the  past.

then you're thinking along the lines of algorithms like collaborative filtering. Those see the tastes of people as missing values in a matrix and try to complete it using a low-dimensional representation. Now these can be overly generic, depending on how low you choose that dimension to be.

Another, older, approach is something like association rules, which can be deducted using algorithms like apriori, which however rely on the product combinations seen in the past, so they won't ever be able recommend product combinations not seen in the past.

While the former set of algorithms can deal with what under the definition above would be called 'missing data', the latter cannot. Those however tend to give better recommendations. The paper you quote uses neural networks in what looks like a successful attempt to get the best of those two worlds.

If hover you mean by 'missing data' completely new products, than you're out of luck using any of the algorithms described above. If you don't have meta-data about those products, it's near impossible to recommend anything. Just imagine you have a list of products A, B, and C and who bought them. Now you get product D. Who will buy it? Without any additional information on D, you're dead in the water.


How to determine if dataset is a suitable representation of the context?

Before collecting the data, domain knowledge is used to determine plausible variations the task may present. It is not necessary to capture in data all the variations experts may point out (for example, some may be syntheticaly constructed). But domain knowledge indeed dictates the variations a certain task may have to take account for.
That being said, one then gathers the data trying to match plausible variations in the percent that domain experts dictate. Again some may even be syntheticaly constructed a-posteriori. Then we have a representative dataset.
In case the dataset proves inadequate, this means domain knowledge needs to be updated with new information about the task.


What does the repeated message "No further splits with positive gain, best gain: -inf" mean?

I don't know the internal specifics about LGB behavior in this case but I don't think it is stuck in an infinite loop.
From Github's LightGBM issues we can find that:
What's the meaning of "No further splits with positive gain, best gain: -inf" message?

It means the learning of tree in current iteration should be stop, due
to cannot split any more. I think this is caused by
"min_data_in_leaf":1000, you can set it to a smaller value.

That is, LGB is trying to split the data in leaf but it can't.
The greater min_data_in_leaf is the more conservative the algorithm is.
Its optimal value depends on the number of training samples and num_leaves.


Using dependency parsing alone will not give you what you need. You may be able to get your answer by interpreting the dependency tree. For instance, in this case ABC-EFG GROUP Inc. is a pobj of between, which could infer that it is party 1. In this particular case, the dependency parsing isn't fully correct and party 2 (Rob Cummins) is difficult to find.

I would recommend you try the following things:

It seems that the parties are either organization or people, so you could narrow down your candidates to proper nouns and you could use NER to see whether any ORG or PER are detected. That will not tell you exactly whether they are the parties, but they could give you useful hints.

In this case, it feels like a plain regular expression could work. Whatever proper noun occurs after between seems to be party 1, and whatever follows and seems to be party 2. Of course, a regular expression would only work for cases written with the same pattern, but it is always wise to consider. Perhaps, you are already a fair amount of your scenarios with a solution that will always work. You could also use the presence of words like "between" and "and" as a feature.

If you don't have a lot of training examples, you could take a look at libraries like mimesis and faker. They allow you to generate company names, addresses, people names, etc.


Best Python NLP library for supervised topic classification

The recommendation is going to be still Huggingface transformers. You extract the features from BERT, then, some dense layers and then, feed them into sigmoid layer with unit equal to number of classes. Pose it as a multi-label classification.


Identify patterns leading to error in structured log files

There are several different ways to model this.
One option is to simplify the problem ignoring time. Fit a machine learning model that find the features that are associated with presence of errors.
Another option is treat this as event-prediction in time series data. In this case, it would be multivariate time series.


What is my training score the mean_train_score or mean_test_score?

The mean_test_score is actually the mean score of the validation step for each fold. The "test" word is probably not well chosen by sklearn in that case, if you want to make the distinction between validation and test.

However, I don't think you are totally finished here, and you should therefore not compare mean_train_score with the test core.

Indeed, the cross-validation phase gives you the best set of hyperparameters (case with maximal "test" score, which is actually a validation score), but you should not keep the corresponding model, especially (but not only) if the training set has few observations or there are few folds.
You should instead re-train your model a last time, with this set of hyperparameters, but over the entire train set (not only the $\frac{k-1}{k}$ cases used during cross-validation). This trained model will give you the training score (over the whole training set) and the test score (over the test set).


It all depends on the specific algorithm you're using. Some of them support incremental learning, while others don't.

For example, in the case of scikit-learn, using fit() more than once on the same model will simply overwrite the model's weights each time (see here for more details).

What you can do however, is look for algorithms that implement the partial_fit api, and use this to retrain your existing models - see the documentation for a list of algorithms that support incremental learning and thus implement this api.

An alternative solution is to look for algorithms that support the warm_start parameter, e.g. LogisticRegression. Note that warm_start might also be influenced by other parameters, so you need to pay attention to their values, too - e.g. in the case of LogisticRegression, warm_start won't work if you use the liblinear solver (which is the default).


Why are eigenvectors produced by np.linalg.eig different than the PCA components stored in the instance of the PCA object?

Two problems:

Your sorting is incorrect:

eVec = eVec1[np.flip(np.argsort(eVal1))]

sorts the rows of the matrix, but you want to sort the columns.  Replacing this with
eVec = eVec1[:, np.flip(np.argsort(eVal1))]

fixes this issue.

The sign of the eigenvectors are sometimes opposite.  (That's fine, being an eigenvector is scale invariant, and while both np.linalg and sklearn use unit eigenvectors, that still leaves a choice of two.  I'm not sure how the packages end up picking one.)


If you had two different functions $z_1$ and $z_2$, I would agree with Piotr: you could have two parallel networks, one for each part of the input, that join together at the end.  This would minimize (though not eliminate) information from $x_{i,1}$ bleeding into the fitting of $z_2$.

But with $z_1=z_2=z$, we'd like to enforce that those two parallel networks are the same.  The idea of using the same weights for different parts of the input remind of convolutions, and I think that will work here: let $x_{i,1},x_{i,2}$ be the columns of a $3\times 2$ input, apply 1-dimensional width-1 convolutional layers instead of dense ones, and add a "Sum Pooling" layer at the end.  (Without writing something custom, you could manage that in Keras with average pooling and manually deal with the factor of 2.)


How to determine which features matter the most?

One common approach is to use Principal Component Analysis (PCA) and drop the directions with less variance. See here for instance:
Using principal component analysis (PCA) for feature selection

The latest version of sklearn allows to estimate the feature importance for any estimator using the so-called permutation importance:
Permutation feature importance
Random forest in sklearn also have other methods implemented to estimate the feature relevance:
Feature importances with a forest of trees


Calculating correlation for categorical variables

According to The Search for Categorical Correlation post on TowardsDataScience, one can use a variation of correlation called Cramer's association.

Going categorical
What we need is something that will look like correlation, but will
work with categorical values — or more formally, we’re looking for a
measure of association between two categorical features. Introducing:
Cramér’s V. It is based on a nominal variation of Pearson’s Chi-Square
Test, and comes built-in with some great benefits:

Similarly to correlation, the output is in the range of [0,1], where 0 means no association and 1 is full association. (Unlike
correlation, there are no negative values, as there’s no such thing as
a negative association. Either there is, or there isn’t)

Like correlation, Cramer’s V is symmetrical — it is insensitive to swapping x and y

def cramers_v(x, y):
    confusion_matrix = pd.crosstab(x,y)
    chi2 = ss.chi2_contingency(confusion_matrix)[0]
    n = confusion_matrix.sum().sum()
    phi2 = chi2/n
    r,k = confusion_matrix.shape
    phi2corr = max(0, phi2-((k-1)*(r-1))/(n-1))
    rcorr = r-((r-1)**2)/(n-1)
    kcorr = k-((k-1)**2)/(n-1)
    return np.sqrt(phi2corr/min((kcorr-1),(rcorr-1)))


Suggestions for improvement? Time series of variation in relative frequency of emotion-related words in academic psychology over time

I am aware this is incredibly basic, but any input/ recommendations would be much appreciated (e.g., is anything unclear?).

In general it is strongly recommended to communicate knowledge in the most simple way possible, as long as it's accurate. Yes a line plot is simple but there's nothing wrong with that, in this case I can't think of any better way to convey the observation accurately.

My main concern is whether I have adequately displayed the data and
whether I can do anything useful to improve (e.g., moving average)?

It perfectly shows the increasing trend which is your main point:

A moving average would not really help in this case: it's useful only when there's too much noise/variation in the data and the general trend is hard to see. Here you apparently have enough data and/or the data is stable enough, so there's simply no need.
You took care of making the plotted value a proportion per 10,000 tokens, and this is also a good idea since an increase in absolute value could be biased: if the total number of abstracts increases, the proportion of abstracts containing the target terms would naturally increase as well even though this doesn't show anything.

Overall this looks very good to me. Just a small note, in your description you could mention that the rate by year is an average: "approximately 0.037 units per year in average".


User-based nearest neighbour implementation in R?

Maybe the best answer for Your question is recommenderlab package for R available from CRAN. Here's the link to vignette. It has CF algorithms implemented and Jaccard coef. for 0/1 user feedback case.

Only similarity(distance) computation in base stats R package there is dist function. But it's limited to few distance measures. I know that when I search for distance measures in R package called vegan claimed to have many methods implemented. But I have never used it.

IMHO most basic recommender systems algorithms are really simple. It worth spending few minutes and implement it to get a better understanding and intuition how it works.
Nevertheless, below my quick and dirty user-based kNN implementation for Your case. The code can be rewritten in better way (Adv.R is really good resource) - but this is only example for learning purpose and I do not take responsibilities for any damages :-)

require(reshape2)

ratings <- read.csv(con <- textConnection('
User,Item,Rating
Anna,Apple,2
Bob,Apple,4
Carlos,Apple,1
Ethan,Apple,1
Gene,Apple,2
Anna,Banana,4
Dana,Banana,3
Ethan,Banana,5
Frank,Banana,5
Bob,Carrot,1
Ethan,Carrot,2
Gene,Carrot,4
'), header = T)

R <- ratings %>% dcast(Item ~ User)
ProductNames <- R[,1]
R <- R[,-1] # only ratings

# tanimoto similarity (~Jaccard coefficiet)
tanimoto <- function(v1,v2) length(intersect(v1,v2))/length(union(v1,v2))

# return top-k similar users
getKNN <- function(R, i, k, sim = tanimoto) {
  similarity <- array(0,length(R))
  for(j in 1:length(R)) {
    if(i!=j) similarity[j] = sim(which(!is.na(R[,i])),which(!is.na(R[,j])))
  }
  idx <- order(similarity, decreasing = T)[1:k]
  data.frame(idx = idx, similarity = similarity[idx])
}

kNNRecommender <- function(R, k) {
  reco <- data.frame()
  for(u in 1:length(R)) {
    userItems <- which(!is.na(R[,u])) #items rated by user
    nn <- getKNN(R,u, k) # get user neighbours

    # rating prediction - wiegted by similarity
    r <- array(0, c(dim(R)[1], k))
    for(i in 1:k) {
      r[,i] = R[,nn[i,]$idx] * nn[i,]$similarity
    }
    r[is.na(r)] <- 0
    userReco <- rowSums(r) / sum(nn$similarity)
    userRecoIdx <- order(userReco, decreasing = T)

    # remove items already rated by user
    userRecoIdx <- userRecoIdx[-userItems]

    # add to recommendations result
    reco <- rbind(reco, data.frame(
      User=u,
      Item=(if(length(userRecoIdx)==0) NA else userRecoIdx),
      Prediction=(if(length(userRecoIdx)==0) NA else userReco[userRecoIdx])
    ))
  }
  reco
}

kNNRecommender(R, 2)


Term for calculated values that lose pertinence when changing scale

Let us think of sales ($S$) as a property of a system.  You could imagine Week 1 to be a system and Week 2 to be another system.  Call the Week 1 system $W_1$ and the Week 2 system $W_2$.  Then,
$$
  S(W_1) + S(W_2) = S(W_1 + W_2)
$$
A property that has this additive character is called extensive.

Now consider the temperature ($T$) as the property with values $T(W_1)$ and $T(W_2)$ for weeks 1 and 2.  Assume that the two $T$ values are identical.  Merge the Week 1 and Week 2 systems together to get another system $W$ for the two weeks together.  Then
$$
  T(W) = T(W_1+W_2)
$$
Properties that follow this rule are called intensive and cannot be added.

These definitions were invented for physical properties and systems and one has to be careful in applying the definitions to general situations.  I'm not sure how much thought along these lines has gone into the social sciences.

For more, see https://en.wikipedia.org/wiki/Intensive_and_extensive_properties.


How to divide MNIST training dataset into subparts?

I think you can just read each line of the file which gives you one data point and write it into an output file. Run a loop for the number of data points you want.


Comparing 2 columns from separate dataframes and copy some row values from one df to another if column value matches in pandas

You may merge two dfs:
df1.merge(df2, on=['id'], how='left')


I have used R, SAS Base and H2O. First, I do not think that H2O seeks to be either R or SAS.

H2O provides data mining algorithms that are highly efficient. You can interface with H2O using several APIs such as their R API. The benefit of combining R and H2O is that H2O is very good at exploiting multi-cores or clusters with minimal effort of the user. It is much harder to achieve the same efficiency in R alone.

The reason why H2O is much faster is that they have a very good indexing of their data and their algorithms are written such that they exploit parallelism to the fullest. See http://h2o.ai/blog/2014/03/h2o-architecture/

R with the default matrix dynamic libraries can only use one CPU core. Revolution R community edition ships with the Intel Math Kernel Library. This allows for some matrix computations in parallel but definitely not as efficient as H2O. For SAS it is a bit harder to say anything considering it's closed source but based on my CPU utilization I would assume that they have a similar approach as Revolution R. Their matrix algebra exploits parallelism but they algorithms are not as efficient as H2O. Their data storage is also not as efficient as H2O.

Lastly, H2O with R comes at a very different price tag than SAS.

Hope this clarifies a bit.


Tensorflow CNN not predicting correctly for a well trained model

I dont think you are rescaling the image after reading it. Because you have rescaled it during training.

img = image.load_img(path_to_file, target_size=(150,150))
img = image.img_to_array(img)

img = img/255 # this must be done.


How to visualise a large correlation matrix?

Try this (note that I didn't add error checking, so it'll crash if your threshold removes all values). Also, I made it an absolute high pass rather than a normal high pass because I assume you'd be interested in strong negative correlation as well? If you're not, just remove the abs() in the filter function.
from numpy.random      import randn
from pandas            import DataFrame
from seaborn           import heatmap
from matplotlib.pyplot import show
from itertools         import combinations

def absHighPass(df, absThresh):
    passed = set()
    for (r,c) in combinations(df.columns, 2):
        if (abs(df.loc[r,c]) >= absThresh):
            passed.add(r)
            passed.add(c)
    passed = sorted(passed)
    return df.loc[passed,passed]

labels = [chr(x) for x in range(65,91)]
corrDf = DataFrame(randn(26,26), index=labels, columns=labels).corr()

#heatmap(corrDf,cmap="YlGnBu")
heatmap(absHighPass(corrDf,0.5),cmap="YlGnBu")
show()

This is the filtered heatmap:

And this is another run but with the unfiltered heatmap:


Removing sensitive information from a dataset while retaining its usefulness for machine learning tasks?

There are lots of solutions to this problem and both of them would allow you to release the dataset. There are in general, two factors that are of importance, the first, whether the method is error prone or not and secondly, whether it is time consuming or not.

The first, being a simpler and dependent solution. If you have a dictionary of names that you maintain, in your database, what you can do is search for whether the given word is a substring of a name (probably through a trie search) and then, check to see if it's a name in your database. If it is, replace it with some string, something even as easy as "Name101" would be easy to accomplish. The same can be done for geographical locations and other possibly sensitive data. This is a non-error prone and time saving method.

If you haven't done the former, there is a solution, however, it's a natural language processing technique called Named Entity Recognition (https://en.wikipedia.org/wiki/Named-entity_recognition), where you can detect if a given word is a name, location or anything that could be worth hiding. However, please understand that this method is not guaranteed to succeed, it may hide most of your data, but it is not guaranteed to hide all of it. This is time saving but certainly error prone.

The last possible alternative is to actually comb through the data yourself and make sure that you've changed all of the names. This is an error prone and time consuming technique.

Ultimately, you can use any of these to accomplish the tasks, but some of them are possible given your method of data collection, or your data pipeline.


sklearn FeatureUnion vs ColumnTransformer

I've tried this with an sklearn builtin dataset rather than yours, but the only difference appears to be the order of the columns.  Switching the order of the elements in the transformer lists produces the same results.  (In both cases, the numeric columns and categorical one-hot encoded columns are separated from each other, but are placed in the order that they appear in the transformer list.)


Why is correlation between my independent variables helping my linear regression model?

It seems that you are dealing with the problem of Multicollinearity. Multicollinearity happens when your predictors are correlated with other predictors in the model.

Moderate multicollinearity may not be problematic. However, severe multicollinearity is a problem because it can increase the variance of the coefficient estimates and make the estimates very sensitive to minor changes in the model. The result is that the coefficient estimates are unstable and difficult to interpret.

You can use adjusted R-squared to see if the new added variable is actually helping your model to better explain the variance.


Are time series algorithms immune towards collinearity?

The fact that some variables may be correlated among each other cannot be the source of your unsactisfactory results. Think about it: let's say that I'm predicting the wheather, having as variables v1 the number of fingers in my hand v2 in my foot (highly correlated variables, right?). Removing v2 from the model may help a little bit because my model won't consider a noisy variable to predict the wheather.

If you are asking about colinearity, I would say that it's better to remove correlated variables, but it's not mandatory. If you are looking for better results in your time series predictor, you may want to understand how forecasting models, such as ARIMA, works in-depth. Often, this kind of models uses the past (with time series "lags", i.e. previous observations) to predict the future. Try to model lag variables (also referred to autoregressive components), moving averages (trends) and seasonality components. Although they can be very correlated between each other (look for ACF plots), they are also suitable variables for forecasting models.


For a categorical variable, if the fitted model encounters a previously "unseen" category, i.e. it did not exist in the training set when the model was trained; then, you should skip that record. You could also opt to throw an error so that you're notified of the existence of new categories and can re-train the model based on that trigger.

If you skip the records with new categories, you would still be able to evaluate the pipeline successfully. That may be the preferred option for a fully automated production setup.

The one-hot encoding creates a new "column" of data for a new category and if this hasn't been used for training, the ML algorithm has no way of knowing how to use the new dummy variable.

In your sample code, each of the categories are encoded into new variables e.g. is_category_a = 0/1, is_category_b = 0/1 , is_category_c = 0/1, etc. If such a model is sent data with a new category "d", then it would be encoded in another column called is_category_d = 0/1 , but the model would ignore this column (or throw an error) since it doesn't expect its input matrix to contain is_category_d.

Lets assume you've fit a linear regression model with the coefficients as:
$$ y = 1 + 2.is\_category\_a + 3.is\_category\_b + 4.is\_category\_c  $$
Now, when you try to evaluate a record with new category = "d", then, the model is not able to use the new dummy variable since it doesn't have a coefficient for is_category_d .

Hence, so you should not "tolerate" new categories and process such records.


How do we decide which kernel needs to be used in SVM?

Without knowing the geometry of your data, determining a kernel is generally achieved through trial and error.  The radial basis function (RBF) kernel is a good starting choice because most data are not linearly separable.  Fortunately training an SVM is fast, so brute-forcing the kernel search is not a terrible method.

For selecting the C and gamma values, we generally use a grid search:

Every cell in the graph represents a SVM model trained on your data given the respective C and gamma parameters.  The color represents its accuracy depicted by the scale on the right.  Note that our scales are logarithmic, so we are able to search a broad solution space while only training 169 models.

From this example, we can predict that the optimal (C,gamma) value combination would be in the region around $( 10, 0.1 )$ or $( 10^{10}, 1e^{-7}) $

We can further refine our C and gamma parameters by "zooming" into the regions with greatest validation accuracy.  In other words, we can repeat the grid search with finer granularity and with boundaries determined by the regions of interest until we find the optimal C/gamma combination.


PCA - Error minimization and Variance Maximization

Yes, the two formulations where you either capture the maximum amount of variance of the data while reducing the dimension or where you try to minimize the distance of the data from the selected subspace (i.e. reconstruction error) are equivalent.


How to train neural network using maximum likelihood principle

This is pretty common.

The last step for a simple deep neural network would be to determine a way to choose values of $\theta$ that maximize the likelihood of your training data. So, what do you do? Typically, you write a log-likelihood function, and then find the values that maximize it.

In fact, it's relatively trivial to prove that minimizing the loss is equivalent to maximizing the likelihood of the input. Typically, in academic papers, you may see people prefer maximizing likelihood simply for the mere fact that some proofs are easier that way.

Example

Consider the Binary Cross Entropy formula $$CE(y,x,\theta) = - \sum_{i=1}^{n} y_{i} log{p_{\theta}}(y \mid x_{i})+ (1-y_{i})log(1-p_{\theta})(y \mid x_{i})).$$  Naturally, you would optimize this with respect to $\theta$.

We can instead choose to take a probabilistic approach and maximize the likelihood of the data under some probabilistic model.  A natural choice would be to use a Bernoulli distribution, such that $$p(y \mid \pi) = \prod_{i=1}^{y_{i}} \pi_{i}^{y_i}(1-\pi_{i})^{1-y_{i}}.$$

Now, the task here is to train a neural network to estimate $\pi$. The likelihood function is simply $$p(y \mid x, \theta) = \prod_{i=1}^{y_{i}} p_{\theta}(y \mid x_{i})^{y_i} (1-p_{\theta}(y \mid x_{i}))^{1-y_{i}}.$$  We want to maximize the function w.r.t $\theta$.

Now, if you take the log, and we find $$\sum_{i=1}^{n} y_{i} log{p_{\theta}}(y \mid x_{i})+ (1-y_{i})log(1-p_{\theta})(y \mid x_{i})).$$  Thus there is literally no difference in minimizing a cross entropy or maximizing the log likelihood.

Therefore, the only difference is the approach. At the end of the day, you're doing the same thing regardless what you choose to call it.


Very simple real-valued time-series dataset for RNN prototyping

You can take a look at some of these datasets in UCI:

https://archive.ics.uci.edu/ml/datasets.php?format=&task=&att=&area=&numAtt=10to100&numIns=&type=ts&sort=attUp&view=table

These are quite straight forward, although they might contain more columns that 4 or 5. But they are quite simple.

Or you can take a look at this:

https://www.kaggle.com/uciml/electric-power-consumption-data-set

This is also based on UCI data.

Overall you can take a look at datasets available on Kaggle, you can probably find something there.

https://www.kaggle.com/datasets

Hope this helps.


Artificial Neural Networks and Efficient Parameter Optimization

If you were fitting a large number of different models, and you had sequences of training data for each different model (with the params already known in those cases), then you might be able to use a recursive neural network (RNN) to provide param estimates for new data sequences. However, that does not seem to be your situation.

As I understand your question, you have a mathematical model with some free parameters, and one set of data that you would like to use to estimate those parameters.

A neural network is not really of any use to you here. That is because the neural network would replace your semi-empirical model with the NN, and the free parameters become the weights of the NN. The trained NN would not match your desired model, but would generically fit to the data. It could be used to predict more function outputs given the inputs - the NN might even do better than your preferred model, which would be an interesting result implying that your model is incomplete.

Often we don't know (or perhaps even care) about an underlying parsimonious model. In this case, ML techniques that fit params of a generic model to data using some objective function like squared error can be very useful tools. However, when we do have a good idea about a simple or explanatory  model, those same techniques are less useful, and instead we can use optimisers direct on the model - in some cases these could be the same optimisers used to train neural networks, provided the model is differentiable.


100% accuracy on both train and test after feature engineering

Yes - Getting 100% accuracy is possible for neural networks compared to tree-based models. Neural networks can learn non-linear relationships through the activation function. Tree-based models are restricted to piece-wise linear relationships.


How to set the Reconstruction error threshold for anomaly detection using autoencoders?

If one considers prediction of anomalous status as binary classification (i.e., if reconstruction error < threshold, classify as normal, else classify as anomalous), one can find the threshold that maximizes some appropriate metric of classification performance (e.g., F-beta) by optimizing the metric over a suitable validation set containing normal and anomalous data. See Malhotra et al., 2016 for an example of how to do this for time series.


Is there a problem of over fitting in my dataset?

It is not possible to tell whether a machine learning algorithm is overfitting based purely on the training set accuracy.

You could be right, that using more features with a small data set increases sampling error and reduces the generalisation of the SVM model you are building. It is a valid concern, but you cannot say that for sure with only this worry and the training accuracy to look at.

The usual solution to this is to keep some data aside to test your model. When you see a high training accuracy, but a low test accuracy, that is a classic sign of over-fitting.

Often you are searching for the best hyper-parameters to your model. In your case you are trying to discover the best number of features to use. When you start to do that, you will need to make multiple tests in order to pick the best hyper-parameter values. At that point, a single test set becomes weaker measure of true generalisation (because you have had several attempts and picked best value - just by selection process you will tend to over-estimate the generalisation). So it is common practice to split the data three ways - training set, cross-validation set and test set. The cross-validation set is used to check accuracy as you change the parameters of your model, you pick the best results and then finally use the test set to measure accuracy of your best model. A common split ratio for this purpose is 60/20/20.

Taking a pragmatic approach when using the train/cv/test split, it matters less that you are over or under fitting than simply getting the best result you can with your data and model class. You can use the feedback on whether you are over-fitting (high training accuracy, low cv accuracy) in order to change model parameters  - increase regularisation when you are over-fitting for example.

When there are a small number of examples, as in your case, then the cv accuracy measure is going to vary a lot depending on which items are in the cv set. This makes it hard to pick best hyper-params, because it may just be noise in the data that makes one choice better than another. To reduce the impact of this, you can use k-fold cross-validation - splitting your train/cv data multiple times and taking an average measure of the accuracy (or whatever metric you want to maximise).

In your confusion matrices, there is no evidence of over-fitting. A training accuracy of 100%* and testing accuracy of 93.8% are suggestive of some degree of over-fit, but the sample size is too low to read anything into it. You should bear in mind that balance between over- and under- fit is very narrow and most models will do one or the other to some degree.

* A training accuracy of 100% is nearly always suggestive of overfit. Matched with e.g. 99% test accuracy, you may not be too concerned. The question is at worst "could I do better by increasing regularisation a little"? However, matched with ~60% test accuracy it is clear you have actually overfit - even then you might be forced to accept the situation if that's the best you could achieve after trying many different hyperparameter values (including some attempts with increased regularisation).


How to compare topics generated from topic modeling from different datasets?

I'm not aware of any standard options to do that, but I could suggest a couple directions. Note that in both ideas it's possible to compare topics individually or globally.
Applying evaluation measures on the instances
In this idea you would apply the model obtained from dataset A on all the instances of dataset B (and of course you can do the converse as well). This would give you a prediction of topic for every instance in B (typically using the max probability topic for the instance) from both model A and model B.
Then you would need a method to match the topics between A and B, i.e. convert topics from A to B. This can be done using the number of instances they have in common, I know that there is some literature and several methods which have been used in supervised evaluation measures for word sense induction, I would guess that there are similar things for topic modeling (not sure). Once the matching is done, various evaluation measures can be used to compare the topics. The idea is to consider the topics obtained from A as the predictions to be compared against the gold-standard topics from B. Note that most measures are symmetrical, so it doesn't matter which one is considered the prediction or the gold.
Comparing words probabilities
I'm not familiar with top2vec but usually it's possible to obtain the top words for every topic together with their associated probability. Assuming this is possible here, the idea would be to compare either:

The top N words for every pair of topics (every topic from A against every topic from B). This can be done very simply by counting the number of words in common between the two top N words: the more the topics are similar, the more they have words in common in their top N.
the whole probability distribution of the words for every pair of topics, using a distance measure such as KL divergence (there are other options). This value will represent how closely related two topics are.


Mismatch between optimum features and grid scores using RFECV?

RFE stands for Recursive Feature Elimination, meaning that the search starts off with a full set of features and with each step some features (in your case, 5) are dropped in an attempt to improve the predictive power of the model. If we take a closer look at sklearn's RFECV reference, it states:

  grid_scores_ : array of shape [n_subsets_of_features]

  The cross-validation scores such that grid_scores_[i] corresponds to the CV score of the i-th subset of features.

The i-th subset of features being referred to is the group of features being evaluated at that particular step of the search. Judging by your output (and the plot), only 12 steps were needed to figure out which 30 features are optimal and that no further improvements could be found using the chosen approach - that's why you see an increase in CV score followed by a plateau. Your assumption in q1 is almost correct, but you are looking at algorithm steps, not features. This is simply an issue of misinterpretation, so is q3.

As for q2, you need to look at support_ attribute of the RFECV object. It returns a boolean mask that corresponds to X.columns (if True, then that feature is selected, if False, then not). You can also look at ranking_, but it provides information on the "goodness" of discarded features only (all selected features are ranked 1). The importances of selected features are model-specific. For RandomForestClassifier, you can look at feature_importances_. For LinearSVC, look at coef_.

Hopefully this answers your question. Good luck with the selection!


Solution: Residual Plots

 What is R2

The definition of R-squared is fairly straight-forward; it is the percentage of the response variable variation that is explained by a linear model.

R2 = Explained variation / Total variation

R2 is always between 0 and 100%:

0% indicates that the model explains none of the variability of the response data around its mean.
100% indicates that the model explains all the variability of the response data around its mean.

Limitations

R2 value has limitations. You cannot use R2 to determine whether the coefficient estimates and predictions are biased, which is why you must assess the residual plots.

R2 does not indicate if a regression model provides an adequate fit to your data. A good model can have a low R2 value. On the other hand, a biased model can have a high R2 value!

Interpreting Residual Plots

A residual is a difference between the observed y-value (from scatter plot) and the predicted y-value (from regression equation line).

A residual plot is a graph that shows the residuals on the vertical axis and the independent variable on the horizontal axis. If the points in a residual plot are randomly dispersed around the horizontal axis, a linear regression model is appropriate for the data; otherwise, a non-linear model is more appropriate.

 Good Fit

 Bad Fit

An unbiased model has residuals that are randomly scattered around zero. Non-random residual patterns indicate a bad fit despite a high R2.

High R2 and Bad Fit Example

Refer fitted line plot and residual plot below. It displays the relationship between semiconductor electron mobility and the natural log of the density for real experimental data.

Here R-squared is 98.5%. However, look closer to see how the regression line systematically over and under-predicts the data (bias) at different points along the curve. You can also see patterns in the Residuals versus Fits plot, rather than the randomness that you want to see. This indicates a bad fit. Always check residual plots!

 Source and References

Stattrek.com. (2010). Residual Analysis in Regression. [online] Available at: https://stattrek.com/regression/residual-analysis.aspx.
Roberts, D. (2019). Residuals - MathBitsNotebook(A1 - CCSS Math). [online] Mathbitsnotebook.com. Available at: https://mathbitsnotebook.com/Algebra1/StatisticsReg/ST2Residuals.html.
Coursera. (2018). Model Evaluation using Visualization - Model Development | Coursera. [online] Available at: https://www.coursera.org/learn/data-analysis-with-python/lecture/istf4/model-evaluation-using-visualization [Accessed 9 Jan. 2020].
Minitab Blog Editor (2013). Regression Analysis: How Do I Interpret R-squared and Assess the Goodness-of-Fit? [online] Minitab.com. Available at: https://blog.minitab.com/blog/adventures-in-statistics-2/regression-analysis-how-do-i-interpret-r-squared-and-assess-the-goodness-of-fit.
Frost, J. (2019). Jim Frost. [online] Statistics By Jim. Available at: https://statisticsbyjim.com/regression/interpret-r-squared-regression/.


how to convert multiple columns into single columns in pandas?

Let's say you have the following data:

import pandas as pd
import numpy as np
df = pd.DataFrame({'values': ['1', '2', '3', '4', '5', '6'],
                  'month1': ['January', 'March', np.nan, np.nan, np.nan, np.nan],
                  'month2': [np.nan, np.nan, 'February', 'April', np.nan, np.nan],
                  'month3': [np.nan, np.nan, np.nan, np.nan, 'May', 'October']})

  values   month1    month2   month3
0      1  January       NaN      NaN
1      2    March       NaN      NaN
2      3      NaN  February      NaN
3      4      NaN     April      NaN
4      5      NaN       NaN      May
5      6      NaN       NaN  October

You can use the following solution:

df['month']=np.nan
for month in [col for col in df.columns if 'month' in col]:
    df['month'].fillna(df[month],inplace=True)

It first creates an empty column named "month" with NaN values, and you fill the NaN with the values from the "monthX" columns, concretely it gives you:

  values   month1    month2   month3     month
0      1  January       NaN      NaN   January
1      2    March       NaN      NaN     March
2      3      NaN  February      NaN  February
3      4      NaN     April      NaN     April
4      5      NaN       NaN      May       May
5      6      NaN       NaN  October   October


How to determine Nonnegativity in Matrix Factorization?

Yes, the parameter nonnegative is for constraining values for both matrices: user features and item features to not have values below zeros while optimizing using ALS algorithm. It drive which solver to use: CholeskySolver or NNLS(conjugated gradient). Both are implemented in Spark mllib.

The flag is of course connected with the input. If You have ratings below zeros and You directly pump all of Your data to model, the prediction of negatives values while You have nonnegative constrain check will be somehow hard :-)

In Your case, what's important is that You have implicit feedback. So, You should set appropriate flag or use trainImplicit method. The easiest way to find out what works for Your case best with the nonnegative flag is to threat it as another hyper-parameter which needs cross-validation check. And for me, the measurement of the model while validating is a more crucial aspect. If You are trying to do top-N recommendations, go for ranking measures like Recall@N, MRR@N or AUC.


From pairwise comparisons to ranking - python

Pairwise comparison models such as that of Bradley and Terry can easily be extended to your case, when you have pairwise comparison probabilities instead of binary outcomes.

Let $N$ be the number of items, and let $p_{ij}$ be the probability that query $j$ is better than query $i$.
Then, the log-likelihood of the Bradley-Terry parameters $\lambda_1, \ldots, \lambda_N$ given probabilities $\{ p_{ij} \}$ is

$$
\sum_{i,j} p_{ij} [\log(\lambda_j) - \log(\lambda_i + \lambda_j)]
$$

This can be reparametrized into a convex function, and the maximum-likelihood parameters can be found by one of many convex opimization methods.

Here is a simple Python algorithm that will find the ML estimate, using a minorization-maximization approach.

import numpy as np

def mle(pmat, max_iter=100):
    n = pmat.shape[0]
    wins = np.sum(pmat, axis=0)
    params = np.ones(n, dtype=float)
    for _ in range(max_iter):
        tiled = np.tile(params, (n, 1))
        combined = 1.0 / (tiled + tiled.T)
        np.fill_diagonal(combined, 0)
        nxt = wins / np.sum(combined, axis=0)
        nxt = nxt / np.mean(nxt)
        if np.linalg.norm(nxt - params, ord=np.inf) < 1e-6:
            return nxt
        params = nxt
    raise RuntimeError('did not converge')

Example usage:

import itertools

# Generating pairwise probability matrix.
pmat = np.zeros((10, 10))
for i, j in itertools.permutations(range(10), r=2):
    pmat[i][j] = (j + 1) / (i + j + 2)

# Estimating Bradley-Terry model parameters.
params = mle(pmat)

# Ranking (worst to best).
ranking = np.argsort(params)

Source: I am an author of a Python library for parameter inference in various statistical comparison models, choix.


Pandas - Avoid boolean result when using groupby()

First filter your results:
filtered_df = df[df['investment_in_millions']>10]

And then group it by company_sector
import numpy as np

sectors = filtered_df.groupby(['company_sector']).agg({"investment_in_millions":np.mean})

You can do it in one line:
sectors = df[df['investment_in_millions']>10].groupby(['company_sector']).agg({"investment_in_millions":np.mean})


I am building on the first part of @Dylan's answer:

For general items like "dogs" pre-trained models are easily available. A good starting point is ImageNet. There are plenty of pre-trained models available for this dataset, e.g. see here for PyTorch. Since ImageNet includes multiple categories for a given item you can check this list to see which indexes correspond to which items and define a mapping (e.g. dogs are classes 151 to 268).

Once you have done that, manually check the cases for which the labels and the transfer model disagree.


How does SelectKBest order the best features?

No, SelectKBest and other *Select* transformers from sklearn.feature_selection do not change order of features, only drop not selected ones.
Anyway, generally, machine learning models do not utilize relative order of a feature.

If you need to check and reorder features, you can use scores_ and/or pvalues_ attributes of a fitted transformer (e.g. SelectKBest) object.

Under the hood those classes use a boolean mask to identify which features to keep. The boolean mask is composed using the first (or the only) set of scores. Those would be t-values in your case. p-values are not used by SelectKBest to select features for now, according to the source code.


If you visualize the lateness of time posting over the revenue, you will see the clear picture and understand the correlation.

What you need is to create the corresponding columns in your tables.


Goodness of fit metric to compare topic models NMF vs LDA

NMF and LDA models produce topic-word and document-topic distributions, so you can compare these models on evaluation tasks of topic coherence (i.e. evaluating topic-word distributions), document clustering/classification (i.e. evaluating document-topic distributions) or information retrieval (i.e. evaluating topic-word and document-topic distributions together).

See some examples in "Improving topic models with latent feature word representations", "LDA-Based Document Models for Ad-hoc Retrieval".


You either need to create a new figure before plotting by plt.figure(), or you need to use subplots. Otherwise, plt.show() will act on the last-created figure.


Gradient Boosted Decision Trees How to Find Prediction of Each Tree?

Sklearn's GradientBoostingClassifier is not implemented using trees of DecisionTreeClassifiers. It uses regressors for both classification and regression. You can read it here:

  GB builds an additive model in a forward stage-wise fashion; it allows
  for the optimization of arbitrary differentiable loss functions. In
  each stage n_classes_ regression trees are fit on the negative
  gradient of the binomial or multinomial deviance loss function. Binary
  classification is a special case where only a single regression tree
  is induced.

This means it will not be as simple as calling predict on the tree estimators. I previously suggested it could be implemented using sklearn's private methods, but as BenReiniger pointed out sklearn has already implemented this for us in the method staged_predict.


First, let's be clear about the fact that all these measures are only for evaluating binary classification tasks.

The way to understand the differences is to look at examples where the number of instances is (very) different in the two classes, either the true classes (gold) or predicted classes.

For instance imagine a task to detect cities names among the words in a text. It's not very common, so in your test set you may have 1000 words, only 5 of them are cities names (positive). Now imagine two systems:

Dummy system A which always says "negative" for any word
Real system B (e.g. which works with a dictionary of cities names). Let's say that B misses 2 real cities and mistakenly identifies 8 other words as cities.

System A gets an accuracy of 995/1000 = 99.5%, even though it does nothing. System B has 990/1000=99.0%. It looks like A is better, that's why accuracy rarely gives the full picture.

Precision represents how correct a system is in its positive predictions: system A always says negative so it has 0% precision. System B has 3/11 = 27%.

Recall represents the proportion of true positive instances which are retrieved by a system: system A doesn't retrieve anything so it has 0% recall. System B has
3/5 = 60%.

F1-score is a way to have a single value which represents the harmonic mean of the precision and recall. It's used as a "summary" of these two values, which is convenient when one needs to order different systems by their performance.

The choice of an evaluation measure depends on the task: for instance, if predicting a FN has life-threatening consequences (e.g. cancer detection), then recall is crucial. If on the contrary it's very important to avoid FP cases, then precision makes more sense (say for instance if an automatic missile system would mistaken identify a commercial flight as a threat).
The most common case though is certainly F1-score (or more generally F$\alpha$-score), which is suited to most binary classification tasks.


How to deal with exception handling in python?

In an interactive program,
If no matching city name found
then shows the input value again and ask the user to enter another value

In data preparation,
select unmatching city names
and find patterns of mistakes
use regular expression accordingly to replace the mismatching city names with the right ones
for example, new_df = df.replace(to_replace ='Londo.*', value = 'London', regex = True)

That is how I would do.


Why do we need convolutions over volume in convolutional neural networks for image recognition?

I'm not sure if the information in the colours is as redundant for real images as you say. I could imagine a scenario where the colours around an edge are relevant, e.g. training a network to tell brown bears from polar bears. In terms of edges and shapes they are very similar, the colour however is not. So while the edges show up in all channels, I would guess that they show up with different contrast, thus propagating colour-information into the classifier.

If you check out this blog where the author visualized some of the learned features, it seems that some features are fairly similar in terms of shape, but not in colour.

There might be a work-around, as you describe it, where you separate shapes and colours, but I think that would involve a lot of prior knowledge and it might be easier to just let the network figure this out during training. This is true even more so if you go to multi-spectral images as they come from EO-satellites for example. There you definitely have features popping up only in certain channels, but not so much in others.


No gradients provided for any variable: ['Variable:0', 'Variable:0']

tw = tf.Variable(initial_value=np.random.randn(), dtype=tf.float32, trainable=True)
tb = tf.Variable(initial_value=0, dtype=tf.float32, trainable=True)

cost =  lambda :tf.reduce_mean(tf.square(tx*tw+tb-ty))
optimizer = tf.keras.optimizers.SGD(0.01)
train = optimizer.minimize(cost, var_list = [tw,tb])

Reference to tw and tb directly in cost function rather than through yhat. It will work.


Applying the same changes to the test set

Most transformations are available as built in classes in scikit learn. You can assemble the classes into a scikit pipeline and have your train data pass through it as part of the pipeline's fit operation. When you are ready to evaluate your test data you can simply run the pipeline's predict operation. This ensures that the test data passes through the exact same transformation workflow as your train data.


Essentially, you only want to look at your test accuracy. The training accuracy is only relevant in comparison with the test accuracy. In your situation, it seems that your test accuracy is roughly the same across the board beside a slight peak around 0.620. So I would pick this as the best value for C.

My answer is related to the one I gave on this post, here is a summary:

  Overfitting is not when your train accuracy is really high (or even 100%). It is when your train accuracy increases and your test accuracy decreases
  is low.
  it is not abnormal that your train accuracy is higher than your test accuracy. After all, your model has an advantage with the train set
  since it's been given the correct answer already.


Feature Normalization/Scaling: Prediction Step

I have learned what the correct answer is: you have to transform your prediction location in precisely the same way you do for the columns of matrix $X$: first, subtract the means element-wise from each component of the prediction location $x$. Second, divide element-wise by the standard deviation. Third, prepend a $1$ to allow for the bias. Finally, perform the dot product with your $\theta$ vector. Since the $\theta$ vector was calculated on transformed data, it's meant to operate on transformed data. It will contain units and scaling appropriate to produce an answer in engineering units. Reference: see Week 2, FAQ, Question 8 in Andrew Ng's Machine Learning course on Coursera. (Login is probably necessary.)


Why does my linear regression model converge to a non-zero gradient value?

those x values are way larger than it should. If you normalize your x and increase the learning rate, it will converge easily. General assumption is that the input comes from a normal distribution so we transform the input.

Here is the code(notice that I only subtracted the mean from x and divided it by its standard deviation and I changed the rate to 0.05):

import random, math
import matplotlib.pyplot as plt
import numpy as np

def linreg(x, y):
    """ Performs linear regression: input x, output y. """
    n = float(len(x))
    m = random.random()
    c = random.random()
    dm, dc = [], []
    rate = 0.05
    epoch = 100
    for run in range(epoch):
        d_m = 0
        d_c = 0
        for i in range(len(x)):
            d_m += (y[i] - m*x[i] - c)*x[i]
            d_c += (y[i] - m*x[i] - c)
        d_m *= -2/n
        d_c *= -2/n
        m -= d_m * rate
        c -= d_c * rate
        dm.append(d_m)
        dc.append(d_c)
    return m, c, dm, dc

x = [i for i in range(400)]
x_mean = np.mean(x)
x_std = np.std(x)
x = [(i - x_mean) / x_std for i in x]
y = [4*i + 3 for i in x]

m, c, dm, dc = linreg(x, y)

print(m, c)

plt.grid()
plt.scatter(x, y)
plt.plot(x, [m*i + c for i in x], color='red')
plt.show()

plt.grid()
plt.plot([i for i in range(len(dm))], dm)
plt.plot([i for i in range(len(dc))], dc, color='red')
plt.show()

the output in my case:

3.999898446167822 2.999938454995635


You have got nothing to lose by framing it as a multi-label detection task. All you have to do is remove the final Softmax layer and format your ground truth as multi-hot, and you can regress multiple labels directly. Mean Squared Error and Sum of Absolute Differences are neutrally biased with respect to false positives and false negatives if the data are balanced, whereas categorical cross-entropy is biased for higher recall and lower precision and usually does not work as well once the Softmax is removed, so choose your loss function accordingly.

Definitely include more objects in your training set that you don't want to detect--this will reduce overfitting by forcing the detection template(s) to improve their specificity.


after grouping to minimum value in pandas, how to display the matching row result entirely along min() value

In case this can help anyone else. Here is a solution that is more computationally efficient.

TL;DR version

If each row already has a unique index, then do this:

>>> df.loc[df.groupby('A')['C'].idxmin()]

If you've already indexed by 'A', then convert 'A' back into a column first.

>>> df2 = df.reset_index()
>>> df2.loc[df2.groupby('A')['C'].idxmin()]

Step by Step explanation:

Step 1.

First, make sure each row in your dataframe is uniquely indexed. This is the default when importing csv data. e.g.

    >>> df = pd.read_csv('questionData.csv'); df
        A       B       C
    0   196512  1325    12.901051
    1   196512  114569  12.926770
    2   196512  118910  12.898335
    3   196512  100688  12.950509
    4   196795  28978   12.780517
    5   196795  34591   12.899411
    6   196795  13078   12.913575
    7   196795  24173   12.876965
    8   196341  118910  12.898335
    9   196341  100688  12.950509
    10  196641  28972   12.780517
    11  196641  34591   12.899411
    12  196346  118910  12.898335
    13  196346  100688  12.950509
    14  196646  28980   12.780517
    15  196646  34591   12.899411

Aside: If you already converted column 'A' into an index, then you can turn the index back into a column (https://stackoverflow.com/questions/20461165/how-to-convert-pandas-index-in-a-dataframe-to-a-column) by doing: df.reset_index()

Step 2.

Use the pandas.DataFrame.idxmin function to retrieve the indices of the minimum of each group.

The semantics of the example below is this: "group by 'A', then just look at the 'C' column of each group, and finally return the index corresponding to the minimum 'C' in each group.

>>> indices = df.groupby('A')['C'].idxmin; indices
A
196341     8
196346    12
196512     2
196641    10
196646    14
196795     4
Name: C, dtype: int64

Step 3.

Finally, use the retrieved indices in the original dataframe using pandas.DataFrame.loc to get the rows of the original dataframe correponding to the minimum values of 'C' in each group that was grouped by 'A'.

>>> df.loc[indices]
    A       B       C
8   196341  118910  12.898335
12  196346  118910  12.898335
2   196512  118910  12.898335
10  196641  28972   12.780517
14  196646  28980   12.780517
4   196795  28978   12.780517

Note: The groupby('A') operation returns groups sorted by A. Thus 'indices' is sorted by A. If we want the original order, we just have to do

>>> df.loc[indices].sort_index()
    A       B       C
2   196512  118910  12.898335
4   196795  28978   12.780517
8   196341  118910  12.898335
10  196641  28972   12.780517
12  196346  118910  12.898335
14  196646  28980   12.780517


How to interpret training and testing accuracy which are almost the same?

Why are the training and test accuracy almost identical?

Nearly identical performance on the training set and test set is a good outcome, it means the model is doing what it's supposed to do. To give an intuitive comparison:

The performance on the training set is equivalent to how well a student can redo the exercises which have been solved by the teacher during class. The student might just have memorized the answers by heart, so it's not a proof that they understand.
The performance on the test is equivalent to how well the student can solve some similar exercises that they haven't seen before in a test. This is a much better indication that the student truly understands the topic.

Does this mean that there is no overfitting?

Yes, it proves that there's no overfitting. To keep with my comparison, overfitting is equivalent to memorizing the answers.
However there can be other problems which bias the result:

The performance on the test set is 2 points higher than the performance on the test set. This probably means that the test is very small, because if it was a large enough sample the performance wouldn't be higher. If the test set is too small, the performance is less reliable (any statistics obtained on a small sample is less reliable).
Accuracy can be a misleading evaluation measure. It only counts the proportion of correct predictions, so if a large proportion of instances belong to the same class then the classifier can just predict any instance as this class and obtain high accuracy. For example here if the majority class is around 63-65%, then it's possible that the classifier didn't learn anything at all. Looking at precision/recall/F1-score gives a more accurate picture of what happens.

[edit] Important note: as Nikos explained in a comment below, my answer assumes that you have a proper test set, i.e. that the train and test sets are sufficiently distinct from each other (otherwise there could be data leakage and the test set performance would be meaningless).


You could look at scikit-learn or Orange module in Python.
Scikit-learn has a SGD classifier and regressor that could do a partial fit data in case of online learning.
In R, take a look at caret package


How to use Non Negative Matric Factorization (NMF)'s transform method to project topics on new text data

This is how you should get your topics and corresponding words. You first get the NMF transformation of your new input and sort its components. The biggest components tell you which features to look at in nmf.components_, which links topics to words (your tfidf features). Finally, you print the k most important words of the topics. Topics and words scores are just calculated by dividing their value by the sum of all values.

    import numpy as np
    # Transform new_feedback to NMF space
    nmf_new_feedback = clf.transform(vectorizer.fit(new_feedback))
    # top 10 topics of the new feedback
    compo = np.argsort(nmf_new_feedback)[-10:][::-1]
    # Connection between indices and words of tfidf
    feature_names = vectorizer.get_feature_names()
    for topic_idx in compo:
        # Current nmf topic
        topic = nmf.components_[topic_idx]
        print("Topic number: ", topic_idx)
        print("Topic score: ", nmf_new_feedback[topic_idx]/sum(nmf_new_feedback))
        # Top 15 words of current topic
        for k in topic.argsort()[-15:][::-1]]
            print("Word: ",feature_names[k], "  Score: ", topic[k]/ sum(topic))


How to run KNN (or other) on nested features? Image metadata

You could try to apply a dimension reduction technique to map all of the variables into 2, such as t-SNE or UMAP (I recommend this one), and then apply a clustering algorithm based on densities, such as HDBSCAN.

Be mind that each run of the HDBSCAN might return different results.
Scale all the variables before starting to apply the algorithms (scale)

Links to these algorithms in R:
UMAP:
HDBSCAN


What is meant by Distributed for a gradient boosting library?

It means that it can be run on a distributed system (i.e. on multiple networked computers).

From XGBoost's documentation:

  The same code runs on major distributed environment(Hadoop, SGE, MPI) and can solve problems beyond billions of examples. The most recent version integrates naturally with DataFlow frameworks(e.g. Flink and Spark).


Building a machine learning model to predict crop yields based on environmental data

For starters, you can predict the yield for the upcoming year based on the daily data for the previous year. You can estimate the model parameters by considering each year's worth of data as one "point", then validate the model using cross-validation. You can extend this model by considering more than the past year, but look back too far and you'll have trouble validating your model and overfit.


If you can, go for some non-linear dimensionality reduction technique. The most powerful are Autoencoders, but you can also use t-SNE or other manifold techniques.

The problem of PCA is that it can extract only latent factors that are linearly associated with your variables. Using non-linear techniques, less variables can let you capture more of the original variance.


Why decision tree needs categorical variable to be encoded?

...why is encoding needed on categorical variables?

That isn't true; decision trees can be built on both continuous and categorical features.  (Why don't tree ensembles require one-hot-encoding? )  Some implementations, however, do not support categorical variables (notably sklearn (for now, update) and xgboost (their politics)).

Now, there is a question of efficiency: the number of bipartitions of the set of categories is exponential in the number of categories, so a complete search of the possible splits is only practical for categorical variables with few categories.
There turns out to be a (surprising?) simplification though: if the underlying problem is a regression with MSE, or a binary classification with cross-entropy or Gini index, then the optimal split can be found by ordering the categories according to their average response value and treating it now as an ordinal variable split.  (That said, still having many categories, especially small ones, might lead to heavy overfitting.)  See Elements of Statistical Learning, section 9.2.4.

Some implementations perform the exhaustive bipartition search but cap the number of categories allowed.  LightGBM and rpart perform the ordered search.  (some R discussion, LightGBM).


how to balance the data set with different number of observations

Clustering the data depends on what your objective is

In this case I am assuming you would want to see similarity among customers.

This would mean that the data should be unique at a customer level. This will help find similarities in some customer attributes.

For ex
Number of transactions
Gender
Age group
Weekend or weekday txns etc

Hence you would need to first aggregate your data at a user level and create features of interest like the examples above and the try to cluster the customers.


XGBoost increase the error when changing evaluation function

If your goal is to minimize the RMSLE, the easier way is to transform the labels directly into log scale and use reg:linear as objective (which is the default) and rmse as evaluation metric. This way XGBoost will be minimizing the RMSLE direclty. You can achieve this by setting:

dtrain = DMatrix(X_train, label=np.log1p(y_train))

where np.log1p(x) is equal to np.log(x+1).

When you want to make your prediction in the original space, you will need to compute the inverse transform of np.log1p, that is to say np.expm1:

predictions = np.expm1(bst.predict(dtest))

If you are just interested into monitoring the RMSLE through the training of your XGBoost which actually is minimizing the RMSE, then you should expect to see the RMSLE behave a little strangely as it is not what you are minimizing.


Is ensemble learning using different classifier combination another name for Boosting?

Ensemble learning combines predictions from multiple learners. Boosting methods are one way to form an ensemble. Stacking is another. The important difference between boosting and stacking (and other ensemble methods) is that boosting applies a number of weak learners sequentially and then produces a final result via a weighted majority vote.

The learners in stacking can also be combined as a weighted average or vote (or by another "meta" model) but they can be more or less independent.

In boosting, each weak learner (usually the same, yes) modifies the data for the next learner in the sequence. E.g. in AdaBoost the weight of each learner and also each sample in the data depends on the misclassification rate of the previous weak learner to the effect that the next learner focuses more on previously misclassified samples.

Boosting methods typically vary by how these weight updates are performed.


Spacy - Ignored (essential) entities during labeling - how important is this aspect?

Yes, it matters. A lot. You need to label every entity you encounter in each sentence. As long as they are non-overlapping, you can add as many entity types and entities per document as you'd like.
First of all, it matters because, in your example, your model receives one example of "Germany" that is a LOC and one that isn't. So, it must mean that the context around "Germany" determines whether it is an entity. Not labeling your entities in some sentences works against the very task you are trying to learn.
In their documentation, they show how to define training data with multiple entities of the same type per document:
TRAIN_DATA = [
    ("Who is Shaka Khan?", {"entities": [(7, 17, "PERSON")]}),
    ("I like London and Berlin.", {"entities": [(7, 13, "LOC"), (18, 24, "LOC")]}),
]


One-hot encode multi-class multi-label sequences

You may need to try cat2vec which converts categorical features into vector representation using Word2Vec approach. Check also this link for multi-feature inputs into LSTM.

For the target y, one-hot is a better technique for NN-based models.


Why do we seem to disregard the weights of the second layer (in blue)?
-----because activation function is after that block, so differential with the o/p y is as good as with the weight.
The red circle, should this not be ∂y2 (followed by ∂y2∂P3)?
-----y2 and y1 is the o/p of the same block i.e. both must be same


How to sum values in a column grouped by another column

Answering the last part of your question:
df.groupby('COUNTRY')['CONTRCNT'].agg(sum)