Arsenal,

We have to reshape the input data to fit in the LSTM's input layer like:

   trainX= np.reshape(trainX, (-1, 16, 4096))
   testX = np.reshape(testX, (-1, 16, 4096))

When one layer's output goes to the following layer's input, the earlier layer must produce a sequence like:

    net = tflearn.lstm(net,128, dropout=0.2, return_seq=True)
    net = tflearn.lstm(net,128)
    net = tflearn.fully_connected(net, num_classes, activation='softmax')

Hope this helps.


Need to prepare the data to Link Analysis project?

When referencing code examples built to analyze phenomena that differs from what I am studying, I find it best to learn as much as I can about the procedures themselves.

In this case, I would look at the procedures defined in the example code you have found, run a few tests on a subset of your own data to make sure you know how the procedures are functioning, and then, if you are satisfied with the results, proceed to use the code to in your analysis (making sure to cite code sources where required or applicable).


Model accuracy when training on GPU and then inferencing on CPU

The results you get will be identical* for identical inputs. So for any practical purposes, the accuracy you get will not depend on whether you use a cpu or gpu.
*up to floating point precision. From this post on the pytorch forum:

Both are implementing the floating point number computation standard.
So they are both correct (even though [they may be] different)


Synthetic time series generation according to some distribution

If I understand you correctly, I think I have a simple way and you do not need to overthink it.

Take a Gaussian (Normal) distribution random number generator, it could even be just downloading a set from https://www.random.org/gaussian-distributions/
Multiply by the desired standard deviation, add the desired mean.
From a certain timestamp, change the standard deviation and mean to a new value to simulate the distribution change. This is the same as concatenating smaller sets with different mean and standard deviation.
See if/when your model detects it.


Multi-label classification for text messages (convert text to numeric vector)

What you want to do is find a vector representation of those strings which are in your $X$ vector. Two such techniques are Bag-of-Words and $n$-grams.
Bag-of-Words (BoW)
This technique will build a dictionary with all the words that exist in your training set. Then we will build a vector with the count of each word in each instance. For example let's consider these three separate instances:

'hi'
'i am bussy'
'how are you doing?'

Then we can see that the following "words" in this training set are: hi, i, am, bussy, how, are, you, doing. So the vector representation of the above strings would be:

[1, 0, 0, 0, 0, 0, 0, 0]
[0, 1, 1, 1, 0, 0, 0, 0]
[0, 0, 0, 0, 1, 1, 1, 1]

There are ways to make this technique more effective by removing tenses from verbs or plurality of words. This is called stemming and should be used with BoW.
n-grams
n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...
The segmentation works as follows:
The String: Hello World
2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld"
3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld"
4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"
Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features. Then you can vectorize the results of the $n$-gram in the same way as BoW.
Term Frequency-Inverse Document Frequency (TF-IDF)
Both of the above techniques can be enhanced with TF-IDF. This removes words that appear too often and do not have much information about the string, for example: the, like, a, ab, is...
For a term $t$ in a string $s$, the weight for that term is given by
$W_{t,s} = TF_{t,s} log(\frac{N}{DF_t})$
where N is the number of strings in your corpus, $TF$ is the number of times the term appeared in the given instance string and $DF$ is the number of documents in which the term appears.


How to fit integer output in neural network

You may check a multitarget model like this one. Basically Keras‘ functional API gives you a lot of flexibility to model multitarget problems.

You have three classes per target, so make sure you use 'categorical_crossentropy' as loss function with three terminal layers.


What is representation in optical character recognition?

The representation step is before the feature extraction step for exactly the reasons that they state.

If you take the full image representation and go right to feature extraction, you have much more data to extract features from (added complexity) and the features you do extract will be much more noisy. This is why one at least reduces the representation to grey level (as the authors suggest).

Feature extraction is choosing properties of the image which you will then perform recognition on. It is important to already have thrown out unimportant details of the image before doing this. That is why the authors, and people in general, reduce to a proper representation before performing feature extraction.


You can understand whether your model is overfitting or underfitting  by the difference in the graph of your train and validation score. If your train score(performance not loss) is low and so is val score then your model is underfitting. On the other hand if your model is overfiiting you will have high training accuracy but your validation score will be low and the train and val graph will be far from each other. A perfect model just has high training score with the validation  curve as close as possible and the two graphs will be very close like the graph you provided.


How to use text as an input for a neural network - regression problem? How many likes/claps an article will get

from what I have understood, it's a method used when attempting to predict the next word in a text.

Not really, Word2vec is a technique to represent the (relative) meaning of words in a way that can be fed into a ML model. You can use them as language models, i.e. for the prediction of the next word in a sequence, but that's just one of the possible uses of it. You can train a model with word embeddings for whatever other task. Word2vec is suitable in this case.

Learning word embeddings creates vectors for words that are similar to each other syntax-wise, and I fail to see how that can be used to derive the weight/impact of each word on the target variable in my case.

I don't know your problem well enough, but I'd say you need additional information. For example: informations on the account such as number of followers, likes, shares/retweets, you name it.

In addition, when I tried to generate a word embedding model using the Gensim library, it resulted in more than 50k words, which I think will make it too difficult or even impossible to onehot encode.

Gensim requires a list of words/tokens (in the form of strings), and the word2vec model will take care of anything for you. You don't have to manually one-hot encode all the words. Don't worry about that.

I will have to one hot encode each row and then create a padding for all the rows to be of similar length to feed the NN model, but the length of each row in the new column I created "clean_text" varies significantly, so it will result in very big onehot encoded matrices that are kind of redundant.

I don't really know what you did here, but I'm pretty sure it's not correct. You don't have to manually one-hot encode anything. More importantly, one-hot encoding rows doesn't make sense, why would you do that?


How does batch normalization work for convolutional neural networks

Method 2:
 This is original batch Normalization as suggested in the paper [Ioffe & Szegedy, 2015].
It is the most common approach. It is very well explained here [d2l.ai]

Similarly, with convolutional layers, we can apply batch normalization after the convolution and before the nonlinear activation function. When the convolution has multiple output channels, we need to carry out batch normalization for each of the outputs of these channels, and each channel has its own scale and shift parameters, both of which are scalars. Assume that our minibatches contain  m  examples and that for each channel, the output of the convolution has height  p  and width  q . For convolutional layers, we carry out each batch normalization over the  m⋅p⋅q  elements per output channel simultaneously. Thus, we collect the values over all spatial locations when computing the mean and variance and consequently apply the same mean and variance within a given channel to normalize the value at each spatial location

I'm not sure what the authors mean by "per feature map", does this mean per channel?

Yes, two trainable parameters per Channel/Feature map.
Method 3:
 This was the idea suggested as "Layer Normalization". [Paper]
 It fixed the issue of BN i.e. dependence on a Batch of data and also it worked for sequence data. But the paper didn't claim anything great for CNN.

We have also experimented with convolutional neural networks. In our preliminary experiments, we observed that layer normalization offers a speedup over the baseline model without normalization, but batch normalization outperforms the other methods. With fully connected layers, all the hidden units in a layer tend to make similar contributions to the final prediction and re-centering and rescaling the summed inputs to a layer works well. However, the assumption of similar contributions
is no longer true for convolutional neural networks. The large number of the hidden units whose receptive fields lie near the boundary of the image are rarely turned on and thus have very different statistics from the rest of the hidden units within the same layer. We think further research is needed to make layer normalization work well in ConvNets

Method 1:
 This is averaging across the Feature Maps on every pixel. I am not sure of its aplication.


How to use scikit-learn normalize data to [-1, 1]?

Scaling between 0 and 1 is simply written for an array of values arr = $[x_{1}, x_{2}, ...., x_{n}]$ as scaled_array = (arr-arr.min())/(arr.max()-arr.min())

But scaling between two values can be inherited from normalisation in Images when values are scaled between 0 to 255 (normally).

You can write scaling as scaled_values = ((val - min)*(new_max - new_min)/(max - min)) + new_min

where new_min, new_max are the maximum and minimum values you are scaling in between and max and min are the maximum and minimum values in your array.

For example if you want to scale values between -1 and 1 for an array [2, 5, 0], it reduces to ((arr - 0)*(1-(-1))/(5-0)) - 1 = [-0.2, 1, -1]

Check this wiki page.

Edit:

As pointed out rightly in comments by @AnkitSeth, MinMaxScalar do have feature_range argument you can pass on. But I am sure the internal job done by feature_range is the same thing as I had mentioned above.


Why would you use word embeddings to find similar words?

It depends on how similarity is defined. If similarity is defined as human-defined semantics, then a synset (i.e., synonym set) is most appropriate. If similarity is defined as frequent co-occurrence, then word embeddings are most appropriate. Even within semantic similarity, there are many approaches beyond synsets.

One advantage of word embeddings over synsets is the ability to automatically find similarity with multi-word term vocabulary. For example, the common word analogy - Man is to king as woman is to queen.


How to replace NA Data using a regression?

Set the column with missing values as your target.

Remove rows where your target is null.

Train a regression model that predicts your target using the rest of your features.

Replace missing values with this regression model's predictions.


Standardizing Vegas odds for a randomForest

This is directly related to the idea of calibrating probability values produced by a random forest. Aside from the link, there is substantial literature on how to do this. The simplest approach basically amounts to fitting a logistic regression on the outputs of the random forest to change the response surface into a logistic form.

Once the predictions have this form, you've demonstrated that you have the knowledge to turn the probability estimate into Vegas odds, so that should be a straightforward process for you.


I think a pseudometric should be fine.

The BallTree's ability to limit the number of distances to compute relies heavily on the triangle inequality, and (perhaps to a lesser extent) symmetry.  And nonnegativity certainly seems critical.  At the very least, the code in sklearn will probably in places tacitly assume symmetry and nonnegativity.

In a pseudometric, the relation "distance 0" is an equivalence relation, and the distance naturally extends to the quotient space, where it is an honest metric.  So I think for BallTrees, it really shouldn't matter; any ball that contains a given point will also automatically contain all its "equivalent" points.


Making Prediction on logistic regression using SAS

proc surveyselect data=work.data method=srs seed=2 outall
samprate=0.7 out=work.data_subset;

data training;
set work.data_subset; *dataset in your work directory;
if selected = 1;
run;

data testing;
set work.data_subset; *dataset in your working directory;
if selected = 0;
run;

ods graphics on;
proc logistic data=work.training descending plots=roc;
class Gender /param = effect ref = first; *categorical variable;
model default = x1 x2 x3 x4 x5
/ link=logit;
score data=work.testing out=work.logisticOoutput;
run;
ods graphics off;

Using proc surveyselect to split the dataset 70% 30%, we can split our dataset into train and test. Then, we can run logistic regression on train data. see the performance on the test dataset.

score data=work.testing

This command is running the regression on the test set. see the result in the output.


Multilingual data handling for text summarisation

There are two basic approaches to summarization: abstractive and extractive.

Extractive summarization is simpler and consists of selecting informative pieces/sentences from text. It is commonly performed using algorithms like TextRank (a variation of PageRank). An example implementation can be found in gensim. There are other approaches, for example check out sumy library.

Abstractive summarization on the other hand consists of summarizing documents with shorter sentences which do not have to come from the input text (an abstract may paraphrase some fragments of text). It is commonly performed using Deep Learning Seq2Seq methods.

Which approach is better for start? Extractive wins by far. First of all, you don't have to train any model, as these methods rely on information found solely in summarized text. Second, deep learning models, even if they exist and can be easily found *, are trained on specific texts, so they might not generalize well across application domains.

  But How should be the approach , when the dataset also has documents
  in french, chinese, hindi etc...

If you have good tokenizers and stemmers for these languages you should be fine with TextRank, as it only relies on word counts (I'm not sure about Chinese though). BTW language detection is by far easier than summarization - just count number of stopwords of different languages that document contains.

*They're not, for example Google published a paper sometime ago on the topic and didn't even disclose model, since the training dataset was proprietary. I was trying to find something for abstractive summarization sometime ago but I couldn't find anything that had good pretraining and worked out of the box.


Keras multi-label time-series classification considering time-series as an input image vector

model.predict(X)

Will return an array of probabilities across your classes, the effective discrete prediction can be achieved by:

np.argmax(model.predict(X))

Accuracy is only one measure of the fit of a classifier, and can be strongly affected by properties of the evaluation set such as class imbalance. Does one class represent a large proportion of your observations? If so, this accuracy measure could be misleading you as to its performance.

Otherwise, with the predictions gathered as shown above, you can further analyze the performance of your network using confusion matrices, muticlass log loss (cross-entropy) and/or others.


NN Making Poor Averaging Fit whilst LGBM Regressor Fits Perfectly

It looks like your neural network has a high bias. There are a couple of things you can do as a first step to attack this:

increase number of hidden layers
increase number of neurons
increase number of training epochs

Generally, neural networks have a lot of hyperparameters and so would be good to optimise using a gridsearch for example.
In terms of your first two models, it is of note that you need to double check they are not overfitting the training set.


On fitting a Poisson distribution to make sense of data

Unlike other graphs, the degree distribution is a function of N. Specifically, the number of nodes with k=2 is constant (4), the  number of nodes with k=3 grows linearly with the length of one side $4l-4$, while the number of nodes with k=4 grows exponentially with the length of a side, $l^2-4l-4$. In the large $l$ (or $N$) limit, the degree distribution is 4.

You can get the exact distribution by normalizing this piecewise defined function (divide by $l^2=N$).


Gradient descent around optimal loss surface

Based on how the gradient descent optimization algorithm works (which you can find extensevily explained in this link), find below the answers to your questions:

Has all the loss surface flat area around the optimal thus causing small gradient updates?

as far as the updated weights approximate the optimal weight values (and this happens on what you call flat area of the loss function surface), the weights updates are smaller and smaller, and this is because the derivative of such loss function on each component becomes smaller (i.e. that nearly flat area is less steep):

source
and in terms of the gradient descent algorithm details below:

source

Most of loss curve when I trained some model shows some drastic decrease in loss and shows the decreasing become slow and the loss graph become flatten. Why the loss updates (amount of the gradient) become small?

this second question (if I got it right you are asking about the learning curves) is a direct consequence of the point above: when the weights become near the optimal values when converging, the amount of weight update become smaller and the loss curve becomes nearly (with some oscillations) flat at around the minimum value, something like:

source


Automated bug finder through machine learning?

Is a difficult task for a machine learning method. I would think that there are methods proper to software development that are much more appropiated for the task.

With machine learning, I could think that process analytics is what you are looking for:

I imagine you may have a dataset like this:

[step1: click in LOGIN, step2: Click in Button A, step3: Click in Button B]  -> [No Bug]
[step1: Click in LOGIN, step2: Click in Button Z]                            -> [Bug]
...

Business Process Analytics is the family of methods and tools that can be applied to these event streams in order to support decision-making in organizations. The analysis of process events can focus on the behavior of completed processes, evaluate currently running process instances, or focus on predicting the behavior of process instances in the future. Process Analytics

Is the most similar process (to your question) in Analytics I know


How to prevent vanishing gradient or exploding gradient?

Vanishing gradient and exploding gradient are two common effects associated to training deep neural networks and their impact is usually stronger the deeper the network.

As you know, two fundamental operations when training neural networks are Forward-propagation and Back-propagation. When we carry out Back-propagation, that is, moving backward in the Network to calculate gradients of the loss function with respect to the weights, gradient values tend to decrease or increase dramatically, the further we get back in the network. This happens in cases when we have activation functions like Sigmoid, or TanH, whose non-linear regions below 0 (i.e. x << -5) and over 0 (i.e. x >> 5) return gradient values in the saturation regions. This means  (x << -5) that neurons in earlier layers will learn in a very slow pace as compared to those layers located later in the network (vanishing gradients problem). Exploding gradients are the other side of the coin, i.e. when activation functions are saturated (with x >> 5), gradient values tend to increase dramatically, making updates to weights unstable, thus not being able to converge.

Some possible techniques to try to prevent these problems are, in order of relevance:

Use ReLu - like activation functions: ReLu activation functions keep linearity for regions where sigmoid and TanH are saturated, thus responding better to gradient vanishing / exploding. You can also use different types like Leaky-ReLu, Randomized ReLu, etc.
Use Batch Normalization (BN): this is another solution you could use in order to make your network more robust against gradient vanishing / exploding, especially if you are using sigmoid or TanH as activation functions. Actually, BN gives you more flexibility during the selection of the activation function for your network. The obtained architecture gets more robust at training, given that it is less prone to diverging due to initialization values or from higher learning rates.
Reduce learning rate: if you increase your learning rate without considering using a ReLu-like activation function and/or not using BN, your network can diverge during training much more easily. By reducing your learning rate you can reduce the chance of suffering vanishing / exploding gradients problem, but your network will take longer to learn. That is why the first two options are located first in the list.
Change your architecture: If you are using Convolutional Neural Networks, for example, and you are suffering from vanishing / exploding gradients, it might make sense to move to a new architecture like ResNETs. In comparison to other networks, these structures connect different layers between each other, i.e. the so-called skip connections, acting as gradient highways, allowing the gradient to flow between the different layers unhindered.
Use proper weight initialization: you could use, for example, Xavier initialization Xavier et al. to reduce the chance of suffering vanishing / exploding gradients. By itself this option does not guarantee you will resolve these issues, but it makes your network more robust when combined with other methods.
Gradient clipping: this can be used when having exploding gradient problem. Firsthand, we select a threshold value, and in case the value returned by the function of a gradient is greater than this threshold, we set it to a different value. You can check more info here.


Gradient Boosted Trees or Neural Networks Using Model Averaging?

I believe in your case, predicting claim is more important than no claim. As you said you have you got 70% Accuracy on the training data, most of the time you might be doing wrong predictions in claim case because of less records, comparatively, what I would suggest is to make the data set balance or select a random balanced data set (20% each of claim and non-clan) and train a model using previous techniques you have applied and test it on the remaining data. If possible use different error measures with respect to your business case such as giving weights to the outcomes. If the accuracy is not improved, you can implement GBM techniques on this data. Most of the times GBM makes better predictions because it increases the randomness (white noise) in residuals by decreasing the similarity among residuals. You can apply many different models on this data and check if the accuracy is improved, eventually we should be able to understand the model to explain someone why they should use this model. Moreover, if you use feature engineered data with different models, there is a high probability that you will do better than the previous models. However, this depends on your business understanding.


Understandable and explainable machine learning model

Neural networks are convenient for automatically building features from the data, but as far as I know this is not compatible with interpretability since we don't know what the features represent. So the only way I'm aware of to obtain predictions that can be explained and analyzed is to use traditional models, and this would certainly require a bit of feature engineering in this case. I would recommend decision trees which are completely transparent in their decision process and can be understood even by non-experts. With this kind of model the idea would be to provide the features which represent the evolution across time, e.g. min/max/average over the last N years, difference current-N, etc., possibly for several values of N.


Why is rpart not splitting this data even when there is gain in gini?

It seems that rpart is actually using accuracy rather than gini in the cost complexity pruning, see e.g. https://stats.stackexchange.com/a/223211/232706
Since your split doesn't improve the misclassification rate, rpart doesn't make it even with cp=0.  Setting cp=-1, the split is performed.


Cross Validation - Why does more folds increase variation?

This has been a discussion for quite a while. For a more theoretical point of view you can find a good summary here.

From a practical point of view I'd look at it as follows. With increasing $k$ two things happen:

Your $k-1$ training folds increase in size
Your validation folds decreases in size

From the first point you can draw the conclusion that your $k$ models become more similar since your training data becomes more similar since you splitt off less data for the validation sets in the $k$-th fold. Which might lead to less between model-variance.

Moreover, since you are training on more data, the relative model complexity (i.e. compared to your data) decreases. In the bias-variance-trade-off graph (taken from [1])

this means we are moving towards the left, i.e. we trade less model variance for more model bias (please note that model variance here has a more general and conceptual meaning than the calculated variance or standard deviation between folds). The reason is that we are fitting a model with constant complexity to more data as $k$ increases, i.e. it becomes harder to learn the training data.

However, we are not only increasing the size of our training set with increasing $k$. We are also decreasing the validation set in size (see point two from above). Therefore, there might be higher between-fold variance with regards to our validation sets. This might be more relevant if the overall dataset carries higher variance or more outliers.

[1] "The Elements of Statistical Learning" by Hastie et al


The meaning of γt−t0 in Reinforcement learning with pytorch

I have no experience with reinformcement learning, however looking at the figure I think I understand what is meant. Gamma is the discount factor, which is taken to the power t-t0, i.e. the number of episodes starting from t. This gives the discount factor for a specific episode, which is then multiplied by the return of that episode, r_t, to get the discounted reward for one specific episode. The total return is then computed by summing all the future rewards for future episodes.


Something similar to what you describe is frequently used in some domains and it is called gradient accumulation. In layman terms, it consists of computing the gradients for several batches without updating the weight and, after N batches, you aggregate the gradients and apply the weight update.
This certainly allows using batch sizes greater than the size of the GPU ram.
The limitation to this is that at least one training sample must fit in the GPU memory. If this is not the case, other techniques like gradient checkpointing can be used.


Why is r squared lowered when adding polynomial features?


How should I construct a binary classifier for thousands of positive data and millions of unlabeled data?

I think the problem you mentioned is sometimes countered as a deep one-class classification problem. You may want to Google for latest researches in this field.

One paper I've been reading is: Learning Deep Features for One-Class Classification.

Available on ArXiv.


Would like to know some Machine Learning applications for Civil Engineering

This is something I've experimented with substantially.  Predicting the compressive strength of concrete based on it's mix design is a great example, and fairly challenging - I used the concrete compressive strength dataset available at the UCI ML repository to help me better understand how to manipulate ML algorithms to get good predictive results.  Here's a guy who took the same data set and used BigML's tools to do an analysis.

I've also used ML to predict payitem bid prices on highway construction contracts to help improve cost estimation.  For example, I set out to determine the unit price for a specific item, like guardrail, that contractors are likely to bid given a contract's location, time of year, total value, relevant cost indices, etc.  It took a lot of time and experimentation, but I achieved surprisingly good predictive results.

Regarding your specific topics of interest, I imagine you'll have to just do a lot of searching and reviewing master's theses on related topics, taking time to learn the mathematical models and theories that they employ.

If you want to know where to get started, my learning path involved the following:

Learning whatever I could about neural networks and machine learning
theory (guys like Jeff Heaton proved helpful, here)
Taking Andrew Ng's Machine Learning course on Coursera.
Downloading and getting to know the R scripting language
Reading whitepapers / theses on applying ML to specific
applications.
Developing a solid, rigorous understanding of basic statistics theory, especially linear regression.

There's a lot of different ways to approach the topic, though.  Websites like BigML.com are a good way to get your feet wet doing ML without having a rigorous mathematical / statistical understanding, but it'll only get you so far.

In the end, you'll really need to devote time to getting your head wrapped around the math and programming in languages like R or Python.  It's also important to remember that while ML seems to favor complexity over simplicity (in many cases non-linear problems benefit from novel and sometimes non-intuitive approaches), sometimes the simplest approach is best.  It's really a question of how accurate the final model really needs to be.


Predicting missing data. Looking for good data predicting technique

there are several ways how to interpolate the missing data depending on how long history record do you have available:

you can compute ration of GDPs r_i (where all r_i sums to 1) in year Y-1 of each state with missing values in year Y; then you know the amount of GDP in year Y which should be spread among missing (GDP_total - GDP_known) and multiply by r_i
a more sophisticated way would be computing the gradient from the timeline Y-1, Y-2, Y-3, ... and you predict a value for year Y for each missing state; then you would again compute ration r_i among missing and distribute remaining GDP accordingly. This regression can be done also backward if you know Y+1, Y+2, Y+3, ... or interpolation from Y-1 and Y+1.


Issue with backpropagation using a 2 layer network and softmax

I think it might be a relatively trivial bug in your cost function for softmax:

J = -(sum(sum((Y).*log(h))) + lambda*p/(2*m))

should be

J = -sum(sum((Y).*log(h)))/m + lambda*p/(2*m)

I.e. for softmax only, you have effectively subtracted the regularisation term from the cost function instead of adding it. Also, you forgot to divide the error term by the number of examples in the batch (and you are taking this average when calculating the gradients)

Your back propagation calculations look correct to me if you correct this miscalculation for J.


Big Note: Unless you do not define what "interesting" is, Machine Learning can not do anything for you. Clustering only tells you which documents are similar to eachother but YOU need to define which cluster is interesting (BTW, clustering in terms of context is called Topic Modeling as each cluster of written text is about some topics).

Now my suggestions:

Topic Modeling

Use a simple LDA to detect topics of each document and cluster docuemnts based on topics. For each topic, print first 5 dominant keyword and if they interest you, then rad the articles.

Keyword Search

Set up a list of keywords you would like to read about. Using a simple TF-IDF model you can find "high score" docuemnts according to your favorite keywords.

Keyword Extraction

You may extract keywords of each document usng RAKE with a score higher than a manual threshold. See if keywords interest you or not. (conceptually close to LDA but here you can extract keywords from each individual document, while LDA makes sense if you have a corpus of text.)

Update

According to the additional info in the comments, the question turned even more towards classic recommender systems.

We have the number of clicks per article and we use it as the interestingness score. Now one may say (a very basic idea to be honest):

Crawl all the articles and Use RAKE to extract keywords from n most interesting articles. Use those keywords as search queries.
Use BM25 algorithm to search those queries and get bm25 score for each article according to queries.
Aggregate those bm25 score with interestingness score (e.g. simply multiply them) and rank articles from most interesting to least interesting.


day is out of range for month import from csv file in python

error time data '8/6/2018 6:45' does not match format ' %d/%m/%Y %H:%M'

Your problem is you have a leading single space. Change the match to '%d/%m/%Y %H:%M'


Python library to compute some metrics for multioutput-multiclass classification task

Late answer!  Sorry if I am repeating things you know...

I am sure somebody has worked on this formally, but least in my short industry experience, it might just be easier to roll your own solution here.  I recently had to do this for classifying businesses (into restaurants, or dentists, or doctors, or amusement parks etc). Because calling a dentist a doctor was a relatively minor error compared to calling a doctor an amusement park, certainly all mistakes are not equal.  My colleague decided to make a different ROC curve for each true class, and call things a "hit" if the classification was right, and a "miss" if the classification was wrong, and so he could leverage typical scikit-learn stuff, but it still only told part of the picture.

With all that in mind, we can still be quantitative.  Perhaps you could decide on a misclassification matrix pentaly, where cell (i,j) is the penalty for calling an "i" a "j", with 0 penalties on the diagonal for correct classifications.  Then you could at least compare models (at one particular set of thresholds) by their cumulative error or squared error etc.

It's already a little tough striking a tradeoff between false positives and false negatives in a binary classification problem - and it depends very heavily on your subject matter and what risks you are willing to tolerate. Do you want to be able to say "we aren't confident and therefore we won't guess"?  This complicated things further for me, because we were trading off accuracy with automation, so "not sures" were also unfortunate, but in a different sense.


In which predictive tasks are the ARIMA model useful?

A popular and widely used statistical method for time series forecasting is the ARIMA model. ARIMA is an acronym that stands for AutoRegressive Integrated Moving Average. It is a class of model that captures a suite of different standard temporal structures in time series data. You can read about it more here.

Based on your problem statement, it definitely looks like ARIMA model can help as you are predictions are based of previous time series data. However, as a rule of thumb in Machine learning, there is no one model that can solve the problem out of the box. There is a saying in statistics that goes something like this :

  All models are wrong, some are useful.

What this implies is, you can start with ARIMA but you need evaluate your model to see if ARIMA can solve the problem for you or you need to explore further.


I have 3 graphs of a binary Logistic Regression that I want to understand better what is happening and learn of a strategy to make the model better

Given a confusion matrix:

            predicted
            (+)   (-)
            ---------
       (+) | TP | FN |
actual      ---------
       (-) | FP | TN |
            ---------

we know that:

Precision = TP / (TP + FP)
Recall = TP / (TP + FN)

  As can be seen in the images, as the threshold is increased there's a
  certain point it stops predicting

Thats not true, it stops predicting accurately only one class. What is udnerstandable because you moved threshold to make all your predictions on the other class.

Dont optimise this yourself, for example random forest will do this for you imiplicitly, (determining cut-off level) or just do hyperparameter optimisation youself.


Is it appropriate to run correlation on continuous y value with discrete x value?

Your feature "Distance from the edge" is represented as a factor (), which means it is interpreted as a categorical variable. You need to make sure that it is interpreted as a numerical (either integer or double) and this should label it appropriately in your plot and also correctly display the distance between them (as they are not evenly spaced).
This might help to get the type right.

Check this answer for a discussion on what correlation methods to use for continuous and discrete variables.


Find ID for the name in the list from csv file in python

This question should be posted in StackOverflow.

Store your CSV into a Pandas DataFrame and operate on that.

import pandas as pd

df=pd.read_csv('nameofyourcsv.csv')
names=('er','rtf')

for each in names:
    print(df.loc[df['Name'] == each, 'ID'].iloc[0])


Logistic Regression: Is it viable to use data that is outdated?

Your thinking is sensible. Indeed, in a perfect world, your training data should be completely representative of the data you'll encounter. However, in practice, you often find that "unrepresentative" data may still have some value.

Ultimately, whatever you do is good if it improves your model, so if using "outdated" data helps, then do it!

Here is what you could experiment with:

Let the data speak

You could compare models using more or fewer data and it might give you an idea of the ideal cutoff point.

Implement "time decay"

Maybe, 1870 data is useful but likely, it's less useful than 1871 and even less useful than last year's. You could weight your training instance based on how old they are so that your recent data points have a bigger impact.

Create time-insensitive features

By this I mean, you could find a way to turn your problem such that the number of playoffs team doesn't matter. Instead of a binary "playoff/no playoff", your problem could be to instead rank your teams, then you can select the playoffs team based on the cutoff at this specific year.

You could also add how many playoff teams there are as a feature so that the learning algorithm is aware of how many spots there are.


In Part 1 of this "tutorial/discussion" ther is a good explanation of the problem and a possible solution using OpenCV built in Haar Cascades  or HOG for Human Detection (this is the key-word you should use to search info about your task).

They are both really fast compared to NN and require little training.

In Part 2 are explained more modern approaches using Deep Learning. A notable mention would be AlexNet.

At the end there is even a tutorial on Setting up a Basic Human Detector with Tensorflow.


Text classification: how to assign 1 or more classes to each document?

I think what you are looking for is multilabel classification.I used mlr package in R ; I followed this link to achieve it.


How to find out the percentage of contribution of a variable for another variable/feature?

To explicitly get the percentage of contribution for one variable to another feature, you can take their (pairwise Pearson) correlation and square it. That is the percentage of variation in one explained by variation in the other. (This is the R-squared of a linear regression, if you just regressed one variable on the other.)

Similarly, if you want to control for other factors (e.g. you want to control for day of the week), you can run a linear regression with just the control variables, then one with those control variables and another one of your predictors. The difference in adjusted R-squared between those models will be how much of the variation in Revenue is explained by the other predictor, beyond what's explained by your controls, and with a slight overfitting penalty for adding another variable.


How Dummy Variables Should Be Modeled In A Linear Regression Model?

Since your task is to predict something, the better variable is the one that gives you a higher prediction accuracy. So you can simply test both and choose the one with which your model performs better.

However, I would suggest considering to engineer your own feature that incorporates information of both variables. For example, you could create three dummy variables: workday, weekend and holiday and include two of them into your model (to prevent falling into the dummy variable trap). Another option would be to only include the interaction terms between isWorkday and weekday.


acf function shows error while fiiting time series

Logarithm cannot handle numbers less than and equal to zero. You can exclude them from your time-series and then add them to the graph as points with different colors (if they are not many that would make the graph ambiguous and interpretation of it hard) or just simply exclude them but have the original graph side by side as well.

EDIT:

As proposed in the comments this would be a better option to avoid zeros:

log(x+1)

or for negative numbers as well:

log(|x|+1)


You can use nltk.util.ngrams for ngram extraction.
See an example below:

To extract bigrams:

dataset = ['sample1', 'sample2', 'sample3', 'sample1', 'sample2', 'sample3',\
'sample3', 'sample2', 'sample2', 'sample3', 'sample1', 'sample2']

from nltk.util import ngrams
import collections

bigrams = ngrams(dataset, 2)
result = collections.Counter(bigrams)
result.most_common()

Out[1]:
[(('sample1', 'sample2'), 3),
 (('sample2', 'sample3'), 3),
 (('sample3', 'sample1'), 2),
 (('sample3', 'sample3'), 1),
 (('sample3', 'sample2'), 1),
 (('sample2', 'sample2'), 1)]

To extract trigrams:

trigrams = ngrams(dataset, 3)
result = collections.Counter(trigrams)
result.most_common()

Out[2]:
[(('sample1', 'sample2', 'sample3'), 2),
 (('sample2', 'sample3', 'sample1'), 2),
 (('sample3', 'sample1', 'sample2'), 2),
 (('sample2', 'sample3', 'sample3'), 1),
 (('sample3', 'sample3', 'sample2'), 1),
 (('sample3', 'sample2', 'sample2'), 1),
 (('sample2', 'sample2', 'sample3'), 1)]

Fourgrams:

fourgrams = ngrams(dataset, 4)
result = collections.Counter(fourgrams)
result.most_common()

Out[3]:
[(('sample2', 'sample3', 'sample1', 'sample2'), 2),
 (('sample1', 'sample2', 'sample3', 'sample1'), 1),
 (('sample3', 'sample1', 'sample2', 'sample3'), 1),
 (('sample1', 'sample2', 'sample3', 'sample3'), 1),
 (('sample2', 'sample3', 'sample3', 'sample2'), 1),
 (('sample3', 'sample3', 'sample2', 'sample2'), 1),
 (('sample3', 'sample2', 'sample2', 'sample3'), 1),
 (('sample2', 'sample2', 'sample3', 'sample1'), 1)]
 ...

You only need to specify the length of your n-grams and what number of repetitions is representative in your case.

Hope this helps!


What is the best model for a recommendation system using implicit ratings?

There are different approaches for your case, but I believe you can employ approaches that are based on the self-attention mechanism. There numerous studies that show they perform better than the latent outcomes that can be achieved with SVD. For instance, you can see the paper of SAIN which its code is also open source. In the paper, you can see that they have shown their approach has better performance than SVD and usual self-attention mechanisim. By the way, it is based on self-attention.
If I want to summarize the method, it uses the contents of the products and the items and finds the interactions between the words, and it outputs the score that a user may assign to a product. It has good performance.
Due to the fact that you have the contents of mails, this approach may help you.


Clustering of variants of similar news articles

You don't want clustering.

What you are looking for is near duplicate detection.

Use minhash. Apparently that is what Google News uses for exactly this purpose.


Interpreting decision tree results after target encoding

I believe after encoding I have lost information about fruit type and I can only say that if fruit_target <= 0.841 then the fruit is rotten if smaller, else not rotten. But then how do i interpret 0.841; what does it mean?

Recall what the target encoding actually is in this example: it is the share of rotten fruits per fruit type, e.g. $75 \%$ of data points with fruit == pear are estimated to be rotten (I say "estimated" because it depends on the type of target encoding whether this an exact number or an estimate).
Accordingly, you can infer from the decision tree that a data point will be classified as rotten iff its fruit type has more than $0.841 = 84.1\%$ rotten data points in the training set.


How can I make seaborn do PairPlot such that it wraps around?

You should not use sns.pairplot which is for plotting all-by-all comparisons.

You should use sns.regplot which can specify particular comparisons to make. Then use subplots can control the tiling.


The object with the lowest mean distance (= the object with the lowest distance sum) is known as the medoid and the basis of k-medoids algorithms such as PAM. Because you must not use k-means with arbitrary distances.


What are the assumptions of linear regression

Here are the links I have found:

https://en.wikipedia.org/wiki/Linear_regression#Assumptions

https://www.statisticssolutions.com/assumptions-of-linear-regression/

http://r-statistics.co/Assumptions-of-Linear-Regression.html


Recognizing circled numbers on a piece of paper

The problem you are describing is known as the Object Localization problem. The main DNN architecture used today includes R-CNN and related algorithms as well as YOLO algorithm. It's a well-studied problem in computer vision. you can found plenty of information online.


Stateful LSTM for time-series prediciton - should each input sequence be shifted by 1 time step or by `sequenceLength` time steps

The second batch should be [t=25,t=49] because a Stateful LSTM must be sequential, because as you rightly say, the state of the network is at the state of t=24, if you then feed t=2 (or anything other than t=25) then the sequencing is messed up.

With stateless LSTM you can create sequences from any starting point, but with stateless LSTM the sequence must be observed.

In subsequent epochs you could change your start index so that different sequences are inputted. Nothing wrong with that. I personally generate a random number between 0 and the sequence length.


Assume you clustering results are store in R. You have the partition id part_id and the real clust_id, here in a small example

df <- read.table(text = "obj,  part_id, clust_id
X, 1, 1
Y, 2, 2
Z, 3, 3
U, 1, 3
V, 2, 3
W, 2, 3"

Than the table will do the work for you

> table(df[,c("part_id","clust_id")])

       clust_id
part_id 1 2 3
      1 1 0 1
      2 0 1 2
      3 0 0 1

UPDATE

For completeness you may also use SQL (as proposed in a comment), a sample query is shown below on a table identical to the data-frame above.

You will see imediately, that there is a limitation because you must define the exact column list in the query - so the query must be adjusted based on the cluster / partition list.

with clust_agg as (
select part_id, clust_id, count(*) cnt from clust
group by part_id, clust_id
)
select * from clust_agg
pivot(sum(cnt) clust  for (clust_id) in
     (1 as "1",
      2 as "2",
      3 as "3"))
order by 1;
, header = TRUE, sep=",")

.

   PART_ID    1_CLUST    2_CLUST    3_CLUST
---------- ---------- ---------- ----------
         1          1                     1
         2                     1          2
         3                                1


logistic regression - why exponent (log ratio) is linear

p is a probability so it is strictly between 0 and 1. So ln(p/(1-p)) is:

for p = 0: ln(0/1) = -Inf

for p = 1: ln(1/0) = +Inf

So now you've rescaled the probability to +/- Inf. In the GLM framework you can use practically any function that scales probability to +/-Inf (see any GLM textbook, or https://stats.stackexchange.com/questions/20523/difference-between-logit-and-probit-models).

How did it come about? Well, mathematicians and statisticians realised they needed a function with the above properties, had a think, came up with a few, decided on the ones that had tractable asymptotic properties, explored how they worked with real data and decided the sensible ones were logit (as above), probit (see references) and a few others. Of course you should always test your model assumptions against your data, and choice of link is just another one of those assumptions, just like assuming linearity in a covariate.


When and where dummy.data.frame would be used?

The original df looks like:
     title salary
1 engineer  51000
2  manager  33700
3    sales  26800
4 engineer  53700
5    sales  36800

Using dummy.data.frame(df), the character column "title" is recoded as "one-hot" (aka as dummy variable). Each column =1 if title=TRUE (or zero otherwise).
  titleengineer titlemanager titlesales salary
1             1            0          0  51000
2             0            1          0  33700
3             0            0          1  26800
4             1            0          0  53700
5             0            0          1  36800

An alternative encoding would be to use "title" as a factor with three levels
df$title = as.factor(df$title).
However, in many cases "one hot" can be digested better by ML algorithms.
Simple example:
title <- c('engineer','manager','sales','engineer','sales')
salary <- c(51000, 33700, 26800, 53700, 36800)
df = data.frame(title, salary)
df

library(dummies)
df2 = dummy.data.frame(df)
df2

OLS model with "factor":
df$title = as.factor(df$title)
ols = lm(salary~., data=df)
summary(ols)

Output:
Call:
lm(formula = salary ~ ., data = df)

Residuals:
         1          2          3          4          5
-1.350e+03  5.684e-14 -5.000e+03  1.350e+03  5.000e+03

Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept)     52350       3662  14.295  0.00486 **
titlemanager   -18650       6343  -2.940  0.09882 .
titlesales     -20550       5179  -3.968  0.05804 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 5179 on 2 degrees of freedom
Multiple R-squared:  0.8992,    Adjusted R-squared:  0.7983
F-statistic: 8.918 on 2 and 2 DF,  p-value: 0.1008

OLS model with "dummies":
ols2 = lm(salary~titlemanager+titlesales, data=df2)
summary(ols2)

Output:
Call:
lm(formula = salary ~ titlemanager + titlesales, data = df2)

Residuals:
         1          2          3          4          5
-1.350e+03  5.684e-14 -5.000e+03  1.350e+03  5.000e+03

Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept)     52350       3662  14.295  0.00486 **
titlemanager   -18650       6343  -2.940  0.09882 .
titlesales     -20550       5179  -3.968  0.05804 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 5179 on 2 degrees of freedom
Multiple R-squared:  0.8992,    Adjusted R-squared:  0.7983
F-statistic: 8.918 on 2 and 2 DF,  p-value: 0.1008

Summary:
Results are the same. Dummies are a different representation of R's factors. Sometimes you need to pass content explicity as dummies. In this case model.matrix() often is useful. (docs)


keras flow_from_directory returns 0 images

Keras generator alway looks for subfolders (representing the classes). Images insight the subfolders are associated with a class.

So whan you work on C:\images\ and you have two classes, say C1, C2, you need to create subfolders C:\images\C1\ and C:\images\C2\. The directory insight the generator function should point to C:\images\.

See this post for the case of image prediction: https://stackoverflow.com/a/55991598/9524424


Great question, @gsamaras! The way you've set up this experiment makes a lot of sense to me, from a design point of view, but I think there are a couple aspects you can still examine.

First, it's possible that uninformative features are distracting your classifier, leading to poorer results. In text analytics, we often talk about stop word filtering, which is just the process of removing such text (e.g., the, and, or, etc.). There are standard stop word lists you can easily find online (e.g., this one), but they can sometimes be heavy-handed. The best approach is to build a table relating feature frequency to class, as this will get at domain-specific features that you won't likely find in such look-up tables. There is varying evidence as to the efficacy of stop word removal in the literature, but I think these findings mostly have to do with classifier-specific (for example, support vector machines tend to be less affected by uninformative features than does a naive bayes classifier. I suspect k-means falls into the latter category).

Second, you might consider a different feature modeling approach, rather than tf-idf. Nothing against tf-idf--it works fine for many problems--but I like to start with binary feature modeling, unless I have experimental evidence showing a more complex approach leads to better results. That said, it's possible that k-means could respond strangely to the switch from a floating-point feature space to a binary one. It's certainly an easily-testable hypothesis!

Finally, you might look at the expected class distribution in your data set. Are all classes equally likely? If not, you may get better results from either a sampling approach, or using a different distance metric. k-means is known to respond poorly in skewed class situations, so this is something to consider as well! There is probably research available in your specific domain describing how others have handled this situation.


Feauture selection for clustering regarding zero-correlated feature

Lack of correlation with other features is not a reason to omit a feature. On the contrary, it is usually a reason to keep the feature because it may provide unique information. Typically, highly correlated features provide redundant information and feature reduction techniques (e.g., Principal Components Analysis) are used to remove the redundancy.

While it is possible that the uncorrelated feature is noise, you should not make that assumption. It could be that the uncorrelated feature is the only one containing information and the other 4 features are all correlated noise.


Confusion with using different classes in neural networks (training vs testing)

You are correct, most models in general don't perform well when asked to predict something they haven't seen before. Normally when you are splitting the data into Train and Test datasets, you want to include all examples in each dataset. In the train_test_split library, stratify is used to do this for you.
In the real world, sometimes the new class is not available during training, such as if you are predicting future data that may have a new label. When this occurs, typically your model won't categorize that correctly until you have built up several examples of that data and then rebuilt your model including these new examples in your dataset.
In the meantime, you can try to create a category named "other" and train your model to use this category when it can't categorize into a known class.


What happens when the vocab size of an embedded layer is larger than the text corpus used in training?

tf.keras.layers.Embedding(..., embeddings_initializer="uniform"*,..., *kwargs)

All the weights are initialized with the init strategy
All learn the optimum values  with the backprop
Weights for which there is no input will have zero output every time, hence no learning.
Hence these extra weights will remain at their initialization value

You may check these extra weights before and after.
weight = model.layers[0].get_weights() # Save before training
history = model.fit(x, y)

# These two should be same
weight[0][-1]  # Last weight - Before
model.layers[0].get_weights()[0][-1] # Last weight - After


Random accuracy results on validation set (Binary classification)

Ah, I think I've found the problem with the discrete values at least.  In your random batching, you overwrite X and Y to be the remainder of the dataset here:

if m % batch_size != 0:
    X = shuffled_X[num_batches * batch_size : m,:,:]
    Y = shuffled_Y[num_batches * batch_size : m,:]

These datasets only have size $200-3*64=8$.

It's surprising that the training accuracies work out nicer then; is the training dataset perhaps a multiple of 64?

(After you get this fixed up, your original question is still interesting; maybe it should be posted as a separate question?)


Should the bias value be added after convolution operation in CNNs?

Short answer: the bias is added once after the convolution has been calculated.

Long answer: discrete convolution that you see in CNNs is a linear function applied to pixel values in a small region of an image. The output of this linear function is then jammed through some nonlinearity (like ReLU). For a region $\mathbf{x}$ of size $i \times j$ of an image and a convolutional filter $\mathbf{k}$, and no bias term, this linear function $f$ would be defined as:

$$
f(\mathbf{x}, \mathbf{k}) = \mathbf{x}*\mathbf{k} = \sum_{i,j} k_{i,j} x_{i,j}
$$

Without a bias term, this linear function $f$ must go through the origin. In other words, if $\mathbf{x}$ or $\mathbf{k}$ is all zeroes, the output of $f$ will be zero as well. This may not be desirable, so we add a bias term $b$. This gives the model more flexibility by providing a value that is always added to the output of the convolution, regardless of the values of $\mathbf{x}$ and $\mathbf{k}$ -- in other words, it's the intercept value.

$$
f(\mathbf{x}, \mathbf{k}, b) = b + (\mathbf{x}*\mathbf{k}) = b + \sum_{i,j} k_{i,j} x_{i,j}
$$

If this value was added to each entry of the convolution, it would not achieve its purpose as $f$ would still necessarily go through the origin.


Changing the value of a column based on the condition on another column

There is a way that is not optimum for performance, but clear to read and understand. Also, you can use more complex logic and update with different values as you wish.

def myfunc(row):
    if row['incident_zip'] in [this is a list of zip codes]:
        return new_value
    else:
        return row['borough'] #return old value

data['borough'] = data.apply(myfunc, axis=1)

If your logic of update is quite simple, then you can do it like that

mask = df['incident_zip'].isin([this is a list of zip codes])
df[mask]['borough'] = new_value


Creating a Feature to determine popularity

You need to put your shoes into business people and communicate with people from that department on regular basis, trying to understand what the business's needs are. In e-commerce your second metric is Conversion Rate and is defined as:

CR = Orders/Views*

Please note there that there are two variations of Views. One that comes from Product List Page (aka PLP) that is mostly know as Impression, or the one comes from Product Detail Page(aka PDP) is where the user actually lands on the product detail page. The latter is more common in the CR calculation.

Your first one i.e. the popularity, that somehow I liked, can be used. Only be careful that it is going to be affected by number of items available in that category.

In general, I find it useful to think about edge cases when coming up with new measures and to see whether the resultant is desirable. What do I mean by edge cases, let's take your popularity score, by your definition, for example. Imagine the following two scenarios:

popularity item 1 = 1/2 (very limited items in this category where item 1 is)
popularity item 2 = 1000/2000 (large number of items in this category where item 2 is)

What you see here looks quite normal by the original definition from mathematics perspective. However, what it means from recommendation/business perspective that you have similar scores from two items from two different categories. Here the business may jump in and say looks we have a lot of more assortments in the category where item 2 is and often items in this category are in the warehouse or have faster delivery etc. etc., then we prefer our products from there to be recommended first. This is the point you realize that while results seem reasonable, at least according to the popularity definition, falls short on certain aspects.

Other ideas: I noticed you are leaning towards using multiple scores. That is a great idea. Having multiple factors may catches these corners and address multiple business needs at the same time. Like having both CR and popularity. Or may think of each item's profit contribution to the business and take that into consideration. Imagine you have a score in place that gives equal chances of recommending an iPhone or Wireless Mouse, however iPhone contribute way more to the business (it is more profitable), then you may want to push for that factor too. You may also like to look at Impression I mentioned if it is available from your tracking. Item's rating, delivery, Add_To_Cart, Wish_List, or Time_Spent_on_Item are other things to think about. Check this post to get some other ideas how things are roughly done in Amazon. Hope these help.


How to interpret axis of Histogram and distribution curve?

First figure shows Frequency(Y-axis) distribution over varied values of Line data(X-axis). Similar information gets conveyed by your second figure as well, but second one provides a deeper insight to frequency fluctuation over smaller bins of Line data. Additionally in second figure, various types (Lognorm, Exponential, etc.) of distribution gets line traced on the population distribution. This additional line helps to figure out co-ordinates on this grid where there is a spike or dip in Frequency values based on Line data.
Every distribution has a spread or else mathematical computations won't be feasible for any kind of inference/analysis. For Line data, in both your plots, X-axis takes that responsibility. Unit (m specified by you) associated to values shall entirely depend on nature of your Line data feature.

Also, I remember writing a series of articles and creating a GitHub repository few years back on Statistical data visualization with Python and Seaborn which might be of some help. So please feel free to check, if required.


training and Predicting probabilities using logistic regression model

Since your output variable is continuous it can't be used directly in classification. There are some steps through which you can achieve inference.

Based on the probability, convert your 'y' (output) variable into a class label.

for example, if your class A & class B has a probability of 0.6, 0.4 respectively. convert it into a single variable y and it should have the value that represent the respective classes.

Now you have the features mapped to their respective class label. Now we can train the model with the available data.
After training the model, the prediction will give you class probabilities on your test data.

Note : Use softmax in case of neural networks to squash the output values to probabilities.


Amalgamating multiple datasets with different variables coding

I don't know of an automated tool for this, but as a suggestion, you could look to using feature selection or correlation of all variables in a particular dataset against all other variables.

It depends on the direction that you are wanting to take your analysis later, but the second method will enable you to see variables that are highly correlated, and so might be good candidates for checking that they are duplicates?

Also, not sure if you are aware, but there are a few R packages (memisc is particularly popular) that enable you to read SPSS files directly and then work on the data directly.

Sorry wont save you months of work though ... but might save you days ...


Loading large numpy array (DAIC-WOZ) for LSTM model causes Out of memory errors

Looking at your code, it looks like you're using Keras w/ a Tensorflow backend -- or Tensorflow functions using its Keras submodule. In that case, you can train your model using its fit_generator function.
As the name implies, if you convert your training data into a generator, it will substantially reduce the memory cost needed to train your model. Here's an example of training a Keras model using fit_generator and how you can preprocess your data to utilize it.


Everything looks good. However, what I doubt about is the myDimensionReduction function. When you apply the PCA on the test data, you have to multiply $X_{test}$ by the matrix $C$ that has been generated from the training data, not by the test data. This is to stress that the test data should not be used in any step of the training process. If this is what the myDimensionReduction function does, that is alright. Otherwise, if this function creates the matrix $C$, then it is not right.

I know you didn't ask for the library, but I think it makes sense in here: The whole philosophy of scikit-learn is to create objects that correspond to algorithms, use the training data with the fit method and apply the transform method to the test data.


How to measaure the similarity between two series?

A classical approach for time series similarity computation is Dynamic Time Warping (DTW).

From your description, it may suit your use case:

  In general, DTW is a method that calculates an optimal match between two given sequences (e.g. time series) with certain restriction and rules:

  Every index from the first sequence must be matched with one or more indices from the other sequence, and vice versa
  The first index from the first sequence must be matched with the first index from the other sequence (but it does not have to be its only match)
  The last index from the first sequence must be matched with the last index from the other sequence (but it does not have to be its only match)
  The mapping of the indices from the first sequence to indices from the other sequence must be monotonically increasing, and vice versa, i.e. if $j>i$ are indices from the first sequence, then there must not be two indices $l>k$ in the other sequence, such that index $i$ is matched with index $l$ and index $j$ is matched with index $k$, and vice versa

Some alternatives to DTW can be found in this literature review on time series dissimilarity measures.


Discovering cross category sales using transactions history (Clustering?)

Note that your data can be re-ordered to look like this:

Transaction ID | Items
1                {Apple, Banana, Carrot}
2                {Carrot, Ice Cream}

This kind of data set is trivial for association rule mining. A very simple and well-known algorithm of this kind is the Apriori. I'm certain there are packages for executing this algorithm in R.

For the restriction of "discover cross category sales only", you can just post-prune the generated rules, ie. let the algorithm generated inter-category sales and then remove those later, which should be trivial.


Multiple Object recognition in image using deep Learning

It falls under the multi label object detection problem. There have been lots of advancements done in this field and you should definitely look at implementations like :

Yolonet: https://pjreddie.com/darknet/yolo/
Squeezenet: http://songhan.github.io/SqueezeNet-Deep-Compression/


What visualization I should choose for Monte Carlo simulations in timeline events?

Instead of  (or additionally to) histogram you may check the density plot of the distribution of the intervals between the
events. In the r snipped below the process id identified with  ID and the interval length is INT_LEN .

densityplot(~INT_LEN,  groups = ID,
data =  df,
scales=list(x=list(rot=90, cex= .9),y=list(cex=.9)),par.strip.text=list(cex=.8), plot.points = F,
ylab="density", xlab="Interval between Events", main=paste("Interval Length Density per Process"  ) )

This show the distribution (in my simple case there are two different distribution). One group of processes has a normal distribution of event with a mean of 1,2. So you may argue that there are 50 events per minute on average.

But this may not be true as the time sequence is lost in this view.

To get the time series information I select a basic interval length, where the distribution of the event doesn't play a role any more.
I use a minute, but if this is to coarse go down. The point is there is no difference to get all events at the beginning of the interval (minute)
or uniformly distribute within the interval.

This selection allows to aggregate the data on the interval level and present it as a time series. The graph below shows the count of event per minute.

xy <- xyplot(CNT ~ MINUTE_NO , groups = ID,
data =  df ,    type=c("o","g"),
scales=list(rot=90, x=list( cex= .9 ),y=list(rot=0, cex=.8, alternating = c(1,1), tck=c(1,0), relation = "free")),par.strip.text=list(cex=.8),
ylab="event count per minute", xlab="time", main=paste("Time Series - Event Count per Minute"  ) )
print (xy)

Here you can spot a possible trend in the distribution of the events.


Train model for predicting events based on other signal events

For somebody new to ML you're starting with a quite complex problem!
Your first job is to formalize the problem: what exactly do you want to predict and from which information?

Do you want to predict whenever a specific signal will occur, or whenever any one of several specific signals occurs, or detect any unusual pattern...? In the first two cases your task is supervised classification, detecting unusual patterns would probably be unsupervised outlier detection.
In the case of supervised classification, which seems more compatible with your description, you're going to use the specific signal to detect as target variable: your features are made of a sequence of data until time $t$, and the target variable is 0 or 1 depending on whether the signal occurs in the next 30 mns.

There's a lot of good advice that  you can use in the related question you mention. You could consider sequence labeling methods, but there are certainly a lot of other options.
I'm not so sure you will need dimensionality reduction in your case: the signals you're using as features are binary so your features are much less complex than in the related question.


Try predicting log-probabilities that would give you bigger seperation (i.e. values between -inf and 0) I Would clip these values (so that you dont take log from 0 which would be -inf but something like if 0 than log(0.0001)
For example:

y_pred = estimator.predict_proba(X.drop(columns=self.drop_columns))
log_prob = lambda x: np.log(np.clip(x, 0.001, 0.999)).reshape(-1, 1)
y_pred = np.hstack([log_prob(y_pred[:, 0]), log_prob(y_pred[:, 1])])


Extracting tokens from a document: applying Deep Learning or Classification?

For your case, it might be easier to use text classification or regular expressions.


One major difference is that the F1-score does not care at all about how many negative examples you classified or how many negative examples are in the dataset at all; instead, the balanced accuracy metric gives half its weight to how many positives you labeled correctly and how many negatives you labeled correctly.

When working on problems with heavily imbalanced datasets AND you care more about detecting positives than detecting negatives (outlier detection / anomaly detection) then you would prefer the F1-score more.

Let's say for example you have a validation set than contains 1000 negative samples and 10 positive samples. If a model predicts there are 15 positive examples (5 truly positive and 10 it incorrectly labeled) and predicts the rest as negative, thus

TP=5; FP=10; TN=990; FN=5

Then its F1-score and balanced accuracy will be

$Precision = \frac{5}{15}=0.33...$

$Recall = \frac{5}{10}= 0.5$

$F_1 = 2 * \frac{0.5*0.33}{0.5+0.3} = 0.4$

$Balanced\ Acc = \frac{1}{2}(\frac{5}{10} + \frac{990}{1000}) = 0.745$

You can see that balanced accuracy still cares about the negative datapoints unlike the F1 score.

For even more analysis we can see what the change is when the model gets exactly one extra positive example correctly and one negative sample incorrectly:

TP=6; FP=9; TN=989; FN=4

$Precision = \frac{6}{15}=0.4$

$Recall = \frac{6}{10}= 0.6$

$F_1 = 2 * \frac{0.6*0.4}{0.6+0.4} = 0.48$

$Balanced\ Acc = \frac{1}{2}(\frac{6}{10} + \frac{989}{1000}) = 0.795$

Correctly classifying an extra positive example increased the F1 score a bit more than the balanced accuracy.

Finally let's look at what happens when a model predicts there are still 15 positive examples (5 truly positive and 10 incorrectly labeled); however, this time the dataset is balanced and there are exactly 10 positive and 10 negative examples:

TP=5; FP=10; TN=0; FN=5

$Precision = \frac{5}{15}=0.33...$

$Recall = \frac{5}{10}= 0.5$

$F_1 = 2 * \frac{0.5*0.33}{0.5+0.3} = 0.4$

$Balanced\ Acc = \frac{1}{2}(\frac{5}{10} + \frac{0}{0}) = 0.25$

You can see that the F1-score did not change at all (compared to the first example) while the balanced accuracy took a massive hit (decreased by 50%).

This shows how F1-score only cares about the points the model said are positive, and the points that actually are positive, and doesn't care at all about the plathero points that are negative.


Any issue with "overlapping" sliding windows in time-series data analysis?

In general I don't see anything wrong with overlapping windows, it might make perfect sense depending on your task. In fact some learning models (e.g. for sequence labeling) do use features based on past data points, which is conceptually similar to having overlaps between them.

However you need to be careful about the fact that this makes data points depend on each other, so of course the preprocessing must be done separately for the training and test set.


I have hourly data of a metric for 15 days, Can i predict the outcome values for same metric for the next 15 days?

If its a timer series data, then according to me you should go for RNN based model, typically LSTM.

Also you are looking an output for next "n" days, in this case you have to develop "Multi-Step LSTM Models".

  Please refer a very good link on TimeSeries problems

  Also you can refer TensoFlow code of very basic timeseries problem on
  github.

Note: Please modify code for Multi Step problem.


Keras Early Stopping: Monitor 'loss' or 'val_loss'?

Validation loss. Overall training loss should keep decreasing so monitoring it isn't as meaningful. If your issue is noise in the validation loss, increase patience.


ValueError: Found input variables with inconsistent numbers of samples [43,19]

The line that gives the error is:

pls.fit(X_train, X_test)

The second argument in fit should be your labels, the value you want your model to be able to predict. Instead you are inputting X_test, which contains the same features as X_train but with another length, causing your error. You want an y_train with your target value and make it look like this:

pls.fit(X_train, y_train)

But to get this value you will need to get your labels from you dataframe. It is hard to know how without knowing the data, but this might give you an idea:

X = df.drop(['name of label column'], axis=1).iloc[:, :-1].values
y = df['name of label column'].iloc[:, :-1].values

# Split data to train and test on 70-30 ratio
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

pls = PLSRegression(n_components=2)
pls.fit(X_train, y_train)


Patch wise training vs Full Convolutional Training in semantic segmentation

In Patch wise training, the CNN is trained on a small patch of the image instead of the whole image. For example, if the input image is of size 6000-by-6000 pixels, it would be difficult to train a CNN on it. You could downscale the image but then you would lose relevant information. Instead, if you perform patch based semantic segmentation, the model would pick up patches from the image, say of size 200-by-200 pixels and train on those patches as if they were separate images.

This feature retains information as well as provides good accuracy.A link to a paper discussing the relevance of patch-based CNN can be found here.


Can we use both validation loss and cross validation at the on CNN?

Yes, you are right validation loss is used to show when an ML model is overfitting it's training data. However this only occurs when your training data doesn't fit your model, especially when it's quite low. cross validation is iterating your validation data over your training data because there is no sufficient data to account for both training, validation and test data.
Thus its possible to used both of them to evaluate and improve your models performance although they are different concepts.


NLP: what are the advantages of using a subword tokenizer as opposed to the standard word tokenizer?

Subword tokenization is the norm nowadays in NLP models because:

It mostly avoids the out-of-vocabulary (OOV) word problem. Word vocabularies cannot handle words that are not in the training data. This is a problem for morphologically-rich languages, proper nouns, etc. Subword vocabularies allow representing these words.  By having subword tokens (and ensuring the individual characters are part of the subword vocabulary), makes it possible to encode words that were not even in the training data. There's still the problem with characters not present in the training data, but that's tolerable in most of the cases.

It gives manageable vocabulary sizes. Current neural networks need a pre-defined closed discrete token vocabulary. The vocabulary size that a neural network can handle is far smaller than the number of different words (surface forms) in most normal languages, especially morphologically-rich ones (and especially agglutinative ones).

Mitigates data sparsity. In a word-based vocabulary, low-frequency words may appear very few times in the training data. This is especially troublesome for agglutinative languages, where a surface form may be the result of concatenating multiple affixes. Using subword tokenization allows token reusing, and increases the frequency of their appearance.

Neural networks perform very well with them. In all sorts of tasks, they excel: neural machine translation, NER, etc, you name it, the state of the art models are subword-based: BERT, GPT-3, Electra,...


In a neural network, is it possible to gradient descent with more than one input?

As to what I understand from your question, yes you can have multiple features (variables/input) and use gradient descent to minimize the loss function which has the input variables. Here is an article that gives you a basic vector calculus idea as to what exactly does a gradient is and what it does in the algorithm!
https://towardsdatascience.com/wondering-why-do-you-subtract-gradient-in-a-gradient-descent-algorithm-9b5aabdf8150


What is neural structure learning in tensorflow?

It seems that you actually mean "neural structured learning".

From the Tensorflow webpage on their neural structured learning framework, it seems it is just an umbrella term to two types of regularizations: Neural Graph Learning and Adversarial Learning.

Basically you add an extra term to your training loss where you force the internal representations for certain input to be close to the representation of its "neighbors".

The neighborhood is expressed either as a graph (hence leading to neural graph learning) or as a neighbor/no neighbor relation (hence leading to adversarial training).

Therefore, neural structured learning refers to normal neural networks that are trained with extra knowledge on what inputs are "close" to each other.


What are the key differences between a MLP with lagged features and a RNN

Let's assume you have 3 features + considering 5 timestamps for each feature. That means you have 15 features as X values.

Machine learning techniques other than RNN can utilize the features as individual columns i.e. 15 columns.

RNN can use these data as 5 arrays containing 3 feature in them. We can feed this 5 array in RNN cell at each time step for a single record. The way RNN cell works when we feed second-time stamp data or subsequent time stamp data it gets processed important information from last time stamps.


Which approach to select category based on keywords

Does anyone have a better suggestion or are there just complete algorithms for this already available?

Apparently you want to do information retrieval (IR) but without the information part: normally an important part of the IR process is the set of documents that the user is searching for (for example text, images video). This is important because knowing the documents which correspond to a query gives semantic information about the keywords.

Now, if you only have the keywords available then there's no semantic to help with the clustering. So you are right that the only thing you can use is the spelling. i assume that by " simple fuzzywuzzy" you mean "fuzzy matching", i.e. using string similarity measures. I can think of two options:

You can compare each keyword with each other with some string similarity measure such as Levenshtein edit distance, Jaro, or characters n-gram based Jaccard, cosine etc.
You can represent each keyword as a vector based on the char n-grams in the word, then you can apply clustering with k-means for example.

Topic modeling techniques wouldn't work because you don't have large text documents.


How to build a model on a dataset having 40% missing values in most of the variables?

Generally speaking, you should investigate the process by which your values are missing and try to deal with it. I assume you checked that :

There is no meaningfull way to fill those missing values. Sometimes, typically with companies data that often represent amount of money, missing values means 0$. For exemple values missing by block may correspond to a tax form they didn't fill because they don't pay it and most features could be set to 0.
Similarly you don't have enough data to fill the blanks. Sometimes you can do missing value imputation based on what information you have. Specifically with companies data, you might be able to calculate financial ratios based on other metrics. Basically, you might want to check that your missing values do not correspond to some calculation that hasn't been made (then either correct that calculation or do it yourself in your pre-processing). There are more advanced imputations techniques, but I don't think they would be usefull when values are missing by blocks.

If you checked that, you can proceed to the next step : modeling. How you build your model depends on what is its purpose, what you want to do with it.  Here, you should ask yourself some practical questions about those 40% records with missing values.
The first one is : do you actually need to output something for those instances with a lot of missing values ?
In some cases, notably when the output is public (or can be contested individually) it might just be best to output nothing or something along the line of 'we do not have enough data' for those instances. Practically that means removing them from your data set and building your models on your other instances.
On the contrary, when you look for performance (and not explainability to someone) you might want to use that pattern as an info. The best way to do so would be to introduce some feature that count the proportion of missing value for that instance. It might help a general model (see below) deal with that info.
If you need to output something, because you need some sort of an average, or because management asked for it, then consider the second question : should you make multiple models ?
One approach is to use just one model. Some models, mainly trees, can handle missing values. More advanced methods like XGBoost (eXtreme Gradient Boosted trees) are even considered state of the art for tabular data like companies data. Generally speaking this works (and might works very well with XGBoost). However, having done that myself, you'll end up with something quite hard to explain and overall you'll obfuscate that a significant chunk of your population is missing data. More specifically with data missing in blocks like yours, you'll probably end up with a very early split in your tree and actually have two models inside of one, with possible bizare interactions.
The alternative is to devise two models: one, very performant (say XGBoost), on your data with few missing values and another, way simpler on your other data. This will help you deal with instances with lot of missing values quite efficiently. For companies data with lots of missing data, you might just want to build a simple table of average of the outcome by your main descriptors (size, industry, geographical zone). This will leave you with most of your time to deal with the more complex model and its explanation (which is probably why you are paid for and is way more fun).


Word2Vec: Why do some dimensions of an embedding have an interpretation, and why does addition/subtraction of embedding vectors work?

The dimensions for an embedding space are only be accidentally interpretable.
However, vectors through the space can be interpretable. That is why word analogies are possible in an embedding space . The addition / subtraction of word vectors describe another vector through the embedding space. For example, "king" - "man" + "woman" approximate the "queen" vector.
Words are consistently used in relationship to other words. Word embeddings can model these consistent relationship by finding which words co-occur together and projection the words into lower dimension space that retains the most common occurrences.


Pandas - Get feature values which appear in two distinct dataframes

you can use numpy.intersect1d() method:

In [277]: np.intersect1d(a.user_id, b.user_id).tolist()
Out[277]: [758, 1024]

or pd.core.common.intersection() method, but it seems to be slow (at least on my notebook for aa and bb DataFrames [see setup below...]):

In [307]: pd.core.common.intersection(a.user_id, b.user_id).tolist()
Out[307]: [1024, 758]

Timing for aa DF (50K rows) and bb DF (60K rows):

In [294]: aa = pd.concat([a] * 10**4, ignore_index=True)

In [295]: bb = pd.concat([b] * 10**4, ignore_index=True)

In [296]: aa.shape
Out[296]: (50000, 3)

In [297]: bb.shape
Out[297]: (60000, 3)

In [298]: %timeit aa.ix[aa.user_id.isin(bb.user_id),'user_id'].tolist()
10 loops, best of 3: 41.8 ms per loop

In [299]: %timeit np.intersect1d(aa.user_id, bb.user_id).tolist()
100 loops, best of 3: 5.36 ms per loop

In [300]: %timeit pd.merge(aa, bb, on='user_id').user_id.tolist()
...
skipped
...
MemoryError:

In [308]: %timeit pd.core.common.intersection(aa.user_id, bb.user_id).tolist()
10 loops, best of 3: 52.8 ms per loop

PS original answer


How can I use a class variable with many possible values in logistic regression?

This may not be the answer you are looking for, but I think this is a telling part of your challenge

  My hypothesis is that the propensity to say 'no' will be stronger in some individual respondents than others, be it for psychological or geographical reasons, and that these individual differences will be a stronger determiner for the dependent variable than any other class variable.

Unless I misunderstood you, it seems you want to have an input parameter that indicates how reflexively a respondent will be to just respond with 'No'.  The problem is, as I see it, you don't know the answer to this.  You have no way to measure this directly.  No?

If you have more data about the respondents, maybe you can back into this.  Huge assumption here, but if you could get a count of the number of recommendations made vs the number of times they have said 'No' you could come up with some factor:
$$f_{(No)} = {{Number\ of\ "No"\ Responses}\over{Number\ of\ Recommendations}}$$

This may get you what you are looking for, but it assumes you can tie back the data to the respondent and that the respondent is, in fact, the Branch Manger or Sales Representative, not the customer.  The problem with this approach is that it assumes that all the recommendations were equally competitive.

HTH


My ADALINE model using Gradient Descent is increasing error on each iteration

I think something is wrong here.

self.w_ += self.eta * X.T.dot(errors)

You are going to the positive to the gradient while you should be doing is going to the negative direction of it

self.w_ -= self.eta * X.T.dot(errors)

or

self.w_ += -self.eta * X.T.dot(errors)

see this for more clarification.


a neural network can be used as an optimizer?

Theoretically yes, pratically no.
just think About what you Need for this second Network. You Need a dataset X of real numbers representing loss and you Need y representing weight plus bias.
You can already see that weights of a slightly different Architecture will be different. So creating such dataset is just infeasable given that you Need a lot of datapoints for your Network to learn.


The typical way to address this is to transform those characteristics into a representation where you compose a vector with one component for each possible characteristic, with a $1$ in the position of each characteristic owned by the individual, or zero otherwise.

This way, you would have:

name | greedy | smart | funny | sad | prideful | stupid | ... | score
----------------------------------------------------------------------
John |   1    |   1   |   0   |  0  |    0     |   0    | ... | 0.8
Ann  |   0    |   1   |   1   |  1  |    0     |   0    | ... | 0.6
Joel |   1    |   0   |   0   |  0  |    1     |   1    | ... | 0.2


Predicting products to be sold in a store - problem formulation

To answer your question, in that dataset the first thing to do is to determine the types of products in that product sold column or whatever. If you have like 3 or more different kinds of products to be sold. Then you can rename those types and use a classifier algorithm to make your prediction. So the product sold will serve as your target variable and other closely correlated variables will now serve as your predicting variables.
So to add to the above, you want to predict the number of products to be purchased by a customer, then I suggest you look out for good multi-target regression models.


Is it a good thing if my model overfits in TensorFlow?

Generally, you should pick the model that has the highest performance on the validation dataset.

The number of features is a hyperparameter and should be tuned during the model fitting process.


Reason of use of Product rule of Probability in Predicting Error

The general idea is that there is a true joint probability $p(x,y)$ which can be factorized like
$$p(x,y) = p(y|x)p(x)$$
$p(x)$ basically measures the probability of observing the input $x$, while $p(y|x)$ represents the true relationship between $x$ and $y$.
We would like to learn a model distribution $p_\theta(x, y)$ which is factorized accordingly
$$p_\theta(x,y) = p_\theta(y|x)p(x)$$
$p(x)$ can be considered the same in both cases, so what we are trying to find is $p_\theta(y|x)$ as an approximation to $p(y|x)$ by optimizing the model parameters $\theta$.
In the case of regression, we make the assumption that there is a function $f_\theta(x)$ and the observed $y$ is distributed normally around $f_\theta(x)$ with noise variance $\sigma^2$, i.e.
$$p_\theta(y|x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left[-\frac{(y-f_\theta(x))^2}{2\sigma^2} \right]$$
Since $p_\theta(x,y)$ should approximate $p(x,y)$ as closely as possible, we want to minimize the KL-divergence between the two
$$
\begin{align}
&D_{KL} \left[ p(x,y) | p_\theta(x,y) \right] \\
&= \int \int p(x,y) \log\left[ \frac{p(x,y)}{p_\theta(x,y)} \right] dx dy \\
&= \int \int p(x,y) \log p(x,y) dx dy - \int \int p(x,y) \log p_\theta(x,y) dx dy
\end{align}
$$
Note that the first integral does not depend on $\theta$. The second term can be further simplified using the facotization
$$\int \int p(x,y) \log p_\theta(x,y) dx dy = \int \int p(x,y) \log p_\theta(y|x) dx dy + \int \int p(x,y) \log p(x) dx dy$$
Here the second term does not depend on $\theta$. Using all this, we can write the optimization problem for the KL-divergence as
$$
\begin{align}
\theta^* &= \text{argmin}_\theta D_{KL} \left[ p(x,y) | p_\theta(x,y) \right] \\
&= \text{argmin}_\theta \left[ - \int \int p(x,y) \log p_\theta(y|x) dx dy \right]
\end{align}
$$
Plugging in the model assumption about $p_\theta(y|x)$ this gives
$$
\begin{align}
\theta^* &= \text{argmin}_\theta \left[ \int \int p(x,y) \frac{(y-f_\theta(x))^2}{2\sigma^2} dx dy + \log\sqrt{2\pi\sigma^2}\right] \\
&= \text{argmin}_\theta \left[ \int \int p(x,y) (y-f_\theta(x))^2 dx dy \right]
\end{align}
$$
where we use that neither the constant shift nor the constant scale change the solution of the $\text{argmin}_\theta$.
This way we have derived the mean squared error as the loss for regression from probabilties and their factorizations. The final step would be that we use the empirical risk to approximate the expectaion / integral, i.e.
$$
\int \int p(x,y) (y-f_\theta(x))^2 dx dy \approx \frac{1}{N} \sum_{n=1}^N (y_n - f_\theta(x_n))^2
$$
which repsoduces the mean square error as a loss.


The solution is to install an archived version of the package.


Explanation of how DeepExplainer works to obtain SHAP values in simple terms

From https://en.wikipedia.org/wiki/Shapley_value, it is possible to understand that direct computation of Shapley values is difficult with their general formula :

$$ \varphi_i(v) = \frac{1}{\text{number of players}} \sum_{\text{coalitions excluding }i} \frac{\text{marginal contribution of }i\text{ to coalition}}{\text{number of coalitions excluding } i \text{ of this size}}$$

Basically beacuse the number of coalitions excluding i grows in complexity with $\sum_{k=1}^{n-1} k!$, where n is the number of variables. Some progresses has been made in the direction of evaluating this sum with Monte-Carlo techniques (as mentionned in https://christophm.github.io/interpretable-ml-book/), but those calculations are still intensive.

In their article (http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions) Lundberg and Lee proposes two new approaches, relying on SHAP - (these are the Shapley values of a conditional expectation function of the original model) :

A model agnostic approach basically rewriting the problem as a linear regression problem, which is intuitively less expensive to compute. Basically saying that SHAP is a function of the weight of the model and trying to approximate it.
A model specific approach. Assuming input independence (which is rarely true ...) they show how to compute SHAP value directly from model weights. Starting with linear models they devise similar relations for NN with usual propagation techniques.

As to which method is used exactly for MLP, I don't know exactly, but the second one seems more appropriate (model specific, exact method).


Feature extraction from resume using Python without rule based logic

Yes there is. Look at this paper of general information extraction

Where you can construct features and generalise around extracting, without any hard coded rules.


Equally sized topics in Latent Dirichlet allocation

It is normal, the best explantation I found for it, is from physics. Since Gibbs Sampling was known in physics long before LDA and LDA can simply be seen as a kind of matrix factorization.
There is a system which has particles (words) and the particles can be in different states (topics). States with lower energy have a higher probability of being occupied than states with higher energy.
Or simply: Since LDA is a dimensity reduction method that just compresses a words x documents matrix into a topics x documents and a words x topics matrix, the most effective way to do it is by maximizing the entropy which is done if the clusters get equal size.

Just noticed that you can simply derive the above assumption from the dirichlet distribution itself: Take a look at the density (PDF) function and ignore the normalizing factor at the front and only assume symmetric alpha parameter. This results in a product term of X1...XK, that product term gets it's maximum if all X1...XK's have equal size, since the sum X1+X2+...+XK = 1, it's the same case as entropy above.