First, note that in logistic regression, using both an L1 and an L2 penalty is common enough to have its own name: ElasticNet.  (Perhaps see https://stats.stackexchange.com/q/184029/232706 .)  So using both isn't unprecedented.

Second, XGBoost and LightGBM have quite a number of hyperparameters that overlap in their purpose.  Tree complexity can be controlled by maximum depth, or maximum number of leaves, or minimum sample (count or weight) per leaf, or minimum criterion gain.  Any combination of these might be optimal for some problem.  Overfitting can also be combatted with the learning rate vs. number of trees (and early stopping), subsampling rates, and either of the regularization penalties.

Finally, since L1 regularization in GBDTs is applied to leaf scores rather than directly to features as in logistic regression, it actually serves to reduce the depth of trees.  This in turn will tend to reduce the impact of less-predictive features, but it isn't so dramatic as essentially removing the feature, as happens in logistic regression.  You might think of L1 regularization as more aggressive against less-predictive features than L2 regularization.  But then it might make sense to use both: some L1 to punish the less-predictive features, but then also some L2 to further punish large leaf scores without being so harsh on the less-predictive features.

Toy example: https://github.com/bmreiniger/datascience.stackexchange/blob/master/trees_L1_reg.ipynb

Possibly useful:
https://github.com/dmlc/xgboost/blob/release_0.90/src/tree/split_evaluator.cc#L118
https://xgboost.readthedocs.io/en/latest/tutorials/model.html#model-complexity


Python - Calculate Cost profitability and benefit of the model

Confusion Matrix

A Confusion Matrix is an important tool to measure accuracy of a classification algorithm. It compares predicted class of an outcome and actual outcome.

Scenario 1: Credit Risk

Based on a credit risk scorecard, application for credit card are classified as “Good” and “Bad”. “Good” indicates applicants paying back dues on credit card and “Bad” indicates customers defaulting on the dues. Now, the customers are compared against actual performance of the customer payment behaviour after say 18 months. So, comparison of Predicted Class (“Good” or “Bad”) to actual customer behaviour state (“Defaulted” or “Regular”).

There is always a trade-off between Type I error/False Positives (accepting Bad Customers) and Type II Error/False Negatives (Rejecting Good Customers).
We generally require to optimize between False Positive Rate (Type I Error) and False Negative Rate (Type II Error).
So, role of cost matrix comes in picture to find the optimal cut off value for a classification rule.  Now, going back to Credit Risk Model. The cut off value optimize between cost of an opportunity loss (miss to accept a good customer/Type II Error) and cost of accepting a potential defaulter (involved in loss due to default).
Referencing Example is from a blog and image from Google..

Cost Matrix

Cost Matrix is similar of confusion matrix. It’s just, we are here more concerned about false positives and false negatives .There is no cost penalty associated with True Positive and True Negatives as they are correctly identified.
The goal of this method is to choose a classifier with lowest total cost.

Total Cost = C(FN)xFN + C(FP)xFP

where,

FN is the number of positive observations wrongly predicted

FP is the number of negative examples wrongly predicted

C(FN) and C(FP) corresponds to the costs associated with False Negative and False Positive respectively.
Remember, C(FN) > C(FP).

Hope this Helps..


Your dataset is not [1000x5], one of your documents is that size. LSTMs can deal with variable sized input directly or by padding (this is the case with Keras). In Keras you can use the parameter return_sequences as False in your LSTM, which means only after the last part of your sequence has been parsed it will transform it with a Softmax and run the classification. What I would do is pad your documents to a certain size, for example 1000 (though this is high for LSTMs) so that you will have a [n x 1000 x 5] feature set and a [n] target set. You can feed this to your model now.


Is my model to big? I am trying to predict orders for a company, and I don't know if there are typical values for macroparameters

Theoretically, a model should be big enough to have a low bias (avoid underfitting), but not too big as to have too high of a variance (avoid overfitting), called the bias-variance tradeoff

Whether the bias is too low for you or not depends on the true distribution you are trying to model. Practically, this means whether your model is getting a low training loss or not.

Whether the bias is too high or not depends on the true distribution and how much data you have. If your model achieves low training loss but high validation loss, then it's overfitting and thus has too high of a variance. In that case you need to either lower the complexity of the model or get more data.

If you ever reach a point where

1- reducing the complexity of the model will make training loss too high, and

2- increasing the complexity of the model will make validation loss increase

Then the only way to improve is to either change your model architechture/hyperparameters in some way or get more data.


This questions is best posed in its original domain, ie medicine, because it will require significant domain knowledge about the problem structure and the nature of the data to reason about how well a single model might do. I doubt the analogy to job applications translates perfectly here.

Right now you have one model per review topic which is able to generalize to different publications with different criteria for inclusion. You are asking whether a single model will also be able to generalize to different review topics.

Of course you could build the model and evaluate its accuracy. If that will require significant effort, then it is fair try and reason about how well you might do first, but that will only get you so far.

Again you will have to apply your own domain knowledge here, but I can at least try and give you a framework for thinking about this problem. In order to answer the main question of how well one model will generalize, you need to investigate:

Do different review topics interact with publications / inclusion criteria in similar ways?
Do any of my existing features capture the interaction between review topics and publications / inclusion criteria?
Are there any additional features I can design to capture these interactions in a general way?

To give a very simple example, if having the topic keyword x number of times in the paper was sufficient for inclusion for paper X, topic Y, publication Z with review criteria A,B,C,..., and you think this data helps you predict inclusion for a completely different paper, topic, etc. example, then you might reason that one model will generalize here.


What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?

Here's the list of difference that I know about attention (AT) and self-attention (SA).

In neural networks you have inputs before layers, activations (outputs) of the layers and in RNN you have states of the layers. If AT is used at some layer - the attention looks to (i.e. takes input from) the activations or states of some other layer. If SA is applied - the attention looks at the inputs of the same layer where it's applied.
AT is often applied to transfer information from encoder to decoder. I.e. decoder neurons receive addition input (via AT) from the encoder states/activations. So in this case AT connects 2 different components - encoder and decoder. If SA is applied - it doesn't connect 2 different components, it's applied within one component. There may be no decoder at all if you use SA, as for example in BERT architecture.
SA may be applied many times independently within a single model (e.g. 18 times in Transformer, 12 times in BERT BASE) while AT is usually applied once in the model and connects some 2 components (e.g. encoder and decoder).
SA is good at modeling dependencies between different parts of the sequence. For example - understand the syntactic function between words in the sentence. AT on the other hand models only the dependencies between 2 different sequences (for example, the original text and the translation of the text). While still the SA may be very good in translation task (see Transformer)
AT can connect 2 different modalities (i.e. text and image). SA is usually applied within a single modality but you still can join activations from 2 modalities into a single sequence and apply SA on it.
Generally SA mechanism looks like a more general to me as it can do more than AT. You can simulate AT with SA by just replacing/concatenating the input sequence with the target sequence that you want your attention to look at.

Some more notes

The term Multi Head Attention is often used with SA. But theoretically you can apply Multi Head approach to AT also.
The following terms: content-base attention, additive attention, location base attention, general attention, dot-product attention, scaled dot-product attention - are used to describe different mechanisms of how inputs are multiplied/added together to get the attention score. All these mechanisms may be applied both to AT and SA.
Key/Query/Value approach of attention calculation is usually applied to SA. But you can use it for AT also.


What is the difference between the training and testdata set?

In Machine Learning, we basically try to create a model to predict on the test data. So, we use the training data to fit the model and testing data to test it. The models generated are to predict the results unknown which is named as the test set. As you pointed out, the dataset is divided into train and test set in order to check accuracies, precisions by training and testing it on it.

The proportion to be divided is completely up to you and the task you face. It is not essential that 70% of the data has to be for training and rest for testing. It completely depends on the dataset being used and the task to be accomplished. For example, a simple dataset like reuters. So, assume that we trained it on 50% data and tested it on rest 50%, the precision will be different from training it on 90% or so. This is mostly because in Machine Learning, the bigger the dataset to train is better. You can refer this paper, which tells the precision values based on the dataset size. It now depends on you, what precision or accuracy you need to achieve based on your task.

This said so, how would you predict the results for which you do not have the answer? (The model is ultimately being trained to predict results for which we do not have the answer). I would like to add on about validation dataset here.

Sets:

Training Set: Here, you have the complete training dataset. You can extract features and train to fit a model and so on.
Validation Set: This is crucial to choose the right parameters for your estimator. We can divide the training set into train set and validation set. Based on the validation test results, the model can be trained(for instance, changing parameters, classifiers). This will help us get the most optimized model.
Testing Set: Here, once the model is obtained, you can predict using the model obtained on the training set.

Refer this for more info.


Unsupervised Learning upon a column of dataset (graph shown)

A few observations/questions/considerations:

Wikipedia: Unsupervised learning is a type of self-organized Hebbian learning that helps find previously unknown patterns in data set without pre-existing labels.
Being an anomaly is a label.
Why not just Z-score it? https://en.wikipedia.org/wiki/Standard_score#Z-test . Calculate distribution: $\mu$ and $\sigma$, and check if value is
within $\mu$ +/- 1.96 $\sigma$, $95\%$ chance it is from the normal distribution of the numbers in the column.
Can the Value column be something else than numbers? (E.g. color names)
Or could it be a time series with drift?

Edit:

Under the hood, Machine Learning is basically a set of smart statistics and decisions on subsets of columns and rows. But with only one column and no time series, there is nothing smart to subset or decide. Then essentially, it is an exceptional Z-score for the elements. You have to decide on how many $\sigma$ you want.


how to optimize the weights of a neural net when feeding it with multiple training samples?

But when we feed the NN with more than one training example then which formula do we use ? Do we calculate the average of all dw (derivative of the loss function wrt the weights) or do we sum all of'em then multiply by the learning rate and substract them from the initial weights or what ?

This is batch gradient optimisation, so you average the gradients over all training examples in your batch.

Side note: In principle, you could use several training epochs (= several iterations over your entire dataset) and update the weights once per epoch. However, for large datasets (like ImageNet), you divide your dataset into batches and update the weights once per batch. You could also further subdivide each batch into minibatches and update once per minibatch. The subdivision is entirely up to you and is kind of ad hoc, but a recent paper suggests that consistently higher performance can be obtained with small minibatches of size between 2 and 32. Also, your learning rate should be scaled appropriately depending on the batch size.


How do I encode time in high dimensional space?

I don't know if it's a good technique or not, but I saw this paper last year that proposes a time2vec, analogous to doc2vec, et al:
S. M. Kazemi et al., “Time2Vec: Learning a Vector Representation of Time,” arXiv:1907.05321 [cs], Jul. 2019.


Which service could I use to train my networks?

You can use AWS EC2, or the GCP Compute Engine, or the GCP Machine Learning Engine. You pay based on how many CPU/GPUs you employ and on training times.


Are stationarity and low autocorrelation the prerequisite of regression model?

For (linear) regression models, the following assumptions have to hold:

Linear Relationship
Multivariate normality including your error term with zero mean + some infinite variance following a normal distribution
No-autocorrelation
No (strong) multicollinearity
No homoscedasticity

Non-stationarity (caused by changing mean or/and variance over time) is prevailing in most level data (e.g. stock prices) - as it has a unit root or is trend stationary.

To cure your data from non-stationarity in most of the times, it is sufficient to use the relative change (percentage change - or log changes). Taking logarithmic changes, can also cure of homoscedasticity a bit.

To test for stationarity, you can use the Augmented-Dickey-Fuller-Test where the Null hypothesis states that a unit-root is present in your data set, thus you have a non-stationary variable.

Coming to your second question about time-series and deep learning. Unfortunately, I haven't found a good paper/article yet that discusses the assumptions on the main data properties that have to be satisfied to get correct statistical results. See my own post - not answered yet

However, as deep learning models make use of the same underlying statistical assumptions - e.g. probability distributions etc. or even the underlying models (see e.g. linear activation function) - I guess, in a time-series framework, the same assumptions have to hold, otherwise, there is the chance of spurious regression.


It is possible to cluster the objects based on their labels by treating the labels as features. Typically, labels are treated as targets which would frame the problem a supervised machine learning problem.
Since labels are nominal valued, you will need to use an appropriate distance metric. Jaccard index is one option.


Can you learn an algorithm from a trained model?

In this article https://arxiv.org/abs/1711.09784 the authors the fits decision tree to a neural network in order to understand what is the neural network does. Is it what you are looking for?


Orange Web Services - Problems installing imblearn module for Python Script

The one in Orange's Python Script widget and the one in cmd are obviously not the same Python environments. Confirm by inspecting:

import sys
print(sys.executable)

Fix the issue by installing imblearn into the Orange's Python environment.


how does word variety depend on total words?

The simple answer is to use a sample of equal size from every book, or even better to randomly extract several samples of equal size from every book and then use the mean across samples.

  I tried simply dividing the number of unique words in each book by the number of total words in each book

This is known as the type/token ratio, the simplest way to measure lexical density. I think it makes perfect sense in the case you describe, as far as I know usually it's not too biased.


Given a t-SNE plot, how can I infer the "most correct" labels? How does one understand its structure?

With t-SNE none of the input parameters are weighted more than any other parameter so the differences you want to see like students forming islands by grade level will not happen because there is so much other data present to pull those students/data points in different directions.

I highly encourage you to have a specific question in mind and tailor your input categories so that your question can be answered by the t-sne map.

You might try asking a specific question and changing which input categories you ask t-SNE to look at. For example, do taller students get better grades? Feed in height and grades  categories while leaving out grade level and age. This is a silly example but I hope it gives an idea of how you can use t-sne to help you learn about your data.

You might also find that there are categories that mask meaningful findings. Height might not be very useful for pulling out meaningful information and since the height range is going to be much larger than the A-F grade range, it will likely influence the t-sne map more.

Looking at the data as you described on a color scale for each category is a good place to start with each new t-sne run.

It is okay to run multiple t-snes with different parameters. To be sure you are getting a meaningful answer, I recommend running t-sne multiple times with the same parameters as well.

If you have enough students, it would be great to included half in your training set where you explore and figure out what questions to ask and then when you think you've found something meaningful, apply those conditions to the other students in the testing set to see if it holds true.


Is classifier able to say there's no-such-case?

Taking your questions one after the other:

  Assume that I have a classifier which can classify left hand / right
  hand well. I am curious whether it can decide whether there's a hand
  in the image?

No. It cannot. A model which is trained on hand/no-hand can do that. Not this. At most, it shall predict one of the either (left/hand) with a very low confidence score. But, it definitely is not a nice idea to use it for a hand/no-hand identification.

  But there's so many things that are "no-hand" such like dogs, cats,
  flies... I don't think it is possible to collect all those possible
  "no-hand" data

You are trying to do build a binary hand.no-hand classification model, not a hand/cat/dog/... style multi-label classification model. So, you just need to have a bunch of images with a hand in it, and some more which doesn't have one.

  Would it be reasonable that we use some activation functions like
  [sigmoid] as the final outputs for the classifier(neural network) and
  we consider the outputs for both left/right classes as the scores that
  classifier think how far the input is from the left/right hands
  classes.

Possible, but not advisable.


You'll need to perform two steps:

You get ride of the id column in the pandas dataframe:

df=df.drop(columns=['id'])

Add the index=False param: to the .to_sql() method:

selected.to_sql('dataiswa', engine, index=False)


I haven't attempted this, but I'm pretty sure it would work reasonably well.

Identify all points that are connected to more than two lines. We know these must be vertex points.
For for all the points that are attached to only one line, identify their nearest neighbor and draw a line between them (this would connect that hole on the right wall of your house). All points should now be either labeled as a vertex or touching only two lines.
For each point touching two lines, calculate the angle between the two lines.
Find the min and max of these angles.
Use these as prototypes to classify angles into two proximity clusters using agglomerative clustering. Alternatively, pick a fixed angle value as your threshold (maybe around 120 degrees) and use that as the decision boundary for this classification.
The angles that clustered with the min correspond to the missing vertices.
Infer the vertex-vertex connectivity graph through the non-vertex points.(i.e. find all paths between pairs of vertices that do not pass through a third vertex).
Draw the graph.
Find the cycle's in the graph and color each one separately.

Everything doesn't have to be a big fancy neural network :)


The NLTK book is by far the best tutorial on basic NLP I have seen(in Python).

The Coursera course on NLP is also fairly good. It takes off from the basics and takes the student to a novice level.


Reverse engineering a distance metric from the output of a k-NN

Interesting question. If you supposed that the distance metric was some common one, of course you could just try all the common ones and see what is consistent with the nearest neighbors it returns.

I can't think of a way to learn this reliably, because you don't have any info about the actual distances. Maybe treat all nearest-neighbor pairs as having 'small' distance and the rest 'large' and training a deep model that tries to learn to predict it across pairs of points. Whatever it comes up with as a metric would approximate it.


As rightly pointed out by @erwan, it is a bad idea to use data augmentation with 'text data'
The problem of 'training with less data' can be approached in many ways, here I enlist two ways which helped me with significant impact:
(a) One approach would be to use semi-supervised approach. There are open sourced language models trained on insanely massive datasets, that can be used to perform a specific task like custom NER or sentence classification. Transfer learning is more useful today then ever.
(b) Anyways if we want to go ahead with Data augmentation, 'Sentence‐Chain Based Seq2seq Model for Corpus Expansion' is one of the proven methods to proceed. Please find the link to paper here.
I will strongly recommend to experiment with BERT before moving to Corpus Expansion. For introduction to transfer learning, BERT etc. please visit here.


Try to use the split method as enumerate argument, instead of kf (e.g.:for i, (train_index, test_index) in enumerate(kf.split(X)):

Hope it helps!


Understanding clusters after applying PCA then K-means

PCA removes the connection with the original features,so the interpretation of the visualisations in the principle component space is therefore not very meaningful.
E.g. cluster A has higher values of PC1, where cluster B has higher values of PC2.
If you can clearly see that PC1 is only representative of Feature X, then fine, but this isn't often the case.
Instead use PCA to discover which features best represent the data, and use this to remove unnecessary (unrepresentative) features so that you can visualise the original data using its most important original features. And therefore describe the clusters with meaningful differences.
You might want to experiment with a scatter matrix, to explore which features spaces have the cleanest distinction between clusters.


Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?

One of the benefits of using scikit-learn's Pipeline is to appropriately apply the operations to both training and prediction datasets. The "whole dataset" you mention that is only the training data you currently have available. Scikit-learn's Pipeline can automatically handle additional data, such as production data.
Having two different pipelines can create unnecessary overhead and might lead to bugs in the system. It is best practices to have a single pipeline that handles all operations.


Dummy coding a column in R with multiple levels

You can use either sparse matrices or feature hashing.

Sparse Matrix

I suppose that using a sparse matrix is the only choice. I suspect that this line of code will work. This uses the Matrix package.

sparseProducts <- sparse.model.matrix(~ product, data=salesdata)

Take my example:

sparseDiagonalMatrix <- sparse.model.matrix(~., data.frame(V1 = as.factor(seq(1, 10))))

each column represents a different factor, this will yield:

1  1 . . . . . . . . .
2  1 1 . . . . . . . .
3  1 . 1 . . . . . . .
4  1 . . 1 . . . . . .
5  1 . . . 1 . . . . .
6  1 . . . . 1 . . . .
7  1 . . . . . 1 . . .
8  1 . . . . . . 1 . .
9  1 . . . . . . . 1 .
10 1 . . . . . . . . 1

> class(sparseDiagonalMatrix)
[1] "dgCMatrix"
attr(,"package")
[1] "Matrix"

alternatively you can remove the intercept and have all zeros represent class 1

sparseDiagonalMatrix <- sparse.model.matrix(~., data.frame(V1 = as.factor(seq(1, 10))))[, -1, drop=FALSE]

10 x 9 sparse Matrix of class "dgCMatrix"
V12 V13 V14 V15 V16 V17 V18 V19 V110
1    .   .   .   .   .   .   .   .    .
2    1   .   .   .   .   .   .   .    .
3    .   1   .   .   .   .   .   .    .
4    .   .   1   .   .   .   .   .    .
5    .   .   .   1   .   .   .   .    .
6    .   .   .   .   1   .   .   .    .
7    .   .   .   .   .   1   .   .    .
8    .   .   .   .   .   .   1   .    .
9    .   .   .   .   .   .   .   1    .
10   .   .   .   .   .   .   .   .    1

> class(sparseDiagonalMatrix)
[1] "dgCMatrix"
attr(,"package")
[1] "Matrix"

You will need a package that supports sparse matrices for measuring the net revenue though. Fortunately, most modern mainstream packages support sparse matrices.

Feature Hashing

Here is a great explanation of feature hashing in R (among other techniques) which is also an alternative, specially useful when you have hundreds of thousands or millions of multiple levels.

https://amunategui.github.io/feature-hashing/


Real time noise removal using Savitzky-Golay Method

The original Savitzky-Golay paper addressed smoothing, meaning that you estimate a value for some point in the past, using values from its past and future. It sounds like what you're interested in is filtering:  at time step k, you want to reduce noise and come up with a better estimate of the value at the same time step k, without knowing the future.  That's important in many fields such as process control, and some fault diagnosis where delays can't be tolerated.

The SG approach of calculating the constants for least squares fitting for smoothing has also been done for filtering.  A curve is fit through n recent points, and instead of estimating the middle point, the end point is estimated by a least squares fit, reduced to simple multiplications by constants. There are tables of the necessary coefficients and an example at
https://gregstanleyandassociates.com/whitepapers/FaultDiagnosis/Filtering/LeastSquares-Filter/leastsquares-filter.htm

There is no time delay in the calculations, unlike with smoothing.  Also, if the input signal is a ramp, the filter will converge to the correct value once all of the points in the time window reflect that ramp.  On the other hand, there are some side effects that may or may not matter to you.  Even in the linear case, for a step change in the input, there is overshoot in the filter output.  Higher order least-squares filtering can be quirkier, so it is rarely used. But the linear version overall is quite effective, assuming the little bit of overshoot isn't a real problem  (for instance, if you don't see step inputs in the data anyway).

The link above is part of a more general "Guide to Filtering", that covers some other common filters as well, such as the exponential filter and moving average filters in particular.  (Filters don't have time delays, but most do lag the inputs - meaning smear out changes over time.  That's a byproduct of filtering out the noise, essentially cancelling out some new noise by a certain amount of averaging against old noise).  It also covers general issues for filtering, such as aliasing.

There are two other filters that are not covered in the Guide that might be considered close competitors to the SG-style filtering:  the alpha-beta filter and double exponential smoothing.  These are covered in Wikipedia:  alpha-beta filtering has its own page, and double exponential smoothing is part of the page on exponential smoothing.  The goal of both is to filter/estimate both the values and the derivative, resulting in a filter that will give correct outputs to a ramp input signal.  I've long suspected that these two filters are essentially the same, but never taken the time to look into it.


How to create a classification model for multi output dataset?

So the question asks how to create a model such that you take in input X and translate this into two simultaneous predictions, one which predicts the breed_category and pet_category.
Depending on the number of the classes within the two targets (breed_category and pet_category), you could simply concatenate the outputs as one-hot encoded vectors (https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/).
So, you would encode the categories for breed_category and pet_category into one-hot encoded vectors and then concatenate them together.
By representing the output like this, you can use a regular feedforward neural network architecture, with a final sigmoid output layer, which maps the output into values between 0 and 1. In this case you cannot use softmax, since you are now predicting two targets simultaneously.
If the number of classes from both targets is large, it might be better to have separate feedforward models, one for breed_category and pet_category.
For this, you can then use a softmax final output layer.


Machine Learning models in production environment

I think this is a good approach in general. However:

Fine-tuning your model (online learning) depends a lot on the algorithm and model how well this works. Depending on your algorithm it might be wise to retrain the whole thing
Your sample space might change overtime. If you have enough data maybe retraining every few days/weeks/months over only the last year worth of data might be better. If your old samples don't represent the current situation as well having them included might hurt your performance more than the extra samples help
The biggest condition is if it's tested and how much downtime it involves, but in general swapping more times is better, and this can be automated


Since the students know about eigenvectors and eigenvalues, I will assume that they know about Lagrangian multipliers as well.

Let's start with computing the variance of data $\vec{x}$ along a given direction $\vec{e}_a$. Since we only care about the direction, $\vec{e}_a$ is a vector of unit length, i.e.
$$\vec{e}_a^T \vec{e}_a = 1$$

So first, the component of the data along $\vec{e}_a$ is given by
$$ \vec{e}_a^T \vec{x}$$
The variance $\sigma_a^2$ of this component is given by
$$ \sigma_a^2 = E[(\vec{e}_a^T \vec{x} - \vec{e}_a^T \vec{\mu})^2] $$
where $\vec{\mu} = E[\vec{x}]$.
If we re-write this expression a little bit
$$ \sigma_a^2 \\
= E[(\vec{e}_a^T \vec{x} - \vec{e}_a^T \vec{\mu})(\vec{e}_a^T \vec{x} - \vec{e}_a^T \vec{\mu})] \\
= E[\vec{e}_a^T(\vec{x} - \vec{\mu})(\vec{x} - \vec{\mu})^T\vec{e}_a ] \\
= \vec{e}_a^T E[(\vec{x} - \vec{\mu})(\vec{x} - \vec{\mu})^T ] \vec{e}_a \\
= \vec{e}_a^T \Sigma_x \vec{e}_a
$$
where in the last step we use the covariance matrix $\Sigma_x = E[(\vec{x} - \vec{\mu})(\vec{x} - \vec{\mu})^T ]$.
So the variance along a given direction $\vec{e}_a$ is computed using $\Sigma_x$. So far, so good.

Let's now try to find the direction $\vec{e}_a^*$ where the variance is maximal. Since $\vec{e}_a^*$ is also supposed to only indicate the direction, we demand
$$\vec{e}_a^{*T}\vec{e}_a^{*} = 1 $$
With this we can formulate a constrained optimization problem using the Lagrangian multiplier $\lambda$
$$ L(\vec{e}_a) = \vec{e}_a^T \Sigma_x \vec{e}_a + \lambda (1 - \vec{e}_a^T \vec{e}_a ) $$

Taking the derivative with respect to $\vec{e}_a$ and equating it to zero, we end up with
$$ \frac{\partial L}{\partial \vec{e}_a} = \Sigma_x \vec{e}_a - \lambda \vec{e}_a = 0 $$
i.e.
$$ \Sigma_x \vec{e}_a = \lambda \vec{e}_a $$
This is a cool result. The direction of optimal (maximal) variance is a solution to the eigenvalue problem of $\Sigma_x$. Assuming that we found a normalized eigenvector $\vec{e}_b$, it is easy to interpret the eigenvalue $\lambda$

$$ \vec{e}_b^{T} \Sigma_x \vec{e}_b = \lambda = \sigma_b^{2} $$

So, to summarize:

the variance along a given direction is computed using the covariance matrix
the direction of maximal variance is a normalized eigenvector of the covariance matrix
the eigenvalues are the variances along the direction of the eigenvectors

This means, that the problem of finding the direction of maximal variance reduces to finding the eigenvector corresponding to the maximal eigenvalue. This is about as intuitive as it gets, I believe.

(I skipped the part where we need to show that all eigenvalues are non-negative and thus can be interpreted as variances, but this follows from $\Sigma_x$ being positive semi-definit.)


In a lot of cases, you deal with too many features and you either try to reduce the dimensionality of your feature space with techniques like PCA or you try to select only the relevant ones. The latter is known as feature selection. However, what you have is a situation where you don't have a lot of features and the information you get from those does not seem to be enough for a solid classification.

Generally speaking, the best options you have in this case are cleaning your data as good as possible and gathering more features. This can either be done by using new sources of information or by deriving new information from existing sources.
Usually, you derive the best feature candidates for your model from the literature of the respective subject area. Domain knowledge helps a lot!

In your case, I would guess that variables like time of day, weekday and day of the month could provide some valuable information, apart from the mail content of course. But these are simple guesses. In the end, you will have to dig into the literature and find the right features by trial and error.

Since you are trying to classify incoming emails, I assume you have access to the mail subject and the mail body. In this case, I would recommend starting with a simple Naive Bayes algorithm, since it is easy to use, effective and there are many implementations and guides how to use them.  Also, Naive Bayes can easily be applied to multiclass problems like yours.


How to order a python dataframe by adding the row values?

#create data
DF=pd.DataFrame(columns=["M1", "M2", "M3", "M4"])
DF.loc[0]=[45, 46, 54 , 57]
DF.loc[1]=[32 ,36 ,29 ,56]
DF.loc[2]=[56,44 ,40 ,55]
DF.loc[3]=[57 ,43 ,42 ,54]
DF=DF.astype(int)

#main function

DF["sum_"]=DF.apply(lambda x: sum(x), axis=1)
DF.sort_values("sum_", ascending=False)


Taking average of multiple neural networks?

I think even this method is also called Ensemble Method.

How could I conclude that?

You might have heard about this algorithm named Random Forest,
what does it do? It take data randomly at row level and column level
builds different trees and takes an average of it. It is also
considered as one of the best algorithm for Prediction and
Classification. Can go through this explanation for better
understanding. Random Forest is called Ensemble model(of trees).

A suggestion, as you have mentioned in the question that you have very less data, at that time the models cannot generalize well and you cannot achieve good results. If you have any way to increase the dataset size by collecting more data, it could aid you in achieving better accuracy. This is also explained the link attached(explanation). Do go through and let me know if you have any additional questions.


How to compute AUC in gridsearchSV (multiclass problem)

Metrics are independent from ML algorithms, so it doesn't matter which algorithms did you use.

To calculate multiclass AUC you could use lib pRoc in R or use code this link(in Python).

Sources:

https://web.expasy.org/pROC/
https://medium.com/@plog397/auc-roc-curve-scoring-function-for-multi-class-classification-9822871a6659


First, let's look at the term normalization.

  In statistics and applications of statistics, normalization can have a range of meanings. In the simplest cases, normalization of ratings means adjusting values measured on different scales to a notionally common scale, often prior to averaging. In more complicated cases, normalization may refer to more sophisticated adjustments where the intention is to bring the entire probability distributions of adjusted values into alignment.

Your NHL goals example probably does not need "adjusting values measured on different scales to a notionally common scale."

Think of the handful of players who had a high-scoring beginning of the season, and then was injured and sat out the majority of games. Would you want to normalize his goals up proportionally to the rest of the league?


Algorithm for multiple extended string matching

Do you change the patterns often? If not, then you can use Aho-Corasick method, whose idea is, first, to build a finite automaton based on your patterns and, then, to make a single pass over the text with this automaton to find matches (if the automaton visits a "matching" state, then there is a match). The complexity of the automata building should be linear in the length of the patterns (# of patters * max pattern length), and the matching phase should be linear in the size of the text you search in.


I think I got my answer from the following post:
http://unixnme.blogspot.com/2018/04/dimensionality-reduction-and-scattered.html


I am not sure ANOVA is the best and easiest way to find correlation between these categorical features and your target. You may see this great post where they propose many other methods along with ANOVA. If you persist to use ANOVA test or Kruskal-Wallis H Test, you need to know how it works to give you that notion of correlation (variation of variance among groups of categoricals). It is nicely explained in that post:

  ANOVA estimates the variance of the continuous variable that can be
  explained through the categorical variable. One need to group the
  continuous variable using the categorical variable, measure the
  variance in each group and comparing it to the overall variance of the
  continuous variable. If the variance after grouping falls down
  significantly, it means that the categorical variable can explain most
  of the variance of the continuous variable and so the two variables
  likely have a strong association. If the variables have no
  correlation, then the variance in the groups is expected to be similar
  to the original variance.

Once you understand how it works, implementing it and automating it is not difficult. In fact scipy and statsmodels have ANOVA. Check this post out, where they demonstrate in details how to perform ANOVA test on an actual dataset and estimate the correlation between categorical variable and continuous target. It is just a matter of putting these pieces together and change a bit to make it work for your own dataframe.


How to statistically prove that a column in a dataframe is not needed

With respect to the other answers i observed your point about having features which are strings, you must first find a way to encode them as numerical features which can then help you with PCA and the problem with the heat-map will be resolved too. Also if your problem is of regression then you can use methods such as L1-Regularization which help you in feature selection and you don't have to remove the features while pre-processing.


What it means for a neural network to memorize data

Memorization is the same as overfitting.  The memory is implicitly represented by your weights. If your network does have enough parameters it will be able to memorize/overfit.

In order to understand what I mean by overfitting and memorization let us look at the polynomial regression

$$y_n=w_0+w_1x_n+w_2x_n^2+ \varepsilon.$$

We have three coefficients. If we only had three data points (which do not perfectly lie on a line) we could fit a quadratic regression equation without any error. Hence, the model would memorize the data by using three coefficients.
We would have a loss of zero, but as you know this result would also be very likely overfitting the model to the data.


Basis expansion for regression using neural network?

I guess expanding basis is somehow looking at the problem using linear algebra perspective. For explaining it using a trivial example, in linear algebra, for the 3D space, there are three bases namely, $i$, $j$ and $k$. It means for referring to any vector in 3D space you just need a linear combination of these three vectors to construct any desired vectors. Each basis for each space has a number of properties like being orthogonal or not or maybe being linearly independent and some other important properties. By adding extra features, you increase the number of features if they are linearly independent.

In machine learning, there are situations that comparing each feature with the output directly may not lead to a good decision boundary while using some features alongside each other can make decision boundaries which can lead to a better decision boundary. All of that means that if you change the input space to space with more number of features which the features are extracted using the current feature space, you can find a space that the data can be separated better. Finding a feature space such with properties can be done using feature extraction methods. Kernel methods are some of them.

Kernel trick is a kind of mathematical approach for reducing the difficulties of transforming the data in the current feature space to the appropriate feature space. Suppose you have one million rows of raw data and the feature space belongs to $R^{3}$. Suppose you find out that a feature space that can have a good  generalisation belongs to the space which belongs to $R^{95}$. Trying to transform the data in hand to the desired feature space is a time-consuming task if possible. Instead of transforming the data directly, the kernel trick tries to calculate the outcome of the transformed data without transforming the data into the new space.


Multivariate outlier detection with isolation forest..How to detect most effective features?

One important difference between isolation forest and other types of decision trees is that it selects features at random and splits the data at random, so it won't produce a nice feature importance list; and the outliers are those that end up isolated with fewer splits or who end up in terminal nodes with few observations.

I guess if you dig into the code you might be able to rank the features according to levels of the splits that use them and the number of observations in the nodes, but AFAIK neither the scikit-learn implementation, nor Zelazny's R implementation, have such a thing built-in (although the R one has some functions for individual nodes: https://github.com/Zelazny7/isofor/blob/master/R/interpret.R).

Nevertheless, after you identify the outliers from your sample, you can use that as labeled data for a classification problem, and from there fit a model such as xgboost or random forest that could give you the feature importances.


How to Deploy your trained ML model in client VM without they getting access to code

You can transform your python code into a standalone binary file, which could for example take as input some data and return the predicted labels. If you only put this file in the client's VM and not the .py files, the client could never see the code, but just execute it.
Several libraries exist for this purpose like pyinstaller


In neural networks in general, and in deep learning algorithms (CNN, DNN, etc.) that are also based on neural networks, learnable parameters are parameters that will be learned by the model during the training procedure such as weights and biases.
You can generate learnable parameters for each layer of your model.
DNN is a deep neural network. What I understand that when a neural network becomes deep, it can be said that it's a deep learning model.
So, to compute a DNN or CNN learnable parameters, you can build your model using Keras. It can automatically generate the models learnable parameters for each layer. When you complete building your model, type model.summary(). It generates a list of learnable parameters when you run it.
This link builds a DNN model using using Keras.


Running multiple random forest and combining them

I'm very surprised you're running into this error with only 2mil rows and 38 variables. I would encourage you to have a go doing this in python using SKLearn and see if you run into the same issue. More generally though, if you have too much data (imagine you had 200 million rows), the right thing to do would not be to build multiple forests, but to build each tree with a smaller fraction of the data.

I don't have a suggestion for where to find such an implementation (there might be some way of combining sklearn and dask, I've seen such things for XGBoost), but the general principle would be that you wouldn't hold all of your data in memory, you'd read a subset of it from disk, train a tree, read a new subset, etc.


How to scale outputs from AutoEncoder from multiple models?

I always recommend to use StandardScaler() rather than MinMaxScaler(). The reason is that the latter is too sensitive to outliers. Imagine shrinking the given vector in the [0, 1] interval:
[1, 2, 3, 4, 999]

Most of your data would be squeezed in a tiny subspace of the [0, 1] interval, causing absurd compression. The problem doesn't exist if you use the StandardScaler(). This looks particularly relevant given that you're trying to detect abnormal observations.
(I would use min-max scaling only if I'm sure that a given variable is bounded within a given range of values.)
But most importantly: What are the problems of your output? As far as we know, they might not be due to scaling.

UPDATE:
Lately, a scaling technique that I found very effective is robust min-max scaling. It works just like a min-max, but instead of being done between min and max value, it is done between min and a certain percentile (usually in 95-99). That's very effective when your dataset suffers from very few (but influent) outliers. This method makes sure that 95-99% of your dataset is properly scaled in [0,1], while few outliers take reach out-of-range values.


Best way to search for a similar document given the ngram

You could use a hashing vectorizer on your documents. The result will be a list of vectors. Then vectorize your ngrams in the same way and calculate the projection of this new vector on the old ones. This is equivalent to the database join on an index, but may have less overhead.


What is required in Affinity Propagation

According to sklearn.cluster.AP (in case in AP chosen affinity = "precomputed") the method "fit" requires X  where X is matrix of similarities / affinities, that is, if distance matrix is estimated, use as:

X $ = matrix\ of\ similarities = -distance\ matrix$.


categorizing articles using deep learning

Yes, this is classic sentiment analysis. You'll need training data in order to write this program.

Typically, people will build a language modeling using the text data in the article, tweets, posts, etc. Most folks specify the language model using a Recurrent Neural Network with a Long Short Term Memory. Here's a blog post that has code linked to their github.

Before diving into a more advanced deep neural network like an LSTM, it may be easier to start with a multilayer perceptron using n-grams.

It seems like you may be asking for a couple of different things. So, let me be very explicit. The models I listed above take as input the articles and a label that evaluated the articles as either positive or negative sentiment, the model then returns the likelihood of a given article being positive or negative sentiment. It in no way returns a topic.

To return the "topic" your best bet would be to run Latent Dirichlet Allocation, which would then return the likelihood of a given article being generated from a given topic that was previously learned. It's important to note that these will only be from topics learned on previous training data and not on new topics.

There are standard algorithms available to visualize topic models (e.g., LDAVis) and the embeddings of language models (e.g., word2vec visualizations).

It's worth noting that there is research being done in learning these items jointly (i.e., the latent topic and the classification of good/bad). I'd recommend reviewing this paper.

It may be worth reviewing these items more thoroughly before actually diving into applying them. It sounds like you may not have a complete picture of what your goal is and how deep learning can help you.


In R's randomForest, is predict() non-deterministic?

The model will not change unless you re-train it.
The same input sample should always have the same output value for a given model.
changing the seed, saving and reloading, etc should have no impact on the results.

The training itself is indeed non-deterministic, predict is not.

library(randomForest)
# sample 80% of data to train
split = sample(1:nrow(iris), floor(nrow(iris) * 0.8))
df_train = iris[split,]
df_test = iris[-split,]
# rf model
mod = randomForest(Species ~ ., df_train)
# predictions
set.seed(123)
res1 = predict(mod, df_test)
set.seed(999)
res2 = predict(mod, df_test)
identical(res1, res2)


Input sequence ordering for LSTM network

In order to make this decision, you have to think about what you want the representation to be passed to the next layer (or network output) to represent.

If you want the representation at (after) $t = 0$ to be passed, you should pass the arrays in the $X = [x_{-n} .. x_0]$ order. The LSTM cell will form a representation of the sequence, then, at $t = 0$.

In the reverse case, the representation will indicate the state at (before) $t = -n$

That's not to say that this is always a simple question for LSTM design- frequently in some domains such as NLP (text modeling), a bidirectional LSTM is used. This, basically, means both representations are implemented and used.


Machine Learning: Single input to variable number of outputs

I would try to set a multilabel classification algorithm and make the output standard by adding zeros. So if your data is like this: <1, 1>, <2, [1, 1]>, <3, [2, 1]>, <4, [1, 2, 1, 1]>, <5, [1, 1, 1, 2, 2, 1]>. The maximum number of output is 6. So you could transform your data into something like:
<1, [1,0,0,0,0,0]>, <2, [1, 1,0,0,0,0]>, <3, [2, 1,0,0,0,0]>, <4, [1, 2, 1, 1,0,0]>, <5, [1, 1, 1, 2, 2, 1]>

Another option that occurs to me is to add the limit dynamically. Let say you have your training and test set. You can search for the biggest length and create an algorithm that adds the zeros to both datasets. Then let's say a new data you want to predict has a bigger length, then you'll need to recompute all training and test with for this new prediction. You can even check how extending the limit affects your model.


How to create a historical timeline using Pandas Dataframe and matplotlib

Sorry for the late reply. I don't know whether the below kind of plot suffices for what you are looking.

If yes is the case, you might like matplotlib.hlines. I am providing a sample code for generating a picture like the above.

import matplotlib.pyplot as plt
import random

for x in range(0, 10, 2):
    color = random.choice(['red', 'green', 'blue', 'yellow'])
    plt.hlines(1, x, x + 2, colors=color, lw=5)
    plt.text(x + 1, 1.01, color, ha='center')
plt.show()

Let me know if this was helpful!


The basic answer is yes, what you're describing is possible. But the real question is should you?

I'd like to hear a little bit more about your motivations for doing this. The reality is that data science requires a passion & curiosity that most people (even smart people) just don't have. They just want the money in data science.

I read my fair share of resumes on a daily basis and the most difficult part of a resume with someone strictly with a maths background is that they don't have the business knowledge that is required. For most of us working in the field, data science is not theoretical, these algorithms actually get deployed and people work with them. More importantly, they work to solve core business issues/use cases. Most of the applied data science out there (i.e. projects that pay you) will start with a business use case.

Can you effectively stand in front of a room of executives that haven't taken a math course in decades and explain what you want to do without any formulas or any jargon that will lose the audience? Can you effectively work across multiple teams to create all the infrastructure you will need (data pipelines, storage, processing power, etc)?

From there, you also have to "sell" your solution and implementation because, since it's a use case, you may be asking a large corporation to completely shift their approach from one direction to another. Yes, you know the algorithm works. But to them, you're saying, "turn your business over to me and let this algorithm, that you can't see or touch, run your business". Spoiler alert: you're going to get a lot of executives rolling their eyes at you - it's not an easy thing to do!

So you should be asking yourself if you are an expert in a particular business domain or what are the items that you bring to the table in a business discussion? Those are the parts, and the lack of passion for data science, that will be the biggest roadblocks to making the transition.


How is the weight matrix set and its dimension in CNN

The 1176 is the number of neurons in the layer before the last layer, which is the number of pixels in the convolutional layer but flattened (height*width*n_filters, i.e. 7*7*24=1176). The 3 comes from the number of neurons in the final layer, since you have 3 classes your final layer has 3 neurons, 1 for each class (which will output zero or one). This then gives a weight matrix of 3x1176 for the weights between the second to last layer and the last layer. Using torch we can show the dimensions of the data passed between the layers in the network. In the example below I have omitted the batch normalization and relu layers since they do not affect the dimensions. The dimensions are defined as follows: batch_size * n_channels * height * width.

import torch

x = torch.randn(1, 1, 28, 28)
x.shape
# torch.Size([1, 1, 28, 28])

conv1 = torch.nn.Conv2d(in_channels=1, out_channels=12, kernel_size=3, padding=1)(x)
conv1.shape
# torch.Size([1, 12, 28, 28])

pool1 = torch.nn.MaxPool2d(kernel_size=2, stride=2)(conv1)
pool1.shape
# torch.Size([1, 12, 14, 14])

conv2 = torch.nn.Conv2d(in_channels=12, out_channels=24, kernel_size=3, padding=1)(pool1)
conv2.shape
# torch.Size([1, 24, 14, 14])

pool2 = torch.nn.MaxPool2d(kernel_size=2, stride=2)(conv2)
pool2.shape
# torch.Size([1, 24, 7, 7])

pool2.view(-1).shape
# torch.Size([1176])


Features reduction for the not correlated data set

I believe that what you are looking for is Best Subset, Forward Stepwise, or Lasso?

Here is the R implementation Best Subset

If your dataset is too large, try to subsample your data and then running the feature selection algorithm several times. If you are getting similar results there you go, if the results are different then rank them and select the most frequent ones.

If anyone knows a python implementation for the best subest algorithm, please let me know!


What model should I use for bounding box detection?

There are 2 object detectors that are quite popular:

Haar Cascade Classifier, introduced by Viola and Jones: available on OpenCV (Python, Java and C++) and on Matlab computer vision toolbox (and probably on many other languages) is a great model for when deep learning is not a option.
Yolo, You Only Look Once: It is a real-time object detector and classifier, it uses a DL CNN and if I am not mistaken has cows on it's training dataset (so you can use transfer learning). It is part of the Darknet library and you should totally check the site. Also, check Tiny Yolo is you need some more speed.

You can  should also take a look at:

MMOD (max-margin object-detection): has a great implementation using dlib.
Fast R-CNN and R-CNN: Deep learning classifier, check the paper on this site.

My advice would be to go straight to YOLO and try it out without training for the COW class, then do some fine tunning by retraining few layers with your data. If you need more accuracy than that you can go for R-CNN (probably has some models already trained with cows).

if you lack on computational power you can try MMOD and HCC. But if you got the time you should check BDNN: Binary convolution neural networks for fast object detection, don't know if it is implemented in any libraries already since this paper is relatively fresh (march 2019).


Implementing Frequently bought together using a DB

It is called Association Rule mining. You can implement a very basic version of this algorithm easily which works fine for small  or mid size datasets. What this algorithm does is basically finding the most frequent items in all the transactions. In your example, you can find the frequent items purchased together by customers in your online store. I implemented one in .Net for a course it just took me 2 days to understand and code it.

The format of the data really depends on the implementation, for instance, my algorithm reads data in a text file that each line of the file is a tuple of items:

1,2,3
4,3,5
6,1,4,5,1
7,6,5

Where each line is a transaction, and the numbers can be the items purchased by a user at one checkout. The algorithm needs a number which is called the threshold. It means the algorithm finds the patterns of items that repeated X times (it can be a percentage) in all of the transactions.
The result can be a couple of rows that each is the combination of items that were bought together in a transaction and managed to satisfy the threshold.


Predicting parallel time series with multiple features

Here are two ways to model the problem. The first one is simpler, the second one is more complex but closer to your original statement of the problem.

Store as an input feature

You can consider the store as a feature to pass to your LSTM. With two different stores, just add a binary input feature "store" where store A is 0 and store B is 1, for example.

Then, you can implement it this way.

from keras.models import Sequential
from keras.layers import LSTM, Dense

timesteps = 20
n_features = 5
model = Sequential()
# add +1 to n_features for the store identifier
model.add(LSTM(32, input_shape=(timesteps,n_features + 1), return_sequences=True))
model.add(LSTM(32))
model.add(Dense(1,activation="relu"))
model.compile(optimizer="rmsprop", loss="mse")

One sample of your data at a given timestep will be a vector of the form (feature_1, feature_2, ..., feature_n, store).

Multivariate time-series prediction

Here we input both time series and aim to predict next values of both stores. So you have a shared-LSTM processing store separately, then concatentate both produced embeddings, and compute the predicted values.

from keras.models import Model
from keras.layers import LSTM, Dense, Concatenate, Input

timesteps = 20
n_features = 5

# Store A and B time-series inputs
a_inputs = Input(shape=(timesteps, n_features))
b_inputs = Input(shape=(timesteps, n_features))

# Stacked LSTM
lstm_1 = LSTM(32, return_sequences=True)
lstm_2 = LSTM(32)
# Stacked LSTM on Store A
a_embedding = lstm_1(a_inputs)
a_embedding = lstm_2(a_embedding)
# Stacked LSTM on Store B
b_embedding = lstm_1(b_inputs)
b_embedding = lstm_2(b_embedding)

# Concatenate embeddings and define model
outputs = Concatenate()([a_embedding, b_embedding])
outputs = Dense(64)(outputs)
outputs = Dense(2, activation="relu")(outputs)
model = Model([a_inputs, b_inputs], outputs)
model.compile(optimizer="rmsprop", loss="mse")

Inspired by section 7.1.5 Shared weight sharing, in Deep Learning with Python by F. Chollet


Encoding logic rules in machine learning

I would suggest you apply these rules to the output of your model after it has predicted the results.  In your example, if you are building a DNN classifier that predicts pass or fail, then before you return the prediction back to the calling application, simply check the value and update the model results based on the criteria you have established.

I have performed this in various models where the model may return negative results which are impossible.  In those cases, I would perform a simple update to the results data frame such as:

data.loc[data["y-hat"] < 0, "y-hat"] = 0

You can have as many of these types of 'hard-coded' rules as you need. You can even build a dictionary for them and map the results rather than having a series of statements. That obviously depends on the complexity of the rules.
Hope that helps.


Depends on the dataset, and what you're looking for.

It is possible to do multiple data transformations. Standardization is often the safest choice, but only when the data is normally distributed.

If you have normalized the data, you can always apply standardisation. Standardization does not mutate the data set, but rescaling does create a new data set. From the standardized data set you can go back to the original data set.


TensorFlow / Keras: What is stateful = True in LSTM layers?

This flag is used to have truncated back-propagation through time: the gradient is propagated through the hidden states of the LSTM across the time dimension in the batch and then, in the next batch, the last hidden states are used as input states for the LSTM.

This allows the LSTM to use longer context at training time while constraining the number of steps back for the gradient computation.

I know of two scenarios where this is common:

Language modeling (LM).
Time series modeling.

The training set is a list of sequences, potentially coming from a few documents (LM) or complete time series. During data preparation, the batches are created so that each sequence in a batch is the continuation of the sequence at the same position in the previous batch. This allows having document-level/long time series context when computing predictions.

In these cases, your data is longer than the sequence length dimension in the batch. This may be due to constraints in the available GPU memory (therefore limiting the maximum batch size) or by design due to any other reasons.

Update: Note that the stateful flag affects both training and inference time. If you disable it, you must ensure that at inference time each prediction gets the previous hidden state. For this, you can either create a new model with stateful=True and copy the parameters from the trained model with model.set_weights() or pass it manually. Due to this inconvenience, some people simply set stateful = True always and force the model not use the stored hidden state during training by invoking model.reset_states().


This question is quite open, but nonetheless, here are some:

lemmatization/stemming only makes sense in languages where there is a lemma/stem in the word. Some languages like Chinese have no morphological variations (apart from some arguable cases like the explicit plural 们), and therefore lemmatization and stemming are not applied in Chinese.

Word-based vocabularies are used to represent text in many NLP systems. However, in agglutinative and polysynthetic languages, using word-level vocabularies is crazy, because you can put together a lot of affixes and form a new word, therefore, a prior segmentation of the words is needed.

In some languages like Chinese and Japanese, there are no spaces between words. Therefore, in order to apply almost any NLP, you need a preprocessing step to segment text into words.


After doing some reasearch, I found that OpenMined (a company based on FL and Private AI) Are making strides to develop PySyft. Hence I would suggest continuing with PySyft.


What technique to use in order to identify what position an audio sample is at in a longer audio sample?

In the first step, I think you should start with discretizing your song wave and then take a Fourier Transform for each chunk (in numpy for example you can use numpy.fft()). This Link might be helpful. After that, you can try sequence pattern recognition models.


If you have a whole year data, use the moving window technique, thus you'll have diffrent clusters each month (since users cycles can change), after that, I don't think you should use averaging because it does not take into consideration the times series aspects, which means clusters from the beginning of the year can be very different from the recent ones, you better attribute different weights to different clusters.
I recommend using Subsequence TimeSeries Clustering since it cluster data and it's based on its temporal aspect.
This research paper might help you out Subsequence Clustering
and this is the source code that you can start with Incremental Subsequence Clustering.


Can CBOW model only accept fixed number of words?

That's a good question. The answer is yes and no.

No because the input layer of the CBOW model expects a fixed number of words. So you'll either always input 3 words or 9 words.

Yes because you can however set the sequence length as 9 words and provide just 3 words as context while the remaining 6 words can just be zero vectors. Remember in a CBOW each word is represented by a vector.


is multicollinearity a problem also in classification?

I was thinking about this very issue for some time. It seems like in machine learning, the multicollinearity is usually not such a big deal because it should not mess up the prediction power as such. It is problematic for estimation of the effects, for coefficients of the individual variables - hence, the problem with regression.
This post summarizes it quite well https://stats.stackexchange.com/a/168631/197070.


AttributeError: 'numpy.ndarray' object has no attribute 'columns'

The problem is that train_test_split(X, y, ...) returns numpy arrays and not pandas dataframes. Numpy arrays have no attribute named columns

If you want to see what features SelectFromModel kept, you need to substitute X_train (which is a numpy.array) with X which is a pandas.DataFrame.

selected_feat= X.columns[(sel.get_support())]

This will return a list of the columns kept by the feature selector.

If you wanted to see how many features were kept you can just run this:

sel.get_support().sum()  # by default this will count 'True' as 1 and 'False' as 0


Tools for nutritional value tracking and visualization

I would recommend the Python libraries: matplotlib and seaborn for generating and exporting static plots, or Plotly for interactive plots with hover features that you can export to a web browser.


Classifying time series data that overlap

You could calculate the distance between time series using Dynamic Time Wrapping DTW and then you could cluster them using K-means or so. Here is a python implementation of the DTW or use the dtw package in R.


How to determine best parameters after grid search?

I too was in your place when I started using Neural networks. There are so many hyper-parameters to choose, and each take on many values. Like Emre said, you need to check the model which is giving best metric score on your data (Cross validation set). The parameter values of that model will be your optimized values. You can also check this link


How to manage memory constraint and increase speed for 1 vs rest image similarity comparison for over 100k images for computer vision?

One option is locality-sensitive hashing (LSH) which hashes similar input items into the same "buckets" with high probability. LSH is frequently used for near-duplicate image detection, a variation of your problem.


Machine learning method to predict event date

You can both utilize regression and classification models in here.
Also I suggest log transform your financial features (eg: debt, last credit risk, average loan amount etc) when using algorithms except decision tree based models (since they are not affected by monotonic transformations).

Classification

You can construct a model such that predicts if a customer would pay "x" days after payment day or not. In this model you need to find best "x" by comparing results of different x day models. Also you need to find optimal confidence range in each model in comparison.

Regression

In this setting, you will predict how much day a customer will pay after payment date. Again you should obtain confidence intervals such that you calculate 3 days for a customer which means she will pay 3 days after (it can be earlier for negative predictions). However you know that your model is powerful on +2 -2 day interval. Then you should say this particular customer will pay in 1 or 5 days after payment date.

Algorithms

I suggest xgboost, lightgbm and RNNs. But its not so clear maybe you have a linear space and it converges fast with SVMs, line fits etc. Yet they are pretty good, kaggle winner learners.

I hope it helps. I thinks it will be a good starting point.


What is the quantity sold for a specific fruit & country combination?

Minimize

If I understand your question correctly, you want to solve an optimization problem. Using two countries and two fruits as an example, the problem can be written as a linear program:

Minimize

$$\Delta = \delta_{Apple} + \delta_{Mango} + \delta_{India} + \delta_{China}$$

s.t.

$$X_{India, Apple} + X_{China, Apple} = 508 + \delta_{Apple}$$
$$X_{India, Mango} + X_{China, Mango} = 608 + \delta_{Mango}$$
$$X_{India, Apple} + X_{India, Mango} = 1403 + \delta_{India}$$
$$X_{China, Apple} + X_{China, Mango} = 1370 + \delta_{China}$$

A simple optimization problem like this can be solved using many software packages, including Excel. Note that it is likely that the solution is not unique.


Balancing data-sets for regression problems

First, the histogram you plotted is that of the marginal distribution. The residuals might look perfectly fine once you regress on the variables. Linear regression is an estimation of Y conditional on the variables X; thus, in order to determine whether the distribution is a good fit, you examine the residuals which are assumed to be gaussian.

If you have data that is bounded between an interval, one approach would be modeling it using a beta regression. If the data is constrained to be positive, assuming a gamma distribution could work as well. You must, however, check the diagnostic plots afterwards to ensure that the distribution assumptions that you are making are correct.

See Beta Regression by Ferrari, A Better Lemon Squeezer by Smithson '06, and Extending the General Linear Model by Faraway. Also, for Bayesian implementations of regression models, you could look at Gelman's book.


do it without ravel (or reshaping of any kind).

Or if you going to do than transform dataframe X_train into an matrix also. This is the correct format  fit_sample


Model not learning when using transfer learning

Well the critical step for finetuning/transfer learning any model in tensorflow[1,2+]:

use right preprocessing for images (find the preprocessing which requires mobilenetv2)
start with "imagenet" weights
change the BatchNormalization behavior, in tf the default momentum value is 0.99 which is not best, in pytorch it is set to 0.9 which is better when finetunig your model and you don't have lot of data (see the link and this for more)


How to evaluate the clustering result when cluster numbers are not equal to data set class

If no pair of your classes are overlapping and they do classify each and every element of the data set, then they constitute a partition of that set. So does your clustering.

In such a case, I would suggest to use a distance metric to compare two partitions, instead of any bona fide index, even if it's popular -unless that index happens to be a metric as well.

One such distances for comparing partitions is the $VI$ index, rediscovered by Maria Meila$\phantom{,}^\dagger$ a few years back, but its origin dating back to the 80's in a physics paper by Zurek$\phantom{,}^{\ddagger}$.

How to calculate distance between two partitions P and Q:

Determine $P\wedge Q, $the "intersection" (I always mix up the words join and meet here) of P and Q. This is simply the partition formed by taking the intersection of each cluster of P against each cluster of Q.
For all three partitions $P$, $Q$, $P\wedge Q$ calculate their entropy $H$ defined as follows: $$H(P)=-\frac{1}{N}\sum_{i=1}^Np_i\log p_i\;,\;p_i\equiv\frac{n_i}{N}$$ where $N$ is the total number of elements you clustered/classified and $n_i$ is the number of elements within cluster/class $i$ of a given partition $P$.
The distance $d(P,Q)$ between partitions P and Q is then given by $$d(P,Q)=2H(P\wedge Q)-H(P)-H(Q)$$

The reason for using a real metric is simple: a metric satisfy the triangular inequality and our intuitions of "close" and "similar" heavily relies on that. Any ad-hoc measure of dis-similarity (distance) that doesn't satisfy the triangular inequality will eventually give you results like this: A is "$d$ away" from B (d being concrete number), with $d$ smaller than a given bound, say $D$. Then you find A close to C by an amount $d'<D$. If this distance were not a metric, it could then happen that the distance $d''$ between B and C is say $2D$. For a sound metric distance, it should be
$d''\leq d+d'<2D$. In other words A would be "similar" to B and C, but the last two would be "different"/"not similar" from each other. This is counterintuitive.

The downside is that the VI distance may be computationally more expensive than calculating say a Rand index. The upside is that it's mathematically sound and makes sense.

You can use R to calculate the VI distance. You can also use Partanalyzer a program I wrote some time ago (https://github.com/MASantos/Partanalyzer). It needs a rewrite to make it more efficient, but it's open source, so you can modify it at will.

$\dagger$: Meila, M. Journal of Multivariate Analysis, 98, 873 (2007).

$\ddagger$: W.H. Zurek, Nature, vol.341, p.119 (1989).


Smaller test data set than training data set in machine learning

It's normal (and expected even) to have a Test Set that is smaller than your Training Set.

In general, the more training data you have, the better your performance should be. That is, there'll be less variation in your model parameters if trained on more examples.

This is especially true for Deep Learning approaches (your Convolutional Neural Network, CNN, you mentioned).

Below is a graph that illustrates the above. Where 'Classical Learning' refers to Machine Learning models (SVM, Random Forest, etc.)

From this explanation and graph, it should now make sense why you get a lower test accuracy when you have a smaller training set. There simply isn't enough variation in the training samples, leading to overfitting.

There are numerous ways to avoid overfitting, through data augmentation, adding Dropout layers (in the case of CNN), or regularizing your model. Going into the details is beyond this answer.

However, one of the best and easiest ways to help avoid overfitting is through increasing the the training set.

By increasing the training set, you follow the graph above and decrease variation in parameters but more importantly expose your model to more samples and variation in the dataset.

This helps explain, in part, why you see a higher test accuracy when using a larger test set.

Should you reduce your test set to maintain a ratio? Probably not, in a similar way to your parameters experiencing variation with a small training set the same is true with your test set. You'll want a big enough test set to cover enough variation in samples so the accuracy returned can be trusted.

This StackOverflow answer provides good detail and rationale on choosing the training/test split.

Let me know if you have any questions! I can't comment yet, hence a relatively general answer.


Can one build linear models on "chunks" of the data set, if one can't build them on the entire data set?

If "variables" refers to training examples:
You can use Stochastic Gradient Descent (SGD) where each iteration uses one training example.
Or you could use Mini-Batch Gradient Descent where each iteration uses a partition of the training set.  SGD is Mini-Batch Gradient Descent where the partition size is one training example.

If "variables" refers to features:
You should use dimensionality reduction to reduce your number of features.  For instance, you can use Principal Component Analysis (PCA) to reduce your feature vector size while maintaining high variance.  This would also help your models train significantly faster.


LabelEncoding, if not processed again before passing into the model, will impose an order on the categories (namely the 0<1<3) that you mentioned. Here, since it does not seem like you have a lot of categories, one-hot encoding is the easiest to implement and also more effective (the categorical values are nominal rather than ordinal here).

These two articles are a decent starting point for categorical encoding in Python:
https://pbpython.com/categorical-encoding.html

https://medium.com/@contactsunny/label-encoder-vs-one-hot-encoder-in-machine-learning-3fc273365621

As for the TypeError, can you double check what variable type train_df is?


Cross Validation with Artificial Neural Network in Keras

It is likely that your train variable in kf.split(train): is a list of two lists e.g. train_x and train_y or something similar. I am guessing this because the KFold API is only detecting only two entries in it, which it is unable to partition in 5 subsets (folds).


What's the right way to setup an image classifier by multiple params?

You normally one hot encode your labels so that every possible attribute gets it's own binary representation. So if you have 10 attributes they would be represented as [attr1, attr2, attr3, ..., attr10], with values either 1 if the attribute is present, or 0 if it is not.

Then you train a network with the same number of output neurons as possible categories and use a sigmoid activation function.


Is there a way to overlay stem plot over line plot in python?

I think what you are looking for is called twinx. I just used one of the examples given and slightly modified the input parameters:

import numpy as np
import matplotlib.pyplot as plt

date_name = "Date"
stem_name = "Day_Perc_Change"
plot_name = "No. of Trades"
t = df[date_name]
data1 = df[stem_name]
data2 = df[plot_name]

fig, ax1 = plt.subplots()

color = 'tab:blue'
ax1.set_xlabel(date_name)
ax1.set_ylabel(stem_name, color=color)
ax1.stem(t, data1)
ax1.tick_params(axis='y', labelcolor=color)

ax2 = ax1.twinx()  # instantiate a second axes that shares the same x-axis

color = 'tab:red'
ax2.set_ylabel(plot_name, color=color)  # we already handled the x-label with ax1
ax2.plot(t, data2, color=color)
ax2.tick_params(axis='y', labelcolor=color)

fig.tight_layout()  # otherwise the right y-label is slightly clipped
plt.show()


Training the parameters of a Restricted Boltzman machine

I believe the problem is that log-likelihood is not directly computable, because of exponential in number of units complexity. There exist different proxies to the true log-likelihood, for instance pseudo log-likelihood (more here), and in principle you can train RBM until PLL is not changed too much.

Hovewer, there is a lot of randomness involved in the training process, so PLL most likely will be quite noisy (even true log-likelihood would be I believe).


How not to see the output in the console when adding a new column in a table?

R by default returns the updated dataframe to console.You can assign it to variable by avoiding it.

td <- td[,fv_mc2:=1]

By this way you can avoid the console of 14k data.


How to JUST represent words as embeddings by pretrained BERT?

BERT does not give word representations, but subword representations (see this). Nevertheless, it is common to average the representations of the subwords in a word to obtain a "word-level" representation.

You may try to handle this as a normal tagging problem, where the tag of each word is the class associated with the word, much like part-of-speech (POS) (e.g. this) tagging or named entity recognition (NER) (e.g. this). Normally, you associate the tag to either the first or the last subword token in the word. If you prepare a dataset that way, you could fine-tune BERT to perform word tagging with the classes you need. If you only have the words, you could find some text corpus (ideally of the intended domains) and apply the described data preparation process.


You may use Permutation importance
 - Get your base-line score
 - Permutate a feature values. May replace with Random values
- Calculate the score again
- The dip is the feature importance for that Feature
- Repeat for all the Features

....Breiman and Cutler also described permutation importance, which measures the importance of a feature as follows. Record a baseline accuracy (classifier) or R2 score (regressor) by passing a validation set or the out-of-bag (OOB) samples through the Random Forest. Permute the column values of a single predictor feature and then pass all test samples back through the Random Forest and recompute the accuracy or R

To check the importance for the individual Class i.e. 0/1
Extrapolate the same to check if the increase is more for False-Positive or False-Negative.
Read Beware Default Random Forest Importances for more explanation.
Few other quotes from the page-

Any machine learning model can use the strategy of permuting columns to compute feature importances. This fact is under-appreciated in academia and industry.

The permutation mechanism is much more computationally expensive than the mean decrease in impurity mechanism, but the results are more reliable. The permutation importance strategy does not require retraining the model after permuting each column; we just have to re-run the perturbed test samples through the already-trained model.


You have two scenarios here.

The vectors with the same labels are close to each other.

Then, what you can do is perhaps use the distance between the two closest/furthest elements in the groups with label x and the group with label y.

You could also create an average vector for each label, and then simply get the distance between these.

Your vectors are not close to each other

This is not an issue. Here, what you'll want to see, is if for instance, for each vector with label x, there is a vector with label y close to it.

To do that, you could use the Earth Mover's Distance which gives you a single score when you are trying to see how far two "groups of things" are.


How to change random number of random elements in a column by group in R?

The idea is correct, even if you have a couple of mistakes in your code (like id is not defined, and inside paste you want to use just n, not df$Name). This is not super compressed code but it does the job:

Name = c('Ed','Ed','Bill','Tom','Tom','Tom','Ed','Bill','Bill','Bill')
df = data.frame(Name)

for (n in unique(df$Name)){

    # get indices
    indices = which(df$Name==n)

    # sample size
    samp_size = round(length(df[df$Name==n,])/2)

    # get indices to replace
    samp = sample(indices, samp_size)

    # need to set column as character
    df$Name = as.character(df$Name)

    # set new values
    df[samp,] =  paste(n,'_X',sep='')

    # set again column as factor
    df$Name = as.factor(df$Name)
}

Out:

     Name
1      Ed
2    Ed_X
3    Bill
4   Tom_X
5   Tom_X
6     Tom
7    Ed_X
8    Bill
9  Bill_X
10 Bill_X


Should I use a regression or classification framework to predict rankings?

Sounds like a classification task. Since your $y$ is ordered, you may also consider "ordered probit" or "ordered logit". These methods make sure that the order in $y$ is accounted for.


How do I identify clusters that match on categorical data?

You have just 500 data points...

Excel of course is the worst possible tool though.

Anyway, build a dictionary. Put everybody in there 6 times: 1 with all five strengths, and 5 times with one strength omitted. Then you can easily identify the largest groups, and you can also perform various completion operations easily: if you have identified a group with strengths A B C D E, you can add all that have ABCD etc. using the dictionary.


Understanding text conversion into SVM input

Text can be converted to data via the use of concept clusters (after stemming and stopping), or to count (frequencies) via use of n-grams.  N-grams are basically tabulations of the 1-gram count (frequency) of alphabet characters (a though z) in each document, and counts of 2-grams (aa to zz), 3-grams (aaa through zzz), up to about 5-grams (aaaaa through zzzzz).  Beyond 5-grams, the data will be sparse and less informative.  Thus, a dataset can be constructed for which rows represent documents, and columns represent n-grams. The data values themselves are the total number of occurrences of each gram found in each document.

FYI - n-grams have proven to be the best technique for identifying different languages based on characters.

Regarding SVMs, focus on the SVM literature.


Division of numbers from contours in opencv in python for cnn

After browsing for long hours I have found out that taking the contours as ROI is a considerable option


Are pre-trained models vor CIFAR-10 / CIFAR-100 / SVHN available?

Yes, google the various "Model Zoo" (e.g. one for caffe here: https://github.com/BVLC/caffe/wiki/Model-Zoo), they host a bunch of models including  "tiny" ones.


Changing default values of ANNIE resources in GATE from Java code

For any of these situations the best approach is to construct the application as you need it using GATE Developer, "save application state" to create your own custom gapp file, then pass your new gapp to the PersistenceManager instead of loading the standard ANNIE_with_defaults.


Aleksandr Blekh's answer in this older question provides a lot of interesting reading material for time-series clustering methods and examples. Also, I include below some interesting reading material for calculating similarity among multivariate time-series (the latest 2 are quite old but I think they are very interesting):

An approach on the use of DTW with multivariate time-series (the paper actual refers to classification but you might want to use the idea and adjust it for clustering)
A paper on clustering of time-series
A PCA-based similarity measure for multivariate time-series
A review on feature extraction and pattern recognition methods in time-series data

Before proceeding with any method, I believe it is important to spend some time to think of the following:

Try to select the right step for your input data (e.g. if the time-steps are per second, the time-series might be too long and unnecessarily detailed for this job, while hourly data might catch the patterns better).
Seasonality might be interesting to take under consideration: e.g. if the time-series are hourly and last for several days/months, there might be some daily/monthly seasonality. In this case, you might want to calculate and compare the average day/month from each time-series (if so, you will also need to decide whether weekdays and weekends should be averaged all together or treated in a different way).
Depending on what you are trying to find out, you need to decide
whether Dynamic time warping (DTW) is useful for you. For example, if
2 time-series have the exact same pattern but one of the 2 has a
time-delay, should they still belong in the same cluster? (and how
small/big time-delay is acceptable to put them in the same cluster?)


Single Layer Perceptron with three classes

The standard way to do this is called 'one versus all'... you train three perceptrons. First 1 target = is class a?, 2nd perceptron target = is class b? 3rd = is class c. You just train each perceptron separately, and then take max of the three perceptrons to decide class


Back-propagation through max pooling layers

If this is correct then every "neuron" of the pooling layer has the same gradient?

No. It depends on the weights and activation function. And most typically the weights are different from the first neuron of the pooling layer to the FC layer as from the second layer of the pooling layer to the FC layer.

So typically you will have a situation like:

$ FC_i = f(\sum_j W_{ij} P_j)$

Where $FC_i$ is the ith neuron in the fully connected layer, $P_j$ is the jth neuron in the pooling layer and $f$ is the activation function and $W$ the weigths.

This means that the gradient with respect to P_j is

$grad(P_j) = \sum_i grad(FC_i) f^\prime W_{ij}$.

Which is different for j=0 or j=1 because the W is different.

  And the second question is when the pooling layer is connected to another convolution layer. How do I compute the gradient then?

It does not make a difference what type of layer it is connected to. It is the same equation all the time. Sum of all the gradients on the next layer multiplied by how the output of those neurons is affected by the neuron on the previous layer. The difference between FC and Convolution is that in FC all neurons in the next layer will provide a contribution (even if maybe small) but in Convolution most neurons in the next layer are not at all affected by the neuron in the previous layer so their contribution is exactly zero.

  For the topmost rightmost "neuron" of the pooling layer (the outlined green one) I just take the gradient of the purple neuron in the next conv layer and route it back, right?

Right. Plus also the gradient of any other neurons on that convolution layer which take as input the topmost rightmost neuron of the pooling layer.

  How about the filled green one? I need to multiply together the first column of neurons in the next layer because of the chain rule? Or do I need to add them?

Add them. Because of the chain rule.

Max Pooling
Up to the this point, the fact that it was max pool was totally irrelevant as you can see. Max pooling is just the that the activation function on that layer is $max$. So this means that the gradients for the previous layers $grad(PR_j)$ are:

$grad(PR_j) = \sum_i grad(P_i) f^\prime W_{ij}$.

But now $f = id$ for the max neuron and $f = 0$ for all other neurons, so $f^\prime = 1$ for the max neuron in the previous layer and $f^\prime = 0$ for all other neurons. So:

$grad(PR_{max neuron}) = \sum_i grad(P_i) W_{i\ {max\ neuron}}$,

$grad(PR_{others}) = 0.$


Advance Methods of Understanding Significance of Customer Behaviors

Welcome to DS stack exchange! Some of your questions/statements are not too clear, I'll try to answer to the best of my abilities but try to ask more precise questions in the future to get better answers (and avoid being down-voted).

To your points 1. and 2.: I would say it's highly unlikely that you'd be able to attribute a small fluctuation like 8% to just moving a button, moreover on such a short time span. And the question you should ask yourself is: "Is this the only thing that you've changed?". The fluctuation in point 2. could easily be attributed to effects like it being the weekend. Two things you'll want to look at to get you started are A/B testing and ARMA models.

Use A/B testing to show some of your customers e.g. a button on a new position or a new background color. Now check how likely they are to return vs. the batch of customers who saw the old design. If the new one is significantly better, then you can be confident that you should roll out the new design.

Set up an ARMA model to predict daily aggregate traffic. I'd advice against smoothing of any kind, it would likely be a bad idea. You first need to set a baseline which you do by comparing e.g. a month's worth of predictions with the actual counts. From this you can estimate the probability distribution of the errors you'd expect from your model. Now when you roll out new features/contents, you can see if the predicted traffic (which is based on data for the old website) is significantly lower that the actual traffic. If so, your new feature/content is a success.

Hope this helps!


Possible for batch size of neural network to be too small?

A batch size of 32 is commonly used and referred to as "small," but
  why don't we use a smaller batch size of 1 to guarantee convergence?

The issue is that a small batch size both helps and hurts convergence. Updating the weights based on a small batch will be more noisy. The noise can be good, helping by jerking out of local optima. However, the same noise and jerkiness will prevent the descent from fully converging to an optima at all.

It is a trade off between these factors, the best batch size will depend on the shape of the error manifold. Larger batch sizes are better on convex errors and smaller batch size are good on errors with lots of deeper local optima.


Downstream models are simply models that come after the model in question, in this case ResNet variants. Models for various topics within the computer vision domain often use a backbone to extract features from images, after which a downstream model is used to help to fit the model better to the task at hand. Tables 5, 6, and 7 in the linked paper give a good overview of the different ways backbones are often combined for topics such as object detection and segmentation.


What's the difference between hessian regularisation (min_child_weight) and loss regularisation (gamma)? When to use one over another?

Loss regularization also prevents the trees from growing; it does not penalize the number of trees.
I think you're right that which is better (of course, you could use both) depends on the data and other hyperparameters.  You can think though about whether a given node will split, depending on the type of regularization:
If you have a large node with some potential split that would create two other large nodes, but that split doesn't decrease the loss much, gamma will prevent it but min_child_weight won't.  If you have a node with a potential split with one child being very small but that decreases the loss a lot, gamma will allow it but min_child_weight won't.  So, personal opinion (based on that thought experiment, and not experience tuning these two together or against each other): set min_child_weight large enough that you're comfortable that a node with that many samples (in the squared-loss case, and appropriately transformed in other cases) is big enough to absorb noisiness of your population [by which I mean: one sample is surely quite likely to be random noise; an average of 100 might be better, but might not be enough, depending on your data]; after setting that, tune gamma.  Better, tune both of them together, but still use the above intuition of your data to give a reasonable range for min_child_weight.


Metric to use to choose between different models - Hyperparameters tuning

Essentially, the function of your testset is to evaluate the performance of your model on new data. It mimics the situation of your model being put into production. The validation set is used for optimizing your algorithm.
Personally I would recommend tuning your algorithm using your validation set and using the hyperparameters of the training epoch with the lowest validation loss. The accuracy of the model using these hyperparameters on your testset can be used to estimate the performance of your model in the 'real world'.


Python: advice on which machine learning algorithm to use for a problem which involves lots of randomness

It sounds like you have good data - 50 columns and 100,00 rows!
I would do exploratory data analysis (EDA) and look for variables (columns) that are correlated with the response variable (re-offending) but NOT correlated with each other. If you can find a handful (~10) of these then you can build an excellent regression model.
Other techniques to try could include random forests and cluster analysis. Both can be done really quickly in Python so you can compare lots of different hyperparameter iterations.


I'll be honest, I haven't thoroughly checked your code. However, I can see that the range of values of your dataset is approx [0,12000]. As an engineer, I see that:

sqrt(MSE) = sqrt(437000) = 661 units.
MAE = 400 units.
MAPE = 2.5 which means that MAE can be up to 0.025*12000= 250 units.

All three cases show similar magnitude of error, so I wouldn't say that "MAPE is quite low but you're getting high mse and MAE".

Those 3 values explain the results from similar yet different perspectives. Keep in mind, if the values were all the same, there would have been no need for all 3 of those metrics to exist :)


It seems that you have a domain adaptation problem. The samples from the two sources behaves differently.

I suggest reading Frustratingly Easy Domain Adaptation. As the name hints, the solution is easy , popular (800 citation until now) and a good survey of other directions.

I understand that the classifier that you run on the entire dataset was train on it.
How well does the classifiers trained on the single sources perform on the other sources?
How many of the samples belong to the first source?
Will you have an indication at production of the source of the sample?
The answer to these question might open more directions.