Understanding ConfusionMatrix for Google Prediction API

As explained in the documentation:

  This shows an estimate for how this model will do in predictions. This is first indexed by the true class label. For each true class label, this provides a pair {predicted_label, count}, where count is the estimated number of times the model will predict the predicted label given the true label.

If you are not sure what a confusion matrix, see Wikipedia, where the "actual class" refers to the same thing as the "true class" in the Google documentation.


laptops which are suitable for heavy Image and video processing

The answer to such a question is opinion based and the question itself is very broad.

I have used an HP Envy in the past, with a 4GB Nvidia 950M GPU, which worked well with Linux installed.

In general:

The higher the compute capability of the GPU the better (look here at the list for Nvidia GPUs under GeForce products for notebooks).
Another option worth considering would be to build compute - you will lose portability, but get a vast improvement of power for the same money.
Finally, you could buy a cheap laptop and then use the rest of your budget on cloud services, such as Google Cloud, Amazon, etc. This might be the best option to get started (with some free credit they usually offer for new accounts) - also if you think you might only be using the GPU for your thesis, and not training huge models for days at a time after that period. If you know you'll be training large models over long periods of time, it's generally more cost effective to buy the hardware.


Overall classification is generally better when the decision rules of each component classifier differ and provide complimentary information.

So the question becomes: Can you set up your component classifiers so that their decision rules are different and compliment one another based on the feature space? e.g. Does Post 1 have a significantly different feature space than Post 2?  etc. If so, the ensemble approach should be beneficial.

Which technique?  If you can highly train each classifier and make it an expert in different regions of the feature space, try models:

mixture model
mixture distribution
gating subsystem
winner take all.


How do I best visualize this voltage data for a science project

As per my observation, you need to visualize different configurations. WEKA can be used to visualize this type of things where it automatically stack multiple Graphs side by side to find trends. There are other features as well.

Yo can further explore here

enter link description here


Problem when using Autograd with nn.Embedding in Pytorch

So, this is just half answer, I write it here to be able to format the text clearly. You are facing troubles because you are trying to do something that you shouldn't, which is applying gradient to indices instead of embeddings. When using embeddings (all kinds, not only BERT), before feeding them to a model, sentences must be represented with embedding indices, which are just number associated with specific embedding vectors. Those indices are just values mapping words to vectors, you will never apply any operation on them, especially you will never train them, cause they are not parameters.

['Just an example...'] # text
       |
       | Words are turned into indices
       v
[1., 2., 3., 4., 4., 4.] # indices, tensor type long, no sense in applying gradient here
       |
       | Indices are used to retrieve the real embedding vectors
       v
[0.3242, 0.2354, 0.8763, 0.4325, 0.4325, 0.4325] # embeddings, tensor type float, this is what you want to train

If you want to fine tune BERT for a specific task, I suggest you to take a look to this tutorial BERT-fine-tuning


How to deal with features that have widely different dimensionalities

One solution is using the same technique that is used in sequence learning. In that technique we use from Recurrent Neural Network and LSTM module.

  we can train recurrent neural networks to learn very specific outputs for an arbitrary sequence of inputs, which is very powerful.

To know more about this, you can follow this link.


Approach 1: create features before transforming

If you want to create a categorical variable based on a numerical variable and then treat it in cat_pipeline, you need to do create it before the column transformer.

Implement a transformer (called "bucketer" ?) that takes p variables and transforms it into p+1 (if you want to add the categorical representation and keep the initial numerical feature). This transformer is the FIRST step of your pipe.

Then, create a ColumnTransformer (I think it is more suited for your case, but have not enough details to be sure. I suggest you read this to be sure). This second transformation is the second step in your pipe.

Each branch is fed with respect to what it should output because feature creation (bucketing) was done prior to column transformer.

Approach 2: create features in transforming

Otherwise, you can create two main paths :

One branch will output only categorical features, whatever the input type.
The other will output only numerical features, whatever the input type.

You may want to re-use your feature selector in some branches, I just wanted to illustrate the two possible approaches

Hope this helps


Excerpts from the very same source tell the answers:

In order to localize, high resolution features from the contracting path are combined with the upsampled output.

--

A successive convolution layer can then learn to assemble a more precise output based on this information.

--

The cropping is necessary due to the loss of border pixels in every convolution.

So, the combined data adds precision of U-net (reduces error) and cropping filters blurry borders out of that combined data.
Edit:

For the pixels on the border of image matrix, some elements of the kernel might stands out of the image matrix and therefore does not have any corresponding element from the image matrix.

(kernel is the matrix with which the convolution of image is done to produce the output)
Source of edit


Your job seems like a map-reduce job and hence might be good for Hadoop. Hadoop has a zoo of an ecosystem though.

Hadoop is a distributed file system. It distributes data on a cluster and because this data is split up it can be analysed in parallel. Out of the box, Hadoop allows you to write map reduce jobs on the platform and this is why it might help with your problem.

The following technologies work on Hadoop:

If the data can be represented in a table format you might want to check out technologies like hive and impala. Impala uses all the distributed memory across a cluster and is very performant while it allows you to still work with a table structure.
A more new, but promising alternative might also be spark which allows for more iterative procedures to be run on the cluster.

Don't underestimate the amount of time setting up and the amount of time needed to understand Hadoop.


Pandas, R, and interoperable data frame formats

Is your data in a data frame?  If so, feather is what you're looking for.


Correlation between time to event data and continuous data

There is nothing wrong with using Pearson's correlation coefficient (frequently called THE correlation coefficient) with these variables. I would also have a look at Spearman's rank correlation coefficient. It is less sensitive to extreme values.

http://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient

However, both coefficients have the drawback that they only measure linear relationships. Hence it is worthwile to have a look at the scatterplot of the variables. Adding a smoother to the graph can be worthwile too, especially  if the plot is clutterd and you have trouble to spot a relationship with the naked eye.


Imbalanced dataset - Undersampling & multiple classifiers

Maybe your (3) can be complemented to an oversampling of the other classes. I don't think every oversampling generates overfitting. I agree with you, it will make you lose information. But the only way to know if it's that bad is by checking it.

Regarding (4), I don't see how you'll manage the predictions of these multiple classifiers. You could test binary classifiers for each class, like classifier1 for A/not-A, classifier2 for B/not-B. In this case, undersampling could be applied.

However, all these advices are just speculation. You must test on your data, and see the results with more evaluation tools, such as learning curves and feature importance analysis.


There is a field of online machine learning where incremental updates can be made to a model, including a variation of stochastic gradient descent called incremental stochastic gradient descent.

Generally, several rows of data are batched together to perform an update to the parameters. Training on a single row of data at a time can lead to unstable updates.


Interpretable xgboost - Calculate cover feature importance

A more detailed explanation of cover can be found in the code

  cover: the sum of second order gradient of training data classified to the leaf, if it is square loss, this simply corresponds to the number of instances in that branch. Deeper in the tree a node is, lower this metric will be

You can find this here: cover definition in the code

This basically mean that for each split the second order gradient of the specified loss is computed (per sample). Then this number is scaled by the number of samples.


Is there a way to manually remove support vectors from a scikit-learn model?

I was having the same problem. I managed to do it with Scikit's OneClassSVM; maybe it can also help in your case.
I adjusted some of the parameters you mentioned plus a few more (just in case) including:

_dual_coef_
dual_coef_
_intercept_
intercept_
support_
support_vectors_

Ex: For OneClassSVM, if you obtained 10 support vectors (after fitting dataset X) and wanted to prune the last one (index -1) the code would be as follows:

model = OneClassSVM()
model.fit(X)

model._intercept_ = model._intercept_ + temp_model.dual_coef_[0][-1]
model.intercept_ = model.intercept_ + temp_model.dual_coef_[0][-1]
model._dual_coef_ = np.delete(model._dual_coef_, -1, axis=1)
model.dual_coef_ = np.delete(model.dual_coef_, -1, axis=1)
model.support_ = np.delete(model.support_, -1, axis=0)
model.support_vectors_ = np.delete(model.support_vectors_, -1, axis=0)


Why are we taking the square root of the gradient in Adagrad?

We are taking the square root of $G_i$  because $G_i$ is the sum of squares of the gradient of weight $i$ from iterations $1$ to the current iteration. Since we squared all the gradients of all timesteps, $G_i$ was very large, and so we take its square root. Now you may think that, then why don't we just add the gradients instead of squaring the gradients, then adding. The answer to this question is that we square it to prevent negative gradients, so that negative gradients and positive gradients don't cancel each other out. Now you may think that why don't we take absolute value of the gradient then, the answer to this question is
here
That post was about root mean squared error or mean absolute error.
If you think carefully, these both issues are the same.
So, we take the square and not the absolute value. Now you may think that why don't we  power the gradient to some even power, why only square? Now, I leave this question to the consideration of the reader


Yes, LSTMs are fairly used in time series prediction. They can even handle missing data which is quite common in timeseries: Learning to diagnose with LSTM recurrent neural network. Still, I don't recommend starting directly with LSTM as the training takes time and you have to try many parameters to find the best that fits your data. You should start with some basic regressors like RandomForest or XGBoost.


Why do we need the sigmoid function in logistic regression?

If you go and define the hypothesis function as h(x) = p^T * x, and claim that y = 1 if p^T * x >= 0, you are complicating the things even more.

This is due to the fact that during training time you will have to calculate the derivative of the function h(x). Now your function can lead to exploding or vanishing gradients depending on your input and initialization.

Logistic Sigmoid solves this problem and it also has a nice gradient g(x)(1 - g(x)). So rather that complicating things, sigmoid actually simplifies the calculation.


How to run a model in an application using gpu (without CUDA)

I'm pretty sure that you will need CUDA to use the GPU, given you have included the tag tensorflow. All of the ops in tensorflow are written in C++, which the uses the CUDA API to speak to the GPU. Perhaps there are libraries out there for performing matrix multiplication on the GPU without CUDA, but I haven't heard of a deep learning framework that doesn't use CUDA, when executing on the GPU.

What you can perhaps do it find a solution that works using Docker. The latest version doesn't require any additional Nvidia software to be installed. It would mean your customer doesn't need to mess around with CUDA themselves, they just need docker installed, which is generally less than 5 minutes work. Here is the Tensorflow documentation for a starter.

In addition, using Docker is perhaps the most professional and isolated way to deliver code to somebody else's machine. There are many benefits and it works on Windows, Mac, Linux. Any changes you make can simply be pushed to their machine, without them having to change anything. It can also scale arbitrarily to multiple machines, running locally or in the cloud, the list goes on and on...


[edited, I misread the question in the first version]

The fact that the label is determined from a combination of values from this feature is not a problem in itself: if it makes sense, it's always better to give the best indicators to the learning algorithm. So the only questions are:

whether it makes sense for your problem to have the feature provided as input for any new instance: if yes, then there's no reason to remove it.
whether it's useful to apply ML to your problem: if the label can be determined directly from a single feature, it's simply not useful to train a model.

You mention that the label is based on information from the previous/next two records. Keep in mind that the model needs to predict its target for any individual instance as input, unless you're using a sequential model (for instance with times series).


What are good resources to learn the basics of graph theory

I can think of two very good resources that may help you to acquire the foundational knowledge on graph theory and complex networks:

Network Science by Barabasi is you prefer a web tutorial: http://networksciencebook.com/
The Structure of Complex Networks by Estrada if you prefer a book: https://oxford.universitypressscholarship.com/view/10.1093/acprof:oso/9780199591756.001.0001/acprof-9780199591756


Memory error when using more layers in CNN model

More layers mean more parameters for your network, which in turn means more required space in memory to store those parameters.
The only solution (besides increasing the memory of your computer) is reducing the size of your network. A few pointers on this: Firstly, 3-D CNNs require much more space than 2-D ones. One idea could be to shift to a 2-D one. Other than that, the layers with the most parameters are the FC ones. These are sometimes redundant. I'd suggest reducing the size of those first.


MAE and MSE are Nan for regression with Neural Networks?

Possibilities:

There are missing values in your dataset.
You are introducing missing values with your scaling.  (In particular, were any of the features constant?  And you seem to be misusing the scaler: the fit method sets the max and min from the data, so running fit(x) then immediately after fit(y) probably isn't what you meant to do.)
If neither of the above is the problem, then presumably Keras is introducing the problem.  This is trickier to diagnose, but often seems to be from vanishing/exploding gradients?  Do you get real scores initially, or NaN's from the very start of training?  See also https://stackoverflow.com/questions/37232782/nan-loss-when-training-regression-network
and its Related questions.


Gradient Descent is an universal method, you can us it with basically every loss function you can find in known ML algorithms.

In your case, you have only to derive the logarithmic cost function. You can find a detailed calculation at

https://math.stackexchange.com/questions/477207/derivative-of-cost-function-for-logistic-regression


Randomforest parameter tuning R(caret) and Python(scikit learn)?

I guess what you are looking for is sklearn.model_selection.GridSearchCV() or a similar function regarding the type of "search" (grid, random, ...) you would like to conduct. This function does hyperparameter tuning and uses cross validation when doing so and the cv parameter will allow you to specify the number of folds.


Which result is right or wrong here depends on your objective. You have created two variables (vectors) $x_1, x_2$.
Assuming $x_1, x_2$ are two samples of i.i.d random variables $X_1$ and $X_2$, respectively. Now, with some more assumptions, you want to test the null hypothesis: $\mathbb E(X_1) = \mathbb E(X_2)$.
For this, your second output is the correct one. However, based on the data that you have generated, this is not applicable because each of your samples, $x_1, x_2$ are not coming from the same distribution, as the mean of your first five values is different from the last five.
Ignoring your data, this analysis can be done using the formula approach as well. Join the two vectors $x_1,x_2$ to $x$ and add another column (say, y) which identifies which data point is coming from which sample. Call this new data frame df1. Then an equivalent way of doing the above mentioned test is t.test(x~y, data = df1)
The second approach is helpful when your data is organized in such a format. For example, say, you have data frame with two columns: height ($x$) and gender ($y$). Then running t.test(x~y, data = df1) will test whether the mean height is different between genders.
Your first approach can be considered right only when your $x_2$ is a factor variable which identifies the group or sample of the data point in vector $x_1$.


That's a wonderful question, and beautifully posed.  I love how you capture the essence of the issue in such a clean way.  Unfortunately, the answer is that your data set does not have enough information to help you decide which cats you should feed.  Without a properly controlled experiment, you don't have a way to infer causality; you can't rule out the possibility of confounding factors.

Suppose that 70% of the cats he feeds make it to the top, and 20% of the cats he doesn't feed make it to the top.  Let me make up four alternative explanations for why that might happen:

Hypothesis #1: There's only one kind of cat in the world.  The characteristics of the cat are irrelevant.  If you feed them, they will have a 70% chance of making it to the top; if you don't, they'll have only a 20% chance.  This is true for all cats, regardless of their attributes.
Hypothesis #2: There are two kinds of cats in the world: the strong, and the weak.  The strong have a 70% chance of making it to the top of the mountain, regardless of whether you feed them or not.  The weak have a 20% chance of making it to the top, regardless of whether you feed them.  You can tell the two kinds of cats apart by their characteristics, and your friend happens to like the strong ones and dislike the weak ones.
Hypothesis #3: There are two kinds of cats in the world: the athletes, and the champion nappers.  The former have a 70% chance of making it to the top of the mountain if you feed them, or a 50% chance if you don't.  The latter have a 40% chance of making it to the top if you feed them, or a 20% chance if you don't.  You can tell the two apart by their characteristics, and your friend happens to like the athletes and dislike the nappers.
Hypothesis #4: There are two kinds of cats in the world: the hungry, and the overweight.  The former have a 70% chance of making it to the top of the mountain if you feed them (they're pretty fit, but they already have all the food they need, so why bother exploring?), or a 100% chance if you don't (hunger drives them to greater heights).  The latter have a 21% chance of making it to the top, if you feed them, or a 20% chance, if you don't (either way, they probably won't make it, because they are so overweight and out of shape).  You can tell the two apart by their characteristics, and your friend happens to like the hungry ones and dislike the overweight ones.

All four hypotheses are equally consistent with the data.  The data set is useless at helping you distinguish between them.  Yet each hypothesis leads to a different prediction about which cats to feed: the first two say it doesn't matter, the third says you should feed the ones your friend likes, and the fourth says you should feed the ones your friend dislikes.  No amount of statistics or machine learning is going to enable you to make a decision based solely on the data.

To distinguish between these possibilities, you either need some kind of prior that will give you a basis for choosing among the models that are consistent with the data (perhaps based on your domain knowledge), or you need to conduct a controlled experiment (where the choice of whether to feed a cat is randomized, rather than being selected based on whether your friend likes the cat).

What if we have a prior that says that feeding a cat can only help its chances of making it to the top, and not hurt its chances?  Well, that's an example of the first of two options I suggest in the last paragraph.  However, it's still not informative enough to let us decide which cats to feed.

The problem is that we need to tell whether feeding the cats your friend likes improves their odds more than it improves the odds of feeding the cats your friend doesn't like.  We just don't have any information on that.

The chances could be 70% vs 69% for cats he likes and 90% vs 20% for cats he doesn't (i.e., cats he like have a 70% chance if fed or a 69% chance if not fed, etc.); or it could be 70% vs 0% for cats he likes and 21% vs 20% for cats he doesn't.  In the former case, you should feed the cats he doesn't like, as they see a 70 percentage point improvement from feeding, while the ones he likes see only a 1 p.p. improvement.  In the latter case, you should feed the cats he does like, because they see a 70 p.p. improvement, while the ones he dislikes see only a 1 p.p. improvement.  You just can't distinguish between these two possibilities from the dataset you have.  So your prior would need to be considerably more detailed than that, before you can make any headway.

If you'd like to learn more about the subject and under what circumstances we can draw conclusions about causality despite the lack of a controlled experiment, you might start by reading Judea Pearl's work.


Training accuracy is not considered because it can give falsified evaluation value for over-fitting 100% in your case. Hence it is always computed against the batch not entertained for training. This corresponds to the generalized error.
For more information see K-Fold cross validation.

This might help: https://thebayesianobserver.files.wordpress.com/2012/02/debugging-ml.png


Connection between Embedding and LSTM and Dense layer

The embedding layer has an output shape of 50. The first LSTM layer has an output shape of 100. How many parameters are here? Take a look at this blog to understand different components of an LSTM layer.

Then you can get the number of parameters of an LSTM layer from the equations or from this post.

Now, between LSTM(100) layer and the Dense(100, activation='relu') layer, there should be 100*(100 + 1) parameters. The additional 1 is for the bias.


Consensus clustering: how to choose the final cluster based on the consensus matrix?

Consensus clustering is an additional evaluation technique (i.e. how much consensus is there between clusters), like the Silhouette score, rather than explicit clustering.
The clusters themselves would drop out earlier in the processing. The consensus metric is used to test for the best K value, in the K-means clustering algorithm.


How to place multiple graphs in same coordinate system (pandas,mathplotlib) -> look pic

Problem is with the dtype of your y values. It's "object" and thus matplotlib thinks two object containing same float value are different.
Use,
y  = a.astype(float).values
y1 = b.astype(float).values
y2 = c.astype(float).values


I got the answer.
I have to install some of the packages in terminal . I installed it and it works.

sudo yum install gtk+-devel gtk2-devel


Keras model evaluation accuracy vs. observation

Actually, I think accuracy is not a good metric for this case as y_actual and y_expected have most values 0 and the length of y_actual is quite big too.

So the accuracy calculation using an equation like K.mean(K.equal(K.argmax(y_true, axis=-1), K.argmax(y_pred, axis=-1))) (Reference) is not a good metric on model performance


checking model stability - Performance for different class

A consideration: I dont think A is stable since it has a huge difference between validation and test results.

Some questions before answer:

Are you using cross validating with how many folds? The best result are the mean between folds results? What is the stddev? If stddev is big, it can tell us a history.
Your folds are shuffled? I see lot of people folding when using scikit learn without shuffle since its not the default behavior.
Is your test data unbalanced? if so, its normal the difference between test and validation results.
How many examples are you using for test and how many fot test? Sometimes the stability of the model requires more examples.


How do I calculate the maximum likelihood (machine learning statistics) of this table of data?

You might have mistaken the equation for maximum likelihood , and it should be like this :

  $p^y*(1-p)^{1-y}$

while y is your label , it should be 0 or 1 , 0 for negative and 1 for positive , and p is your sigmoid function value.

so , in your case it should be :

  p(2.5)*(1 - p(0.3))*p(2.8)*p(0.5)

Hopes this will help you , good luck !


Is it possible to achieve activity recognition on a single image?

The short answer, yes and even with a good performance, although not with same accuracy as with multiple frames.
There are actually many different architectures that rely on classifying each single frame as part of their process (you can treat the overall model as an ensemble-model of multiple frames).

There is a great article by "Schindler et al."  about exactly this question:

Action Snippets: How many frames does human action recognition require?


How can I turn user audit data into behavior clusters?

Scaling

Standardize you data by $(x - mean(x)) / std(x)$

Most K-mean implementation by default uses Euclidean distance which assumes the equal importance of all features. This requires proper scaling to prevent one action dominates the others.

Dimensionality

K-mean is not robust against the curse of dimensionality (see this post). So as always it good to reduce you dimensionality before feed it to the algorithm.

Do some feature engineering first. For example, login and page load can be group together as a measure of passive engagement, page edit and page created could be considered as a single activate engagement feature.

Also, you can try using some standard dimensionality reduction algorithm like PCA


What does 'contextual' mean in 'contextual bandits'?

A contextual bandit algorithm not only adapts to the user-click feedback as the algorithm progresses, it also utilizes pre-existing information about the user's (and similar users) browsing patterns to select which content to display.

So, rather than starting with no prediction (cold start) with what the user will click (traditional bandit and also traditional A/B testing), it takes other data into account (warm start) to help predict which content to display during the bandit test.

See: http://www.research.rutgers.edu/~lihong/pub/Li10Contextual.pdf


How can I determine how many signals there are in a mixture?

It is fourier transformation problem. All you need to do is to do a fourier transformation of your discrete signal and it will tell all the different frequencies it consists of. Numpy.fft class will be helpful.


Two options :

Use pre-built libraries for OCR + Bounding Box detection   (E.g.: https://www.pyimagesearch.com/2018/08/20/opencv-text-detection-east-text-detector/ for Bouding Box detection and then OpenCV / Tessract for OCR)

Train a Deep learning model for Text Detection in scene . Examples : https://github.com/qjadud1994/CRNN-Keras  and https://github.com/mvoelk/ssd_detectors


Feature Scaling of Training Set and Test Set

You do feature scaling for accelerating learning process. Features may have different scales. One maybe from 1 to 10 and one may be from -100 to 1000. Using normalization, you make the scale of them the same as each other, helps accelerate the learning process. You should find the mean and variance for each feature separately on your training data. then during training and testing each feature should be reduced by the corresponding mean and be divided by the corresponding standard deviation. So yes, for each feature during testing and training you have to provide same values for mean and std which are obtained using training data. I suggest you taking a look at here.


attach "1" for the bias unit in neural networks. what does it mean?

It allows you to account for some static offset in your learning process. To illustrate, consider the output of your first hidden layer without the bias unit.

h = sigma(W * x)

Here W is the matrix of your layer weights, and W * x is the matrix-vector multiplication between these weights and your input. sigma is your nonlinearity operating on each element of the result. Now consider what this looks like With the "bias unit".

h = sigma(W' * [x, 1]) ~ sigma(W*x + b)

Here, W' is your weight matrix with new entries associated with the "bias unit", and your new input is your original input with a 1 concatenated to it. You can think of this as being equivalent to your original matrix multiplication W * x plus some bias term b.

I will have to dig for the sources on this, but conventional wisdom is that you don't need the bias unit in modern architectures. If it's for a class though, I'd say use it if that's what you are instructed to do.


Predict part of the input for a fixed target

I am not sure what is the scale of your dataset, but I guess what you're looking for conceptually is Multi-task learning. You can use this concept to carry out multiple tasks, or in your case regression predictions using a single model.What you can try doing is use the four features, and the target variable as input features to predict the two variables.

If you have a large dataset, you can look into deep learning multi-taking approaches here. If you are familiar with python and sklearn, you can try some basic multi task regression algorithms available as a starting point. This algorithm might be helpful, its easy to use. If you are looking for a more deep learning approach, there are many architectures available online that you can try as starting point.


The example you find is calculating the gradient for the ReLU function, whose gradient is
$$\text{ReLU}'(x)=\left\{
\begin{array}{c l}
     1 & \text{if } x>0\\
     0 & \text{if } x<0
\end{array}\right.$$

Therefore when x<0, you make the gradient 0 by grad_x[x < 0] = 0.


Automatic labelling of text data based on predefined entities

The task you have is called named-entity recognition. From wiki:

  Named-entity recognition (NER) (also known as entity identification, entity chunking and entity extraction) is a subtask of information extraction that seeks to locate and classify named entity mentions in unstructured text into pre-defined categories such as the person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc.

Since this is a common NLP task there are libraries that are made to do NER out of the box. One such library is spaCy which can do NER as well as many other NLP tasks using Python.

You will not be able to perform NER without first training a model on your custom labels/entities. You need to have some labelled data to train on, maybe you already have this or you can label it manually. SpaCy wants yo have the data labelled with location of each entity on the format:

[("legal text here", {"entities": [(Start index, End index, "Money"),
                                   (Start index, End index, "Judge"),
                                   (Start index, End index, "Tribunal"),
                                   (Start index, End index, "State")]}),
("legal text here", {"entities": [(Start index, End index, "Money"),
                                  (Start index, End index, "Judge"),
                                  (Start index, End index, "Tribunal"),
                                  (Start index, End index, "State")]})
...]

Example on how to training a spaCy model for NER (taken from docs):

from __future__ import unicode_literals, print_function

import plac
import random
from pathlib import Path
import spacy
from spacy.util import minibatch, compounding

# training data
TRAIN_DATA =   Insert you labelled training data here

@plac.annotations(
    model=("Model name. Defaults to blank 'en' model.", "option", "m", str),
    output_dir=("Optional output directory", "option", "o", Path),
    n_iter=("Number of training iterations", "option", "n", int),
)
def main(model=None, output_dir=None, n_iter=100):
    """Load the model, set up the pipeline and train the entity recognizer."""
    if model is not None:
        nlp = spacy.load(model)  # load existing spaCy model
        print("Loaded model '%s'" % model)
    else:
        nlp = spacy.blank("en")  # create blank Language class
        print("Created blank 'en' model")

    # create the built-in pipeline components and add them to the pipeline
    # nlp.create_pipe works for built-ins that are registered with spaCy
    if "ner" not in nlp.pipe_names:
        ner = nlp.create_pipe("ner")
        nlp.add_pipe(ner, last=True)
    # otherwise, get it so we can add labels
    else:
        ner = nlp.get_pipe("ner")

    # add labels
    for _, annotations in TRAIN_DATA:
        for ent in annotations.get("entities"):
            ner.add_label(ent[2])

    # get names of other pipes to disable them during training
    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != "ner"]
    with nlp.disable_pipes(*other_pipes):  # only train NER
        # reset and initialize the weights randomly – but only if we're
        # training a new model
        if model is None:
            nlp.begin_training()
        for itn in range(n_iter):
            random.shuffle(TRAIN_DATA)
            losses = {}
            # batch up the examples using spaCy's minibatch
            batches = minibatch(TRAIN_DATA, size=compounding(4.0, 32.0, 1.001))
            for batch in batches:
                texts, annotations = zip(*batch)
                nlp.update(
                    texts,  # batch of texts
                    annotations,  # batch of annotations
                    drop=0.5,  # dropout - make it harder to memorise data
                    losses=losses,
                )
            print("Losses", losses)

    # test the trained model
    for text, _ in TRAIN_DATA:
        doc = nlp(text)
        print("Entities", [(ent.text, ent.label_) for ent in doc.ents])
        print("Tokens", [(t.text, t.ent_type_, t.ent_iob) for t in doc])

    # save model to output directory
    if output_dir is not None:
        output_dir = Path(output_dir)
        if not output_dir.exists():
            output_dir.mkdir()
        nlp.to_disk(output_dir)
        print("Saved model to", output_dir)

        # test the saved model
        print("Loading from", output_dir)
        nlp2 = spacy.load(output_dir)
        for text, _ in TRAIN_DATA:
            doc = nlp2(text)
            print("Entities", [(ent.text, ent.label_) for ent in doc.ents])
            print("Tokens", [(t.text, t.ent_type_, t.ent_iob) for t in doc])

if __name__ == "__main__":
    plac.call(main)

Then when you have a trained model you can use it to get your entities:

doc = nlp('put legal text to test your model here')

for ent in doc.ents:
    print(ent.text, ent.start_char, ent.end_char, ent.label_)


Can Boosted Trees predict below the minimum value of the training label?

Yes, gradient boosted trees can make predictions outside the training labels' range.  Here's a quick example:
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingRegressor

X, y = make_classification(random_state=42)

gbm = GradientBoostingRegressor(max_depth=1,
                                n_estimators=10,
                                learning_rate=1,
                                random_state=42)
gbm.fit(X,y)
preds = gbm.predict(X)
print(preds.min(), preds.max())

outputs -0.010418732339562916 1.134566081403055 (and make_classification gives outputs just 0 and 1).
Now, this is unrealistic for a number of reasons: I'm using a regression model for a classification problem, I'm using learning rate 1, depth only 1, no regularization, etc.  All of these could be made more proper and we could still find an example with predictions outside the training range, but it would be harder to construct such an example.  I would say that in practice, you're unlikely to get anything very far from the training range.
See the (more theoretical) example in this comment of an xgboost github issue, found via this cv.se post.

To be clear, decision trees, random forests, and adaptive boosting all cannot make predictions outside the training range.  This is specific to gradient boosted trees.


Does overfitting depend only on validation loss or both training and validation loss?

In my opinion, only case 3 should be considered overfitting. As @stans has mentioned, there is not a very rigorous definition of overfitting so other people might think differently.
I wouldn't say the point where the validation loss stops decreasing is where bias and variance are minimized since there is a trade-off between bias and variance:

A constant model will have very low variance, but very high bias.
An overfitting model will have very low bias, but very high variance.

The point where the validation loss starts increasing can be considered optimal in terms of the sum of squared bias and variance, that is, an optimum of the generalization error.


How to predict future sales based upon last year sales data?

You can not predict sales for the whole next year by training your model with only one year data.

But still one year data is better than having nothing. And you can analyze one year data to find trends and patterns. On top of those trends and patterns, you can apply domain knowledge and make some very some useful strategic decisions.

Problem you are trying to solve is called Time Series Forecasting as temporal order of input matters while training.

Here are reference points you can your start with :
1) https://machinelearningmastery.com/how-to-get-started-with-deep-learning-for-time-series-forecasting-7-day-mini-course/
2) https://www.analyticsvidhya.com/blog/2018/02/time-series-forecasting-methods/
3) https://towardsdatascience.com/an-end-to-end-project-on-time-series-analysis-and-forecasting-with-python-4835e6bf050b


On simple 1D dataset, LogisticRegressionCV selects terrible hyperparameters, resulting scores are nonsensical

First I did a visualization of what your code is doing (see code at the bottom)

The model seems completely fine. The coefficient of the linear regression is close to 0, where it should be given how you have created the data.

You are misunderstanding

  regularized_logistic_regression_model.scores_

    {1: array([[0.47058824, 0.47058824, 0.47058824, 0.47058824, 0.47058824],
            [1.        , 1.        , 1.        , 1.        , 1.        ],
            [0.63636364, 0.63636364, 0.63636364, 0.63636364, 0.63636364]])}

I quote from sklearn documentation:

  scores_dict: dict with classes as the keys, and the values as the grid of scores obtained during cross-validating each fold

This is the prediction of one of the folds (note that if you increase the cv parameter it changes length)

The data created is sorted! If you plot the values of X vs the Index you can see

You are getting those results because the data is sorted! Not just because of luck. If you shuffle the data you won´t have perfect predictions.

Your result is around 0.7 that is actually what makes sense by looking at the image I attached.

I shuffled the data, not in the most elegant way but you get now different results.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.linear_model import LogisticRegressionCV
from sklearn.linear_model import LogisticRegression
plt.style.use('seaborn-whitegrid')

def gen_y(x):
    p1 = np.clip(x + 0.5, 0, 1)
    v = np.random.uniform(0, 1)
    if v < p1:
        return 1
    return 0

np.random.seed(6)

x_data = np.sort(np.random.normal(0, 0.3, 100))
y_data = np.array([gen_y(x) for x in x_data])

df = pd.DataFrame(x_data.reshape(-1, 1),columns=['X'])
df['y']= y_data

plt.figure()
plt.title('Values of X vs Index')
df.X.plot()
plt.savefig('x')
plt.show()

df = df.sample(frac=1)

regularized_logistic_regression_model = LogisticRegressionCV( fit_intercept = False, cv = 3)

regularized_logistic_regression_model.fit(df['X'].values.reshape(-1, 1) , df['y'])

plt.figure()

df['results'] =regularized_logistic_regression_model.predict(df['X'].values.reshape(-1, 1))
plt.scatter(df[df['results']==0].y,df[df['results']==0].X,label='pred=0')
plt.scatter(df[df['results']==1].y,df[df['results']==1].X,label='pred=1')
plt.legend()
plt.hlines(y=regularized_logistic_regression_model.coef_.squeeze(),xmin=-0.1,xmax=1.1)
plt.savefig('ex')
plt.show()

regularized_logistic_regression_model.scores_


How could I go about finding the weights or importance of inputs based on outputs?

Pearson correlation can be used for this purpose. The Pearson correlation between two entity shows that how mush the values of these two are  linearly related to each other.

  According to the Cauchy–Schwarz inequality it has a value between +1
  and −1, where 1 is total positive linear correlation, 0 is no linear
  correlation, and −1 is total negative linear correlation.

According to the values you reported here, there is a strong correlation between doc and mrr so the role of doc in the prediction of mrr is more important than others.

But on the other hand none of the features doesn't have any considerable linear correlation with ra. In this case you can test some other correlation methods.

For further information visit here. It can be helpful to you.

Conclusion: the most important feature in predicting an output is the most correlated one with it which has a considerable correlation.


What is purpose of the [CLS] token and why is its encoding output important?

CLS stands for classification and its there to represent sentence-level classification.
In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog where this is also covered in detail.


What arguments should I pass to input_shape parameter of LSTM function in Keras?

Following example shows that by printing shape of your train data you can pass the right value to input_shape(). Also it depends you want to feed your data in 2D or another way. As it can be seen in snippet:

trainX.shape[1]=10

trainX.shape[2]=1440

and in output based on test shape Dense(units = 960)

#Shape of train and test data
trainX, testX, trainY, testY = train_test_split(trainX,trainY, test_size=0.2 , shuffle=False)
print("train size: {}".format(trainX.shape))
print("train Label size: {}".format(trainY.shape))
print("test size: {}".format(testX.shape))
print("test Label size: {}".format(testY.shape))
#train size: (23, 10, 1440)
#train Label size: (23, 960)
#test size: (6, 10, 1440)
#test Label size: (6, 960)

# create and fit the Simple LSTM model

#model_LSTM = Sequential()
model_LSTM.add(LSTM(units = 1440, input_shape=(trainX.shape[1], trainX.shape[2])))
model_LSTM.add(Dense(units = 960))

Edit: Following tips based on this example to help you when preparing your input data for LSTMs:

The LSTM input layer must be 3D.
The meaning of the 3 input dimensions are: samples, time steps, and features.
The LSTM input layer is defined by the input_shape argument on the first hidden layer.
The input_shape argument takes a tuple of two values that define the number of time steps and features.
The number of samples is assumed to be 1 or more.
The reshape() function on NumPy arrays can be used to reshape your 1D or 2D data to be 3D.
The reshape() function takes a tuple as an argument that defines the new shape.


Keras intuition/guidelines for setting epochs and batch size

Actually it's controversial and it can differ from one problem to another. the best thing to do is to search the hyperparamteres space for an optimal result.

For Question 1

Here I quot the father of CNN (Yann Lecun) :
"
Training with large minibatches is bad for your health.
More importantly, it's bad for your test error.
Friends dont let friends use minibatches larger than 32.

Let's face it: the only people have switched to minibatch sizes larger than one since 2012 is because GPUs are inefficient for batch sizes smaller than 32. That's a terrible reason. It just means our hardware sucks.

What's worse is that the easiest way to parallelize training is to make the minibatch even larger and distribute it across multiple GPUs and multiple nodes.
Minibatch sizes over 1024 aren't just bad for your health. They cause brain tumors.
They learn quickly, but the wrong thing.
...

Update2: I'm pointing to this particular paper as a way to make a general point. But I'm not necessarily endorsing the particular results nor the methodology in this paper (some folks have pointed out a few flaws in the methodology). Still, the main point is that the only reason to use a batch size larger than 1 is the limitations of the limitations of the hardware at our disposal."

In this paper a value for batches between 2 and 32 is recommended

For Questions 2 & 3:

Usually an early stopping technique is used by setting the number of epochs to a very large number and when the generalization error gets worse we just stop. please see

For more details and explanations please see

At the end I would recommend using "ReLu" activation function for the hidden layers rather than the "sigmoid". it learns much faster


Which algorithm to use to classify plots based on graphical features?

Your question is unclear. Please provide more information about the dataset, the dataset size, the target, the motivation.
Until then I can only assume and try to help nontheless.

Assumptions

Since you speak of classification I assume that you have labels.
I further assume that you have access to the underlying data used to produce the plot.
In that exact case, the following will be useful:

Solution

Instead of classifying the plots, you could classify the underlying two-dimensional datasets treating them as time-series.

For your specific case, Dynamic Time Warping distance could be useful. You could therefore use this code:

import numpy as np
from scipy.spatial import distance
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report

#toy dataset
X = np.random.random((100,10))
y = np.random.randint(0,2, (100))
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

#custom metric
def DTW(a, b):
    an = a.size
    bn = b.size
    pointwise_distance = distance.cdist(a.reshape(-1,1),b.reshape(-1,1))
    cumdist = np.matrix(np.ones((an+1,bn+1)) * np.inf)
    cumdist[0,0] = 0

    for ai in range(an):
        for bi in range(bn):
            minimum_cost = np.min([cumdist[ai, bi+1],
                                   cumdist[ai+1, bi],
                                   cumdist[ai, bi]])
            cumdist[ai+1, bi+1] = pointwise_distance[ai,bi] + minimum_cost

    return cumdist[an, bn]

#train
parameters = {'n_neighbors':[2, 4, 8]}
clf = GridSearchCV(KNeighborsClassifier(metric =DTW), parameters, cv=5)
clf.fit(X_train, y_train)

#evaluate
y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred))


How to predict past test set on time series data using LSTM

The best answer I found is here

Unfortunately I couldn't get it to work for my solution but reworked my solution and just incremented the sliding window, appended that answer to my test data, and iterated like that.


How would be plotted the logistic functions associated to a multiclass logistic classifier in a X-Y plot?

If you are asking how to reproduce those plots, the following technique works to visualize the decision boundaries for any model:

Generate a dense grid of coordinates that fill your plotting area
Score each point on your grid
Plot your results, coloring observations based on your model's predictions. This serves as the background.
Overlay your test data as a scatter plot

In your link, they generate the grid using np.meshgrid and constrtuct the background as a contour plot. The lines representing the decision boundaries of each respective 1-vs-all classifier is plotted using a closed form solution for logistic decision boundaries.

EDIT: To answer the question in your comment, you don't have a single $X$ dimension, and consequently your model output doesn't correspond to a curve as simple as this: it's a 3D surface. The simplest solution would be to just apply the strategy I suggested earlier (and which is also described in your link) but instead of calling .predict() to construct the background coloration, you'd call .predict_proba() and use a sequential color map (e.g. the default veridis) so color intensity corresponds to your class likelihood, giving you something which should look like this. Alternatively, you could plot surface curves. Both of these solutions are projections of the $Y$ axis, $P(Y|x1,x2)$, onto the $X_1$-$X_2$ plane. If that isn't satisfactory, you could pass scored results through PCA to combine your $X$ dimensions into a single $X$ feature -- call this $PC_1$ -- and then plot $PC_1$ vs. $P(Y|x1,x2)$. Personally I think this is significantly less informative, but it would give you an x vs. y plot.


Result of uniform weight initialization in all neurons

I'll make this as explicit and non handwavy as possible, at the risk of being boring.
Let us take a small toy example: a network with two inputs $x_1$ and $x_2$, two hidden layers with two units each and one output. For simplicity, we will assume that the activation function is the identity function (i.e. no activation) and that the objective function is the output itself (not very realistic, but for our purposes this could mean our cost function is meant to drive the output to the zero function).

Below are the outputs of each unit expressed as a function of their inputs.
$$y = w_9y_3 + w_{10}y_4$$
$$y_3 = w_5y_1 + w_7y_2$$
$$y_4 = w_6y_1 + w_8y_2$$
$$y_1 = w_1x_1 + w_3x_2$$
$$y_2 = w_2x_1 + w_4x_2$$
Take a look at $w_9$: $\frac{\partial y}{\partial w_9} = y_3 = w_5y_1 + w_7y_2 = w_5(w_1x_1 + w_3x_2) + w_7(w_2x_1 + w_4x_2)$. The update of $w_9$ depends on $w_1$, $w_2$, $w_3$, $w_4$, $w_5$ and $w_7$: if we consider the diagram above to be a directed acyclic graph, the weights of all the edges on paths leading to $y_3$.
In our toy example, updates are simple: $w_{i, n+1} = w_{i, n} - \eta \frac{\partial y}{\partial w_i}(w_{i, n})$, where $i$ denotes the weight in the diagram and $n$ denotes the iteration.
If all weights are initialized to a constant $a$, this amounts to $w_9 = a$ and $\frac{\partial y}{\partial w_9} = a(ax_1 + ax_2) + a(ax_1 + ax_2)$.
Let us also take a look at $w_{10}$: $\frac{\partial y}{\partial w_{10}} = w_6(w_1x_1 + w_3x_2) + w_8(w_2x_1 + w_4x_2)$. In this case, $w_{10} = a$ and $\frac{\partial y}{\partial w_{10}} = a(ax_1 + ax_2) + a(ax_1 + ax_2)$.
As can be seen, the initial value of both $w_9$ and $w_{10}$ is the same and their updates are the same too. On the first iteration, they will have changed after the update, but they will still be equal to each other. Not necessarily a bad thing, but if they're equal, you might just as well keep only one of them in memory. If you're keeping both, they should be different.
What happens on the next iteration? Looking at the derivatives with respect to $w_9$ and $w_{10}$ previously derived, we notice that if $w_5 \neq w_6$ or $w_7 \neq w_8$ the two updates will be different, so let's take a look at those then!
$$\frac{\partial y}{\partial w_5} = \frac{\partial y}{\partial y_3} \frac{\partial y_3}{\partial w_5} + \frac{\partial y}{\partial y_4} \frac{\partial y_4}{\partial w_5} = w_9y_1 + w_{10} \cdot 0 = w_9(w_1x_1 + w_3x_2)$$
$$\frac{\partial y}{\partial w_6} = w_{10}y_1 = w_{10}(w_1x_1 + w_3x_2)$$
$$\frac{\partial y}{\partial w_7} = w_9(w_2x_1 + w_4x_2)$$
$$\frac{\partial y}{\partial w_8} = w_{10}(w_2x_1 + w_4x_2)$$
We can see that for the same reasons as before, after the first iteration, $w_5$ and $w_6$ will be the same, just like $w_7$ and $w_8$ will be the same. This, combined with the fact that $w_9$ and $w_{10}$ were equal after the first iteration, means that $w_9$ and $w_{10}$ will still be equal after the second iteration.
We can already say that $w_5 = w_6$, $w_7 = w_8$ and $w_9 = w_{10}$ ad infinitum.
Finally, let us take a look at the weights of the first hidden layer, $w_1$, $w_2$, $w_3$ and $w_4$.
$$\frac{\partial y}{\partial w_1} = (w_9w_5 + w_{10}w_6)x_1$$
$$\frac{\partial y}{\partial w_2} = (w_9w_7 + w_{10}w_8)x_1$$
$$\frac{\partial y}{\partial w_3} = (w_9w_5 + w_{10}w_6)x_2$$
$$\frac{\partial y}{\partial w_4} = (w_9w_7 + w_{10}w_8)x_2$$
Here, $x_1$ and $x_2$ don't have to be equal, so $w_1$ and $w_2$ can be different from $w_3$ and $w_4$. However, if we initialize all weights to $a$, $w_1 = w_2$ and $w_3 = w_4$ after the first iteration, which means that $w_5 = w_6 = w_7 = w_8$ after the second iteration (not just the first).
To conclude, we can say that $w_1 = w_2$, $w_3 = w_4$, $w_5 = w_6 = w_7 = w_8$ and $w_9 = w_{10}$ ad infinitum!
Generalizing this toy example, we can state that if we initialize all weights to the same value $a$, the weights of a layer will all be equal forever, except for the first layer, which keeps a semblance of normalcy. In other words, using only one weight per layer is enough.
A special case of initializing all weights to the same value is initializing them all to 0. If we look once more at the partial derivatives (which we use to update our weights), we can see that they will all be zero, always! In this case, the weights never get updated at all, so our model becomes entirely useless.
A very pretty visualization for the effects of different basic weight initialization techniques can be viewed here.


This question on stackoverflow might help you. To sum up, some deep learning researchers think that padding a big part of the image is not a good practice, since the neural network has to learn that the padded area is not relevant for classification, and it does not have to learn that if you use interpolation, for instance.


Constructing function - f(x,y) for the given minimums (Python)

One way to go is to define functions that are non-zero only near your 'minimal' points, and add them. The goal is to avoid overlap, such that a function associated with a point won't modify the value of your global function around other points.

import numpy as np

def func0(x1):
    lim = 0.01
    dist = x1*x1
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

That will give you a 1D function with nice properties (value 1 at 0, 0 outside the neigborhood of 0) :

x = np.linspace(-1, 1, 1000)
z = np.array([func0(i) for i in x])
Z = z.reshape(1000)

plt.plot(Z)
plt.show()

Modify it to get a 2 D function centered in (a,b) :

def func(x1,x2):
    a = 0
    b = 0
    lim = 0.01
    dist = (x1-a)*(x1-a) + (x2-b)*(x2-b)
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

This will give you a 2d function :

x = y = np.linspace(-1, 1, 100)
z = np.array([func(i,j) for j in y for i in x])
Z = z.reshape(100, 100)

x, x = np.meshgrid(x, y)

plt.imshow(Z,extent=(x.min(), x.max(), y.min(), y.max()))
plt.colorbar()
plt.show()

Then build the threes functions associated with your points.

def func1(x1,x2):
    a = 0.2
    b = 0.3
    lim = 0.01
    dist = (x1-a)*(x1-a) + (x2-b)*(x2-b)
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

def func2(x1,x2):
    a = 0.6
    b = 0.8
    lim = 0.01
    dist = (x1-a)*(x1-a) + (x2-b)*(x2-b)
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

def func3(x1,x2):
    a = 0.85
    b = 0.5
    lim = 0.01
    dist = (x1-a)*(x1-a) + (x2-b)*(x2-b)
    if dist<lim:
        value = np.exp(-dist/(lim-dist))
    else:
        value = 0
    return value

And add them (up to coefficients that determine the value on points of interest) to form a global function :

def global_func(x,y):
    return 1-0.3*func1(x,y)-0.2*func2(x,y)-0.4*func3(x,y)

This will give you :

x = y = np.linspace(0, 1, 1000)
z = np.array([global_func(i,j) for j in y for i in x])
Z = z.reshape(1000, 1000)

x, x = np.meshgrid(x, y)

plt.imshow(Z,extent=(x.min(), x.max(), y.min(), y.max()))
plt.colorbar()
plt.show()

Finally you can check that :

print(global_func(0.2,0.3))
print(global_func(0.6,0.8))
print(global_func(0.85,0.5))

returns :

0.7
0.8
0.6

You have a continuous function with all expected properties by construction.

If you are unhappy with it being constant on a lot of the domain, you can modify the individual functions so that they don't stop at the limit with :

value = np.exp(-dist)

And in your global function, determine which 'minimal' point is the nearest and only apply a function of the form 1-coef * funci based on which point is nearest. This way the individual functions won't overlap and thus won't modify value of the global function around other 'minimal' points.


Entropy loss from collapsing/merging two categories

I ended up coming up with my own solution. I'm not sure if it's correct so I won't mark it as an answer.

Let's call the original distribution P. Suppose we will be collapsing A and B into a new category X. The entropy of P will be: $S_P = - \sum P_i\ln P_i$ where $i$ is A, B, C, ... Notably, $P_A > 0, P_B > 0, P_X = 0$ .

After collapsing, we get a new distribution $Q$. Now $Q_A=Q_B=0$ and $Q_X =Q_A+Q_B$. This will have entropy $S_Q$.

The entropy we lost by collapsing is $S_P-S_Q = P_A \ln P_A + P_B \ln P_B - Q_X \ln Q_X$. All other terms, such as $P_C\ln P_C$ for category $C$ which is unaffected, are present in both distributions and cancel out.


Feature selection in R too large dataset

Technically, it will be alright to use a sample of the data. One important assumption is that most of the features don't have outlier. In such case your sampling might miss out on those.

But, I don't see a necessity of sampling with the size of data that you have given. 150mb is not large data at all, given that you have 8 GB ram at disposal. Rather than sampling the data, first check if there is alternate way of selecting the features. Since there are only 18 columns, look at the summaries for each columns, how do the features correlate with your outcome variable, which variables add very little information. If you have not done more exploratory analysis of the 18 variables, I'll suggest you do that first.

Secondly, why not build your random forest directly with 18 features? If you had, say, 180 features, it would have made sense to do automated feature selection first using Boruta etc. If there are features that do not add information, their variable importance in forest model will be low. You can drop them in subsequent analysis


Is there a way to; easily, merge data into a CSV (or Excel) file without having to edit each file individually?

You can either load the excel files directly into Python, e.g. using the pandas package, or you can use a package that hits the data where it lives i.e. a package that directly modifies the Excel files.

Python and Pandas

import pandas as pd

data = pd.read_Excel('/path/to/your/file)

Data is basically a table that will look like the Excel itself. You can add fields (usually referred to as columns) like so:

data['new field'] = new_data

new_data will contain all values for all rows in the table, e.g. as a list:

new_data = [12, 24, 13, 66, 72, 45]    # assuming there are 6 rows (a.k.a. samples)

You will have to alter the code for each of your files. You will also need to think about how to get your new data into that list (or a similar construction).

Caveat: this way might have problems if you have some very specific formatting or special Excel features in use (more than just conditional formatting - think dropdown lists and hidden/folded rows and columns).

Excel interface

There are a few good packages that will allow you to speak directly with the excel file and make use of things such as sheets/tabs and references to cells e.g. A1 is the top left cell in Excel.

Have a look at the XlsWriter package or the python-excel package. Both let you access the fill and add/remove data as you please. I have used both for simple tasks and they both got the job done quite nicely.

Follow those links to see some nice examples of how to manipulate Excel programmatically.

It is worth mentioning that the first option above, using something like Pandas, will offer a lot more functionality and would likely not present a steeper learning curve (in my opinion!).


Is that what you want?

In [261]: df
Out[261]:
  names  number
0   bob       5
1  sara      10
2   bob       8
3   foo      12
4   moo      16

In [262]: def my_op(ser):
     ...:     return ser.sum()
     ...:

In [263]: df.groupby('names').agg({'number':my_op})
Out[263]:
       number
names
bob        13
foo        12
moo        16
sara       10


what is the difference between euclidean distance and RMSE?

Let's say that RMSE and euclidean distance are not of the same type, even though their formulations are close. RMSE is a loss function, while euclidean distance is a metric. See this question on Cros Validated to better understand the difference between a loss function and a metric: a loss function is generally based on a reference metric.

Euclidean distance is a metric, so it quantifies the distance between two observations.
RMSE is, as the name suggests, the root of the mean of the squared error between a true value and a predicted value, over a range of observations. RMSE is generally intended for model performance assessment.

Your confusion probably comes from the fact that, if you apply the RMSE formula to your set of coordinates (longitude and latitude), it gives the euclidean distance. But this would not be called RMSE in that case, rather euclidean distance.

In your case, you may use the RMSE of euclidean distance as loss function. The error made by your predictor is the euclidean distance, and your loss function would be the RMSE of these errors.

Defining a loss function is strongly problem-specific. First, you need to determine which metrics to use as error function. In your case, the euclidean distance between the actual position and the predicted one is an obvious metric, but it is not the only possible one. For instance, you could use the squared or cubed euclidean distance in order to give more weight to cases that are not well predicted. You could also design an ad-hoc metric to consider:

assymmetry, e.g. to be more tolerant on errors towards the east side than the west side
anisotropy, e.g. to be more tolerant on errors in the latitude axis than the longitude
uncertainty tolerance, e.g. error is null if the euclidean distance is less than a tolerance, and is equal to the euclidean distance otherwise
etc. (depending on your problem)

The subsequent metric will allow you to evaluate the performance of the trained model over a test set, and will therefore give you a disitrbution of errors. You need to shrink this distribution to a single scalar value used as global model performance: you can pretty much choose any scalar that represents something about the distribution:

Mean squared error (MSE) or its root (RMSE) - MSE is faster computed but RMSE has the advantage or having the same dimension as the error function (a distance in your case)
Other types of $\mathcal{L}_p$ norms (RMSE is the $\mathcal{L}_2$ norm)
Any quantile of the distribution (median, 75%, 95%, maximum value)
Whatever you think is relevant!


How to build a classification pipeline that will pass to another model?

You can use Microsoft Azure, to make two different pipelines. If I am not wrong, I understand that you want to compare the two methods, using a pipeline, therefore Microsoft Azure ML Studio is an easy and quick method to test your ideas.


Fitting as much points as possible on a line

I think as suggested in the comments, you should split this up in several linear regressions.

If the datapoints are ordered you could just do repeated linear regressions, always adding one point until the added point leads to a significant reduced goodeness of fit. Then you start the next linear regression until that one goes bad and you start the next one.
At the end you can compare the angles between the two lines and see if they are (roughly) 90 degrees.


This is a class imbalance problem. Your data has more number of 0's, thats why the model is also predicting 0's only. There are many solutions to this problem, like over/under sampling, SMOTE etc. Here are the links which show how to tackle the problem-

Binary classification with strongly unbalanced classes

Imbalanced classification problem


Is there a quick way to speed up ICP in python using a cached KD-tree

Try using spatial.cKDTree, in my ICP implementation switching from KDTree to cKDTree 30 iterations went from ~160 seconds to ~50 seconds


You are thinking about this correctly. If data doesn't vary between your outcomes then it doesn't need to be included.
That being said, if you are using time series techniques such as trend decomposition to feature engineer, then changing the structure of your data could complicate interpretation (ie: what is a moving average if you've removed data points?).
In that light I'd say that you should not build these quirks into the code you write, and that you should keep it as general as possible except where absolutely necessary. This is related to the concept of writing "DRY" code where you don't repeat yourself.
Personal opinion: Classifiers, like software, should try not to make assumptions about data if possible. This gives you a better chance of being able to reuse it or share it.


The complexity in this instance is discussing the smoothness of the boundary between the different classes. One way of understanding this smoothness complexity is by asking how likely you are to be classified differently if you were to move slightly. If that likelihood is high then you have a complex decision boundary.

For the $k$-NN algorithm the decision boundary is based on the chosen value for $k$, as that is how we will determine the class of a novel instance. As you decrease the value of $k$ you will end up making more granulated decisions thus the boundary between different classes will become more complex.

You should note that this decision boundary is also highly dependent of the distribution of your classes.

Let's see how the decision boundaries change when changing the value of $k$ below. We can see that nice boundaries are achieved for $k=20$ whereas $k=1$ has blue and red pockets in the other region, this is said to be more highly complex of a decision boundary than one which is smooth.

First let's make some artificial data with 100 instances and 3 classes.

from sklearn.datasets.samples_generator import make_blobs
X, y = make_blobs(n_samples=100, centers=3, n_features=2, cluster_std=5)

Let's plot this data to see what we are up against

Now let's see how the boundary looks like for different values of $k$. I'll post the code I used for this below for your reference.

$k$ = 1

$k$ = 5

$k$ = 10

$k$ = 20

The code

The code used for these experiments is as follows taken from here

from sklearn import neighbors

k = 1
clf = neighbors.KNeighborsClassifier(20)
clf.fit(X, y)

from matplotlib.colors import ListedColormap
import matplotlib.pyplot as plt

cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA', '#AAAAFF'])
cmap_bold = ListedColormap(['#FF0000', '#00FF00', '#0000FF'])

# Plot the decision boundary. For that, we will assign a color to each
# point in the mesh [x_min, x_max]x[y_min, y_max].
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure()
plt.pcolormesh(xx, yy, Z, cmap=cmap_light)

# Plot also the training points
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=cmap_bold)
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.show()


What is the common practice for NLP or text mining for non-English?

Translation as a pre-processing step is usually sufficient for many tasks (e.g. sentiment classification), but naturally undesirable for other tasks e.g. grading someone in written Dutch fluency.
Hence, for these tasks, the objective is:

Be able to train a language model for your specific language
However, you want to be able to do this with minimal resources

Thus, people in research view this as few-shot learning. One of the most common approaches to this is to exploit meta-learning, i.e. use models that have been pre-trained using many tasks (languages in this case) and they have been trained so that only a few gradient steps and little data is required for them to be applicable to a new task (Dutch in your example). An example of one such model you could finetune (transfer learning) is called multiliingual BERT... this was back in 2019, so I expect since many other multilingual language models have emerged - these can be good initialisation points for any bespoke model you want to train for some new language...


Why do you need to learn statistics for machine learning?

You need to learn statistics to be able to make sensible decisions about the ML models you plan to use, for example:

which model to use?
is the training-set large enough?
how to measure the model performance (which error measure)?
how to judge if the result is good enough?
how to (correctly!) compare different models and correct for multiple-testing?
how to avoid/identify selection bias?
how to deal with correlated features?
how to interpret the coefficients of the multiple linear regression model you just trained?

And most importantly: you need to learn statistics to understand that these questions (and many other ones) are important to ask.

If you want to be a software developer you may not need to learn statistics (although it will still be super useful!). But if you want to be a data-scientist, machine learning engineer or AI expert you cannot do your work properly without it.


How we can Identify Specific Feature from a larger amount of Dataset?

I dont think there is one correct way, but what you can do is

Use PCA if you have many features. This will reduce some number of features based on the amount of variance in each feature. You may use other dimensionality reduction techniques.
You can use models like Lightgbm or random forest and know which feature are important. 3. You may use Lasso Regression for feature selection.
You may use intuition to see if some features are just does not make any sense.

These are some of the methods to understand important features.
You may read this article:
https://towardsdatascience.com/the-5-feature-selection-algorithms-every-data-scientist-need-to-know-3a6b566efd2


Can someone clarify what the linear assumption of PCA is?

PCA is the best (in the mean-squared error sense) linear decomposition method.

PCA is defined as an orthogonal linear transformation that transforms
the data to a new coordinate system such that the greatest variance by
some scalar projection of the data comes to lie on the first
coordinate (called the first principal component), the second greatest
variance on the second coordinate, and so on.
Wikipedia

The term "Linear" in PCA means:
a. That any data point is simply a linear combination of the principal components.
b. That the data matrix ($A$) can be decomposed via linear similarity transformations to diagonal matrix ($\Sigma$).
Ie
$$A = U \Sigma U^T$$
or
$$AU = U \Sigma$$
$\Sigma$ is the diagonal matrix of variances for each basis vector.
One can see at once that the linear algebra of the above formula makes clear the meaning of Linearity in PCA.
On the other hand decomposition methods like ICA (Independent Component Analysis) cannot be expressed via linear algebra as PCA above, since they require not only decorrelated components but independent components which is a stronger condition requiring non-linearities.
See also: https://datascience.stackexchange.com/a/80361/100269


How to find the driver features towards a particular result in Classification problems

If you want to know what features are important to predict some outcome, you speak about feature importance (FI). FI can differ between models, e.g. between a random forest and logistic regression. This is because the models take a different perspective on how to predict some outcome.

A mathematically sound way to assess FI in binary classification would be to run a logistic regression with a lasso or ridge penalty. This is easy and the results (the log odds) can be directly interpreted. Larger positive or negative values indicate a larger positive or negative FI et vice versa.

Here is a good R example: https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html

See „Introduction to Statistical Learning“ Ch. 6.2 for some background and demo: https://www-bcf.usc.edu/~gareth/ISL/


Check Imputation Efficiency - How To Compare Data Frames?

If we want to check the imputation efficiency, we should probably compute the performance for the imputed values only. Indeed, the performance depends on the percentage of missing values.


The score decreasing and the standard deviation of the scores increases, as sample size increases, indicates overfitting to the smaller sample size.


Geometric Interpretation of Whether SVMs are performing well or not

A most interesting paper! Following Opper and Haussler$\dagger$, the authors define a version space; the set of unit vectors which separate the training samples:

$\mathcal V \equiv \{ \mathtt w | y_i f(\mathtt x_i) >0 , i= 1, \ldots, n, \| \mathtt w \|_2 = 1 \}$

Remember that we are dealing with a classification problem where $y_i \in \{-1, +1\}$, so we want our classifier $f(x_i)$ to have the same sign as $y_i$. It might help to recall that $\mathtt w = \sum_i \alpha_i \phi(\mathtt x_i)$ and $f(\mathtt x) = \left< \mathtt w, \phi(\mathtt x) \right> = \sum_i \alpha_i k(\mathtt x_i, \mathtt x)$

Normally $y_i \left( \left< \mathtt w , \phi(\mathtt x_i) \right> + b \right) \ge 1$. What they've done is to set $b=0$ and eliminate the margin (RHS). The length constraint is to ensure uniqueness.

  The version space is illustrated as a region on the sphere as shown in
  Figs. 5 and 6. If the version space is shaped as in Fig. 5, the SVM
  solution is near to the optimal point. However, if it has an elongated
  shape as in Fig. 6, the SVM solution is far from the optimal one.

The reason there is an inscribed sphere is that:

  the SVM solution coincides with the Tchebycheff-center of the version
  space, which is the center of the largest sphere contained
  in $\mathcal V$. However, the theoretical optimal point in version
  space yielding a Bayes-optimal decision boundary is the
  Bayes point, which is known to be closely approximated by
  the center of mass.

In other words, they are saying that the SVM classifier is not always Bayes optimal. Please see the references for proof. The hypersphere comes about when you consider the margin $\gamma \equiv \min y_i f(\mathtt x_i)$. The support vectors are then the boundaries of the version space which are tangent to the hypersphere.

A cited paper* follows this idea to develop an algorithm to actually estimate the center of mass "by averaging over the trajectory of a billiard ball bouncing in version space." If you're interested in diagnosing SVM failures, perhaps it would help to read that paper too, since it claims to offer a better algorithm instead.

$\dagger$ M. Opper and D. Haussler, “Generalization performance of Bayes optimal
classification algorithm for learning a perceptron,” Phys. Rev. Lett.,
vol. 66, p. 2677, 1991.

* T. Graepel, R. Herbrich, and C. Campbell, “Bayes point machines: Estimating
the bayes point in kernel space,” in Proc. IJCAI Workshop Support Vector Machines, 1999, pp. 23–27


Looking for a rough explanation of additive hidden nodes and radial basis functions

Albeit not wrong, Huang seems to be the only person in the world to use the term "additive hidden nodes". By this, he means that the neuron computes the sum of weighted inputs. In other words, the kind of neural network you're already used to.

An RBF neuron, on the other hand, computes a distance (usually the Euclidean distance) from input to some center (which can be thought as the weights if you see them as a vector) and applies a exp(-dist²) function in order to obtain a Gaussian activation. Thus, RBF neurons have maximum activation when the center/weights are equal to the inputs.


First, I wouldn't use the word "noisy" here because if you know which instances are "wrong" then these are not noise, they are negative examples. In my opinion "noisy" is when positive and negative cases are mixed together in a way that makes it difficult (or impossible) to distinguish between them. I think this matters because you're more likely to find similar use cases and relevant methods using this terminology.

I don't have a precise method to suggest but I would check the state of the art in machine translation: it's also a sequence-to-sequence task in which there are potential positive/negative cases. In particular there has been some work done in MT quality estimation, where the goal is to predict the quality of a translation for a sentence. This might be related because it's about labeling or quantifying how good a translation is, and I would assume that there are works which re-use labelled/scored translations (including potentially wrong ones) in order to obtain a better model. Unfortunately I don't have any pointers since I haven't followed the field recently.


Ordering a material science dataset (properties names, properties scalars, formulas)

Given that your Dataframe is:
df2 = pd.DataFrame({
    "Chemical_Formula":["He", "NO_2", "CuO3", "CuO3", "He", "NO2"],
    "Property_name":["Electrical conduc.", "Resistance", "Hardness", "Fluorescence", "Toxicity", "Hardness"],
    "Property_Scalar":[1, 50, 10, 300, 39, 80]
})

Chemical_Formula
Property_name
Property_Scalar

0
He
Electrical conduc.
1

1
NO_2
Resistance
50

2
CuO3
Hardness
10

3
CuO3
Fluorescence
300

4
He
Toxicity
39

5
NO2
Hardness
80

You can use pivot to "unmelt" this in a wide format
df3 = df2.pivot(index="Chemical_Formula", columns="Property_name")

Chemical_Formula
('Property_Scalar', 'Electrical conduc.')
('Property_Scalar', 'Fluorescence')
('Property_Scalar', 'Hardness')
('Property_Scalar', 'Resistance')
('Property_Scalar', 'Toxicity')

CuO3
nan
300
10
nan
nan

He
1
nan
nan
nan
39

NO2
nan
nan
80
nan
nan

NO_2
nan
nan
nan
50
nan

From then on you can drop columns you don't need and plot them.


Impact of sparse features on tree-based models

There are of course different split algorithms and losses that change the behaviour in these cases a little bit. However if we take a look at the most common choice in modern tree based algorithms we can deduce what the impact would be.

Let's have a scenario where we have two binary features that we have to decide between and their corresponding target distribution.

$$P(f_1=0) = 0.95$$

$$P(y=0|f_1=0)= 0.5$$

$$P(y=0|f_1=1)= 0.05$$

$$P(y=0) = 0.5225$$

This means feature 1 is mostly 0, it is 50/50 whether the target is 0 in this case but if it is 1 then the target is almost always 1 as well.

Now we have a second feature which is a bit different.

$$P(f_2=0) = 0.5$$

$$P(y=0|f_2=0)= 0.4225$$

$$P(y=0|f_2=1)= 0.6225$$

In this case the feature is split evenly and there is predictive impact but it is less extreme.

Gini impurity

By reducing the gini impurity we hope to minimize the number of splits neccessary. The current gini impurity is:

$$G_I(p) = 1 - 0.5225^2 - (1-0.5225)^2 = 0.4989875$$

When splitting on a feature we need to take the weighted average gini impurity and check the reduction.

$$G_{If1=0} = 0.5$$

$$G_{If1=1} = 1 - 0.05^2 - 0.95^2 = 0.095$$

Given that most splits (95%) will go the left we weight that one much more:

$$G_{If1} = 0.95*0.5 + 0.05*0.095 = 0.47975$$

In the second feature we have:

$$G_{If2=0} = 1 - 0.4225^2 - (1-0.4225)^2 = 0.4879875$$

$$G_{If2=1} = 1 - 0.6225^2 - (1-0.6225)^2 = 0.4699875$$

Given that the splits are equal we get a normal mean:

$$G_{If2} = 0.4879875*0.5 + 0.5*0.4699875 = 0.4789875$$

In this case it is a bit close but it would choose the less sparse feature. The reason is that while the minimum impurity is much worse, because the split is completely balanced the impact in reduction on both sides has a bigger effect. In the sparse case the split to the right has a major reduction but only counts for 0.05 of the cases while the impurity on the left is at it's highest. As you can see there is a trade-off between the impurity on either side and the amount of cases where the impurity is reduced.


Which classification algorithms can handle 24000 features

Deep Learning Algorithms and Graphical model algorithms can handle that scale of features.

For example a typical parsing algorithm using CRF++ computes millions of features.
In case of Deep Learning, A typical image of 256*256*3 has to deal with 196608 number of features where each pixel in image is a feature.


How to train model to predict 1 value from multiple input samples

Is there really any benefit than just creating three different models, and combining them at the end? If the end use case is just to predict the best final models, then I would just train on them individually.

You could have also do a NN with 3 output nodes, for each target. If you're just looking to do regression or something, I don't see why you wouldn't just predict them independently. There is some literature on multi-output regression (good summary here), but these involve scales of predicting high dimensional vectors(or even matrices), rather than just 3 numbers, so I don't know how useful this literature would be for your case, but that's where I would look if you're insistent on having a single model predict 3 outputs.


There are some packages for anomaly detection such as Linkedin's Luminol (https://github.com/linkedin/luminol) or Microsoft's TagAnomaly (https://github.com/microsoft/TagAnomaly).
Also you can use clustering algorithms for your features and detect outlier clusters. Or you can tag your previous anomalies to train your data (however its not possible most of the cases)

So my suggestion is prepare your data with features such that 5_min_window_processcount_mean, 5_min_window_processcount_std, 5_min_window_processcount_std and 5_min_lag_processcount_std etc.(get lagged and windowed features and then calculate their mean,std,median etc)

After that try clustering and check it if it can find anomalies or not. If not try to label your data with those features and try classification algorithms. Meantime you can use above packages with your features. (time based lag/window features are very important).


What type of neural network could emulate a binary to HTML conversion tool?

In short your friend is correct, seq2seq is a reasonable match to the problem.

However, from your numbers this could be too complex to use current machine learning libraries on.

Despite you calling it "not feasible", you are far better off with the reverse engineering effort in my opinion.

If you really want to have a try using ML, you could start with some shorter files with the generated output and attempt to train a LSTM-based seq2seq model to match them. It may help, if the HTML is strongly templated and driven by data in the binary file, to reverse-engineer just the templating, and use seq2seq model to derive a more raw data sequence as opposed to HTML.

You should expect to spend several weeks learning how seq2seq works, followed by an unknown amount of time in the attempt to apply it to your problem. You will need a large number of sample files to train your model - hundreds of thousands, maybe millions. There might be exceptions if the generated HTML is very repetitive/formulaic, or if short segments of binary data can also generate short segments of HTML (so you could train on smaller segments of the larger files).

If any of this would break the terms and conditions for use of the software, I advise you not to attempt either reverse engineering or machine learning.


What are the tools to plot cluster results?

ELKI has some very nice cluster visualizations. You could also use tSNE or MDS.

But as you used cosine, I assume you have text data, and that is not easy to visualize. You probably first need to figure out how to visualize your input data, then you can think about adding cluster information to this.


How to learn to score new documents based on a existing set of scored documents?

First come up with some features for the document. Stuff like frequency of some popular words associated with that topic might work. In this way get the features for all the documents and then apply some algorithm. Some ways you could apply them are:

1) k means - cluster the documents on the basis of the features. Each cluster should be predominantly associated with a particular score value. Then see which cluster a new document will be assigned to.

2) Supervised learning - use neural networks, multiclass SVMs etc to classify the new document to a particular class (score) using the model you would have generated.

All of these are examples of classification to a discrete score value. However, since you are dealing with a large score range (0-2000), you could also try something like regression which will give you a continuous value, but could be rounded off to the nearest discrete one.

Check out the Coursera course on Machine Learning for a great introduction!


Technically this is possible but most approaches combine localization with some sort of classification so that the network can get better at scoring regions, and also removes the problem of having to use the [0, 0, 0, 0] vector to encode a null region since if the classifier associated with the region gives a low probability for that object you can ignore the bounding box. That way the bounding box regression can focus on finding regions that look like they contain objects, not their type. Note that the classifier needs to be trained so that it recognizes a non-object region (ie, it must have a null/garbage/background class).

See for example:

Object localization and detection

Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks

(I don't have enough reputation to post more links but you should also google the R-CNN paper "Rich feature hierarchies for accurate object detection and semantic segmentation")


CrossValidation using glmnet and very high values of Lambda?

I don‘t think a large lambda is a problem per se. It just means that a lot of regularization is going on (under Ridge). See here:

https://stats.stackexchange.com/questions/212056/ridge-lasso-lambda-greater-than-1

Here is a good tutorial from the authors of glmnet. I suggest you check your approach, i.e by looking at various figures as shown in the tutorial, especially plot(cvfit) might be instructive. Also when you go through the tutorial, you see quite „large“ values of lambda (note that log of lambda is plotted).

https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html


How to deal with a feature that has lot of categorical values?

one idea is to craft some features of this feature - when you classify cars you don't have data like strings "Ferrari 991 year 2014 red" "BMW z4 year 1999 2.0L blue" but you would like to have columns like "Manufacturer", "Type(SUV/Cabrio...)", "Year", "Engine" etc.

you could transform it or just discard


Pandas v. SFrame in learning data science

SFrame is not used much in industry, so I'd stick to pandas or Spark DataFrames. But they're all rather similar, and you should not spend much time thinking about it: it is easy to pick these things up, and employers understand this. Concentrate on the algorithms; that's the real "foundation", not the tools.


Is standardization needed before using scikit-learn SVM?

scikit learn does not standardize data, but it does offer utilities for you to standardize your input data yourself:
http://scikit-learn.org/stable/modules/preprocessing.html

the rule of thumb is to standardize if your data aren't related.  That is, if channel X is not a function of channel Y, you should standardize

Qualitatively, think about it this way, SVM 'creates a hyperplane' to separate data into categories; if the data are skewed too far in one axis, that will make it harder to draw a plane to separate them


Why not use Scaler.fit_transform on total dataframe?

If you would use the scaler on the full dataset you would provide the algorithm with some information about the values in the test set that it would not have otherwise.

This additional information ("data leakage") from your test set to your training set is problematic because it leads to unreliable performance estimates. It is therefore not surprising that you achieve a higher R-squared. Due to the data leakage, this R-squared might be overly optimistic because it could depend on the additional information that you introduced into the training set.

This is particularly true for the MinMaxScaler because it is by definition very sensitive to outliers. The effect will probably less problematic if you use the RobustScaler (or even the StandardScaler).

The same holds for other preprocessing steps like outlier removal, feature selection etc.

If you are worried that your training data does not adequately reflect the true distribution, you can fall back to a cross-validation approach with multiple folds so you can estimate the effect across multiple splits of the data. Again, remember to fit the scaler on all the training folds and apply it to the test fold for every iteration.


Anomaly detection for time series data with only positive samples?

If you want to model it as a regression problem, using only the normal ECG data sounds enough to me. In that setting, abnormal data would be only used for validation of your model, but even those few negative samples that you have could be valid for that.

You have a variety of options for getting that divergence, e.g. the Mahalanobis-Taguchi System. You can also model the error coming from your regression model or go fully Bayesian with something like Gaussian Processes, so that you already have a distribution for your posterior. That way, you can cross check new negative samples.


I would not start with (manual) feature selection. Use Lasso instead to "automatically" shrink/select features (this is Logit with shrinking of features basically). Logit (or Logit with Lasso as here) is for binary cases, but you can also do "Multinominal Logit" (option multi_class='multinomial' in sklearn), which is for more than two classes. Usually you use sklearn in Python for such things. Also see the examples in the sklearn docs.

Make sure you have a test and trainings set. Also make sure that you do not use data from your test set for training. Train on the train set only and use the test set to see how your model performs on data NOT seen during training.

It is not clear what you mean when you say "move to production". This depends on your problem. You simply need to make predictions here, but the implementation is of course contingent on the environment.

It is okay to play around with data. However, if you really want to go for serious data science, you should have a look at the methods behind all this magic. I recommend "Introduction to Statistical Learning". It is a really good book with many code examples and it is not very technical.

Note that there is no silver bullet. Lasso or Logit may be okay, but other methods may be better. This really depends on the problem/data.

Here is a little sample code for Lasso:

# Split test/train
from sklearn.model_selection import train_test_split
xtrain, xtest, ytrain, ytest = train_test_split(X, y, test_size=0.20, random_state=7)

from sklearn.metrics import roc_auc_score
from sklearn.metrics import confusion_matrix
from sklearn.linear_model import Lasso
from sklearn.linear_model import LassoCV

# Perform lasso CV to get the best parameter alpha for regulation
lasso = Lasso(max_iter=10000)
lassocv = LassoCV(alphas=None, cv=10, max_iter=10000)
lassocv.fit(xtrain, ytrain.values.ravel())

# Fit lasso using the best alpha
lasso.set_params(alpha=lassocv.alpha_)
lasso.fit(xtrain, ytrain)

# Look at results (coefficients)
la1 = pd.Series(abs(lasso.coef_), name="lasso")
la2 = pd.Series(X.columns, name="names")
dflasso = pd.concat([la2,la1], axis=1)
dflasso = dflasso.sort_values(by=['lasso'], ascending=False)
print(dflasso)

# Look at AUC
print("AUC Lasso: %.3f" %roc_auc_score(ytest.values, lasso.predict(xtest)))

# Predict probs
lasspreds0 = lasso.predict(xtest)
# Classes
lasspreds = np.round(lasspreds0)

# Confusion matrix
tnlog, fplog, fnlog, tplog = confusion_matrix(ytest, lasspreds).ravel() #y_true, y_pred
print("True negative: %s, False positive: %s, False negative: %s, True positive %s" %(tnlog, fplog, fnlog, tplog))
print("Share false %.2f" %(((fplog+fnlog)/(fplog+fnlog+tplog+tnlog))))

# Look at probs
print("Min. prob. of belonging to class 0: %.3f" %lasspreds0.min())
print("Max. prob. of belonging to class 0: %.3f" %lasspreds0.max())

EDIT:

Please note that the sklearn Lasso as described above does not do a logistic regression, which means predictions can be smaller zero or larger one. To use Lasso with Logit (ensuring predictions are zero or one), one can use LogisticRegression:

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
log = LogisticRegression(penalty='l1', solver='liblinear')
log.fit(X, y)


Use 1 or 2 norm for Voronoi vector quantization?

In general the quality of the quantization is measured using the mean squared error (MSE) between the input vector $x$ and the reproduction value $q(x)$ ($w_k$ in the notation you used in the question). For the MSE the best partition is the one defined by Voronoi regions under the euclidean distance ($l^2$ norm), i.e:
$$V_j = \{ x \in R^d : \|x - w_j \|^2 \leq \|x - w_i \|^2\ for\ all\ i   \}$$

You can find the proof in here. Also here and here the euclidean distance is used.


How to use 5 by 5 or 7 by 7 kernel size for a deep learning network with 3 by 3 kernels?

The main think you need to be mindful about is the size of the output of your convolutional layer, remember that it's going to be
$$
\frac{W-K+2P}{S} + 1
$$
with your K the kernel size, so if you increase your kernel size the size of your output will decrease faster. You might have to reduce the number of layers. If you're using a framework for which you need to specify input size for each layer then in that case you would need to change that as well.


How to prepare the varied size input in CNN prediction

Conventionally, when dealing with images of different sizes in CNN(which happens very often in real world problems), we resize the images to the size of the smallest images with the help of any image manipulation library (OpenCV, PIL etc) or some times, pad the images of unequal size to desired size. Resizing the image is simpler and is used most often.

As mentioned by Media in the above answer, it is not possible to directly use images of different sizes. It is because when you define a CNN architecture, you plan as to how many layers you should have depending on the input size. Without having a fixed input shape, you cannot define architecture of your model. It is therefore necessary to convert all your images to same size.


Ranking of feature sets which will be used for binary classification

You could use some statistical method to rank these feature sets, such as extracting in each features set a discriminative score per feature (kolmogorov smirnof, mutual information, ...) and then take the mean, the median, the p95, ...
But, depending on the classifier that will be used, this score may be useless, because some classifiers act as features selectors (decision trees, neural networks, ...) and others don't (Gaussian NB, kNN,...).
Example
If you have 10 features in your 500 that are really good to discriminate your classification and the other 490 that do not contain information. Using a decision tree based classifier will lead to good performance as probably only the 10 features will be used. kNN in the other hand will lead to bad performance...
However, your features set score would remain the same.


Exploration vs exploitation tradeoff to find a price that maximizes revenue

Without making any underlying assumptions you will not get anywhere. That said, there are multi-arm bandit strategies that try to optimize the rewards, there is a ton of research on this field. It comes down to sampling from a distribution of your options (in your case two) and adapting this distribution based on the rewards.

https://en.wikipedia.org/wiki/Multi-armed_bandit

Once you know that the reward distribution from each bandit comes from a specific distribution, you can deduce optimal sampling strategies. Once you have at least some prior information you can do fairly well although not always optimal. Regardless, most strategies will do better than normal A/B testing if the strategy is not super greedy.