If I'm understanding you correctly, you have a bunch of sets of 20 3D points, and for each set of points you have three metrics associated with it.

Yes, it's definitely possible to learn which points are most predictive for each metric. I would recommend that you split your data such that the point sets are in one numpy array which looks like

x1_1, y1_1, z1_1, x1_2, y1_2, z1_2, ..., x1_20, y1_20, z1_20
x2_1, y2_1, z2_1, x2_2, y2_2, z2_2, ..., x2_20, y2_20, z2_20
...

and then create one 1D numpy array for each of your metrics which contain the value of each metric for every point set. Then, you can use something like a RandomForestRegressor to try to predict the metrics individually from the points. Once you have the random forest trained with the fit(X, y) method, you can extract the importances of each coordinate in the points by getting the feature_importances_. For example, something like this:

rf1 = RandomForestRegressor(n_estimators=1000)
rf1.fit(points, metric1)
print(rf1.feature_importances_)

rf2 = RandomForestRegressor(n_estimators=1000)
rf2.fit(points, metric2)
print(rf2.feature_importances_)

rf3 = RandomForestRegressor(n_estimators=1000)
rf3.fit(points, metric3)
print(rf3.feature_importances_)

Hope this points you in the right direction!


Fisher Scoring v/s Coordinate Descent for MLE in R

The only way to be sure is by benchmarking, but for glm Fisher scoring should be faster than coordinate descent.  Fisher scoring is a special case of Newton Raphson, which has a faster rate of convergence than coordinate descent (Newton-Raphson is quadratically convergent, while coordinate descent is linearly convergent.)  So while the computation of second-derivative information means each step takes more time, it can require many fewer steps than coordinate descent.

For the lasso, the special form of the penalty term makes it a very special case (and in fact absolute value isn't differentiable anyway, though sometimes you can finesse this).  For this special problem, coordinate descent proves to be particularly fast.  There are many other optimization problems where in practice Newton-Raphson is faster.


Decision trees work well with categorical variables because of the node structure of a tree. A categorical variable can be easily split at a node. For example, yes or no or 0 or 1. A continuous variable is just that, continuous along a range which cannot be easily split at a node. For example, a floating point number between 0 and 100.


How to add a new column with labels in a dataframe?

Do you have a working method to get the sentiment label for an individual sentence?

If so, I would think you could just create a method to get the sentiment given a sentence and use it's return value as the row entry.

The apply function from pandas will apply a function to every row if you specify axis=1

def getSentiment(row):
    # Do some sentiment operations here on row['Testo']
    return sentiment_label

df.apply(getSentiment, axis=1)


Build Deep Belief Autoencoder for Dimensionality Reduction

Unlike Autoencoders, Botzmann Machines (restricted or not) do not have an output layer and thus classified as deep generative models.
There is a variety of implementations in Pytorch. This one is GPU compatible (https://github.com/GabrielBianconi/pytorch-rbm) and I have found it particularly helpful in the past.
RBMs can come quite handy in a variety of tasks such as

Dimensionality reduction
Collaborative filtering for recommender systems
Feature learning
and others.

This was an interesting read in case you want to find out more about RBMs. https://heartbeat.fritz.ai/guide-to-restricted-boltzmann-machines-using-pytorch-ee50d1ed21a8


Word Mover’s Distance (WMD) is an algorithm for finding the distance between pairs of strings. It is based on word embeddings (e.g., word2vec) which encode the semantic meaning of words into dense vectors.

  The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

Source: "From Word Embeddings To Document Distances" Paper

The gensim package has a WMD implementation.


How gradients are flown back to Network in siamese architecture? How weights of all CNN models are same even when using different models

You only create ONE model for a siamese network that you pass your inputs to.
(You have created three models in the example)
So in your triplet case you would pass the three inputs seperatly to the network and compute a loss and backprop it.
The gradients are then updated just like training any neural net.


model.fit indefinite execution in Google Colab

I have solved the problem.

In case anybody else deals with similar situation, the solution is to assign a batch_size for training and validation images. Then add the steps_per_epoch and validation_set as parameters within model.fit.

steps_per_epochs= number_of_training_images/batch_size

validation_set = number_of_test_images/batch_size


No specific answer to your question, it all depends on which algorithm you are using or in other words how you will use the normalized data. Based on my experience I found that the zscore normalization performs the best, especially if you are using svm or nn.


Unsupervised Learning::Satellite Images::Single Bands

it fails as it requires a 2D array not 1D

The X in .fit(X) is 2D, where each row is one training point and the columns are the features of that training point.

It should be a list of lists:

X = [
  [feature_1, feature_2,....feature_n], # training point 1
  [feature_1, feature_2,....feature_n]  # training point 2
  .
  .
  .
  [feature_1, feature_2,....feature_n]  # training point 123
]

The K-Means will train with all the examples in X.

However, you are only giving 1 training example, namely img_A. So your X is like this at the moment:

X = [feature_1, feature_2,....feature_n] # img_A

So, to .fit() K-Means, you need more image examples.


Smaller alternatives to sklearn that doesn't require scipy?

Did you check tinynumpy?
Anyway, I rarely found alternatives to famous packages (except scikit-image instead of opencv). What usually works for me is:

Slim the model as much as I can (e.g. weights quantization)
Check in the code which functions I use from each module. Once I have a list of them, I retrieve the corresponding python files and get rid of the rest
Try to split my process in multiple functions (e.g. one function to perform data processing, one function where to implement the model and make the inference)

The second point is crucial. In my experience, one rarely needs entire packages.
However, depending on the case, it could also be that AWS Lambda does not fit your needs.


Machine Learning validation data returns 100% accuracy

I'm assuming this is a classification task as you mentioned accuracy:
100% on the validation set is usually a red-flag. Check there is no data leakage in your code (inspect it at each line). Make sure your validation set has a mixture of both classes (for example: is this a very imbalanced set and your validation set has all negative class?).
There is really no way to tell if or what the problem is without some description of the data or presence of code - but 100% accuracy is usually a big indication to dig into the details to find out if there's a bias you are adding to the system.


Can I install Tensorflow and Keras on Cloud?

I think your best option is to use the cloud, since your PC is not capable, and has limited resources.
You can set up a Google Colab or Kaggle account to run Jupyter notebooks online. They will support most libraries, including Tensorflow, Keras, and Pytorch. Both will also provide limited GPU resources.

Google Colab has a free version you can use, just need to keep an eye on your usage.
Kaggle is free, you don't need to enter a competition to use the service. Kaggle also has discussion boards and public notebooks to look at for ideas.


It's a dimensionality reduction algorithm. Inference is the problem of determining the parameters, or labels, that best fit the model for a given input once the model parameters have been learned, or estimated.


How to best accomplish high speed comparison of like data?

Can't you create a hash for each classes, and then merge rows by rows, field by field only the classes where the hash changed ? It should be faster if most of the classes don't change..

Or a hash of each rows or maybe columns.. depending on how the data normally change..


How do I perform Leave One Out Cross Validation For Top n Recommendation Sytems?

Here is my procedure for computation of the "Hit Rate with leave-one-out cross validation":

leave only one actual user-item interaction (this can be rating, purchase or whatever you are dealing with in your data) out of the train dataset (this gives you a total of U "unseen" interactions provided that you have at least one interaction per user);
Generate top n recommendations for all user with whatever recommendation system you are using;
if a user actually interacted with the recommended item (is the left-out" item was among the top n recommendations for that particular user) => that's a hit
Hit rate = sum of all the hits across the users (this can be at most U since you can have at most 1 hit per user) DIVIDED BY the number of users (U).


Feature importance and deriving rules using tree based classification models

You can use categorical features with decision trees in scikit-learn, but you'll need to encode them as numbers. If your categorical features are ordinal (such as ranking ‘bad’, ‘fair’, ‘good’), they are easy to encode in numbers that respect the underlying ordering (e.g. 0, 1, 2). For nominal features, given the high cardinality you mention, you can try some version of the target encoding or the frequency encoding. With the target encoding, make sure to use some form of regularization in order to avoid overfitting.

After the features are encoded, you can build, for example, a random forest and obtain the impact of each feature with attribute feature_importances_.


Partial answer: Quoting a paper
"The advantage of such deep supervision is evident: (1)for small training data and relatively shallower networks, deep supervision functions as a strong “regularization” for classiﬁcation accuracy and learned features; (2) for large training data and deeper networks deep supervision makes it convenient to exploit the signiﬁcant performance gains that extremely deep networks can bring by improving otherwise problematic convergence behavior"


max_iter hyper parameter in sklearn.cluster.MiniBatchKMeans

max_iterint, default=100
Maximum number of iterations over the complete dataset before stopping independently of any early stopping criterion heuristics.

It's the number of iteration over the full dataset.
Number of partial_fit will depend on batch_size

batch_sizeint, default=100
Size of the mini batches.

partial_fit(self, X[, y, sample_weight])
Update k means estimate on a single mini-batch X.

Ref :
User guide
Class Def.


Correlation/distance between sparse vectors

The first idea that comes to mind is a similarity measure such as cosine. It's often used with sparse vectors (text represented as vectors over the vocabulary).
There are many options for distance/similarity measures:

Basic set measures like overlap coefficient or Jaccard
Entropy-based measures such as KL divergence.
...


Why isn't the neural network updated after every example in the dataset

Weights get updated based on the number of examples you feed in a batch. This is because, a full pass(forward and backward) of matrix computations has to be completed in order for the weights to be updated, after back-propagation and proceeding with next epoch, with batch type you had chose.
Moreover, If you use stochastic gradient descent, where each example will be processed at a time, your weights will updated after having processed every example.


Neural networks, optimization math intuition

Note that $\frac{\partial L}{\partial \theta}$ is different from $\frac{\partial \theta}{\partial L}$. What you tried to describe seems to be $\frac{\partial L}{\partial \theta}$ where $\theta$ is a variable.  If $\theta$ is high dimensional, sometimes we just use the $\nabla$ notation.

Gradient descent is

$$\theta_{n+1}=\theta_n-\gamma \nabla L(\theta_n)$$

Not everything is differentiable and gradient might not be well defined for some optimization problem.
In the event that there are constraints, $L$ might need to take the role of Langragian rather than the objective function.
Gradient descent is just a means to find the parameters for a model. Gradient based approach seems to be the norm for now but things can change.

What you proposed to tattoo is just "gradient" or "slope". Not objection but just want to let you know what you are doing.


Python & Pandas : TypeError: to_sql() got an unexpected keyword argument 'flavor'

Based on the documentation 0.22 and 0.24.1, the flavor does not exist in the argument list of the to_sql method. You're probably running the 0.24.1 version which does not need flavor argument.


what is the meaning of independent and identically distribution of samples in a dataset for neuralnetworks

Once the data is with us, the best we can do is shuffling the samples to make them Identically distributed assuming a hypothetical process behind it that generates all the samples.
Let's take the example of MNIST digits, the hypothetical process is our similar knowledge of digits, so most of us will write a specific digit in a similar way.
Now let's test the MNIST model on Japanese MNIST, it will fail as the assumption is broken.
Again, mix both the MNIST, Shuffle it, Split it and Train it. It will learn the pattern and can predict the digits.
Once again, both the train/test have a similar probability distribution of pixels.
While training the model,
If the process is incremental e.g. Gradient Descent, we try to assure that each batch represents the parent distribution. Again this is achieved by Shuffling esp. Stratified shuffling.


Naive Bayes is almost always my first go to for a problem with text data. This is primarily due to the independent distributions that integrate so well with the document by feature  independent style matrix by nature. I would extend your idea of Ridge with Lasso or ElasticNet. I would not recommend normalize your data if it is sparse, this has been known to cause problems, but StandardScaler is a rather conventional way to get a centered dataset without producing disharmony among the variables in the data. I would consider your error metric of precision or fmeasure under each of the above models as my personal next step, then do some visualizations to argue your best fit.


how to scale a dataset contains a b&w and Grayscale images

$\hat{p}=2\big(\frac{p}{p_{max}}\big)-1$ so that $\hat{p}\in [-1,1]$ where $p\in[0,p_{max}]$
E.g. with 8-bit channels, $p_{max}=255$, black/white correspond to $-1$ and $1$ respectively, and grayscale is linearly mapped in $[-1,1]$.


Why does BERT embedding increase the number of tokens?

A [CLS] token is added to the beginning of the sentence, and a [SEP] token is added to the end.
These two special tokens have specific functions:

[CLS] is needed because it was used in the training loss of BERT to keep the first output position for a different purpose than the rest: it was used for the "next sentence prediction" loss, which trained the model to tell if the 2 text segments passed as input were consecutive in the original text or not. When finetuning BERT, the output of this position is normally used for sentence classification tasks. When BERT is used for feature extraction, the output vector at that position is used as sentence embedding.

[SEP] is used to mark the ending of the sentence.


Does it make sense to repeat calculating AUC in logistic regression?

The calculation of ROC curve and the AUC based off of that curve is simply a comparison of the predictions from your model (logistic regression) and the actual values on some set of data. This can occur with predictions on the training set or predictions on a test set. Best practice is to do this comparison on a test set as this will best represent the performance of the model on new data.
Averaging of the AUC based on repeated train-test splits, with retraining of the logistic regression on the training set and calculation of the ROC curve and AUC on the test set can give a better estimate of the performance of the model. In addition, the distribution of the AUC across each of the splits can give a sense of the stability of the model's performance.
The repeating of train-test splits is commonly called cross-validation and most commonly performed through k-fold cross-validation where you split the data into k sets, use one as the test set and the rest as the training set. You can then repeat this process using each of the k sets as the test set.


If you're in Python, there are a couple of packages that can automatically extract hundreds or thousands of features from your timeseries, correlate them with your labels, choose the most significant, and train models for you.

https://github.com/blue-yonder/tsfresh

https://github.com/rtavenar/tslearn


How to find the average and median values of the column 'Total Traded Quantity' for each of the types of 'Trend'

df.groupby(df.Trend).mean()['Total Traded Quantity']
df.groupby(df.Trend).median()['Total Traded Quantity']

or

df_trend_average = df.groupby(df.Trend).mean()
df_trend_median = df.groupby(df.Trend).median()


is there a way to normalize [-3,1] to ${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\sqrt{10}}\\ \end{bmatrix}}$ with python?

You have already computed that, but you've not bound the output to a variable, also called name in python. Try the following snippet:

result = np.linalg.norm(v1,ord=2,axis=1,keepdims=True)
print(result)

Based on the edit, I update the answer. As you may find answers to your question, a typical way to find what you need is something like the following function:

def normalize(v):
    norm = np.linalg.norm(v)
    if norm == 0:
       return v
    return v / norm

Equivalently, there is a function called normalize in sklearn.preprocessing which can be employed for your task.


Automation of finding a starting point of measurement in a large dataset

Well ... I would do exactly what you did. The derivative on original signal is very noisy. I would probably take derivative out of moving-averaged smoothed signal, however it brings some delay into your detection. See this answer for more info and python code.
The other approach is to detect the point in time-frequency domain. Simply plot the STFT of your signal and see if that helps the accuracy of detection over derivative.


Causes of inconsistent results with neural network?

Before briefing about the other things, I would like to suggest you something good. It is always better to set a seed number in machine learning and deep learning problems to get a reproducible result and by that I mean that every time when you try to run your model anywhere, it can produce the same best result which you got on your machine while experimenting. So, you should always set numpy.random.seed(*anynumber*).

Now how to compare your result to your expected result? There are a number of parameters that you can look upon during training your model. Some of these are:

Validation accuracy: It is the one of the most important parameter to look upon for evaluating your model's performance. This gives you a fair idea if your model is overfitting or not. More the validation accuracy, the better is the model.
Validation loss: If your model is really good, then validation loss must decrease, at least initially until your best epoch hasn't been obtained. Lower the validation loss, better the model.
Optimizers: There are a number of optimizers like 'rmsprop', 'sgd', etc. which can be used for different types of problems but not all give the same performance. You should grid-search for a number of optimizers for your model if you are not specific about which is best for your model.
Number of training epochs: Depending upon the size of dataset, no. of training epochs differ but if you are not giving enough training to your model, then your model performance will suffer.
Training data: If you think your model is fine, but you are not achieving expected results, then you can try to train your model on a large dataset. NN require large dataset for achieving awesome performance.


seq2seq is a type of problem, that can be solved with LSTM, RNN etc...

Generally one thinks about sequences of data that are inter connected and these relationships in sequences should be modeled (which can be achieved with for example LSTM etc.)


How do Bayesian methods in machine learning help with the problem of limited data? Can this be used for image classification?

The discussion of Bayesian reasoning and its advantages over frequentist reasoning is very wide. I refer you to the book “Pattern recognition and machine learning” by Cristopher Bishop which is a great book on Bayesian reasoning.


Getting lower performance metrics when using GridSearchCV

Your confusion seems to stem from this line:

print('Best (Train) AUC Score: {:.4f}%'.format(gsearch.best_score_*100))

The best_score_ is not exactly a training score (nor is it an unbiased estimate of future performance*): as you say, it's the averaged score across different fold splits, but each of the scores that get averaged are the performance of the models on their test fold.  So, this score reflects performance of models on unseen data.

But, when you compute the score of the model retrained on the entire training set, that model has seen the training data, and the score there is inflated (quite a lot) as you'd expect.

* This is discussed at length elsewhere, but in short, while the scores are based on performance on data unseen by the models, you have looked at that data when selecting the "best" model, so to use that score now would be optimistically biased.


Rephrase Neural Network : Find the Fittest Word for Given Meaning/Explanation

In a perfect setting, where you have infinite sentences (definitions).

You could use some kind of attention models by chunking sentences into smaller pieces.

[a spherically shaped] [fruit] which [tastes sour] and [sweet] and gets [red] when matured well.

With a distributed representation framework, every little piece of information is simply like a binary indicator (the feature exists or not).

Maybe each piece is vague by itself and it could not tell you anything about the target yet by combining them together they define useful information, see products of experts

To learn p(Y|X) where x consists of the sentence, y is the embedding of some class, so may similar classes share similar sentences.
If you have a model trained on the sentences and learned to project similar sentences near to each other, and assume you learned the word classes by some word2vec like algorithm, you could generalize to new pair even if the sentence and that class target were never paired (never seen in your training data), their respective feature vectors have been related to each other by the mapping function.

At the end of the day if you have a smooth function and your input is continuous then any small change of the input will introduce small change at the output, yet how could we truly extract the knowledge exists in the data. see Generating Sentences from a Continuous Space


For the 1st question, I think B is not a supervised learning problem. You are already given emails which are spam. You need to analyse them to find if there are subtypes (Clustering.)

For the 2nd question, it is just B. Scaling doesn't make each step less computationally expensive. You do same amount of computation.


How to find the size of a tensor in bytes?

Step 1 :Get the dtype of the tensor. This will tell you about the number of bytes e.g.float64 is 64 bits = 8 Bytes.

Step 2 Get the shape of the Tensor. This will give you the number of place-holders of the dtype. lets's assume shape = m x n x p
Count of the placeholders is C = m * n * p
Memory = 8 * C => Memory = 8 *m * n * p Bytes.

See this excerpt from Deep Learning with Python, by Francois Chollet

  A 60-second, 144 × 256 YouTube video clip sampled at 4 frames per second would have 240 frames. A batch of four such video clips would be stored in a tensor of shape (4, 240, 144, 256, 3). That’s a total of 106,168,320 values! If the dtype of the tensor was float32, then each value would be stored in 32 bits, so the tensor would represent 405 MB


What is the best question generation state of art with nlp?

For your first part of the question as to which question generation approaches are good - Neural question generation is being pretty popular (as of 2018/2019) among NLP enthusiasts but not all systems are great enough to be used directly in production. However, here are a few recent ones which reported the state-of-art performances in 2019 and have shared their codes too:

https://github.com/ZhangShiyue/QGforQA
https://github.com/PrekshaNema25/RefNet-QG

This one is a 2020 one (now that NLP performances have been improved with Transformers)
3. https://github.com/patil-suraj/question_generation
Besides, if you want more control as to understand and fix for wrongly generated questions, I would suggest the more traditional rule-based approach like the below which is more reliable than the above neural ones ones and generates a larger amount of question-answer pairs than the above 2:

http://www.cs.cmu.edu/~ark/mheilman/questions/
https://bitbucket.org/kaustubhdhole/syn-qg/src/master/

To answer your second question, if your QG model is generating an answer, then it makes sense to use cosine similarity. Assuming your question generation is at the sentence level, you will mostly have short answer spans and hence averaging Glove or Paragram word vectors might serve you better results than the Universal Sentence Encoder.


Connection between the first pooling layer and the second convolutional layer

The first dimension is the batch size. From the tutorial you're linking to:

  We store the training feature data (the raw pixel values for 55,000 images of hand-drawn digits)

So it looks like your code is performing each gradient update step using all 55,000 training examples. That's unorthodox, but it's fine as long as your computer can handle it. See this video for an explanation of batching in gradient descent.

Your interpretation of the shape as being 32 14x14 images is fine for now, but it is more proper to refer to these 32 matrices as "feature maps". A grayscale image is/contains a single feature map, while a color image would consist of three of them, one for each red, green, and blue (assuming RGB color space). Also, note that at this point in the network the feature maps are no longer "images" as any human would understand the word.


The new Amazon Machine Learning Service may work for you. It works directly with Redshift and might be a good way to start. http://aws.amazon.com/machine-learning/

If you're looking to process using EMR, then you can use Redshift's UNLOAD command to land data on S3. Spark on EMR can then access it directly without you having to pull it into HDFS.

Spark on EMR: https://aws.amazon.com/articles/Elastic-MapReduce/4926593393724923


Why are NLP and Machine Learning communities interested in deep learning?

Why to use deep networks?

Let's first try to solve very simple classification task. Say, you moderate a web forum which is sometimes flooded with spam messages. These messages are easily identifiable - most often they contain specific words like "buy", "porn", etc. and a URL to outer resources. You want to create filter that will alert you about such suspecious messages. It turns to be pretty easy - you get list of features (e.g. list of suspicious words and presence of a URL) and train simple logistic regression (a.k.a. perceptron), i.e. model like:

g(w0 + w1*x1 + w2*x2 + ... + wnxn)

where x1..xn are your features (either presence of specific word or a URL), w0..wn - learned coefficients and g() is a logistic function to make result be between 0 and 1. It's very simple classifier, but for this simple task it may give very good results, creating linear decision boundary. Assuming you used only 2 features, this boundary may look something like this:

Here 2 axes represent features (e.g. number of occurrences of specific word in a message, normalized around zero), red points stay for spam and blue points - for normal messages, while black line shows separation line.

But soon you notice that some good messages contain a lot of occurrences of word "buy", but no URLs, or extended discussion of porn detection, not actually refferring to porn movies. Linear decision boundary simply cannot handle such situations. Instead you need something like this:

This new non-linear decision boundary is much more flexible, i.e. it can fit the data much closer. There are many ways to achieve this non-linearity - you can use polynomial features (e.g. x1^2) or their combination (e.g. x1*x2) or project them out to a higher dimension like in kernel methods. But in neural networks it's common to solve it by combining perceptrons or, in other words, by building multilayer perceptron. Non-linearity here comes from logistic function between layers. The more layers, the more sophisticated patterns may be covered by MLP. Single layer (perceptron) can handle simple spam detection, network with 2-3 layers can catch tricky combinations of features, and networks of 5-9 layers, used by large research labs and companies like Google, may model the whole language or detect cats on images.

This is essential reason to have deep architectures - they can model more sophisticated patterns.

Why deep networks are hard to train?

With only one feature and linear decision boundary it's in fact enough to have only 2 training examples - one positive and one negative. With several features and/or non-linear decision boundary you need several orders more examples to cover all possible cases (e.g. you need not only find examples with word1, word2 and word3, but also with all possible their combinations). And in real life you need to deal with hundreds and thousands of features (e.g. words in a language or pixels in an image) and at least several layers to have enough non-linearity. Size of a data set, needed to fully train such networks, easily exceeds 10^30 examples, making it totally impossible to get enough data. In other words, with many features and many layers our decision function becomes too flexible to be able to learn it precisely.

There are, however, ways to learn it approximately. For example, if we were working in probabilistic settings, then instead of learning frequencies of all combinations of all features we could assume that they are independent and learn only individual frequencies, reducing full and unconstrained Bayes classifier to a Naive Bayes and thus requiring much, much less data to learn.

In neural networks there were several attempts to (meaningfully) reduce complexity (flexibility) of decision function. For example, convolutional networks, extensively used in image classification, assume only local connections between nearby pixels and thus try only learn combinations of pixels inside small "windows" (say, 16x16 pixels = 256 input neurons) as opposed to full images (say, 100x100 pixels = 10000 input neurons). Other approaches include feature engineering, i.e. searching for specific, human-discovered descriptors of input data.

Manually discovered features are very promising actually. In natural language processing, for example, it's sometimes helpful to use special dictionaries (like those containing spam-specific words) or catch negation (e.g. "not good"). And in computer vision things like SURF descriptors or Haar-like features are almost irreplaceable.

But the problem with manual feature engineering is that it takes literally years to come up with good descriptors. Moreover, these features are often specific

Unsupervised pretraining

But it turns out that we can obtain good features automatically right from the data using such algorithms as autoencoders and restricted Boltzmann machines. I described them in detail in my other answer, but in short they allow to find repeated patterns in the input data and transform it into higher-level features. For example, given only row pixel values as an input, these algorithms may identify and pass higher whole edges, then from these edges construct figures and so on, until you get really high-level descriptors like variations in faces.

After such (unsupervised) pretraining network is usually converted into MLP and used for normal supervised training. Note, that pretraining is done layer-wise. This significantly reduces solution space for learning algorithm (and thus number of training examples needed) as it only needs to learn parameters inside each layer without taking into account other layers.

And beyond...

Unsupervised pretraining have been here for some time now, but recently other algorithms were found to improve learning both - together with pretraining and without it. One notable example of such algorithms is dropout - simple technique, that randomly "drops out" some neurons during training, creatig some distortion and preventing networks of following data too closely. This is still a hot research topic, so I leave this to a reader.


How to show percentage (text) next to the horizontal bars in matplotlib?

Here is a working example to add a text to the right of horizontal bars:

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

df = pd.DataFrame(np.array([['a'], ['a'], ['b']]), columns=['current_status'])
ax = sns.countplot(y="current_status", data=df)
plt.title('Distribution of  Configurations')
plt.xlabel('Number of Axles')

total = len(df['current_status'])
for p in ax.patches:
        percentage = '{:.1f}%'.format(100 * p.get_width()/total)
        x = p.get_x() + p.get_width() + 0.02
        y = p.get_y() + p.get_height()/2
        ax.annotate(percentage, (x, y))

plt.show()

Output:

You can move the texts around by changing the formula for x and y. For example, to place them on top:

x = p.get_x() + p.get_width() / 2
y = p.get_y() - 0.02

Here is a general guideline for annotating axes.


What does normalizing and mean centering data do?

If you don't center before you normalize, you don't take advantage of the full [-1,1] range if your input is non-negative. The combination of centering and normalization is called standardization.

Sometimes one normalizes by the standard variation, and other times by just the range (max-min). The latter is called feature scaling. The effect is much the same. Normalizing by the range is easier computationally. Normalizing by the standard deviation fixes the sample variance, which is nice from a statistical perspective. When using the standard deviation, the subtraction is usually against the sample mean rather than the minimum.

There are several reasons for performing standardization. Sometimes we are interested in relative rather than absolute values. Standardization achieves invariance to these irrelevant differences. By explicitly preprocessing the data to reflect this disinterest, we relieve the model from having to learn it, allowing us to use a simpler one. Another reason is computational; it reduces the condition number -- you can think of this as the skewness or niceness of the loss surface -- making optimization easier and faster.


How to find bias for perceptron algorithm?

Rather than viewing your data as

X = [[0.8, 0.1], [0.7, 0.2], [0.9, 0.3], [0.3, 0.8], [0.1, 0.7], [0.1, 0.9]]
Y = [-1, -1, -1, 1, 1, 1]

You could have treat the problem as

X = [[1, 0.8, 0.1], [1, 0.7, 0.2], [1, 0.9, 0.3], [1, 0.3, 0.8], [1, 0.1, 0.7], [1, 0.1, 0.9]]
Y = [-1, -1, -1, 1, 1, 1]

That is to append a $1$ in every single entry of $X$. The weight corresponding to $1$ would be the bias.

That is $Y=Ax+b$ can be written as $Y=[A, e]\begin{bmatrix}x \\b\end{bmatrix}$ where $e$ is the all one vector.


How to aggregate face embeddings of all photos of the same person?

This question is the first I've heard of FaceNet, but I don't think that the right solution to the question is to aggregate the face embeddings but to ask why you're using an SVM to classify the embeddings. Importantly, many SVM implementations of multiclass classification use a one-vs-rest method to train the classifiers -- if you're using a one-vs-rest implementation with 3000 labels, I suspect that this is the reason your training is taking so long.

You should look into how your implementation is training the classifier. Additionally, How large is your embedding size?


What variance threshold to consider in feature selection?

Usually 3 is taken as a threshold Value by standard deviation

Using z-score we can look for value above or below threshold like in code below:

threshold=3
mean = np.mean(data)
std =np.std(data)

for x in data:
    z_score= (x - mean)/std
    if np.abs(z_score) > threshold:
        ...


What would I prefer - an over-fitted model or a less accurate model?

There are a couple of nuances here.

Complexity question very important - ocams razor
CV - is this trully the case 84%/83% (test it for train+test with CV)

Given this, personal opinion: Second one.

Better to catch general patterns. You already know that first model failed on that because of the train and test difference. 1% says nothing.


Choosing between prepadding and postpadding for variable length sequence data in a CNN

I don't think that it makes a difference. As long as (1) your sequence is long enough to handle a reasonable amount of your sentence/paragraph/whatever and (2) you are consistent in they way you pad between your training cycles and your prediction cycles, then which padding direction you use should be a moot point.


How relevant is Self Organizing Maps in today's science?

One important and well known (in their respective research domains) usage of SOMs today is the usage of SOMs for data visualization and clustering.

The U-Matrix uses a SOM with a lot of neurons in order to achieve emergent behavior. Based upon the SOM, a so called U-height is calculated for each neuron.

For a neuron $n$, a weight $w_n$ of the neuron $n$, as well as a set $NN(n)$ of immediate neighbours of $n$ the U-height is calculated as:

$$\sum_{m \in NN(n)} d(w_n - w_m)$$

$d$ is the same distance as used in the SOM training. This U-height is used to visualize the SOM and can be capable of catching quite interesting
inherent structure. It is visualized by using a color scheme similar to topographical maps:

The U-Matrix uses distances for visualization. It can be combined with the so called P-Matrix, which uses densities, to use the density of data points as well, and allows the combination of densities and distances. Based on the U-Matrix one can do classification or clustering as well. An R package is distributed by the research group of Prof. Ultsch.

I wouldn't say that it's important in Machine Learning per se, but in the fields of Data Mining and KDD it is, at least academically. You can clearly see this if you look on the reception of this or similar work on semanticscholar or researchgate.

Disclaimer: I studied with Prof. Ultsch, the head behind U-Matrix and U*-Matrix.

NB: Sorry for the missing links to the papers, I haven't got enough reputation yet to post more than two links.

Papers:

U*-matrix: a Tool to Visualize Clusters in High Dimensional Data, Ultsch 2004
Self-Organizing Neural Networks for Visualisation and Classification, Ultsch 1993

I even spotted the U-Matrix in the wild here. Also see the sompy package.


Gradient descent multidimensional linear regression - does learning rate affects concurrency?

If you use a convex loss function you always have one optimum point, and you will always be able to find it.

  I have done some calculations and I get correct results but not for all learning rates I tried to specify.

The reason is that you may have selected a large learning rate which may lead to overshoot the optimum. It also may be so small that you don't reach to the desired point and it all depends on your data. If you use the simple gradient descent algorithm, it is an incident that can be seen a lot. You can use other optimisation techniques like Adam which is much more better and you rarely see these problems.


how to avoid tokenizing w/ sklearn feature extraction

You can change the tokenizer instead of the token pattern. The token pattern is used by the tokenizer. You can set as tokenizer any function that returns a list of string. For example, str.split

tokenizer=str.split

From the official documentation,

  tokenizer callable, default=None

  Override the string tokenization step while preserving the preprocessing and n-grams generation steps. Only applies if analyzer == 'word'.


Any good Implementations of Bi-LSTM bahdanau attention in Keras?

Here's the Deeplearning.ai notebook that is going to be helpful to understand it. Neural machine translation with attention


Assign more importance to recent records during training

One option is incremental training. Update the model weights as more recent data is available. This automatically assigns more importance to recent records.
Incremental training works well within the Bayesian framework where historical data is the prior and more recent data updates the prior.


How to make an MNIST classifier work with blank images?

Try thresholding on the image. I believe, you will get ~95% of what is required.
Then try other classical image processing techniques depending on the issue.
import cv2 as cv
import numpy as np
from matplotlib import pyplot as plt
img = cv.imread('/content/sample_data/issue_image.png',0)
ret,thresh1 = cv.threshold(img,127,255,cv.THRESH_BINARY)

_, ax = plt.subplots(1,2,figsize=(12,6))
ax[1].imshow(thresh1,'gray')
ax[0].imshow(img)

Ref - OpenCV py_thresholding


Factors of variation are some factors which determine varieties in observed data. If that factors change, the behavior of the data will change.

The bold phrase said these factors are usually independent and one factor does not change by changing the value of the others.


Downloading a large dataset on the web directly into AWS S3

Since you obviously posses an AWS account I'd recommend the following:

Create an EC2 instance (any size)
Use wget(or curl) to fetch the file(s) to that EC2 instance. For example: wget http://example.com/my_large_file.csv.
Install s3cmd
Use s3cmd to upload the file to S3. For example: s3cmd cp my_large_file.csv s3://my.bucket/my_large_file.csv

Since connections made between various AWS services leverage AWS's internal network, uploading from an EC2 instance to S3 is pretty fast. Much faster than uploading it from your own computer. This way allows you to avoid downloading the file to your computer and saving potentially significant time uploading it through the web interface.


How are the Convolution kernels learned?

Sometimes the kernel function is very difficult to choose in a fixed form even if once found it would be easier. It can be learned to map the data in a features space (Hilbert space) through an Convex Optimisation method such as Semidefinite programming, using Hyperkernels, etc.

Some very usefull papers for your question can be found here:

Learning the Kernel via Convex Optimisation

Learning the Kernel Matrix with Semidefinite Programming

Learning the Kernel with Hyperkernels

Learning the Kernel Function via Regularization

I hope that answers your question


Ratio between embedded vector dimensions and vocabulary size

The ratio of vocabulary vs embedding length to determine the size of other layers in a neural network doesn't really matter. Word embeddings are always around 100 and 300 in length, longer embedding vectors don't add enough information and smaller ones don't represent the semantics well enough. What matters more is the network architecture, the algorithm(s) and the dataset size.

A simple way to understand this concept is that a bidirectional LSTM model with 50 neurons (nodes) followed by a fully connected layer of 70 neurons will outperform a simple MLP of 1000 neurons (nodes) connected to a embedding layer simply due to its architecture. Adding dropout will improve performance as well.

In addition, even if the vocabulary is just 300 words, using pre-trained embeddings will probably yield better results than training the embeddings directly on the dataset. The same applies to data size, a dataset with more samples will make a better classifier than a dataset with just a couple thousand samples.

In summary, it is preferable to try many architectures and cross-validate them (and/or ensemble them depending if you have a large enough dataset) with the smallest number of neurons possible and then start building up in size, depending on what computational resources you have and the speed of development you need. Large models slow down development speed whereas small models speed it up. This goes whether your vocabulary is the size of common crawl or just 300. As usual, try feature engineering (sentence length, special characters, etc.) and increase the dataset size as doing so often helps in whatever task you're trying to predict.


Does matrix shape for training/testing sets have to be in a particular order?

In practice

Andrew Ng seems to be using the convention on the Theano framework. If you had 10 colour images, each 100 pixels high and 200 pixels wide, Theano models would expect input of the form:

  (batch size, input channels, input rows, input columns)

and so dimensions: (10, 3, 100, 200). The three is because of the three RGb dimensions in a colour image.

Tensorflow on the other hand reverse this order to instead use:
(num_obs, height, width, channels) - for the same example as above this becomes:

  (batch size, input rows, input columns, input channels)

meaning the dimensions of the input should be (10, 100, 200, 3).

Keras works with Tensorflow and Theano and supports both conventions by simply allowing the user to set the position in which the number of channels are given. This can be set in a config file or using a specific environment variable upon setup. Have a look at the relevant documentation.

(Possible) justifications

There are different points of view on what is more natural. If you come from a computer vision arena (or image processing in general), then libraries like OpenCV use a coordinate system that has (0,0) in the top left of an image, and you specify a single pixel by given the vertical movement, then the horizontal movement from the origin. This means you give a height, then a width. One justification for this might be that many computer vision algorithms (e.g. colour filters) focus on difference across the channels of an image, but it is just convention.

In general graphing and mathematics, it is common to give the X coordinates first, then the Y, which means talking about horizontal movements, then vertical. Linear algebra in general is a good example of this.

All this being said, the best advice is to make sure you use the correct dimensions according to the documentation of the software you use. If you make your own software - you get to choose! (But make sure to document it!)


use hidden layer of word2vec instead of 'one-hot', to reduce number of weights in other nets?

Are you asking if the 300 dimension 'middle layer' has learned to encode words into an embedding like word2vec?

If this is the case, I'd say no. There's nothing in the one-hot vector that specifies distance between two words. For instance, the one-hot vectors for cat and table will be the same distance away for cat and lion. With embeddings, clearly you want to have the second example closer together.


Q learning - how to use experience replay, when playing against other agent?

I am pretty sure you have to:

wait for enough data in the memory (100 entries, etc), then take out a minibatch (for example, 10 random elements), run backprop 10 times - once for each selected element.

You then take an average of resulted gradients and finally, correct your network with that averaged gradient.

After,

put the elements back in the memory (where you've taken them from)

  Samples that have high priority are likely to be used in training many times. Reducing the weights on these oft-seen samples basically tells the network, "train on these samples, but without much emphasis; they'll be seen again soon."

or throw it out, substituting it with new memories instead.

Then keep playing the game, adding say, another 30 examples before doing another weights-correction session using a batch of 10 elements

I refer to the "Session" meaning a sequence of backprops, where the result is the average gradient used to finally correct the network.

  EDIT: Another question I have in terms of training a neural network against a neural network, is that do you train it against a completely separate network that trains itself, or do you train it against a previous version of itself. And when training it against the other neural network, do you turn the epsilon greedy down to make the opposing neural network not use any random moves.

Consider using just one network.  Let's say our memory bank contains several elements:

...
{...}
{stateFrom, takenActionIndex, takenActionQval, immediateReward, stateNext }  <-- a single element
{...}
{...}
{...}
...

When using each element in your memory during the Correction session (one element after the other), you need to:

pick an element. As seen, it contains stateFrom, taken action (action X), qvalue of action X, reward you received, state that it led to.
Run forward prop as if you were in "next state" (mentioned in that element).  Get its best [action Y, qvalue]. In fact, your action Y doesn't have to be the action with highest q-value. Instead, it could be the "epsilon-greedy decided action" from the "next state" - you would then have SARSA instead of Q-learning
obtain delta by which the "qvalue of action X" differs from the immediate reward + "qvalue of action Y".
get deltas of several other elements that you've chosen from the bank
Using each delta, compute a gradient tensor for your network's weights. Average those gradients into a single tensor. Punish your network's weights with this tensor.

I've intentionally circumvented 1 thing - you don't actually store takenActionQval, because they might become obsolete by the time you fetch its element from MemoryBank. You have to re-compute these scores during backprop.

Therefore, you are training against the version of your network from the similar but previous correction session.

Notice, you don't store ["next state", action Y] because by the time you select it to train (maybe you didn't select it for several minibatches), the network might have a different q-value for that action Y.

You could also copy your network to the second network (target network), but only, say, every 200 timesteps. You would still proceed to punish your network for any differences with the target network in the meantime, after every 30 timesteps.

Notice, the intuition of why this works is because: the Q-values are sort-of "flowing" from finish back to the beginning, by a little with every new journey. And you are always updating the current state towards the successor state. And train the successor state against its own (even further) successor.


Why are positional embeddings learned?
This was asked in the repo of the original implementation without an answer. It didn't get an answer either in the HuggingFace Transformers repo and in cross-validated, also without answer, or without much evidence.
Given that in the original Transformer paper the sinusoidal embedding were the default ones, I understand that during preliminary hyperparameter tuning, the authors of BERT decided to go with learned embeddings, deciding not to duplicate all experiments with both types of embeddings.
We can, nevertheless, see some comparisons between learned and sinusoidal positional embedding in the ICLR'21 article On Position Embeddings in BERT, where the authors observe that:

The fully-learnable absolute PE performs better in classification,
while relative PEs perform better in span prediction.

How does the model handle multiple sentence segments?
This is best understood with the figure of the original BERT paper:

The two sentences are encoded into three sequences of the same length:

Sequence of subword tokens: the sentence tokens are concatenated into a single sequence, separating them with a [SEP] token. This sequence is embedded with the subword token embedding table; you can see the tokens here.
Sequence of positional embedding: sequentially increasing positions form the initial position of the [CLS] token to the position of the second [SEP] token. This sequence is embedded with the positional embedding table, which has 512 elements.
Sequence of segment embeddings:  as many EA tokens as the token length of the first sentence (with [CLS] and [SEP]) followed by as many EB tokens as the token length of the second sentence (with the [SEP]). This sequence is embedded with the segment embedding table, with has 2 elements.

After embedding the three sequences with their respective embedding tables, we have 3 vector sequences, which are added together and used as input to the self-attention layers.
Are the weights in these embedding layers adjusted when fine-tuning the model to a downstream task?
Yes, they are. Normally, all parameters are fine-tuned when fine-tunine a BERT-based model.
Nevertheless, it is also possible to simply use BERT's representations as input to a classification model, without fine-tuning BERT at all.
In this article you can see how these two approaches compare. In general, for BERT, you obtain better results by fine-tuning the whole model.


cv2.error: OpenCV(3.4.3) (-215:Assertion failed) !empty() in function

I think this may be due to the fact that XML file is missing or the path to it is incorrect. Could you please check that? Maybe you coul try with the full path for the haarcascade_frontalface_alt.xml file.


Confusion about Keras' skipgram and sampling table utilities

After looking through the code for the skipgram method, I figured it out: it has to do with the sampling_factor parameter. It defaults to 1e-5, which means that for such a small sampling factor it's almost never going to actually pick a sample unless your dataset & vocabulary size is big enough to actually get some samples. If you have a small vocabulary (like in the toy example), you should set that to something much higher: when I set it to 1.0 (i.e. take a sample as often as possible) I got lots of pairs, even for the toy example.


I have a csv file with time as the datetime format and want to edit the same column into miliseconds, how do I change that?

Familiarize yourself with the pandas library, which allows you to do this:

from pandas import read_csv
from io import StringIO
csv_file = StringIO("""time event earthAcceleration.x earthAcceleration.y earthAcceleration.z
2017-11-22T09:20:13.944 motion -0.006379993 -0.001029071 -0.010781396
2017-11-22T09:20:13.954 motion 0.014739769 0.008146121 -0.001068155
2017-11-22T09:20:13.964 motion 0.008520941 -0.008222823 0.022573506
2017-11-22T09:20:13.974 motion 0.000573836 0.016701656 -0.016170829
2017-11-22T09:20:13.984 motion 0.016282924 0.003181049 0.006969224""")
df = read_csv(csv_file, sep=' ', parse_dates=['time'])
df.assign(delta_t = df.time-df.time.ix[0])

I believe your output is wrong; the difference between each line is exactly 0.01s. Welcome to the site!


Is there a paper and a model that uses positional encodings and outcompetes RNN/LSTM based models for small scale datasets (MBs of text data, not terabytes)?

Yes, there are several. Similar to GPT, they still pre-train on terabytes of data. But the embedding they learn generalize well. Then you can fine-tune on a much smaller dataset. It works much in the same way as transfer learning on a CNN where a model first is trained on ImageNet and then trained on a specific task. It tends to give better results than RNN/LSTMs.

If there are many, which ones are the leading ones in production applications?

The one that sees most use is definitely BERT. Here is a really nice explanation of how it works. This transformers library from Huggingface makes it really easy to work with BERT and other transformers that have already been pre-trained.


What does Theano dimension ordering mean?

Let's say you're working with 128x128 pixel RGB images (that's 128x128 pixels with 3 color channels).

When you put such an image into a numpy array you can either store it with a shape of (128, 128, 3) or with a shape of (3, 128, 128).

The dimension ordering specifies if the color channel comes first (as with theano / "th") or if it comes last (as with tensorflow / "tf").

The code you've posted contains the following line:

inputs = Input((1, img_rows, img_cols))

It specifies the shape of the input as (1, img_rows, img_cols) - i.e. there's one color channel (gray scale) and it comes first. That's why it requires Theano's dimension ordering "th".


How to properly apply CrossValidation and/or split the dataset?

I'm not sure that I understand every part of the process but there is one clear issue with it: because the CV is applied in the inner loop, there is a serious risk of overfitting the model with respect to the other parameters (feature subset, model type, sampling technique). Depending on the goal, this is not necessarily wrong but it's important to interpret the results accordingly. For instance in this case the results will show how different subsets of features perform on the data, but this difference in performance across subsets shouldn't be considered reliable: it's possible that a particular subset happens to be better than another by chance.

It's quite a complex setting and I assume that there are efficiency constraints to take into account. If possible the most reliable results would be obtained by doing several stages of CV (or other techniques, e.g. bagging) using different subsets of data. For instance you could run the whole process a few times, each time using a different random subset of instances: in this way you can average performance and see whether a particular subset of features is constantly better than another (for example, same idea for other parameters).

[edited] Disclaimer: I don't know if there is any standard way to proceed with a complex multi-level setting like this, my advice is based only on the experience I had with a few broadly similar cases.

Generally the idea is that every choice to make can be considered as an hyper-parameter, including the subset of features, the type of model, the architecture, the sampling technique. Therefore I think that ideally one would cross-validate at every level, i.e. put every loop level in a function and call this function k times with a different subset of data. It would look like something like this:

train1, val1 = splitDataset()
iterate each featureSubset:
    train2, val2 = splitData(train1)
    resultTrain2 = kfoldCV(processLevel2, train2)
    resultLevel2 = apply(resultTrain2, val2)
resultLevel1 = apply(resultLevel2, val1)

processLevel2:
    iterate each modelType:
        train3, val3 = splitData(train2)
        resultTrain3 = kfoldCV(processLevel3, train3)
        ...

remark: I'm not 100% sure about the algorithm, maybe I over-complicated it. I think it gives the general idea though.

But of course following this logic the computational complexity becomes way too high, so you will probably have to take a few shortcuts. One thing I've tried successfully in the past is to use genetic learning in order to optimize different parameters at the same time: that would mean having different "genes" which represent the different parameters (feature subset, model type, etc.), each with its set of values, and then run the genetic process which is supposed to converge to a set of optimal values for all the parameters (I was using CV every time a particular combination of parameters is evaluated). But again I don't know if it's a very orthodox method :)

[edit2]

After more thought I think I would try to do something like this:

innerData, valOuter = splitDataset() // keep large amount for validation set, say between 20-40%
train, valInner = splitDataset(innerData)
iterate each featureSubset:
    iterate each modelType:
        model = manuallySearchGoodArchitecture(featureSubset, modelType, train, val)

        iterate each samplingTechnique:
            train_temporary = applySampling(samplingTechnique)
            HPs = HPOptimization(model, train_temporary, valInner)
            result =  k-FOldCV(model, HPs, samplingTechnique, train)
        end
    end
end

bestHPCombinations = select top N HPCombinations from first stage
for each HPCombination in bestHPCombinations
     model = train(innerData)
     result = apply(model, valOuter)
end

It's simpler: the idea is just to re-evaluate the results from the first stage on some fresh data in order to avoid overfiting (here HPCombination includes featuresSubset, modelType, etc.). The second stage could also include CV or bagging for more reliable results. But in general I think this option would be reasonably reliable, since it's unlikely that the best models from the first stage would also be the best models in the second stage by chance only.


How do I build an image dataset for CNN?

This is a very packed question. Let's try to go through it and I will try to provide some example for image processing using a CNN.

Pre-processing the data

Pre-processing the data such as resizing, and grey scale is the first step of your machine learning pipeline. Most deep learning frameworks will require your training data to all have the same shape. So it is best to resize your images to some standard.

Whenever training any kind of machine learning model it is important to remember the bias variance trade-off. The more complex the model the harder it will be to train it. That means it is best to limit the number of model parameters in your model. You can lower the number of inputs to your model by downsampling the images. Greyscaling is often used for the same reason. If the colors in the images do not contain any distinguishing information then you can reduce the number of inputs by a third by greyscaling.

There are a number of other pre-processing methods which can be used depending on your data. It is also a good idea to do some data augmentation, this is altering your input data slightly without changing the resulting label to increase the number of instances you have to train your model.

How to structure the data?

The shape of the variable which you will use as the input for your CNN will depend on the package you choose. I prefer using tensorflow, which is developed by Google. If you are planning on using a pretty standard architecture, then there is a very useful wrapper library named Keras which will help make designing and training a CNN very easy.

When using tensorflow you will want to get your set of images into a numpy matrix. The first dimension is your instances, then your image dimensions and finally the last dimension is for channels.

So for example if you are using MNIST data as shown below, then you are working with greyscale images which each have dimensions 28 by 28. Then the numpy matrix shape that you would feed into your deep learning model would be (n, 28, 28, 1), where $n$ is the number of images you have in your dataset.

How to label images?

For most data the labeling would need to be done manually. This is often named data collection and is the hardest and most expensive part of any machine learning solution. It is often best to either use readily available data, or to use less complex models and more pre-processing if the data is just unavailable.

Here is an example of the use of a CNN for the MNIST dataset

First we load the data

from keras.datasets import mnist
import numpy as np

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.

print('Training data shape: ', x_train.shape)
print('Testing data shape : ', x_test.shape)

  Training data shape:  (60000, 28, 28) Testing data shape :  (10000,
  28, 28)

from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

Then we need to reshape our data to add the channel dimension at the end of our numpy matrix. Furthermore, we will one-hot encode the labels. So you will have 10 output neurons, where each represent a different class.

# The known number of output classes.
num_classes = 10

# Input image dimensions
img_rows, img_cols = 28, 28

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
x_test_reshaped = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

Now we design our model

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

Finally we can train the model

epochs = 4
batch_size = 128
# Fit the model weights.
model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test_reshaped, y_test_binary))


Is it good practice to use SMOTE when you have a data set that has imbalanced classes when using BERT model for text classification?

I don't know about any specific recommendation related to BERT, but my general advice is this:

Do not to systematically use oversampling when the data is imbalanced, at least not before specifically identifying performance issues caused by the imbalance. I see many questions here on DataScienceSE about solving problems which are caused by using oversampling blindly (and often wrongly).
In general resampling doesn't work well with text data, because language diversity cannot be simulated this way. There is a high risk to obtain an overfit model.


How to combine two differently scaled, but equally important "running" signals into a reward function?

Using z-normalisation ensures that they have same mean and Standard Deviation, but ofcourse values will be still different since mean and Standard Deviation depend on the Distribution of the data.
Alternative is to use feature scaling, where you force the values between 0 and 1 for both signals.


Removing constant from the regression model

When you estimate a linear model without constant, you essentially "force" the estimated function to go through the $(0,0)$ coordinates.
With an intercept, you estimate a linear function like:
$$ y = \beta_0 + \beta_1 x .$$
Without intercept, you estimate a linear function like:
$$ y = 0 + \beta_1 x .$$
So when $x=0$, $y$ will be $0$ as well.
You should not only look at $R^2$ since $R^2$ often will go up when you have no intercept. Think about the structure of your model, how the data look like, and what you want to achieve.
Example in R:
library(ISLR)
auto = ISLR::Auto

ols1 = lm(mpg~horsepower,data=auto)
summary(ols1)
plot(auto$horsepower, auto$mpg)
lines(auto$horsepower, predict(ols1, newdata=auto), type="l", col="red")

ols2 = lm(mpg~horsepower+0,data=auto)
summary(ols2)
plot(auto$horsepower, auto$mpg)
lines(auto$horsepower, predict(ols2, newdata=auto), type="l", col="red")

Results:
Model with intercept:
Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept) 39.935861   0.717499   55.66   <2e-16 ***
horsepower  -0.157845   0.006446  -24.49   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.906 on 390 degrees of freedom
Multiple R-squared:  0.6059,    Adjusted R-squared:  0.6049
F-statistic: 599.7 on 1 and 390 DF,  p-value: < 2.2e-16

Model without intercept:
Coefficients:
           Estimate Std. Error t value Pr(>|t|)
horsepower 0.178840   0.006648    26.9   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 14.65 on 391 degrees of freedom
Multiple R-squared:  0.6492,    Adjusted R-squared:  0.6483
F-statistic: 723.7 on 1 and 391 DF,  p-value: < 2.2e-16

Summary:
In this example, excluding the intercept improved the $R^2$ but by forcing the (estimated) function to go through $(0,0)$, the model results are entirely different. In essence, the model without intercept produces bullshit in this case. So be very careful to exclude the intercept term.


ModuleNotFoundError: No module named 'deeplab'

You need to start you program from the models/research repository (cd ./models/research before executing)


Cluster elements that appear in the same lists

That is a data mining problem, specifically affinity analysis.
One common method to solve it is the Apriori algorithm.


Understanding softmax layer in Deep Neural Network

Look at the definition of softmax: https://en.wikipedia.org/wiki/Softmax_function

The exp() gives a positive value. Since you "standardize" each value exp(e^z) in the numerator by the sum of all exp(e^z), the values sum to one in total.

Is whis what you are looking for?

Maybe you should have a look at Hasties et al. "Elements of Statistical Learning" (Ch. 11.2 "Neural Nets").

Little R example:

# For standardization (denominator)
base = exp(-2) + exp(5)

# Single exp() values are positive
exp(-2)
[1] 0.1353353
exp(5)
[1] 148.4132

# Standatdize
exp(-2)/base
[1] 0.0009110512
exp(5)/base
[1] 0.9990889

# Total sum equals 1
exp(-2)/base + exp(5)/base
[1] 1


For prediction step, are you using the same batch size as training ?  batch prediction should bring down memory usage.


Is machine learning successful in solving combinatorial optimisation problems in NP-hard? Discuss problem of scheduling using machine learning

You don't have to get to EXP-complete in order to get a hard problem.
NP-Complete is bad enough...

Cryptographic assumptions (e.g., the existence of one way functions) are also a good way to create hard problems

There is research in the area lead of plenty of results like Cryptographic Limitations on Learning Boolean Formulae and Finite Automata - DFA is not PAC learnable in polynomial time.

However, there is an important difference between these negative results and writing a great chess program.
The negative results means that for a given problem no algorithm will solve ALL instances (let say, in polynomial time).
That doesn't mean that you won't be able to play better the human champion on some/most/all instances.

Note that this complexity observation is indifference of the technique. Machine learning is just another (useful) way to cope with these problems.

As for machine learning ability to cope with these problems, you can find NP-Complete problems all around in machine learning (SVM, Neural Networks, Decision trees). That didn't prevent them from finding many useful models.


Encoding for k-level qualitatative variable

There is always clustering techniques you could do that would help you determine how to group userId into a dummy variable.


Should I accept a model with good validation loss & accuracy but bad training one?

The validation set having just 12 samples is too small to consider deploying a model which performs well on the validation set.
A model that performs well on the training set with 110 samples and then performs better on the small validation set would be preferable.
Since your set is too small, all though it is not prudent, you should consider running the model on the entire 120 samples to see how it performs on the entire dataset.
I presume you can't get more samples, if you can, that would be a good step.


This problem is related to the following standard problems:

topic classification/modeling, which ranges from simple supervised document classification to unsupervised assignment of topics distributions to every document (the advanced option, with Latent Dirichlet Analysis and variants)
sequence labeling, a supervised task which predicts classes for every instance in a sequence, taking into account the order of the instances (e.g. it can leverage the fact that the class of instance $n$ is influenced by the class of instance $n-1$).
text segmentation, more precisely topic segmentation in this case.

I think that one can find a lot of good implementations for the first two problems, which are very common. However adapting these to your case and/or combining them will probably be more complex.

If you have a small set of topics and you have (or can have) a reasonable sample of  messages annotated with these topics, I would suggest starting with sequence labeling with Conditional Random Fields. I think this could give good results given the sequential nature of chat conversations.


Keras custom layer using tensorflow function

you can overwrite tensorflow classes and add you function as layer as colleague mentioned but Keras is just high level API to tensorflow and that layer can be called from keras.


How to visualize (make plot) of regression output against categorical input variable?

One possible first step is to convert the data back to the original coding.
This is called in SQL unpivot, in R melt.

Here an R example

> my.df <- read.table(
+ text = "DistrictA     DistrictB    DistrictC    DistrictD     DistrictE     Price
+         1             0            0            0             0             10000
+         0             1            0            0             0             20000
+         0             0            1            0             0             30000
+         0             0            0            1             0             40000
+         0             0            0            0             1             50000"
+      , header = TRUE)
> my.df
  DistrictA DistrictB DistrictC DistrictD DistrictE Price
1         1         0         0         0         0 10000
2         0         1         0         0         0 20000
3         0         0         1         0         0 30000
4         0         0         0         1         0 40000
5         0         0         0         0         1 50000

> library(reshape)
> subset(melt(my.df, id="Price", variable = "District"),value == 1)[,c(1,2)]
   Price  District
1  10000 DistrictA
7  20000 DistrictB
13 30000 DistrictC
19 40000 DistrictD
25 50000 DistrictE

After that you plot the Price dependent on a factor variable. You may additionally consider to order the factor based on the predicted price.

I provide no details, as you don't tagged your tool, but I would recommend additional to a scatter plot to consider a box plot and/or density plot - always combined with the prediction value from the model for each factor level.


How much time do scikit classifiers take to classify?

I don't see a huge problem here. So, I would try to answer all of your questions from the production-level point of view:

  How do I confirm that classification will not take much time?

Take a subset of the corpus data you have (you can randomly do it, no need of sampling), and test your algorithm on it, and they approximate/generalize it to the overall dataset.

(SVM is comparatively faster. Nevertheless, do the above process just to be sure.)

And do test it in the development environment before pushing to production.

  Is using Python's scikit library for websites/web applications
  suitable for this?

Yes, it is. It is already being used by a nice chunk of companies out there.

The third question about Amazon and Flipkart cannot be answered by someone outside their teams.

In addition, I would advise you to use the mapreduce techniques for training your models. And as already advised, pickle your models so that you don't need to train them with every request.


Gradient Boosting RandomSearchCV or GridSearchCV

I think it depends on the size of your multi dimensional grid. If it is small, then you can afford to be exhaustive and do a grid search. But if it is very large, and your computation time for a grid search extends too much, then definitely go to a random search. In fact, with random search one can explore larger regions than with grid search, and that is an advantage.

In any case, for hyperparameters search there are two keys:

Monitor it while it is running, so that you can stop it when you are happy with the results. This is specially important for random search.
Split your data in three, train, cross validation and test. Evaluate the hyperparameter search in the cv set. Once finished, rank them by their performance there, and then take the best point and re-evaluate in test. It may happen that you do what is called "overfitting to cv set", in which case the performance in test set will deteriorate significantly vs the performance in cv set. If that happens, try with the second best, third best etc until the performance in test set is good. But take into account that the more points you take from the ranking, the higher the chance of "overfitting to test set".

Once you find the best regions, you can do either perform a more constrained there with either grid search or random search again.

Another option, which tends to work very well, is bayesian optimisation. Here the library that you use is important. In Python, after trying several which gave different problems, the best I found was skopt: https://scikit-optimize.github.io/


Dataframe has no column names. How to add a header?

For any dataframe , say df , you can add/modify column names by passing the column names in a list to the df.columns method:
For example, if you want the column names to be 'A', 'B', 'C', 'D'],use this
df.columns = ['A', 'B', 'C', 'D’]
In your code , can you remove header=0? This basically tells pandas to take the first row as the column headers . Once you remove that , use the above to assign the column names


Best approach for this unsupervised clustering problem with categorical data?

K-means is a reasonable approach and a sensible way to understand the data.
I've never used mahout, but I would use R or Python for this sort of analysis because of the nice libraries available to quickly implement K-means.
The clustering approach with the tags is fairly straightforward. You can essentially encode this using an indicator variable (also known as a binary encoding). You can set this variable/feature to 1 if the tag appeared in the list of tags and 0 otherwise. Then you only need to allocate space for the total number of tags that exist. If you have a large set of tags, you can limit them by taking tags with at least some frequency or some other "sensible" way.
You can choose $k$ in a number of ways. Typically people choose K arbitrarily because they want, e.g., 10 groups to segment their customers or data into. In the simulation I've provided, it'll give you a lame way to optimize for K using an incremental improvement.

I've made a notebook with a simulation walking you through how to "tokenize" your tags and represent them with a binary/one-hot encoding. It's worth noting that this tokenization ignores the order of the tags, which may be okay for your use case.

It's also worth noting that K-means certainly isn't the only way to measure the similarity of your data but I think it's a nice intuitive start.

Again, for choosing $k$ the approach outlined in the notebook is exhaustive, since it starts from 1 and goes until each observation is a cluster. This means you'll have to run K-means $n$ times, which is silly in practice to do but useful from a learning perspective here. In general, this isn't ideal because it's expensive and typically you don't want to set $k = n$ but this simulation gives nice intuition about what's happening.

In practice, you can just do it in gaps for a large number of clusters (e.g., 5, 10, 15, 20, .., 100) or something like that and choose the one that has the biggest drop-off by eye-balling it. This is a very arbitrary and unsatisfying way to choose $k$, but it seems to work okay for many people.


Prediction with multiple features (attributes)

There is nothing "wrong" about weighting different actions to get a measure of preference between a user and an item, and then using ALS to compute a latent feature space. This is quite common in my experience. However, the consequence of ALS is that it can detect categories and sub-categories using individual user/item interactions. So, I don't see why you need to be adding higher level preference data into your user/item matrix. The weighting should be done using only behavior that indicates preference between a specific user and specific item, not user/category or user/sub-category interactions - ALS will detect if a user likes a category on its own.

How to weight the preference behavior is somewhat tricky, but you can use heuristics or information from your data. For example, a page view is not as strong as a product purchase and should be weighted less. You might look at the distribution of page views vs purchases to get an idea how common one is vs the other, to develop a weighting scheme.

I am sure that you could also use an RNN in a classification setting, to predict the next item that a user will purchase given an input sequence of the last n items they've purchased or interacted with. The framework you use depends on your preference, for Python based Keras, MXNet, PyTorch, and Tensorflow are all worthy. For easy Spark integration and JVM-based, DL4J is a good choice.


Hindsight Experience Replay, how to define a partially-known End-Goal

I've got an answer from the Author:

  The simplest solution would be to have 2 types of goals - a) specific hp and position; and b) specific hp, arbitrary position. I haven't tried anything like that though.

Based on the response I reason it might work if we do the following:
use a placeholder value, perhaps for the last component of the input vector.
This way the DQN will notice that when this "last bit" is on, only health matters.

<hp,  x1,x2,x3,  q1,q2,q3,  chestID, handsID, headID, feetID,   placeholder0or1>

Edit: if you know a better way, please share it as an answer!


How to save a Numpy array output of an autoencoder as an image

That is right. The size you see is the Frame Size of the image i.e. height and width. It does not refer to the dimensionality of a color image array. To see it try this

rgb = np.zeros((255, 255, 3), dtype=np.uint8)
img = Image.fromarray(rgb, 'RGB')
r = img.getchannel("R")
g = img.getchannel("G")
b = img.getchannel("B")
print(np.array(r.getdata()))
print(np.array(g.getdata()))
print(np.array(b.getdata()))

where output is

[  0   1   2 ... 252 253 254]
[55 55 55 ... 55 55 55]
[  1   0 255 ...   5   4   3]

So you have 3 dimensions (or colors). And the point about noise is in dtype=np.uint8. Convert your array to this and it will work.

You could also simply try

from scipy.misc import imsave
imsave("file_name.jpg", SP)

It does the job.

Good Luck!


How to save google colab notebook outputs and variables

Save your content in the current directory and execute !dir to see the content. Then follow the following code snippet:

from google.colab import files
files.download('your typical text file or what ever.txt')

For more information take a look at here.


Comparison of performance of autoencoder with PCA

The features in your data are probably linear as an Autoencoder should give the same results then (see here and here). The surplus of an AE is that it is efficient to detect nonlinear features which a PCA cannot.


Named Entity Recognition (NER) models should be able to identify money amounts.

Other NLP techniques such as Dependency Parsing or Constituency Parsing can be used to identify the subject of the sentence - or the person the amount is referred to.

For the months amount, I think once you have other informations that's something you could extract with a "hard coded" script.


How would you interpret the following ROC and PRC curves?

My interpretation is that the data can be described as the following three groups, by decreasing order of the prediction to be positive:

a large number of positive instances which can "easily" be predicted as positive, with no false positive
then a large number of negative instances
then a smaller number of positive instances, which are predicted as less likely to be positive than the negative instances.

For instance it might look like this:
<predicted probability> <true class>
       1                     P
      0.9                    P
      ...                   ...
      0.4                    P

      0.38                   N
      ...                   ...
      0.14                   N

      0.11                   P
      ...                   ...
      0.05                   P

This kind of configuration would explain the three stages which are clearly visible. There might be other ways this would happen, but I can't think of anything very different than this explanation (I could be wrong about that).
Note that this is why both precision and recall increase with this third group: since the threshold has already passed all the true negative instances (i.e. all the possible FP errors have been made), adding these true positive cases increases recall (more TP cases) without decreasing precision (no more FP). It's indeed very unusual, but not impossible.
What is interesting here is to investigate why this third group of positive instances are totally misclassified: they're not only confused with the negative (the common case), they are predicted as "more negative" than the true negative instances. This is very strange: it's as if there are two very distinct groups of positive instances that the classifier cannot predict on the same side of the positive/negative continuum. But it's encouraging that the classifier can quite well distinguish the three groups from each other, it means that it's probably something which can be fixed.


Deep Q-learning, how to set q-value of non-selected actions?

Consider these two depictions of a simple Deep Q Learning Algorithm:

Look at the gradient update steps in both images: we are only taking the gradient with respect to the Q value of the one action, $a_i$. The network outputs $N$ nodes, but since we only care about one of them, you can probably see that all of the computations still work out, for the ground truths of the other output nodes are never used in this gradient update step.
I hope this helps.
Edit: looking at you first solution you listed, this is correct, but i feel you aren’t understanding the actual computation of the gradient. If you work it out, the ground truth values of the other nodes aren’t actually used because the loss function is only written in terms of the $i$th node.


Determining extreme outliers from boxplot (by eye)

These images aren't very useful to make decisions by eye, because the scale is far too compressed. In the second and third images the mean, entire box and both whiskers are all represented by a single vertical line, making it impossible to tell if those values are distributed over a range of 0.01, or 0.00001 or 0.000000001. In the third image, the circle looks overlaid directly onto the vertical line, so any attempt to guess how far outside the interquartile range that point is would be just that, a guess. That point in the circle could be just barely outside the whisker, or a million times greater than the mean value, we simply cannot tell from that plot. All we know from the circles is that those point are outside of the whiskers (typically the mean + 1.5 * IQR), but with no meaningful way to discern mean or IQR from the plots, that's not very valuable information.
When making a visual assessment, it's important to put your data on a scale that can actually be visualized. You wouldn't try to visually estimate the distance from your house to the pub on a map of the universe - you'd need a map that can discern distances on the order of miles, not millions of miles.  There's not really any reason to plot the range from 0 to 1 when all your points fall in the range of 0 to 0.01, and it will compress potentially meaningful differences to the point where there is no difference whatsoever on a visual scale. You could also try log-transforming your data, which will compress the high end of the range and stretch the low end of the range, which could be useful for looking at data with a right tail.
Also consider that these might not be outliers at all, depending on the distribution of your data - a boxplot may not tell the whole story. For an exponentially distributed value, for example, it'll be completely expected that you'll have some values skewed to the right which are far from the mean. Those aren't weird outliers, they're just an expected part of the exponential distribution.


How do I stack copies of a 1D tensor in keras?

After many troubles with Concatenate, I simply did:
x = Reshape((5, 1))(x)
t = ones((1, 10))
stacked_x = x @ t

Maths help a lot. Should have thought about it earlier.


Looking for a model architecture that can work with an unordered list of arbitrary length

You can pass all of the input to a MLP layer, and take the aggregation of the results. It would be independent of the order


RAM crashed for XML to DataFrame conversion function

import dask
import dask.bag as db
import dask.dataframe as dd
from dask.dot import dot_graph
from dask.diagnostics import ProgressBar

dask.set_options(get=dask.multiprocessing.get)
tags_xml = db.read_text('data/Tags.xml', encoding='utf-8')
tags_xml.take(10)

Refer this link for complete tutorial Dask with XML


Replace entire columns in pandas dataframe

You are getting a KeyError because 'opcodes' doesn't exist in dataFrame. Can you show us the contents of dataFrame not just df?

EDIT:

You don't need to create an empty DataFrame and then apply the data across. Simply do:

df = dataFrame[["opt", "opcodes"]].copy()


Automatic topic labelling for topic modelling

Topic modelling is an unsupervised task, so by definition there is no gold-standard label. The task is a kind of clustering, i.e. it tries to group together documents with similar topics, but it doesn't label the groups.
Instead people usually use the words which are the most associated with a topic by the model as a kind of description for the topic.


How can I implement my own AR(p) simulation algorithm?

You question boils down to:

should you try to do the simulation in a vectorised manner?
should you iterate over a vector?

In general we should always try to vectorise (option 1) as it's much more efficient, and it is easy to lag vectors in R (embed) and Python Pandas (shift).

However, I can't see any efficient vectorisation-based solution. The autoregressive nature of the simulated values force us to iterate over a vector to simulate each of $X_1, X_2, \ldots, X_n$ individually (option 2).


Suppose you have a categorical feature in your dataset (e.g. color). And your samples can be either red, yellow or blue. In order to pass this argument to a ML algorithm, you first need to encode it so that instead of strings you have numbers.

The easiest way to do such a thing is to create a mapping where:

red --> 1
yellow --> 2
blue --> 3

and replace each string with its mapped value.

However this might create unwanted side effects in our ML model as when dealing with numbers it might think that blue > yellow (because 3 > 2) or that red + yellow = blue (because 1 + 2 = 3). The model has no way of knowing that these data were categorical and then were mapped as integers.

The solution to this problem is one-hot encoding where we create N new features, where N is the number of unique values in the original feature. In our exampel N would be equal to 3, because we have 3 unique colors (red, yellow and blue).

Each of these features be binary and would correspond to one of these unique values. In our example the first feature would be a binary feature telling us if that sample is red or not, the second would be the same thing for yellow and the third for blue.

An example of such a transformation is illustrated below:

Note, that because this approach increases the dimensionality of the dataset, if we have a feature that takes many unique values, we may want to use a more sparse encoding (like the one I presented above).


What is a discrimination threshold of binary classifier?

Classifiers often return probabilities of belonging to a class. For example in logistic regression the predicted values are the predicted probability of belonging to the non-reference class or $\text{Pr}(Y = 1)$. The discrimination threshold is just the cutoff imposed on the predicted probabilities for assigning observations to each class.


AUC ROC Curve multi class Classification

df = pd.get_dummies(pred1)
df.insert(loc=2,column='2',value=0)
#print(df)

add this before the for  loop and instead of using pd.get_dummies(y_test) use only df


Significant difference in testing vs cross validation accuracy

Significant differences between the calculated classification performance in cross-validation and in the final test set appear obviously, when the model is overfitted.

A good indicator for bad (i.e., overfitted) models is a high variance in the F1-results of single iterations in the cross-validation.

Possible strategies to get a better estimation of the model performance would be:

useing more folds (e.g., 10-fold cross-validation, or leave-one-out cross-validation)
considering simpler models (e.g., less variables, more general parameters)
considering other machine learning algorithms


I think the answer mostly lies in the fact that these are just approximations and they're not super exact because of the small data set and nature of decision trees. The prediction was really 1.0 (I'm guessing all trees' leaves agreed entirely on the prediction).

If gill_size != broad, it would still be 1.0, you could say (1.13 is meaningless). But maybe if gill_size != broad and odor != foul you could expect the probability to be about 1.0 + 0.13 - 0.26 = 0.87.

A little more explanation is at https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20continuous%20and%20categorical%20features.html


How do I Combine two CNN models (h5 format)?

You can put a dense layer combining both outputs.

model1 = tf.keras.models.load_model('/kaggle/input/models/model1.h5')
model2 = tf.keras.models.load_model('/kaggle/input/models/basic_cnn.h5')

inputs = tf.keras.layers.Input(shape=input_shape)
combined = tf.keras.layers.Concatenate()([model1(inputs), model2(inputs)])
outputs = tf.keras.layers.Dense(n_outputs)(combined)

model3 = tf.keras.models.Model(inputs, outputs)

Set input_shape and n_output accordingly to your data and targets.

You should then freeze your pre-trained weights and train the final dense layer to correctly choose which weight to assign to outputs of your models.

model1.trainable = False
model2.trainable = False

model3.fit(your_data)


Is it valid to compare SHAP values across models?

Shapley values were designed in the context of game theory (source), to share value created by a coalition of player in a game. It has multiple properties, including linearity. The linearity ensure that if you were to average your models, the resulting Shapley value would be the average of Shapley values for individual models. Shapley values  are comparable in that sense of considering an average model.
I think the general answer would be the opposite. Inutitively, because looking at an individual Shapley value you can't know if the value is due to an 'individual performance' or 'overall performance' of the coalition. So looking at two values in the context of ML, the difference might be both explained by a different contribution of the individual value but also by a difference in the overall performance of the model. So I would avoid doing that in the general case. (But I often do so - along with comparing individual predictions -  to check if two models with similar overall performance have learned the same things or not)
Overall I would suggest you use a more suited criteria (like an information criteria) for your model selection, then use Shapley value to explain the model you selected. Not using Shapley Values to do some sort of model selection.
Note that I am mainly talking about Shapley values and not SHAP, which is an approximation. You need to be cautious with SHAP as the approximation rely on the absence of correlation between your features, which rarely happen in practice.


Obtaining column values for a dataframe from another dataframe based on a common column variable

Just use pandas.merge to combine the two dataframes:
import pandas as pd

df1 = pd.DataFrame({"animal": ["Cat", "Dog", "Rat", "Bull", "Dog", "Bull", "Bull", "Dog", "Cat", "Rat", "Dog"], "lifeSpan": [2, 4, 6, 0, 4, 0, 0, 4, 2, 6, 4]})
df2 = pd.DataFrame({"animal":["Bull", "Cat", "Rat", "Dog"]})

df3 = df2.merge(df1.drop_duplicates(), on="animal")

This returns the following dataframe:

animal
lifeSpan

Bull
0

Cat
2

Rat
6

Dog
4


Error when using MAX in Apache Pig (Hadoop)

The issue was with column addressing, heres the correct working code:

grpd = GROUP features BY featureid;
temp = FOREACH grpd GENERATE group as featureid, MAX(features.featurevalue) as val;


Is it possible to have a default class in multi class classification?

Depends on the model but this would be a post-processing move.

E.g. in a recent problem I trained a random forest classifier and then classified any data point where the probability of the likeliest class was less than 50% as unknown.