How to prepare colored images for neural networks?

Your R,G, and B pixel values can be broken into 3 separate channels (and in most cases this is done for you). These channels are treated no differently than feature maps in higher levels of the network. Convolution extends naturally to more than 2 dimensions.

Imagine the greyscale, single-channel example. Say you have N feature maps to learn in the first layer. Then the output of this layer (and therefore the input to the second layer) will be comprised of N channels, each of which is the result of convolving a feature map with each window in your image. Having 3 channels in your first layer is no different.

This tutorial does a nice job on convolution in general.

http://deeplearning.net/tutorial/lenet.html


XGBoost: predictive, descriptive (or both) model?

You're right that it is a predictive model, but analysing feature importance, partial dependence and other metrics/plots are intended to allow you to gain an understanding of the classification process of models which are often very black-box in nature.

Yes, you can use this to gain some understanding of the underlying data, but I'd be more inclined to say that the primary reason for looking at feature importance is to understand your model rather than to understand your data.


Weighing each label in multi-label classification

Yes, of course, this technique exists. In XGBoost, for instance, you can change the objective function to the multi:softprob which in specifics does:

  multi:softprob: same as softmax, but output a vector of ndata *
  nclass, which can be   further reshaped to ndata * nclass matrix. The
  result contains predicted probability of each data point belonging to
  each class.

From my memory, if it answers your question I think it rounds up to 1.


How to treat column with potentially meaningful NaNs

What you are asking about is called imputation - https://en.wikipedia.org/wiki/Imputation_(statistics)

Imputation basically means replacing NaNs with other data. There are lots of ways to do it and the method usually depends on the application. In this case you are the domain expert so you would be in the best position to decide what imputation method you should use.

Some common imputation methods are:

Mean (replace all NaNs with the average)

Median (replace all NaNs with the "middle" of the data)

MICE (Multiple Imputation by Chained Equations - probably overkill for your application, but worth looking into)

In your case I would go with mean imputation to start and then see what kind of results you get.


Why is predict_generator is returning an empty array?

The code looks OK to me... I can only think that you have somehow exhausted the ImageDataGenerator object, because you use the same on during training and then for the prediction. (I do realise this should never happen).

To rule this out, you could create a separate generator for testing.

This would also have the advantage of using some of your data for a hold-out test set, so your final predictions are out-of-sample, giving you a better idea of performance  :)

For example, you might change your code to be like this:

validation_directory = 'D:\D_data\Rock_Paper_Scissors\Test'
holdout_directory =  'D:\D_data\Rock_Paper_Scissors\Holdout'   # make this new directory

Move e.g. 10 files into the new holdout directory

validation_datagen = ImageDataGenerator(rescale= 1./255)
validation_generator = validation_datagen.flow_from_directory(
    validation_directory,
    target_size = (28,28),
    class_mode = 'categorical'
    )

# new generator for predictions
holdout_generator = validation_datagen.flow_from_directory(
    holdout_directory,
    target_size = (28,28),
    class_mode = 'categorical'
    )

model.fit_generator(training_generator, epochs=20, validation_data=validation_generator)

Set the number N accordingly, e.g. to the 10 files in the new directory

predict = model.predict_generator(holdout_generator, batch_size=N)

print(predict.shape)    # this should return the shape of the numpy array: (N, H, W, C)
print(predict()        # the actual rpedictions


Tips on how to create box whisker plot for huge data set

Hope you are aware of data sampling methods. They exist to solve problems such as these. They are of several types like;
Probability Sampling Methods

Simple random sampling
Systematic sampling
Stratified sampling
Clustered sampling

Non-Probability Sampling Methods

Convenience sampling
Quota sampling
Judgement (or Purposive) Sampling
Snowball sampling

When sampling the data, be careful of the bias.
Bias in sampling
There are five important potential sources of bias that should be considered when selecting a sample, irrespective of the method used. Sampling bias may be introduced when:

Any pre-agreed sampling rules are deviated from
People in hard-to-reach groups are omitted
Selected individuals are replaced with others, for example if they are difficult to contact
There are low response rates
An out-of-date list is used as the sample frame (for example, if it excludes people who have recently moved to an area)

Basis of the above detail, I think there is no need to use the entire dataset for visualization purpose. Consider an example, each country conducts a census survey to estimate its population of people. Such a dataset is huge both in size and in complexity. Do you think those statisticians use the complete dataset for visualization? I strongly doubt it. They use sampling methods.
Edit
Python Code
# load required libraries
import pandas as pd
# create some dummy data
df = pd.DataFrame({'num_legs': [2, 4, 8, 0],
                   'num_wings': [2, 0, 0, 0],
                   'num_specimen_seen': [10, 2, 1, 8]},
                  index=['falcon', 'dog', 'spider', 'fish'])
print("## Original data ##\n",df)

## Original data ##
         num_legs  num_wings  num_specimen_seen
falcon         2          2                 10
dog            4          0                  2
spider         8          0                  1
fish           0          0                  8

Sampling methods:

Simple random sampling: Extract 3 random rows

    print("\n Simple random sample")
    print(df.sample(3, random_state=10))
    Simple random sample
            num_legs  num_wings  num_specimen_seen
    spider         8          0                  1
    falcon         2          2                 10
    fish           0          0                  8

1.1. Simple random sampling: Extract 3 random elements from the Series
df['num_legs']
print("\n Simple random sample of particular column\n",df['num_legs'].sample(n=3, random_state=10))

Simple random sample of particular column
spider    8
falcon    2
fish      0
Name: num_legs, dtype: int64

1.2. Random sampling with replacement
print("\nA random 50% sample of the DataFrame with replacement:")
print(df.sample(frac=0.5, replace=True, random_state=10))
A random 50% sample of the DataFrame with replacement:
     num_legs  num_wings  num_specimen_seen
dog         4          0                  2
dog         4          0                  2

1.3. Random upsampling with replacement
print("\nAn upsample sample of the DataFrame with replacement")
print(df.sample(frac=2, replace=True, random_state=10))
An upsample sample of the DataFrame with replacement
        num_legs  num_wings  num_specimen_seen
dog            4          0                  2
dog            4          0                  2
falcon         2          2                 10
fish           0          0                  8
falcon         2          2                 10
dog            4          0                  2
fish           0          0                  8
falcon         2          2                 10

1.4. Random sampling with weights
print("\nUsing a DataFrame column as weights. Rows with larger value in the num_specimen_seen column are more likely to be sampled.")
print(df.sample(n=2, weights='num_specimen_seen', random_state=10))
# Using a DataFrame column as weights. Rows with larger value in the num_specimen_seen column are more likely to be sampled.
        num_legs  num_wings  num_specimen_seen
fish           0          0                  8
falcon         2          2                 10

Stratified Random Sampling

    print("\nStratified Random Sampling")
    print(df.groupby('num_legs', group_keys=False).apply(lambda x:x.sample(min(len(x), 2))))

Stratified Random Sampling

        num_legs  num_wings  num_specimen_seen
fish           0          0                  8
falcon         2          2                 10
dog            4          0                  2
spider         8          0                  1

This brief example should get you started!


Make Keras run on multi-machine multi-core cpu system

In order to set the number of threads used in Theano (and, therefore, the number of CPU cores), you'll need to set a few parameters in the environment:

import os
os.environ['MKL_NUM_THREADS'] = '16'
os.environ['GOTO_NUM_THREADS'] = '16'
os.environ['OMP_NUM_THREADS'] = '16'
os.eviron['openmp'] = 'True'

This should allow you to use all cores of all CPUs.

This can, of course, also be done in Tensorflow:

import tensorflow as tf
from keras.backend import tensorflow_backend as K

with tf.Session(config=tf.ConfigProto(
                    intra_op_parallelism_threads=16)) as sess:
    K.set_session(sess)
    <Your Keras code>


This 2019 paper states that all the current image classification models are actually doing the same thing, i.e. classifying jumbled image patches. The reason is that they are weak at peaking up long-range dependencies (relations between pixels far away). As a result, breaking these dependencies by shuffling the image patches does not affect them substantially. Here is a post that explains the findings and the BagNet method that suits your need.


Pass your clf object to the function provided in the website. It should work.


What is the best way to match street addresses in a dataset in Python?

You can try using this to help:
https://github.com/openvenues/libpostal

libpostal looks like it can normalize across various geographic styles with the expand addresses functions.


How to use a deep learning algorithm to cluster image *styles* in an unlabeled data set?

You can use a pretrained model, extract features, and try to cluster images based on these features. The success will be contingent on the "style" you want to distinguish (and if the derived features represent the different styles). I had a similar problem and used Kerans and VGG16 features to cluster images using KNN. Find my Python code on Github.


R: Producing multiple plots (ggplot, geom_point) from a single CSV with multiple subcategories

something like this might work:

library(plyr)

# initialize All_Data around here

dlply(All_Data, 'Identifier', function(dataSubset) {
  g <- ggplot(dataSubset) + geom_point(mapping =aes(SampleDate, Total.Result)) + ylim(0,20000)
  file_name <- paste0("Scatter_", unique(dataSubset$Identifier), ".tiff", sep="")
  ggsave(file_name,g)
})

(I didn't test it)


How to use neural network's hidden layer output for feature engineering?

TL;DR:
Yes. You can (iiuc)

Longer Version:
In fact, this is what many popular algorithms like Word2Vec and AutoEncoders do. (With respect to hidden layer outputs)

Word2Vec:
Given an input word ('chicken'), the model tries to predict the neighbouring word ('wings')
In the process of trying to predict the correct neighbour, the model learns a hidden layer representation of the word which helps it achieve its task.

Finally, we just remove the last layer and use the hidden layer representation of the word as its $N$ dimensional vector.

So basically, we feature engineered the word vectors.

For autoencoders,
It takes in $X$ as the input and tries to predict $X$ again, in the process learning a latent representation of the input signal X. The input hidden representation in Layer $L2$ can be used in other tasks. (Note: here X and X hat are the same)

(In fact, you can use features learned by a CNN and feed them into an SVM and get good results)

Relevant to you:
You can train your model on the given data and finally chop off the last prediction layer and use the output of the intermediate layers as features. I believe this should work because it works for so many tasks which I explained above.

Source:

Learn Word2Vec by implementing it in Word2Vec : an article by me explaining word2vec. (Shameless self-advertising here but I feel the article is good and relevant)
Andrew Ng's unsupervised feature learning website


Can we predict when an event will occur in the future from time series data?

I would suggest creating a custom ontology by capturing the ranges of each activity. For example, define in your training data additional parameter for breakfast and put the range of between the minimum and the maximum hour 08:00:10 08:23:30.

Feature engineering is a must if you plan to you RandomForest, LinearRegression, etc..

Additionally, the timestamp itself has very reach data information. You can take the day of the week, the day of the month, the day of the year, week of the year. Based on this you can see whether is the weekend, a weekday(mon-fri), and you can see how this will change over time(maybe during the summer the person is eating later/earlier). Also, you can see the hourly changes. I would also strongly suggest using plots, and this way you will see how the "Breakfast" label is chaining over time(during different periods of the year, different parts of the week/month). This will give you a better understanding of what is happening in your data.

In the end, I would suggest trying RandomForest(s)(maybe a bagging approach). They are really good for time-series data, especially when you can define nice features where the differences are clearly visible. Tunning the RandomForest is a must. In similar problems to this one, the min_sample_split and the max_depth parameter are usually really helpful.

Additionally, Ridge or Lasso regression is also a good way to try your newly constructed features.

At the end, as it was pointed out, LSTM is proven to capture time-dependency in the data. However, I would reconsider using it, without any practical experience.


SMOTE vs SMOTE-NC for binary classifier with categorical and numeric data

You have to keep in mind that machine learning is still largely an empirical field, full of ad-hoc approaches that, while they happen to work well in most cases, they lack a theoretical explanation as to why they do so.

SMOTE arguably falls under this category; there is absolutely no guarantee (theoretical or otherwise) that SMOTE-NC will work better for your data compared to SMOTE, or even that SMOTE will perform better compared with much simpler approaches, like oversampling/undersampling. Quoting from section 6.1 on SMOTE-NC of the original SMOTE paper (emphasis added):

  SMOTE-NC with the Adult dataset differs from our typical result: it performs worse than plain under-sampling based on AUC. [...] even SMOTE with only continuous features applied to the Adult dataset, does not achieve any better performance than plain under-sampling.

The authors proceed to offer some possible explanations as to why they see such not-typical performance with SMOTE/SMOTE-NC on the said dataset, but as you will see this has to do with a deep focus on the dataset itself and its characteristics, and it is itself rather empirical in nature and hardly "theoretical".

Bottom line: there is not really much to be explained here regarding your question; any further detail will require going deep with the specific characteristics of your dataset, which of course is not possible here. But I would suggest to not bother, and continue guided by your experimental results, rather than by any (practically non-existent) theory on the subject...


What is the most common practice of generating (X,Y) from an arbitrary CDF or PDF?

I am no expert in the field but I can imagine 3 different cases :

Variables are independent in which case you can use your formula on each
You have access to some chain rules decomposition that allows you to easily generate each variable one at a time using your formula
If none of the above : https://appsilon.com/how-to-sample-from-multidimensional-distributions-using-gibbs-sampling/


GRU and LSTM does not "take risk" predicting

This is probably because the deltas you are trying to predict are less than 1, so your loss function (I’m assuming MSE) isn’t working as expected.
Squaring an error less than 1 will make it even smaller, so your model is not currently motivated to leave the cosy local minimum of the naïve strategy of always predicting delta as zero.
I recommend rescaling your deltas by multiplying them by 100 and then using MSE as a loss function.


While the other answers aren't wrong, they don't touch anything about bioinformatics. I'll go into details.

In bioinformatic, simply asking for the size of the training set makes no sense. You'll need to understand deeper about bioinformatic to answer this question. There is a good reference: An ensemble approach to accurately detect somatic mutations using SomaticSeq for you.

The goal of the project is to build a gradient-boosting machine for classifying somatic mutation given an alignment file. This is very close to what you're doing.

Now, you want to know the training size? You should ask yourself the following:

Sequencing errors
Reads
Sequencing depth
Sensitivity
F1 score
Allele frequency

In an next-generation sequencing, your sequencing machine always give you errors. How reliable is your pipeline to detect those errors?

How good are your reads for your problem? For example, if you're detecting disease related to structural variant, you might need more samples to compsenate for the short reads.

What's your sequencing depth? In the paper I quote, they have like 30x depth. It'd be pointless to talk about the training size unless you know well about your depth.

Does your sequencing cover well your region of the genome you're interested in?

Sensitivity and F1 score. You can have good idea by drawing an F1 curve something like as in the paper:

What's your expected allele frequency range? If you have something like close to 0.5 (ie: maximum heterozygosity), you'll probably be safe with a smaller sample size. However, if you're trying to detect a rare disease, something like <0.1 as in the diagram, you'll need to have more samples and most likely more features.

Empirically, we generally use spike-in standards to measure the sensitivity. Once you draw the sensitivity against training size, you'll know at what training size to use achieve whatever sensitivity you're looking for. A typical tool would be ROC.


How to fit a KNN and then a linear regression with those neighbors?

KNeighborsTransformer only gives you the indices of the nearest neighbors and the distances.  You need to do more work to retrieve the points to fit your linear regression.
Here's a draft that appears to be working:
from sklearn.neighbors import NearestNeighbors
from sklearn.base import RegressorMixin, BaseEstimator, clone
from sklearn.linear_model import Lasso
from sklearn.utils import check_X_y
import numpy as np

class LocalLinearRegressor(RegressorMixin, BaseEstimator):
    def __init__(self, n_neighbors=10, linear_model=Lasso()):
        self.n_neighbors = n_neighbors
        self.linear_model = linear_model

    def fit(self, X, y=None):
        "Fits the neighbors search."
        X, y = check_X_y(X, y)
        self._fit_X = X
        self._fit_y = y
        self.neighbor_search = NearestNeighbors(n_neighbors=self.n_neighbors)
        self.neighbor_search.fit(X)
        self.local_regressors_ = {}
        return self

    def predict(self, X):
        """Fits linear regressions on the k nearest training points to predict new values.

        We don't fit these linear regressions at fit time because there would be so many.
        However, we do save the regressions as we see them to speed up predictions.
        """
        neighbors = self.neighbor_search.kneighbors(X, return_distance=False)
        ksets, mapper = np.unique(neighbors, return_inverse=True, axis=0)
        for kset in ksets:
            if tuple(kset) in self.local_regressors_:
                continue
            local_X = self._fit_X[kset, :]
            local_y = self._fit_y[kset]
            self.local_regressors_[tuple(kset)] = clone(self.linear_model).fit(local_X, local_y)
        return np.array([
            self.local_regressors_[tuple(ksets[mapper[i]])].predict(X[i, :].reshape(1, -1))[0]
            for i in range(X.shape[0])
        ])

And here's a Colab notebook showing it in action.


Curation of a dataset of audio files in different formats

Keep the files the way they are for long term storage.

For experimenting, it might be beneficial to deal with just one format.

Depending on what resources you have available issues may be:

supported formats
disk space/speed
decompression speed
random access

Depending on the requirements of your analyses, you might want to reduce the quality of your tracks to gain speed advantages.

Supported Formats

If the software you use for analysis supports both MP3 and M4A, you're set. If not, you'll need to convert using tools like ffmpeg or sox.

Disk Space/Speed

If you were to convert your files to uncompressed WAV, this will eat up a lot of disk space. If you don't have this space, WAV is not a good format. Instead convert to another lossless format like FLAC or Apple Lossless. You can do so without loosing any quality.

Besides disk space, you need to pay attention to disk speed. For some deep learning tasks you will have to provide your network with lots and lots of different samples, which you need to read from disk. With WAV being roughly 10 times as large on disk, disk I/O may become a bottleneck, so minimizing file size on disk may become desirable.

Decompression Speed

Using a compressed format like FLAC or MP3 comes at a (small) price. You will have to decompress your audio files on-the-fly before analysis.

Random Access

If you need to quickly access audio starting at a certain timestamp, say at 10.455s, formats like WAV have a distinct advantage, as they allow random access of audio very easily. This is usually not possible (as quickly) with compressed formats.

So what format should you choose for your experiments?

It all depends on your requirements.

Do you need quick random access to the audio? Do you really need stereo quality at 44,100 Hz? You can reduce the problem size a lot, by converting to mono 22,050 Hz. If that's still good enough for your purposes, go for it and save yourself a lot of trouble (but keep your original files around!).

Also, to get more technical, if you are going to analyze your data with something like Python and you know you need spectrograms, there's nothing wrong with preprocessing your MP3/M4A files with a library like librosa and simply store numpy arrays, if that's what you ultimately need.


Here, first I created the column values as a list and then assigned them to be the new dataframe's columns. However, we could do it using the columns and without creating lists. It just seems easier this way.

#Initializing the new_df (df2) dataframe
import pandas as pd
new_columns =  ['GCF', 'Genome']
new_df  = pd.DataFrame(columns = new_columns)
#new_df
#df1.columns
#creating the column GCF of the new dataframe
GCF = [str(df1['BGC Class'][i])+ '_GCF' + str(df1['GCF No'][i]) for i in range(len(df1))]
#creating the new column 'Genome' of the new dataframe
organism = [df1['Organism'][i].replace(' ', '_') for i in range(len(df1))] #substituting ' ' by '_'
BGC_name = [list(df1['BGC Name'].str.split('_'))[i][0] for i in range(len(df1))]
genome = [organism[i] + '_' + BGC_name[i] for i in range(len(df1))]
new_df['GCF'] = GCF
new_df['Genome'] = genome

Output:

    GCF             Genome
0   PKSI_GCF1813    Streptomyces_sp_Hm106
1   NRPS_GCF1813    Streptomyces_sp_MBT13
2   PSKI_GCF1883    Listeria_sp_MBT23
3   NRPS_GCF1887    Streptomyces_sp_MBT36

The second way in which we do not create lists and change the columns' values sort of inplace, is as following:

#Initializing the new_df (df2) dataframe
import pandas as pd
new_columns =  ['GCF', 'Genome']
new  = pd.DataFrame(columns = new_columns)
#new_df
#df1.columns
BGC_name = [list(df1['BGC Name'].str.split('_'))[i][0] for i in range(len(df1))]
new['GCF'] = df1['Organism'].str.replace(' ', '_')+ '_' + BGC_name
new['Genome'] = df1['BGC Class']+ '_GCF'+ df1['GCF No'].astype('str')
#new


Machine learning with sklearn vs. scipy stats

Yes fitting the data and finding the best fitting line is called training the model.
If you look at the source code of scikit-learn linear regression you can find the its using scipy linalg.lstsq module for finding the coefficients and intercept (most cases). See the source code for more details . Machine learning is fancy word for Application of  mathematics (on data mostly) using computers (machines)


The reason behind using a pre-trained model?

I don't understand how a pre-trained model can adapt to my given corpus

You are correct in thinking this way. It is not a magic wand. It learns the embedding values based on the underlying context of the corpus(e.g. news) which may work in the broad sense but not in a specific case.
Two cities may get the embeddings based on their geographical location but that might not be the embedding we may like if we are comparing cities on crime rate/GDP etc.
One of the best posts on Word Embedding [Blog post by Sebastian Ruder] has mentioned all the limitations in detail.
An excerpt from the post,

"...One of the major downsides of using pre-trained embeddings is that the news data used for training them is often very different from the data on which we would like to use them. In most cases, however, we do not have access to millions of unlabelled documents in our target domain that would allow for pre-training good embeddings from scratch...."

If I have new words not present in the pre-trained model will I be able to use this pre-trained model to learn the embeddings for the new words?

It will get a value from a bucket for all the out-of-vocabulary(OOV) list. Sometimes it can be the same value for all the new words. But there is some known strategy to deal with the scenario. Check the post for relevant papers.
From the same posts,

"....One of the main problems of using pre-trained word embeddings is that they are unable to deal with out-of-vocabulary (OOV) words, i.e. words that have not been seen during training. Typically, such words are set to the UNK token and are assigned the same vector, which is an ineffective choice if the number of OOV words is large...."

But this issue will prevail even if you learn embeddings with your own corpus. So I believe it is not just limited to the pre-trained embeddings.


In machine learning there is a fundamental difference between what is known as hyperparameters and parameters.
Parameters
They are the variables that define your model, or in other words the relationship between your inputs and the output you are trying to predict.
For example, in simple linear regression, the goal is to be able to make predictions for a target variable $y$ given some new input value $x$ of the input variable on the basis of a set of training data comprising $N$ input values $\mathbf{x} = (x_1, \dots, x_n)^{\mathbf{T}}$. Relationship between inputs and output is given by
$$
y = w_0 x + w_1
$$
where $w_0$ and $w_1$ are the parameters our algorithm tries to "learn" during training to produce the most accurate possible predictions for $y$. FYI the term $w_0$ is called weight and the term $w_1$ is called bias.
Hyperparameters
These are different from the parameters above in the sense that they act on the training process itself. Hyperparameters can be seen as knobs you would tweak to adjust the learning of your model in the best possible way.
An example of hyperparameter is that of the learning rate $\eta$ used to adjust how far apart from each other values are computed when looking for best possible parameters of a model using Gradient Descent. Another example is the number of decision trees to have in the forest when using Random Forest algorithm.
Hyperparameter tuning
Unlike model parameters, hyperparameters can be set by the user before training a machine learning model. To be fair the Random Search Cross-Validation method you mentioned is pretty efficient for finding most suitable values for hyperparameters given a machine learning problem. You can reasonably use results from your cross validation for hyperparameters.
Next steps should include testing your model on a held-off test data to check that your model parameters determined during training are indeed correctly setting your model up.


Where can I find freely available multi-label datasets online?

Try, Kaggle Toxic Comments Challenge. You have to classify the answer to multiple classes at the same time. It is a multi-label classification problem.


I think the answer to this question is weight sharing in convolutional layers, which you don't have in fully-connected ones. In convolutional layers you only train the kernel, which is then convolved with the input of that layer. If you make the input larger, you still would use the same kernel, only the size of the output would also increase accordingly. The same is true for pooling layers.

So, for convolutional layers the number of trainable weights is (mostly) independent of input and output size, but output size is determined by input size and vice versa.

In fully-connected layers you train weight to connect every dimension of the input with every dimension of the output, so if you made the input larger, you would require more weights. But you cannot just make up new weights, they would need to be trained.

So, for fully-connected layers the weight matrix determines both input and output size.

Since CNN often have one or more fully-connected layers in the end, there is a constraint on what the input dimension to the fully-connected layers has to be, which in turn determines the input size of the highest convolutional layer, which in turn determines the input size of the second highest convolutional layer and so on and so on, until you reach the input layer.


train_test_split ValueError: Input contains NaN

Check to see if you have any null or nan values:

X[X.isnull() == True]

Then you have to decide what to do with those nan values. Something that is commonly done is to forward fill in place.

X.fillna(method = 'ffill', inplace = True)
y.fillna(method = 'ffill', inplace = True)


Why does decreasing the SGD learning rate cause a massive increase in accuracy?

With a higher learning rate, you take bigger steps towards the solution. However, when you are close, you might jump over the solution and then the next step, you jump over it again causing an oscillation around the solution.
Now, if you lower the learning rate correctly, you will stop the oscillation and continue towards the solution once again. That is, until you start oscillating again.
To keep in mind is that a larger learning rate can jump over smaller local minima and help you find a better minima, which it can't jump over.
Also, it is generally the training error that becomes better, and the validation error becomes worse as you start to overfit on the training data.


LSTM Multi-class classification for large number of classes

One way to approach this is to increase model capacity and see if/when your LSTM is able to learn the train data. In this first step it is ok to overfit (also see this recent question and answer for this approach) since you can add regularization/decrease model capacity later.

These are the parameters I would tweak:

find a good learning rate along a log-scale (try at least 0.1, 0.01 and 0.0001)
increase the number of LSTM layers, e.g. to 5
increase the hidden dimension, e.g. to 1024
slightly increase the embedding dimension, e.g. to 300

If training this model takes too long, tune the learning rate based on your current model and then make the other adjustments. However, I suggest to not make the other adjustments in smaller steps or one by one as that can take a lot of time.


Another option would be to apply an established Optical Character Recognition (OCR) system to the raw images. After converting the raw images to plain text, put the relevant data into a tabular dataframe. Once in a tabular dataframe, fit a deep learning or traditional machine learning model can be fit.
It can be tempting to build an end-to-end deep learning system. However, some problems are best solved by combination of subsystems. Those subsystems could include human expertise, rules, traditional machine learning, and just a small amount of deep learning.


How to extract specific information from raw , unstructured text using NLP and Deep Learning?

You can create your own named-entity recognition through a pre-trained model like Spacy. https://spacy.io/usage/linguistic-features#section-named-entities

these keywords in your example should be considered as an age entity

 above 21 years
 65 years or less

try to play its linguistic features to get what you need.
Hope this helps.


What is the difference between data singular and data plural ?

Data is plural, as in 'These data are depicted below'. Singular is 'datum' but it's rare one would have occasion to use the term.


Encoding multiple observations from the same feature space

What you need is Encoding Categorical variables. This topic is discussed in tons of blogposts, it is definitely worth checking out this recent article that nicely and extensively go through most of the methods, since I do not intent to rewrite again what is out there but rather giving you my personal experience.

I asked you earlier what algorithms you are after, simply that could change your choice of encoding method. Some encoding methods like One-hot-encoding makes your feature space very sparse when your categorical variable is very cardinal (usually not recommended!), and it is best to go with sparse-aware algorithms.

In a nutshell, I suggest to start with the following (classic and simple), explanations borrowed from that article:

  OneHot — one column for each value to compare vs. all other values.
  Binary — convert each integer to binary digits. Each binary digit gets one column.
  Hashing — Like OneHot but fewer dimensions, some info loss due to collisions.
  Backward Difference — the mean of the dependent variable for a level is compared with the mean of the dependent variable for the
  prior level.
  Target — use the mean of the dependant variable.

The above-mentioned methods can be used pretty much for all algorithms. And each single has its pros and cons. Some you have more information loss than the other and so on. Good news is that most of these methods are quite easy to use, e.g. via this Python package.

Another method that I found quite interesting and is suitable for Neural Networks is:

Entity Embeddings - An special encoding method borrowed from NLP, in which an embedding space is learned on the fly for each categorical variable.

See these blogpost 1, 2 ,3. I am writing an up-and-running code in Google Colab to fully demonstrate this, but you still find pieces of code in those blogposts to give it a try. I find this method way better than for example OneHot for Neural Networks.

Hope these help!


Deploying the prediction model under missing values for test data

I can think of three ways to deal with the problem:

Treat "missing value" as another feature: Imagine you have a feature like "date of graduation". One possible (likely?) reason why this value is missing might be that the person did not graduate. So you could build a model which as a binary feature "graduation date is available" and the actual graduation date as another feature.
Predict the missing values: If data is missing because of your lack of knowledge of it (in contrast to the first point), then you might think about trying to predict the missing value. You could  also add a feature which encodes the certainty of the predicted value being correct.
Skip the feature: If it is missing very often and if it doesn't add much value to your prediction, you might simply want to remove it.


Continuous Bag Of Words (CBOW) network architecture?

What is the proper architecture to train CBOW encodings?

The original paper by Mikolov et al uses 1 hidden layer. However, for NLP tasks (and deep learning in general), there is no "correct" number of layers to use.
For some tasks, you might find using more hidden layers to be better, for other tasks maybe one hidden layer is sufficient. The number of layers is a hyperparameter, along with other 'design choices' like the dimension of the embedding vectors, the size of the vocabulary, and many, many more.

If this multiple hidden layer approach is correct, how do you not lose semantic information when you only use one of the layers as the encodings?

I think of it this way. When training CBOW, the hidden layers learn some 'relationship function' between the input context words and the output target word. In order for the 'relationship function' to perform well, the embeddings must be arranged accordingly (in N-dimensional space), and it just happens that the optimal arrangement encodes semantic information.
So the hidden layers shouldn't really have anything to do with semantic information. However, because of the immense modeling power of hidden layers (which make up neural networks), the truth is the embeddings will inevitably lose some semantic information to the hidden layers.

If the single hidden layer approach is correct, does anyone have examples of this being implemented using this approach in PyTorch (fine if no)?

Here's literally the 1st github example off Google.


You should probably ask programming questions on StackOverflow as more people will be able to help you with this kind of problems. As suggested here, you can serialize scikit-learn model to PMML and then execute it using jpmml-evaluator. There are different suggestions in that post, so you should take a look at it and see what works best for your use case.


Finding Outliers in Resource Forecast Data

There is not one way of detecting outliers. Most statistical methods assume an underlying normal distribution. If this is the case you could apply for example an Thompson Tau test. You can also define some own criteria. In your use case I think an important question is: what will happen to the quality of the model if you drop or modify missing data. This gives you an idea how to handle these missing data points.


Classification methods using one overlapping feature

Could you plot something like heatmap or 3D plot (x=size,y=location,z=prices) ? I would try something like a simple decision tree to fit your data.

Using only the houses prices as feature, I would first plot prices versus location. If there are two distinct clusters slightly overlaping, maybe fixing just one threshold is good enough.


How to pass common inputs to a merged model in Keras

I found the answer to my question. I need to pass the common input shape to the individual models and concatenate the outputs like this:

input_shape=(img_rows,img_cols,3)
commonInput = Input(input_shape)
out1 = model1(commonInput)
out2 = model2(commonInput)
mergedOut = Concatenate()([out1,out1])


Not entirely sure what you mean by "numpy vectors" but am assuming the question is why each of these methods return essentially (almost but not quite) the same output...
Reference: pandas docs.
df['name'].values is a "Series corresponding to colname". In other words, you're just calling the data from that column and putting the in an array by calling .values.
.iloc is a "Purely integer-location based indexing for selection by position". Same as above but you're calling the indexed location of the column where df.iloc[:, 1] is df.iloc[all rows, col 2]. Probably an easier method to call multiple consecutive columns in a DataFrame then writing out each individual column name.
df.iloc[:,1:2].values <-- creates an array of arrays where the main array is the column that you called (col2) and each row values is contained in a subarray. This is--I think-- because you're slicing the dataframe between column index locations 1 and 2 (rather than just calling loc 1 like above). This would mean that each row is being called individually so that a new array is created for each row that exists between column index locations 1 and 2 (which is the 'parent' array).


If you have enough data and reasonable number of classes, you can definitely train your model. The grouping of words that you have done is similar to an approach called bag-of-words model. You can use that to build a classifier using Naive Bayes or SVM etc.
On a different note, you can also look at the KNN algorithm because it looks fit for your use case. You can have a look at this paper


what is the main difference between GAN and autoencoder?

The main differences are the philosophy that drives the loss metric, and consequently the architecture (the latter goes without saying, obviously).

Autoencoders

The job of an autoencoder is to simultaneously learn an encoding network and decoding network. This means an input (e.g. an image) is given to the encoder, which attempts to reduce the input to a strongly compressed encoded form, which is then fed to the decoder.

The network learns this encoding/decoding because the loss metric increases with the difference between the input and output image - every iteration, the encoder gets a little bit better at finding an efficient compressed form of the input information, and the decoder gets a little bit better at reconstructing the input from the encoded form.

Summarised: An autoencoder learns to represent some input information very efficiently, and subsequently how to reconstruct the input from it's compressed form.

Generative Adversarial Networks

Here, we have a "generator" whose job is to take some noise signal and transform it to some target space (again, images is a popular example). The other component (the adversary) is the "discriminator", whose job is to distinguish real images drawn from the desired target space from the fake images created by the generator. In this case, the network is trained in two alternating phases, each with a different loss:

Firstly, the discriminator is given labelled samples from the real set of images, and fake images generated by the generator (of course, at the start, these images are just noise). Loss is calculated according to some binary classification loss metric (e.g. crossentropy). The discriminator thus begins to learn the difference between a fake image and a real image.

Before it can learn too much, though, we switch over to the generator. To train the generator, we once again feed in some noise and check the output of the discriminator. This time, we want the discriminator to detect that the image (generated by the generator) is real - this means that we have successfully fooled the discriminator, and therefore the generator has begun to learn how to make an image that resembles a "real" image from the training set.

Summarised: A GAN uses an adversarial feedback loop to learn how to generate some information that "seems real" (i.e. looks the same/sounds the same/is otherwise indistinguishable from some real data)

The difference

Hopefully that shows that there is a very fundamental structural difference between the two networks, and the goals that each of them are trying to achieve during the learning process. I don't really know if GANs are necessarily "more successful", but for certain generative problems they have been observed to give more "realistic" outputs. Why? My gut says that a GAN probably learns more about "how can I make an image look real in general" rather than "how can I memorise this particular set of images with the greatest accuracy/efficiency". But there are certainly similarities, in particular between the generator (of the GAN) and the decoder (of the autoencoder).


Are there any Python libraries for predicting the closest value to a correct label out of a variable-size list of possible label values?

This could be treated as a multiclass & multilabel classification problem. If you are able to get your label lists in some good shape, you could check the Keras functional API for multitarget regression/classification models.

The main linitation is that you would need a list of outputs with equal length. I guess the best way is to do one-hot (aka dummy encoding) of your labels and to predict "classes" (labels). However, if the number of possible labels/classes is really large, you may get into trouble with this approach.

Using the Keras functional API is relatively straight forward. You simply need to define your outputs. Here is a minimal example. In order to do classification you would need to change a few things (loss function etc) but this should be no problem:

import numpy as np
import pandas as pd
from keras.datasets import boston_housing
(train_data, train_targets), (test_data, test_targets) =  boston_housing.load_data()

# Standardise data
mean = train_data.mean(axis=0)
train_data -= mean
std = train_data.std(axis=0)
train_data /= std
test_data -= mean
test_data /= std

# Add an additional target (just add some random noise to the original one)
import random
train_targets2 = train_targets + random.uniform(0, 0.1)
test_targets2   = test_targets + random.uniform(0, 0.1)

# https://keras.io/models/model/
from keras import models
from keras import layers
from keras.layers import Input, Dense
from keras.models import Model
from keras import regularizers
from keras.layers.normalization import BatchNormalization

# Input and model architecture
Input_1=Input(shape=(13, ))
x = Dense(1024, activation='relu', kernel_regularizer=regularizers.l2(0.05))(Input_1)
x = Dense(512, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(256, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(8, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)

# Outputs
out1 = Dense(1)(x)
out2 = Dense(1)(x)

# Compile/fit the model
model = Model(inputs=Input_1, outputs=[out1,out2])
model.compile(optimizer = "rmsprop", loss = 'mse')
# Add actual data here in the fit statement
model.fit(train_data, [train_targets,train_targets2], epochs=500, batch_size=4, verbose=0, validation_split=0.8)

# Predict / check type and shape
preds = np.array(model.predict(test_data))
#print(type(preds), preds.shape)
# is a 3D numpy array

# get first part of prediction (column/row/3D layer)
preds0 = preds[0,:,0]
# second part
preds1 = preds[1,:,0]

# Check MAE
from sklearn.metrics import mean_absolute_error
print(mean_absolute_error(test_targets, preds0))
print(mean_absolute_error(test_targets2, preds1))


Clustering users based on buying behaviour

Big Picture:

First of all, the feature set in your data is pretty sparse and uninteresting, so you should not expect to gain much traction from this problem. Use your human mind to think about the types of clusters that could possibly exist and this may help inform the feature engineering that you should perform.  Some people may tend to buy things on payday (15th and 30th of the month), some people might like to shop on Sundays, some people might like sports stores, people who shop on Wednesdays could like to buy woman's clothes, midnight shoppers could tend to focus on "as seen on TV" products.  There might be signal there, but with only 6 features (really 5), it might be tough to find it.

Normalization/Standardization

Make sure you standardize your data before using k-means clustering. Otherwise, your distance metric will loose rotational invariance and the clusters will become nonsense from feature to feature.

Feature Engineering:

You will want to do some feature engineering to help your clustering algorithm find signal in the data:

1) Get rid of userid unless it has some meaningful information like it is a proxy for when they first signed up i.e. ask yourself whether this would help the algorithm cluster the data or provide noise to confuse the algorithm. The answer is likely the latter.

2) Do you have the merchant name only? Have you one-hot encoded the merchant list? Can you turn the merchant name into one or more variables with lower cardinality like groupings for food, clothes, household, sporting, etc?  Can you find other information on merchants (other data), join it into your data and enhance this feature to make it much more useful?

3) Amount can possibly be teased out a bit, but be conservative with this one. You could possibly flag purchases over $100 or $1000 as is_expensive and is_very_expensive.  Also possibly squaring the amount could account for buyer behavior around big ticket items.  This might just add noise, so be careful with this one.

4) Day of week should be first turned into a numerical variable 1-7, but then I would also recommend turning cyclic variables into two spatial dimensions.  I explain how to do this here, and in addition to time, it can be applied to day-of-week also. I recommend keeping both the 1-7 format and the two dimensional spatial format.  Are there other days that you want to specifically highlight?  You could, for instance have an is_weekend variable that is 1 for Saturday and Sunday and 0 for other days. You could have another feature for is_boring_day that is 1 for Monday, Tuesday, and Wednesday and 0 for other days. Play with it, think outside the box... this can be a very useful feature but requires some coaxing at times.

5) Date of purchase is pretty loaded in that you can pull out the months and turn them into cyclic variables.  You can also pull out the day of the month in a similar way.  You can create at payday flag for the 15th and 30th and create a feature like 1/(1+days_since_payday) since people might be more likely to buy on payday. You can add a seasons feature from the date. Finally, you can use raw date in days since your earliest piece of data.

6) Similarly for time, you should turn this into a 2D spatial feature. This will make it easier to pull out the spending behavior of people who shop at certain times of the day.

Clustering Algorithms

k-means can be a bit blunt at times.  I suggest you try at least two more clustering algorithms besides k-means. DBSCAN is a very sophisticated clustering tool and works well for noisy data. It also benefits from not having to specify the number of clusters a priori. Experiment with changing the neighborhood size and how much noise is allowed to see how this effects your clustering. Try at least one more clustering type in the scikit arsenal... they are very easy to swap out once you do it once.

Other Methods:

Finally, you can play with trying to other supervised learning methods. Maybe try separating your data into training and testing sets and try predicting purchase amounts based on the other features.  Or try classifying the type of purchase based on the other information.  Maybe try training a SVM for classification and a Bayesian-ridge regressor for predicting the purchase amount.

Handling changing user data:

Changing data can be handled in a number of different ways.  The most obtuse is simply to retrain your model periodically, while either aggregating all of the data, or forgetting some of the older data due to a lack of relevance. The next step in sophistication is to use an online learning algorithm, which continually updates the model based on new data.

In general, just have fun with it.

Hope this helps!


Is decision tree regression comparable to locally weighted regression

There are two important differences between decision trees and regression:

Decision tree fit a straight a line (mean of the dependent variable for the feature space). Regression fits a sloped line (rise over run).

Decision trees typically do not predict values outside the observed range. Linear regression can predict values outside the observed range.


Neural networks - Find most similar images

I don't think that a high level architecture as such is the best fit but it rather depends on many factors and details. From what I am aware of the first approach is promising especially when extended by additional steps as done in the TiefVision:

An additional bounding box network is used to discriminate the relevant part of the image from the rest
The feature vectors are not simply compared directly but rather used to train a comparison network using triplets (learning similarity based on feature vectors and examples for more and less similar instances).

This work is a lot more recent (2016/17) than what you posted and comes with a nice tool set and a more detailed paper.

Why using triplets aka Deep Ranking?

As stated in the comments: Why should one use triplets for image similarity instead of learning feature vectors and calculate their distance? Triplets are a way of formulating the similarity question as a learning problem instead of learning feature vectors that basically do not care for similarity. This approach makes especially sense in cases where human-perceived similarity is important, which might differ from machine perception.

Triplets work like this: You provide 3 images. One to compare to, one similar (close) and one not so similar (distant) image. This is your training/test/validation data. Training your network on those samples and predicting the correct order (classify similar from non-similar images) overall lets the network learn how to order images based on their similarity.

All in all this approach is comparatively complex. It might be overengineered but you also asked for the best way to do this and Deep Ranking achieves very high precision values.


I think you should first export it with a timestamp and then add a day, month ... other features to the data. This might help in better accuracy.
For more details, you can refer: This article


What is the difference between BatchNorm and Adaptive BatchNorm (AdaBN)?

I think the original batch normalization paper proposes to use mean and standard deviation estimated on the train set. The adaptive batch normalization simply re-estimates them on the target domain (could be the test set, or some unlabeled data from the target domain).

Please correct me if I am wrong.


Shouldn't training and validation loss be approximate before the first epoch is finished?

Is your batch size the length of the full dataset? If you have $N$ samples, and feed in mini-batches of size $k<N$, then batch losses are computed and the model's weights are updated with every $k$ samples. By the end of the first epoch, there may already have been significantly many updates (learning) from the training set. I believe Keras aggregates these batch losses to compute the epoch loss.

If $k=N$ however, it is possible that the distribution of your training and validation sets are different.


How to choose most appropriate Machine Learning model?

Fit a linear model. Using rejected unit % as target. Then see the coefficients of the linear regression to see how much contributes each to the result.

You can adapt your code to this example of sklearn:

https://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html#sphx-glr-auto-examples-linear-model-plot-ols-py

If you are feeling confortable with this, you can move to higher performance/non parametric algorithms as decission trees. Or just use a LASSO regression.


Drawing Neural Network diagram for academic papers

There are few online tools available that give you ability to draw "canned" CNN  diagrams like NN-SVG
EXAMPLE :

Another popular choice seems to be InkScape

There is always PowerPoint


What you describe it is NOT a typical approach for scorecard design.

Typically binary target - "0/1" is used for target variable and logistic regression is typical method to create a scorecard.

Typically to define "default trigger" one fixes two time periods:

1) fix say 1 year from credit start, that means defaults happening after 1 year are not considered as defaults

2) fix say 90 day period - means "default trigger" is 1 if client delays payment for more than 90 days from the payment date, that means you do not consider as "default" if delay is less than 90 days.

There can be some other conditions - like threshold on amount of delinquency - if below - do not consider as "default". And may be some other.

You might consider the book:

Credit Risk Scorecards: Developing and Implementing Intelligent Credit Scoring

by Naeem Siddiqi


AWS EC2 Segmentation Fault (core dumped) issue with fast-style-transfer

I had similar issue.Just use tensorflow-gpu=1.5.0.

Follow these steps:

$ pip uninstall tensorflow-gpu
$ pip install tensorflow==1.5.0
$ pip install numpy==1.14.0
$ pip install six==1.10.0
$ pip install joblib==0.12

Hope this helps!


Why Uniformity of subpixel in image analysis ensures correctness

If the subpixels are not roughly evenly distributed then this means (among others) that certain decimal values are more prevalent than others and this can lead to large gaps in tracking the particle (gaps which can be larger than the particle's size).
So this amounts to saying that the outcome is not very precise.
On the other hand, basic continuity of motion requires that decimal values should be evenly distributed on average.
For example imagine the subpixels have a prevalence of 0.1 and 0.9 decimal values. So for a particle of size 1 its positions may vary by more than 2 pixels (eg 1.1-2.9) so this creates inaccurate trackings on average.


Why multi-class SVM can't detect some classes?

Well, based on how PCA works, and the fact that you're using multi-class SVM, which is a pretty solid family of algorithm, the only possibility i can think of is that the problem comes from your data. It can still depends on the implementation you're using, but i think it come from your data.


Evaluating performance of Generative Adverserial Network?

I think it depends on what exactly you're doing with the GANs. If you're generating images, the two most popular (to my knowledge) are the Inception Score [1] and Frechet Inception Distance [2]. GANs aren't my direct area of study, but I don't think you can miss starting with those two.

Actually, Goodfellow himself endorsed FID (a year ago) on twitter, so I'd start there.


Applying ConvNets to classify motion/video data

Signs are the visual equivalent of "words", and just as words can be decomposed into smaller parts, so can signs be broken down into smaller meaningful/useful units. For example, in American Sign Language (ASL), signs are decomposed into parameters such as handshape, orientation, movement, location on the body. Each parameter can take on a finite set of values, for example handshapes can be closed fist, open hand, cocked index, etc. I would expect a neural network would need to learn these parameters and their values, while learning to ignore distractor and nonsense configurations. Perhaps initial training includes learning to differentiate between signs and non-signs. And just as there are different spoken language, there are different sign languages as well. Moreover, there exist distinct sign language dialects like Black ASL that will need to be taken into account by your system.


The differences between SVM and Logistic Regression

If you use logistic regression and the cross-entropy cost function, it's shape is convex and there will be a single minimum. But during optimization, you may find weights that are near to optimal point and not exactly on the optimal point. This means that you can have multiple classifies that reduce the error and maybe set it to zero for the training data but with different weights which are slightly different. This can lead to different decision boundaries. This approach is based on based on statistical methods. As it is illustrated in the following shape, you can have different decision boundaries with slight changes in the weights and all of them have zero error on the training examples.

What SVM does is an attemption to find a decision boundary that reduces the risk of error on the test data. It tries to find a decision boundary that has the same distance from the boundary points of both classes. Consequently, both classes will have a same space for the empty space which there is no data there. SVM is geometrically motivated rather than statistically.

  None kernelized SVMs are nothing more than linear separators. Therefore, is the only difference between an SVM and logistic regression the criterium to choose the boundary?

They are linear separators and if you find out that your decision boundary can be a hyperplane, it's better to use an SVM for diminishing the risk of error on test data.

  Apparently SVM chooses the maximum margin classifier and logistic regression the one that minimizes the cross-entropy loss.

Yes, as stated SVM is based on geometrical properties of the data whilst logistic regression is based on statistical approaches.

  In this case, are there situations where SVM would perform better than logistic regression, or vice-versa?

Ostensibly, their results are not very different, but they are. SVMs are better for generalization 1, 2.


Dataset to be used to learn NN regressions in classroom experiments

May try these two Kaggle datasets -

camnugent/california-housing-prices
kumarajarshi/life-expectancy-who

Try your question here - https://opendata.stackexchange.com/


Change values of a particular column to value_count()

Try this:

# Do value_count outside of loop to avoid doing it multiple times
vc_item1 = data['Item 1'].value_counts()

data['Item 1'].apply(lambda x: vc_item1[x])

But if understand your comment correctly you want both key and value in each cell:

vc = data['Item 1'].value_counts()

data['Item 1'].apply(lambda x: str(x) + ':' + str(vc[(x)]))


As a long shot, if you have a Python version (or implement it yourself), running it with PyPy instead of Python might make things much faster, as it is well suited to code that uses Python built-ins and also many loops. It optimises these cases very well, through tricks such as garbage collection. The latest version also supports Python3.5 and 3.6 as well as NumPy.

From their website the main benefits:

  Speed: thanks to its Just-in-Time compiler, Python programs often run
  faster on PyPy. (What is a JIT compiler?)

  “If you want your code to run faster, you should probably just use
  PyPy.” — Guido van Rossum (creator of Python)

  Memory usage: memory-hungry Python programs (several hundreds of MBs
  or more) might end up taking less space than they do in CPython.

  Compatibility: PyPy is highly compatible with existing python code. It
  supports cffi and can run popular python libraries like twisted and
  django.

  Stackless: PyPy comes by default with support for stackless mode,
  providing micro-threads for massive concurrency.


Pyspark code is not performant enough when compared to pure python alternative

You have instantiated a sparkContext object with "local" mode configuration. It means you have allocated ressources for a single multi-core Java Virtual Machine on your computer. In this configuration, you can't reach better performs than with python. Because :

With local mode, you have less ressources (cpu, memory...) with Spark than with python (you can't have a JVM with more ressources than your computer).
PySpark code is converted into Scala code before execution.

Spark benefits arise when you use Spark on multiple nodes. In this configuration, spark master is yarn, mesos or standalone. In that case, a spark job would be separated in multiple tasks and each node would be dedicated to different tasks.

For example, if you have 3 nodes and 6 tasks, each node can handle 2 tasks. If you have 6 nodes and 6 tasks, each node will handle a task. With simple python, think that you have only 1 node for "6 tasks". So, for large tasks (big enough datasets), Spark will results in better latencies than simple python.


GAN - am I seeing mode collapse? Common fixes not working

Since this question I have made significant progress, which I have written up as part 3 of the series:

http://makeyourownalgorithmicart.blogspot.com/2019/05/generative-adversarial-networks-part-iii.html

Although my GAN doesn't now have mode collapse - the reasons are still unclear. The post reports visually the impact of combinations of SGD->Adam, none->LayerNorm, Sigmoid->ReLU which I hope others find useful:


Theano logistic regression understanding code

I agree the code may be hard to read at the first glance, but I find the comment pretty clear:

y.shape[0] is (symbolically) the number of rows in y, i.e., number of examples (call it n) in the minibatch
T.arange(y.shape[0]) is a symbolic vector which will contain [0,1,2,... n-1].
T.log(self.p_y_given_x) is a matrix of Log-Probabilities (call it LP) with one row per example and one column per class
LP[T.arange(y.shape[0]),y] is a vector v containing [LP[0,y[0]], LP[1,y[1]], LP[2,y[2]], ..., LP[n-1,y[n-1]]]
T.mean(LP[T.arange(y.shape[0]),y]) is the mean (across minibatch examples) of the elements in v, i.e., the mean log-likelihood across the minibatch.

Everything is Python notation (but sometimes using Theano functions).


Algorithm suggestion for anomaly detection in multivariate time series data

The problem with your data set it that it does contain multiple categorical variables (as far as I can see). Another problem is that the users might do sequences with different lengths and different order (which makes it very difficult to detect suspicious patterns). I would create histograms for each variable and see which categories are common and which are not so common. If you have looked at the descriptives of each variable you should be able to see which variables allow you to discriminate.

A good metric is the entropy (dispersion) $H = -\sum_{l=1}^{L}p_l\ln p_l$ (is 0 if all manifestations of the categorical variable are concentrated at one label; is $\ln L$ if all manifestations are uniformly distributed). and the Gini-index $\text{G}=1-\sum_{l=1}^{L}p^2_l$ (tends to zero if one label is very dominant, becomes larger for uniformly distributed labels for a variable and is bounded by $1-1/L$). The variable $p_l$ is the relative frequency of the $l^{\text{th}}$ manifestation of the categorical variable that we are investigating and $L$ is the number of all possible manifestations of the categorical variable.

The problem with this procedure is that we are not considering the interactions between your variables. But it is the first approach that you could try. If the variables do not correlate that much this might be sufficient.

Without labeled data, it will be very difficult to use machine learning methods to solve this problem.


Reinforcement Learning control with known dynamic equation

Reinforcement learning (RL) is completely based around MDPs, to the point where its definition is essentially "RL is a collection of algorithms that can learn about action choices within a MDP environment".

Outside of RL, you can work with control systems using differential equations more or less directly, and some are solvable analytically. In principle these direct solutions are more robust and require no learning, compared to RL. However, they usually rely on having simple to describe goals - typically static control of maintaining some important value (speed, position, temperature). This works fine for cruise control in cars, thermostats and industrial processes. It also works great for simple environments like inverted pendulum which has been solved without RL for decades.

The analytical non-RL approaches are far more trusted than trial-and-error statistical learners such as RL. But they are limited in terms of describing goals and complexity of environments. They start to fail at the level of the mountain car environment where the correct action may be to move further from a goal state before moving towards it. Mountain car can still of course be fully described by relatively simple differential equations.

If you have equations for a dynamic system in an environment with goals that are too complex to solve analytically, then you can easily convert to discrete MDP form: Use the equations to simulate the environment, choosing a discrete time step for action choice. There are also RL methods that can work with continuous  control and variable time steps, which would also benefit from such a model. You could use such a model to learn in a simulated environment, or use it to help with planning in a real environment (or have a learning + planning algorithm in a simulated environment).

If you are starting with differential equations, then typically you would convert them into some non-differential form in order to apply them. If you can do this fully analytically - e.g. change something in the form $a\frac{d^2x}{dt}  + b\frac{dx}{dt} +c = 0$ into some $x = \alpha e^{\beta t}\text{sin}(\gamma t)$ - then this would make the most accurate predictions and simulations. Otherwise you can use some method of approximation in order to resolve the differential equation into something that predicts next state from current state.


Does Kernels always map data points to higher dimensions

'Another dimension' can be also a lower dimensional space. You are free to choose your kernel. Example is: $k\big((x_1, y_1), (x_2, y_2)\big) = x_1x_2$
Is it true that we can always find a higher dimensions that can linearly seperate data points? Yes. If you have a finite dataset, you can always find a higher dimension that can linearly seperate data points.  The obvious one is mapping n points to a n-dimensional space. If you do any non-trivial mapping (no d points are on a d-1 dimensional hyperspace), then you can always linearly separate any two class assignments.


Pull Random Numbers from my Data (Python)

If what you want is to generate random numbers with the same distribution as your cashflow numbers I recommend you using Python's Fitter package
It is powerful and very simple to use.
You can in this way use it to find the distribution of your data and then generate random numbers with the same distribution.
From documentation:
from scipy import stats
data = stats.gamma.rvs(2, loc=1.5, scale=2, size=10000)

from fitter import Fitter
f = Fitter(data)
f.fit()
# may take some time since by default, all distributions are tried
# but you call manually provide a smaller set of distributions
f.summary()

Also useful resources might be found in stackoverflow


Multivariate time series is an active research topic you will find a lot of recent paper tackling the subject.

To answer your questions, you can use a single RNN. You can input one value for each time step. Nothing keeps you from adding another value at each time step (if your sensor are synchronized). Your model will then learn how to classify with a two dimensional time series.

You check this blog. In your case, only the output is different.

As for the two last points, aggregating the time series into one is risky in the sense that you might lose important information during the process. Finally the main disadvantage of your last point is that you won't be able to use a potential correlation between the two time series for the final classification.


evaluation metrics for multiple values per session

Here is a suggestion:

Standardise everything (if you ommit this than some big number like 9999 can ruin everything), than take average value per user session. Than, optionally, mutliply this number by x/10 for example, where x is the sample size in the use session (think of it like evidence where more samples add more confidence) and finally average by number of sessions for the algorithm.


SparseCategoricalCrossentropy is a class. So you have to define a object first then you can compute the loss using it.
scce = tf.keras.losses.SparseCategoricalCrossentropy()
scce(y_true, y_pred).numpy()

While sparse_categorical_crossentropy is merely a function which can be directly used to compute cost.
loss = tf.keras.losses.sparse_categorical_crossentropy(y_true, y_pred)

If you are to pass the loss to a Sequential API then you must pass the object ,not the function.
model.compile('sgd', loss=tf.keras.losses.SparseCategoricalCrossentropy())


what is correct way to perform normalization on data in Auto encoder?

Min-Max-Scaling will not perform well for your problem, as you already said. For noisy data, scaling according to the quantile range should perform better.

After scaling, you could also try to clip your data to $[-1,1]$, as it it is often done in adversarial learning methods.


This looks quite similar to Named Entity Recognition (NER), which is traditionally done with a sequence labeling model such as Conditional Random Fields. Normally NER is used when:

The list of possible entities is not predefined: the training data might contain "Mr James Smith" but the test data could contain "Mr John Doe". In other words, the classes are open.
It is assumed that the context of the text can help the model predict an entity. For example in a sentence like "Today X said that ...", the word "said" after X should help the model predict that X is either a person or an organization, but it cannot be a location.


Sklearn: applying cost complexity pruning along with pipeline

Pipelines themselves don't generally carry the methods and attributes of the final estimator, aside from basics like predict, predict_proba, transform.  If you need to access a method of a step, you should access the step itself using one of:
pipe[-1]
pipe['decisiontreeclassifier']
pipe.named_steps['decisiontreeclassifier']

However, in this case it's a little trickier, because cost_complexity_pruning_path needs the dataset X, y, but you need your pipeline's transformer to apply to it first.  It's a little cumbersome, but I think this should work and is relatively straightforward:
pipe[-1].cost_complexity_pruning_path(
    pipe[:-1].transform(X),
    y,
)

(Note that pipe[-1] is the final estimator in the pipeline, and pipe[:-1] is every step except the last.)


Mask R-CNN Background Subtraction Implementation

I found this, https://github.com/tony92151/maskrcnn-background-remove
but I didnt use it. I will try using it this week and post more updates here.


Can I apply Dropout In layers other than Fully Connected layers in CNN

In a CNN, each neuron produces one feature map. Since spatial dropout which is normally used for CNN's is per-neuron, dropping a neuron means that the corresponding feature map is dropped.
Pooling usually operates separately on each feature map, so it should not make any difference if you apply dropout before or after pooling.
Yes it should help with preventing overfitting. The added advantage is you can get uncertainty from the network at zero cost if using dropout.


How to create A/B test segements for highly variable data

Your ability to detect a difference depends on the relative standard deviation of your data.  Sometimes the noise is due to seasonal effects, which an AB test does an excellent job in correcting.  However if the metric itself is intrinsically high variance, you will need a large sample size, or adjust your significance and power parameters.
Run a check of your standard deviation and do a power calculation to evaluate your situation.
Keep in mind that there are some statistical distributions which have variances approaching infinite.


Alternative to Flesch for a readability score algorithm

ARI and Coleman-Liau are basically the same thing with different numbers.

Gunning Fog and Dale-Chall are similar in that they're simple functions of how "complicated" the words and sentences are, though at least they appeal to a notion of "complicated words", though that just pushes the problem somewhere else.

There are more and more like this, and I know you're not looking for this.

It may be stating the obvious, but if I were building such a measure, I'd look at things like misspellings and grammar errors. These are fairly possible to check automatically.

Given the success of RNNs in reproducing text in the style of given input, I suspect you could quite successfully apply deep learning to learn what about a text makes it readable, or at least find whether models trained on high or low readability texts seem to find a given new input more probable.


Build a tool for manually classifying training data images

I would recommend building your own database-backed Web app, since you have proprietary data and few (only two?) classes. I would create tables for the images, users, and labels.

user: (id, name)
image: (id, url)
label: (user.id, image.id, time, class)

The label class can be an enum. If you don't want to let users rate the same image multiple times you can drop the time column, and set the first two ID columns as the compound primary key.

If you've never dealt with Web applications and databases it will seem complicated, but it is easy once you get the hang of it. Here is a tutorial. The benefit of this approach is persistence; you can turn off your computer and the start where you left off thanks to the database.

A simpler alternative is to collect all your data in one session using GUI components such as ipywidgets for jupyter, and writing the labels to a file. With this approach you do not get persistence.


Is there a relationship between LDA, linear SVMs and Perceptron?

The brief answers are:

1, 2. No. They depend on different subsets of examples.

3, 4. Yes, but only if it separates classes linearly and you are extremely lucky. Otherwise no.

5, 6. No, because SVM and LDA find only one solution, but perceptron can find many.

Now let me explain.

Decision boundaries of classicl SVM and LDA are calculated offline based on the whole training sample at once. Thus, SVM and LDA will indeed always find exactly one hyperplane, regardless of the order of learning examples.

But these hyperplanes of LDA and SVM are not generally guaranteed to be different. As its name suggests, SVM solution is based only on the support observations, which usually constitute a small fraction of the learning sample. LDA solution, however,  is sensitive to all the learning examples, because it is explicitly based on their classwise means. Thus, you can shift one non-support point a little, and SVM solution will not change, but LDA will. And it means that you cannot expect that SVM and LDA will find the same boundary.

Perceptron, in contrast, is updated online, so its decision boundary depends on the order of learning examples. Moreover, its solutions depends on the initialization of coefficients. Namely, if $w_n$ is vector of coefficients after seeing $n$ learning examples, then:
$$
w_n = w_0 + \lambda \sum_{i=1}^{n} e_i x_i
$$
where $e_i$ equals 1 if $i$'th example was false negative, -1 for false positives, and 0 otherwise, and $\lambda$ is the learning rate. By varying $w_0$ and order of feeding $x_i$ into perceptron, you can achieve many different hyperplanes.

For example, if (by luck!) you set $w_0$ equal to solution of SVM or LDA, and this solution linearly separates your classes, then perceptron will never change this solution, so it will be equal to solution of SVM or LDA.

However, if the solution of SVM or LDA does not fully separate the classes (for LDA, this is may be the case even if the classes are separable), then perceptron will change it with the next misclassified example, so its solution will diverge from SVM or LDA.


Neural network with fixed weights in Python lasagne

I'm not sure what your intention is by setting the weights to zero. Have you looked at dropout layers?

l_hid1 = lasagne.layers.DenseLayer(num_units=200)

l_hid1_drop = lasagne.layers.DropoutLayer(l_hid1, p=0.5)

This should drop 50% of your data from the l_hid1 layer.


Pandas/Python - comparing two columns for matches not in the same row

To clarify, this question is about comparing two columns to check if the 3-letter combinations match.
So, I would approach this in the following manner:
# Extract the 3-letter combinations from column a
df3["a normalised"] = df3["a"].str[:3]

# Then check if what is in `a normalised` is in column b

b_matches = list(df3[df3[“b”].isin(list(df3[“a normalised”]))][“b”].unique())
df3.loc[:, "match"] = False

b_match_idx = df3[df3["a normalised"].isin(b_matches)].index

df3.at[np.array(b_match_idx),"match"] = True

EDIT: The parentheses have now been resolved. Also the .loc warning can now be mitigated.


How do I approach a classification problem where one of the classes is defined by 'not any of the others'

I would use a two-step approach, using the idea of the $\hat{c_4}$ class you mentioned.

In the first step, use a binary classifier(trained on the whole dataset) to decide if a sample belongs to the class $\hat{c_4}$ (i.e. in any non-interesting class). For this, step you could also take a look in outlier detection methods, if the samples belonging in the "interesting" classes are much different than the rest.

If the result is negative, move on to the next step, a new classifier trained only on samples belonging in the classes $c_1,c_2,c_3$ and use that prediction as your final one.

I think that even using a simple clustering approach as a first step (e.g. 4-clustering k-means using as initial centroid values the average centroid $cent_j = \frac{\sum\limits_{x_i\in D: y_i=j}x_i}{\sum\limits_{x_i\in D: y_i=j}1}$  for each $c_1,c_2,c_3, \hat{c_4}$), would still be useful.


Does a neural network continue to change after SGD stops improving?

The evolution of the improvement of the performance of the network if you keep training it depends on:

1) the surface of the loss function with respect to the network parameters.

2) the learning rate, which is the scale factor you apply to the gradient in order to compute the next set of weights.

If the learning rate is too high (with respect to the shape of the loss surface) and you keep training, you may escape a local minimum you just reached.

The problem is that you don't know the loss surface and so we cannot choose the perfect learning rate for it. There are a lot of heuristic tricks to try to improve the exploration behaviour of SGD, like momentum.

So the answer is: it depends. The network could stabilize around a local minimum and so the weights would only suffer small oscillations, or it might ruin everything by escaping the local minimum.


Comparing Non-deterministic Binary Classifiers

A good strategy is to do n times k-fold cross validation, which should give a pretty good estimation of the average performance of both algorithms. What this means is you perform your k-fold cross validation n times, with different random folds each time, and average the results. People commonly use k = n = 10, but the higher the values of k and n the better.


Training models from sklearn using tf.distribute.MirroredStrategy

There are a few approaches that allow you to do basic ML modelling using a GPU.

First of all, in the code as you presented it, the tensorflow MirroredStrategy unfortunately has no effect. It will only work with tensorflow models themselves, not those from sklearn. In fact, sklearn does not offer any GPU support at all.

1. CUML

An Nvidia library that provides some basic ML model types and other things, often offering the exact same API (classes and functions) as SciKit-Learn. Coming from Nvidia, this of course means everything is built with the GPU in mind.

This is part of the larger RAPIDS toolset (incubated by Nvidia). Maybe there are other tools there that can be helpful, like their XGBoost library.

2. Tensorflow / PyTorch + NumPy

These framework are not just for complicated Deep Learning, you can really use them to perform any basic modelling and leverage their GPU support. Their documentation contains examples, otherwise something like Hands On Machine Learning (a book with an accompanying set of Jupyter notebooks) is a nice way to dig in.

These frameworks work well with the normal scientific stack in Python (such as NumPy, Scipy, Pandas) because numpy arrays and the frameworks' Tensor objects are plug-and-play for most cases.

3. Another option:

Stick to sklearn while you are learning about the models, how they work and so on. If you want to just do anything with the goal of learning about about GPU usage, the two options above are the most modern ways to get started.


What is the purpose of multiple neurons in a hidden layer?

To explain using the sample neural network you have provided:

Purpose of the multiple inputs: Each input represents a feature of the input dataset.
Purpose of the hidden layer: Each neuron learns a different set of weights to represent different functions over the input data.
Purpose of the output layer: Each neuron represents a given class of the output (label/predicted variable).

If you used only a single neuron and no hidden layer, this network would only be able to learn linear decision boundaries. To learn non-linear decision boundaries when classifying the output, multiple neurons are required. By learning different functions approximating the output dataset, the hidden layers are able to reduce the dimensionality of the data as well as identify mode complex representations of the input data. If they all learned the same weights, they would be redundant and not useful.

The way they will learn different "weights" and hence different functions when fed the same data, is that when backpropagation is used to train the network, the errors represented by the output are different for each neuron. These errors are worked backwards to the hidden layer and then to the input layer to determine the most optimum value of weights that would minimize these errors.

This is why when implementing backpropagation algorithm, one of the most important steps is to randomly initialize the weights before starting the learning. If this is not done, then you would observe a large no. of neurons learning the exact same weights and give sub-optimal results.

Edited to answer additional questions:

The only reason the neurons aren't redundant is because they've all been "trained" with different set of weights, hence, give a different output when presented with the same data. This is achieved by random initialization and back-propagation of errors.
The outputs from the Orange neurons (use your diagram as an example), are "squashed" by each Blue neuron by applying the sigmoid or Relu function with the trained weights and the output of the orange neurons.


Dealing with normalized regression output

In linear regression, you don't have to normalize the output variable. This is actually why, for example, StandardScaler ignores y input even if you had entered. Also, inverse_transform is for the input variable. The prediction you get should be in your actual output domain.


While I can't tell you why Spark is so slow (it does come with overheads, and it only makes sense to use Spark when you have 20+ nodes in a big cluster and data that does not fit into RAM of a single PC - unless you use distributed processing, the overheads will cause such problems. For example, your program first has to copy all the data into Spark, so it will need at least twice as much memory. Probably even three copies: your original data, the pyspark copy, and then the Spark copy in the JVM. In the worst case, the data is transformed into a dense format when doing so, at which point you may easily waste 100x as much memory because of storing all the zeros).

Use an appropriate - smaller - vocabulary.

There is no use in including every single word, as most of them will never score well in the decision trees anyway!

It's safe to assume that you can omit both very frequent (stop-) words, as well as rare words (using them would be overfitting anyway!). So use min_df=10 and max_df=1000 or so.


How to determine if character sequence is English word or noise

During NLP and text analytics, several varieties of features can be extracted from a document of words to use for predictive modeling. These include the following.

ngrams

Take a random sample of words from words.txt. For each word in sample, extract every possible bi-gram of letters. For example, the word strength consists of these bi-grams: {st, tr, re, en, ng, gt, th}. Group by bi-gram and compute the frequency of each bi-gram in your corpus. Now do the same thing for tri-grams, ... all the way up to n-grams. At this point you have a rough idea of the frequency distribution of how Roman letters combine to create English words.

ngram + word boundaries

To do a proper analysis you should probably create tags to indicate n-grams at the start and end of a word, (dog -> {^d, do, og, g^}) - this would allow you to capture phonological/orthographic constraints that might otherwise be missed (e.g., the sequence ng can never occur at the beginning of a native English word, thus the sequence ^ng is not permissible - one of the reasons why Vietnamese names like Nguyễn are hard to pronounce for English speakers).

Call this collection of grams the word_set. If you reverse sort by frequency, your most frequent grams will be at the top of the list -- these will reflect the most common sequences across English words. Below I show some (ugly) code using package {ngram} to extract the letter ngrams from words then compute the gram frequencies:

#' Return orthographic n-grams for word
#' @param w character vector of length 1
#' @param n integer type of n-gram
#' @return character vector
#'
getGrams <- function(w, n = 2) {
  require(ngram)
  (w <- gsub("(^[A-Za-z])", "^\\1", w))
  (w <- gsub("([A-Za-z]$)", "\\1^", w))

  # for ngram processing must add spaces between letters
  (ww <- gsub("([A-Za-z^'])", "\\1 \\2", w))
  w <- gsub("[ ]$", "", ww)

  ng <- ngram(w, n = n)
  grams <- get.ngrams(ng)
  out_grams <- sapply(grams, function(gram){return(gsub(" ", "", gram))}) #remove spaces
  return(out_grams)
}

words <- list("dog", "log", "bog", "frog")
res <- sapply(words, FUN = getGrams)
grams <- unlist(as.vector(res))
table(grams)

## ^b ^d ^f ^l bo do fr g^ lo og ro
##  1  1  1  1  1  1  1  4  1  4  1

Your program will just take an incoming sequence of characters as input, break it into grams as previously discussed and compare to list of top grams. Obviously you will have to reduce your top n picks to fit the program size requirement.

consonants & vowels

Another possible feature or approach would be to look at consonant vowel sequences. Basically convert all words in consonant vowel strings (e.g., pancake -> CVCCVCV) and follow the same strategy previously discussed. This program could probably be much smaller but it would suffer from accuracy because it abstracts phones into high-order units.

nchar

Another useful feature will be string length, as the possibility for legitimate English words decreases as the number of characters increases.

library(dplyr)
library(ggplot2)

file_name <- "words.txt"
df <- read.csv(file_name, header = FALSE, stringsAsFactors = FALSE)
names(df) <- c("word")

df$nchar <- sapply(df$word, nchar)
grouped <- dplyr::group_by(df, nchar)
res <- dplyr::summarize(grouped, count = n())
qplot(res$nchar, res$count, geom="path",
      xlab = "Number of characters",
      ylab = "Frequency",
      main = "Distribution of English word lengths in words.txt",
      col=I("red"))

Error Analysis

The type of errors produced by this type of machine should be nonsense words - words that look like they should be English words but which aren't (e.g., ghjrtg would be correctly rejected (true negative) but barkle would incorrectly classified as an English word (false positive)).

Interestingly, zyzzyvas would be incorrectly rejected (false negative), because zyzzyvas is a real English word (at least according to words.txt), but its gram sequences are extremely rare and thus not likely to contribute much discriminatory power.


Tfidfvectorizer in R for character n-grams

You should check out the text2vec R package. It has TF-IDF capabilities.

Here's the link to the package info:
text2vec


Data splitting for a binary classification model

My 2 cents:
the number of records in the data set used here is very small. If we have a look into the data set we can see that the target variable split is exactly 50:50 which means the probability is half. Its like flipping a coin to get heads or tail.

The training set contains a known output and the model learns on this data in order to be generalized to other data later on. The dependent variables and the independent variable should be in splatted and  then do a train test fit.

You can use the library from scikit learn as well
from sklearn.model_selection import train_test_split


Training with many CPU cores doesn't improve performance

Have you done any analysis as to what steps in your procedure consume most time? For example, it may be that GPU operations are responsible for most of the time. Then, of course, adding CPU cores will not give any benefit. And have you looked at CPU statistics, i.e. is the full capacity used?


How to do vocabulary estimation based on observed writings?

In my opinion, this estimation cannot be achieved merely based on this plot, because:

From 4000 words onward, the unique words are increasing linearly around 250 per 2K words: (4K, 1.25K), (6K, 1.5K), (8K, 1.75K), (10K, 2K), (12K, 2.25K). So there is not enough evidence to hypothesize an upper-bound for this linear growth,
On average, an adult knows 20K-35K unique words, but this plot goes only up to 2K which is far behind the final expected value. The extrapolation from 2K to 20K is unreliable.

Vocabulary of Shakespeare

The estimation of a person's vocabulary is quite complicated. Below is a paper that estimates the vocabulary of Shakespeare. He had used 31K unique words in all of his writtings. The paper estimates that he knew at least 35K more words which he did not use (at least 66K vocabulary). As you see, the estimated vocabulary is only twice the observation, which sheds light on unreliability of going from 2K to 20K and beyond.

1976 Estimating the number of unseen species - How many words did Shakespeare know


GridSearchCV() to fine tune outputs ValueError and FitFailedWarning

The problem here is that the data train_df or test_df where ever the error is coming has a column which has a string(object) data type. you can't fit string data into a model. Only numerical data is allowed.

try printing
print(train_df.info())
print(test_df.info())

to see which column should be float type but is a string.


What would be the best way to map similar ngrams

You may want to check word2vec algorithm which can map words to a semantic vector space such that words used in similar context will be closer to each other. You can then use cosine similarity on word vectors for calculating similarity.

For extending the similarity comparison to multiple words or sentences, you can look at word mover distance after training word2vec model.

The downside of the above approach is that you need a large text corpus for training word2vec model. You can either use something specific to your domain or use open source datasets (news articles, Project Gutenberg books, wikipedia dumps).


Why is my PCA boomerang-shaped when normalizing?

The problem here is that you are normalizing the output of SVD. It might be clear that this output does not need to be normalized if you consider the output of each step:

Vectorizer: [n_items x n_words] Matrix. Unnormalized.
TFIDF: [n_items x n_words] Matrix with rows normalized to unit Euclidean norm.
SVD: [n_items x dim] Matrix. Columns correspond to each row's loading onto two orthonormal unit vectors.

In short, the output of the third step (SVD) is meaningful as-is and normalization obscures that meaning.

As to why the normalized output appears curved? The math to show this seems a bit involved but my intuition is that normalization is mapping the row weightings onto the ellipsoid manifold used in SVD.


which script can be used to finetune BERT fro squad question answering in hugging face library

A recent PR changed the location of the scripts you are looking for to examples/legacy/question-answering


Do I need to convert booleans to ints to enter them in a machine learning algorithm?

In Python, True and False are cast implicitly into integers:

True == 1     # True!
False == 0    # True!

Although they are not the same objects - you can test this with True is 1, which returns False.

This means that an algorithm running in pure Python should work without conversion. Many libraries/algorithms have some part implemented in C/C++ in the background, in which case you might run into problems.

You could try the model on your Pandas DataFrame as boolean. If it crashes, you know you must convert to integers/floats.

Even if it doesn't crash, you could convert the values to integers or floats and run it for comparison. Here is a short example:

In [1]: import pandas as pd
In [2]: df = pd.DataFrame({'a':[0, 2, 3, 4, 5], 'b':[True, False, True, False, False]})
In [3]: df
Out[3]:
   a      b
0  0   True
1  2  False
2  3   True
3  4  False
4  5  False

Convert everything to boolean

In [4]: df.astype(bool)
Out[4]:
      a      b
0  False   True
1  True   False
2  True    True
3  True   False
4  True   False

In [5]: df.astype(float)
Out[5]:
     a    b
0  0.0  1.0
1  2.0  0.0
2  3.0  1.0
3  4.0  0.0
4  5.0  0.0


Getting prediction labels from TensorFlow 2 ImageGenerator

Using argmax with model.predict is the way to go.
# testdata is the dataframe of Generator
paths = testdata.filenames # Your files path

y_pred = model.predict(testdata).argmax(axis=1) # Predict prob and get Class Indices
classes = testdata.class_indices  # Map of Indices to Class name

from keras.preprocessing import image
a_img_rand = np.random.randint(0,len(paths))   # A rand to pick a rand image
img = image.load_img(paths[a_img_rand])
img = image.img_to_array(img)
from google.colab.patches import cv2_imshow
cv2_imshow(img)
print(f'Class Predicted ---- {list(classes)[y_pred[a_img_rand]]}')