PyTorch offers a range of transformations and you can simply apply a transformation to resize your images when loading your data. For example by using these transforms (using CIFAR10 as an example here which comes with 3x32x32 images too):

# transform for train data
transform_train = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])
    ])

# transform for test data
transform_test = transforms.Compose([
    transforms.Resize(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])
    ])

# load data and apply transforms
train_data = datasets.CIFAR10('data', train=True,
                              download=True, transform=transform_train)
test_data = datasets.CIFAR10('data', train=False,
                             download=True, transform=transform_test)

Both, transforms.RandomResizedCrop(224) and transforms.Resize(224) give the desired size. (the random cropping is applied as data augmentation to further diversify training data) You will find more details on available transforms in the PyTorch documentation.

As a side note: the size requirement is the same for all pre-trained models in PyTorch - not just Resnet18:

  All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225].


Why should I know C++ ,if I am a machine learning engineer?

Many machine learning products emphasize on speed over accuracy, e.g. in autonomous vehicles, a model that predicts in milliseconds with 98% accuracy is more valuable than a model that predicts in seconds with 99.99% accuracy.

While Python provides many great packages for exploring machine learning research and testing proofs of concept, it is significantly slower than C++ and not acceptable for products with real-time requirements. If you want to know why Python is slower, check out this thread.


How gradients are learned for pooling layers in Convolution Neural Network?

1) Will the error subsurface be flat?

One cannot find the hessian or any such error subsurface of the pooling layer because common pooling layers like max and avg pooling do not have any parameters to learn (as you mentioned)!

2) But, we can speak for the effect of the pooling layers on the error surface of the previous layers. The effect is different for different types of pooling layers. For max pooling, it generates a relatively sparser gradient flow for the parameters on the previous layers as only a few of the output from the previous layers are selected during forward propagation. Whereas for average pooling, its allows for a  more smooth gradient flow to all the learnable parameters on the previous layers.


When to enhance a dataset of images with flips and rotations of the images?

Those operations should  be performed on the training data part. We are introducing varieties into the training data set.

If you were to do the rotation and flipping data and then you split the data set with the possibility of the rotated image of the test data being included in the training procedure. You are giving the model a glimpse of the target that you are going to test later, which defeat the purpose of testing it on new unseen data.

The cross validation and test data should be examples of real life data that your model works on. Suppose your test data are flipped in the real life regularly (for example, a car facing different way), then perhaps it can be justified but those flipped images of the test data shouldn't be included in the training procedure.


Apply multivariable linear regression to a dataset in pandas with sklearn

The form of your model is incorrectly specified. You want to model your data so that your independent variables are year (a numeric variable) and state/union (a categorical variable). Your dependent variable will be the population size.

In order to develop such a model you will need to restructure your data so that it is in a long format (see below)

States/Union Territories      Year                Pop
-----------------------------------------------------
Andaman and Nicobar Islands   1951                 31
Andaman and Nicobar Islands   1961                 64
           ...                 ...                ...
Andhra Pradesh                1951              31115
Andhra Pradesh                1961              35983
           ...                 ...                ...
Arunachal Pradesh             1951                307
           ...                 ...                ...

Once the data is transformed into a proper format, you can then fit your model.
I should mention you will need to recode your categorical variable - you can use a method called one hot-encoding to achieve this.

Once your model has been fit, you will be able to predict the expected population size for a given year and Union/State.

The following post Multiple Linear Regression Model in 7 Steps with Python provides a great guide to developing a model.

Hope this helps!


How to analyse the results of cross-validation do determine overfitting

In short, k-fold CV is not about over-fitting. Your samples can never be ideally identical, so you can only conclude that your error is mean±std.
If your model training process is iterative, then you can detect overfitting by checking test score over the course of training.
If you're making hyper-parameter search with k-fold CV, perhaps with many steps, then you can eventually find out that holdout score is much worse than avg. test score. That would be an overfitting too.


How to generate a sentence with exactly N words?

Limit outputs od decoder to N. Not sure how easy it would be, probably a bit digging into official implementation but after that the main "skeleton" of the GPT2 is usable, meaning that all of the pre-training can be reused to produce meaningful sentences.


Yes, it is entirely possible to combine unsupervised learning with the CRF model.  In particular, I would recommend that you explore the possibility of using word2vec features as inputs to your CRF.

Word2vec trains a  to distinguish between words that are appropriate for a given context and words that are randomly selected.  Select weights of the model can then be interpreted as a dense vector representation of a given word.

These dense vectors have the appealing property that words that are semantically or syntactically similar have similar vector representations.  Basic vector arithmetic even reveals some interesting learned relationships between words.
For example, vector("Paris") - vector("France") + vector("Italy") yields a vector that is quite similar to vector("Rome").

At a high level, you can think of word2vec representations as being similar to LDA or LSA representations, in the sense that you can convert a sparse input vector into a dense output vector that contains word similarity information.

For that matter, LDA and LSA are also valid options for unsupervised feature learning -- both attempt to represent words as combinations of "topics" and output dense word representations.

For English text Google distributes word2vec models pretrained on a huge 100 billion word Google News dataset, but for other languages you'll have to train your own model.


The way I see it is because some reason your x limits is different, you could try:

plt.xlim(0,120)


Why I am having odd results with this neural network model

Neural network algorithms are stochastic.
This means they make use of randomness, such as initializing to random weights, and in turn the same network trained on the same data can produce different results.The random initialization allows the network to learn a good approximation for the function being learned.
The most common form of randomness used in neural networks is the random initialization of the network weights. Although randomness can be used in other areas, here is just a short list:

Randomness in Initialization, such as weights.
Randomness in Regularization, such as dropout
Randomness in Layers, such as word embedding.
Randomness in Optimization, such as stochastic optimization.

The Solutions.
The are two main solutions.
Solution #1: Repeat Your Experiment
The traditional and practical way to address this problem is to run your network many times (30+) and use statistics to summarize the performance of your model, and compare your model to other models.
I strongly recommend this approach, but it is not always possible due to the very long training times of some models.
Solution #2: Seed the Random Number Generator
Alternately, another solution is to use a fixed seed for the random number generator.
from numpy.random import seed
seed(1)


Correlation should be as less as possible between different features, because correlated features mean that those features are giving out same kind of information/trend for the predictor to learn. Thus only one of them is actually useful for prediction.

Keeping more number of uninformative features (correlated features) would result in degraded accuracy if your sample size is similar to you feature set size. Feature selection using Recursive Feature elimination or PCA etc. can help you reduce your feature set to optimal size.

We calculate correlation score in predictive analysis between features and Target variable. When using linear regression to model a data set, we first see if the plot between different features and target variable values follow an upward (+ve correlation) or downward trend (-ve correlation) and not scattered randomly. If such a relationship exists then regression modelling on the data would work well.


Getting some data from a csv file into a list of tuples using Python

you can use Pandas library to solve this problem. pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language. further details can be found here:
Pandas Dataframe Documentations


How to model a Bimodal distribution of target variable

To my understanding you should be looking for something like a Gaussian Mixture Model - GMM or a Kernel Density Estimation - KDE model to fit to your data.

There are many implementations of these models and once you've fitted the GMM or KDE, you can generate new samples stemming from the same distribution or get a probability of whether a new sample comes from the same distribution.

In python an example would be like this:(directly taken from here)

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
from sklearn.neighbors import KernelDensity

# Plot a 1D density example
N = 100
np.random.seed(1)
X = np.concatenate((np.random.normal(0, 1, int(0.3 * N)),
                    np.random.normal(5, 1, int(0.7 * N))))[:, np.newaxis]

X_plot = np.linspace(-5, 10, 1000)[:, np.newaxis]

true_dens = (0.3 * norm(0, 1).pdf(X_plot[:, 0])
             + 0.7 * norm(5, 1).pdf(X_plot[:, 0]))

fig, ax = plt.subplots()
ax.fill(X_plot[:, 0], true_dens, fc='black', alpha=0.2,
        label='input distribution')

for kernel in ['epanechnikov', 'tophat', 'gaussian']:
    kde = KernelDensity(kernel=kernel, bandwidth=0.5).fit(X)
    log_dens = kde.score_samples(X_plot)
    ax.plot(X_plot[:, 0], np.exp(log_dens), '-',
            label="kernel = '{0}'".format(kernel))

ax.text(6, 0.38, "N={0} points".format(N))

ax.legend(loc='upper left')
ax.plot(X[:, 0], -0.005 - 0.01 * np.random.random(X.shape[0]), '+k')

ax.set_xlim(-4, 9)
ax.set_ylim(-0.02, 0.4)
plt.show()

In the end the kde model, could be used for sampling new data points or predicting the probability of a new sample to have been generated from this distribution.

You should play around with different kernels in KDE models or number of base distributions in GMMs, along with other parameters to get optimal results for your data.


From the comment specified by the programmer it is clear that he intends to generate negative samples
def get_train_samples(train_mat, num_negatives):
    user_input, item_input, labels = [], [], []
    num_user, num_item = train_mat.shape
    for (u, i) in train_mat.keys():
        user_input.append(u)
        item_input.append(i)
        labels.append(1)
        # negative instances   <============================  FROM THIS COMMENT LINE
        for t in range(num_negatives):
            j = np.random.randint(num_item)
            while (u, j) in train_mat.keys():
                j = np.random.randint(num_item)
            user_input.append(u)
            item_input.append(j)
            labels.append(0)
    return user_input, item_input, labels


Assuming you are doing supervized learning to train a model that when deployed will take text as input and output a label (e.g., topic) or class probability, then what you probably want to do is balanced, stratified sampling. Assuming sufficient labelled data, ensure that your final training set has a balanced number of text examples for each class/label. Depending on your situation, you may need to over/under sample or somehow deal with the problem of highly imbalanced classes (see 8 tactics to combat imbalanced classes).

The simplest NLP approach to use a bag of words technique, simply indicating the presence/absence of a word in the sentence. Thus each sentence becomes represented as vector of length n, where n = the number of unique words in your data set.

data_set <- c("the big dog", "the small cat", "the big and fat cow")
words <- strsplit(data_set, split = " ") #tokenize sentences
words
##[[1]]
##[1] "the" "big" "dog"
##
##[[2]]
##[1] "the"   "small" "cat"
##
##[[3]]
##[1] "the" "big" "and" "fat" "cow"

vec <- unique(unlist(words)) #vector representation of sentences
##[1] "the"   "big"   "dog"   "small" "cat"   "and"
##[7] "fat"   "cow"

m <- matrix(nrow = length(data_set), ncol = length(vec))

for (i in 1:length(words)) { #iterate the index of tokenized sentences
  vec_rep <- vec %in% words[[i]] #create binary word-feature vector
  m[i,] <- vec_rep #update matrix
}

df <- data.frame(m, row.names = NULL)
names(df) <- vec
df
##   the   big   dog small   cat   and   fat   cow
##1 TRUE  TRUE  TRUE FALSE FALSE FALSE FALSE FALSE
##2 TRUE FALSE FALSE  TRUE  TRUE FALSE FALSE FALSE
##3 TRUE  TRUE FALSE FALSE FALSE  TRUE  TRUE  TRUE

Sometimes you can increase performance by adding bi-gram and tri-gram features.

##given: "the big dog"
##unigrams <- {the, big, dog}
##bigrams <- {the big, big dog}
##trigrams <- {the big dog}

Sometimes weighting words by their frequency improves performance or computing the tf-idf.

Another way to increase performance, in my experience, has been custom language feature engineering, especially if the data is from social media sources replete with spelling errors, acronyms, slang, and other word variants. Standard NLP approaches will typically remove stop words (e.g., the, a/an, this/that, etc.) from vector representations (because closed class, high frequency words often don't help discriminate among class/label boundaries). Because vector representations are typically highly dimensional (approximately num of unique words in the corpus/data set), dimensionality reductions techniques can increase performance. For example, one can compute chi-sq, info gain, etc. on a word feature's distribution across classes -- only keep those features/words above some threshold or below some pre-established p value).


Neural Network Learning Rate vs Q-Learning Learning Rate

There is usually only one learning rate active when using a neural network as function approximator in reinforcement learning. The different names $\eta$ and $\alpha$ are just different conventions for the same basic concept.

When you use a function approximator, other than a linear one, in reinforcement learning, then typically you would not use TD error based update like this:

$$\mathbf{w} \leftarrow \mathbf{w} + \alpha[R+\gamma\hat{q}(S', A',\mathbf{w}) - \hat{q}(S, A,\mathbf{w})]\nabla \hat{q}(S, A,\mathbf{w})$$

But you would train your estimator in a supervised learning manner on sampled TD target (which would use $\eta$ param in a neural network):

$$\mathbf{x} = \phi(S, A), y = R+\gamma\hat{q}(S', A',\mathbf{w})$$

You can actually do either if your library supports it - it is certainly possible to calculate $\nabla \hat{q}(S, A,\mathbf{w})$ for a neural network for instance, instead of using an explicit training loss function. However, the two approaches are equivalent ways of expressing the same thing, there is no reason to use both.

There are other parameters used in reinforcement learning that may affect rates of convergence and other properties of learning agents. For example with differential semi-gradient TD learning - which might be an algorithm you would look at for a continuous task - Sutton and Barto present $\beta$ as a separate learning rate for the average reward, distinct from the learning rate of the estimator.

  So are these two parameters redundant? Do I need to worry about even having $\alpha$ as anything other than 1 if I'm already tuning $eta$, or do they have ultimately different effects?

They are essentially the same parameter with a different name.

If you are trying to pass an error value like $R+\gamma\hat{q}(S', A',\mathbf{w}) - \hat{q}(S, A,\mathbf{w})$ (whether multiplied by $\alpha$ or not) into the neural network as a target, then you have got the wrong idea. Instead you want your neural network to learn the target $R+\gamma\hat{q}(S', A',\mathbf{w})$. That's because the subtraction of current prediction and multiplication by a learning rate is built into the neural network training.


Is the F1 Score sensitive to the threshold?

Yes, it is. The formula:

$$F_1 = \frac{2TP}{2TP + FP + FN}$$

There is no symmetry. You have only $TP$ in the nominator which means that $F_1=0$ if $threshold=1$.


It is so called because it classifies the linear sequence, and not because the structure of the graph.


How do I simultaneously select multiple values for k-means in WEKA?

Try writing a program.

It's easy to do a for loop over k.

You will also want to use multiple random seeds.


Is time series forecasting possible with a transformer?

I'm unclear whether transformers are the best tool for time series forecasting. Transformers at the end of the day are just the latest in a series of sequence-to-sequence models with an encoder and decoder. This means that transformers change something to something else.
With time series you aren't changing something to something else, you're trying to find statistical patterns in variations across time e.g do these variations correlate with each other, do they follow trends, cycles, etc. Then you're trying to use these statistical patterns to forecast for future dates. Time series is a lot like regression in that sense, where transformers try to capture relationships between words, and not just in left to right direction (the positional encoding is just extra information for the network).


ValueError: Input 0 of layer sequential is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: [None, 25, 25, 1]

I have solved the problem by changing the shape of my dataset using:
tf.reshape(data, [25, 25])


Multitask learning NN only trains on a few tasks

So, it turns out that my problem had a lot more to do with network structure than anything else. I was trying to share too many lower levels, and then still not leaving enough room for the later branches to learn individual features/locations. In the end, I re-structured my network so that there was only 1-2 shared convolutional layers, then each primary branch shared 1 further convolutional layer before again each branching into 3 sub-branches (one for each output), each with their own convolutional, dense, flatten and then softmax layers. I also possibly had too many filters on each convolutional layer.

This enabled 'training' across each of the 9 outputs, but has left me with some serious overfitting to try to overcome, due to the depth of the network and the small data set. I am currently trialing methods of correcting for this.


How to Write Multiple Data Frames in an Excel Sheet

an example to write in same sheet:

import pandas as pd

data1 = """
class    precision   recall
<18      0.0125         12
18-24    0.0250         16
25-34    0.00350         4
"""
data2 = """
class    precision   recall
<18      0               0
18-24    0.25            6
25-34    0.35            5
"""

#create 2 df for sample
df1 = pd.read_csv(pd.compat.StringIO(data1), sep='\s+')
df1.name = "Dataframe1"
df2 = pd.read_csv(pd.compat.StringIO(data2), sep='\s+')
df2.name = "Dataframe2"
print(df1);print(df2)

writer = pd.ExcelWriter('e:\\test.xlsx',engine='xlsxwriter')
workbook=writer.book
worksheet=workbook.add_worksheet('Result')
writer.sheets['Result'] = worksheet
worksheet.write_string(0, 0, df1.name)

df1.to_excel(writer,sheet_name='Result',startrow=1 , startcol=0)
worksheet.write_string(df1.shape[0] + 4, 0, df2.name)
df2.to_excel(writer,sheet_name='Result',startrow=df1.shape[0] + 5, startcol=0)

output:

if you want to write in different sheets:

import pandas as pd

# Create a Pandas Excel writer using XlsxWriter as the engine.
writer = pd.ExcelWriter('e:\\test.xlsx', engine='xlsxwriter')

# Write each dataframe to a different worksheet. you could write different string like above if you want
df1.to_excel(writer, sheet_name='Sheet1')
df2.to_excel(writer, sheet_name='Sheet2')

# Close the Pandas Excel writer and output the Excel file.
writer.save()


How do I pick "K" for precision at K and recall at K?

Precision and Recall can be dubious at times. It depends on the point on the ROC Curve. Therefore, pick the point as per your need. The best way is to give more information about your results; i.e., Accuracy measure and Confusion matrix.


How to preprocess data in a data table? (e.g. change a value in each row for some feature)

To scale the data, you can use Preprocess widget. To construct new features, e.g.

new_feature = old_feature - some_values

you can use Feature Constructor widget.


Your problem sounds like the classical top-N personal recommendation to me. There're lots of possibilities in the literature, for example:

User-based collaborative filtering
Content-based collaborative filtering
Matrix factorisation

You'll need to read the literature and figure out which one is better for you.


Mathematically prove why sparsity leads to model overfitting

I don't think sparsity in itself cause the model to over-fit, but it can increase the chance of it.
Overfitting is not a state in itself. What I meant is that if I say that my training accuracy is 92%. Can you predict if it's over-fit?
 It's a relative state i.e. you can only know it if I tell the accuracy on test/new data.
It means, farther the new data from the training data more is the chance that the model will Over-fit.
In a high dimension space, let's say a 10 features dataset -
To fill the space evenly you need 1010 record which is 10Bn. Normally you might have 1Mn.
So, it's almost 9999/10000 chance that you will get a new data outside the training data.
Definitely the model will have a very high chance to fail(Assuming no Regularization in place)(I am not saying that it will fail 9999 times)
But
 This is completely based on the assumption that every feature will have values spanning the full space.
 Let's say 6 out of 10 features have only 2 possible value. Then we will have only 6.4Mn datasets required to fill the space evenly. In this case, there is very less chance of the model to be over-fit compared to the last case.
Although the model will be very complex and wiggly(a lot of branches/leaves), but not a lot over-fitted


Classify phrases as biomedical or non-biomedical

Yes. There has been work going on in the healthcare domain which involves NLP.

Resources like Pubmed have been put together for the exact same purpose.

You can make use of the freely available Pubmed vectors for training your models like maybe a sentence tagging style model for identification. We(my team at work) use them for identification of medical terms like diseases, medicines, etc.


Objective function for multi-label classification

What you're looking for is called "cost-sensitive classification". Most methods however don't work with label similarities, but rather with relative penalties for different types of misclassifications.


How can I vectorize this code in R? Maybe with the apply() function?

First of all it should be noted that the code you posted does not actually replicate the output of the dist function, because the line:

distancematrix[i, j] <- sum(abs(myMatrix[i,] - myMatrix[j,]))

does not calculate the Euclidean distance; it should be:

distancematrix[i, j] <- sqrt(sum((myMatrix[i,] - myMatrix[j,]) ^ 2))

Here are two solutions that rely on apply. They are simplified, and in particular do not take advantage of the symmetry of the distance matrix (which, if considered, would lead to a 2-fold speedup). First, generate some test data:

# Number of data points
N <- 2000
# Dimensionality
d <- 10
# Generate data
myMatrix = matrix(rnorm(N * d), nrow = N)

For convenience, define:

# Wrapper for the distance function
d_fun <- function(x_1, x_2) sqrt(sum((x_1 - x_2) ^ 2))

The first approach is a combination of apply and sapply:

system.time(
    D_1 <-
        apply(myMatrix, 1, function(x_i)
            sapply(1:nrow(myMatrix), function(j) d_fun(x_i, myMatrix[j, ]))
    )
)

   user  system elapsed
 14.041   0.100  14.001

while the second uses only apply (but going over the indices, which are paired using expand.grid):

system.time(
    D_2 <-
        matrix(apply(expand.grid(i = 1:nrow(myMatrix), j = 1:nrow(myMatrix)), 1, function(I)
            d_fun(myMatrix[I[["i"]], ], myMatrix[I[["j"]], ])
        )
    )
)

   user  system elapsed
 39.313   0.498  39.561

However, as expected both are much slower than dist:

system.time(
    distancematrix <- as.matrix(
        dist(myMatrix, method = "euclidean", diag = T, upper = T)
    )
)

   user  system elapsed
  0.337   0.054   0.388


Is max_depth in scikit the equivalent of pruning in decision trees?

Is this equivalent of pruning a decision tree?

Though they have similar goals (i.e. placing some restrictions to the model so that it doesn't grow very complex and overfit), max_depth isn't equivalent to pruning. The way pruning usually works is that go back through the tree and replace branches that do not help with leaf nodes.

  If not, how could I prune a decision tree using scikit?

You can't through scikit-learn (without altering the source code).
Quote taken from the Decision Tree documentation: Mechanisms such as pruning (not currently supported)

If you want to post-prune a tree you have to do it on your own:
You can read this excellent post detailing how to do so.


minimization with a negative cost function: works in MATLAB, not in Python

The problem is with division on python 2.x.  In python 2.x, division involving two integers produces an integer result.  So 1/2==0.  Python 3.x does not have this problem, 1/2==.5.

There are two ways to avoid this.  First, you can always convert one value you a float.  So 1./2==0.5 and 1/2.==0.5.  However, you have to remember to do this everywhere, and if you forget it can lead to hard-to-find errors.

The more reliable method is to always put this at the top of your code: from __future__ import division.  This will switch python 2.x to the python 3 behavior, so 1/2==.5.  In python 3.x it does nothing, so it also makes your code python 3.x compatible in this regard.


Is smoothing in NLP ngrams done on test data or train data?

Is smoothing in NLP ngram done on test data or train data?

In short: both.
Smoothing consists in slightly modifying the estimated probability of an n-gram, so the calculation (for instance add-one smoothing) must be done at the training stage since that's when the probabilities of the model are estimated.
But smoothing usually also involves differences at the testing stage, in particular for assigning a probability to unknown n-grams instead of 0.


Training Objective of language model for GPT3

This may be best understood with a bit more of context from the article:

A more fundamental limitation of the general approach described in this paper – scaling up any LM-like model, whether
autoregressive or bidirectional – is that it may eventually run into (or could already be running into) the limits of the
pretraining objective. Our current objective weights every token equally and lacks a notion of what is most important to
predict and what is less important. [RRS20] demonstrate benefits of customizing prediction to entities of interest.

I think that the relevant part of the reference [RRS20] is this paragraph:

Recently, Guu et al.(2020) found that a “salient span masking” (SSM)
pre-training objective produced substantially better results in open-domain question answering.
This approach first uses BERT (Devlin et al., 2018) to mine sentences that contain salient spans
(named entities and dates) from Wikipedia. The
question answering model is then pre-trained to reconstruct masked-out spans from these sentences,
which Guu et al. (2020) hypothesize helps the
model “focus on problems that require world
knowledge”. We experimented with using the
same SSM data and objective to continue pretraining the T5 checkpoints for 100,000 additional
steps before fine-tuning for question answering.

With that context in mind, I understand that the sentence in the GPT-3 papers means that in normal language models, the predictions of every token has the same importance weight toward the computation of the loss, as the individual token losses are added together in an unweighted manner. This as opposed to the salient span masking approach, which finds tokens that are important to predict by means of a BERT-based preprocessing.


One Hot encoding for large number of values

If you really care about the number of dimensions, you still can try to apply a dimensionality reduction algorithm, such as PCA (Principal Component Analysis) or LDA (Linear Discriminant Analysis), after your one hot encoding.

But know that "56 features" isn't really large and it's highly common in the industry to have thousands, millions or even billions of features.


How do I convert an L2 norm to a probability?

I guess you either need to train directly into probability space, or train another model on top of the feature vectors to learn the probability of two vectors belonging to the same person.

All the other metrics you could use on top of the feature vectors are a bit arbitrary, as they were not considered by the cost function used when training the original model.


What is difference between a Data Scientist and a Data Analyst?

Please visit this and read the difference between Data analyst and Data scientist.
You will find the above link very interesting that fulfills your need as you will also find the words by Data scientist of LinkedIn.
Important lines from the above link:

Data analysts are masters in SQL and use regular expression to slice and dice the data. With some level of scientific curiosity data analysts can tell a story from data. A data scientist on the other hand possess all the skills of a data analysts with strong foundation in modelling, analytics, math, statistics and computer science. What differentiates a data scientist from a data analyst is the strong acumen along with the ability to communicate the findings in the form of a story to both IT leaders and business stakeholders in such a way that it can influence the manner in which a company approaches a business challenge.


selecting a number of neurons specifically for RL

Deep reinforcement learning (DRL) methods such as DQN involve deep learning as a large memory that is not expensive. The neural network learns in a supervised way to provide good estimates; and DQN is not an exception.

Hyper-parameters optimization for the neural network in DQN is similar to that of fully supervised learning. you should try various hyper-parameters[ number of layers, neurons,...etc] until obtaining a good solution.

Evolutionary algorithms can help you find appropriate hyper-parameters. Recently there are some published papers reported using evolutionary algorithms to train the neural network in RL.

are we allowed to change the environment and expect agent to perform well, generalizing to new environment?

Yes,the main idea of DQN is to allow generalization, see DeepMind publications such as [1]. And generalization to unseen environments is possible : see [2], [3]

As you can see in the literature, the boundaries of the generalization differ form one work to another. Generalization might mean the ability to perform well for different experiments at the same environment. Some researcher work on building agents that generalize to unseen environments which is more difficult and in this case the design of the neural network and the reinforcement learning algorithm play important role in determining the generalization ability.

is selecting neurons even a thing for RL, can we just give it a max possible number of neurons and layers, letting it learn correct movement of creature's limbs?

Generalization is an issue,  large number of neurons can overfit (not generalize well) DQN involves neural network in order to allow generalization. The ability of the neural network to generalize to new cases (and new experiments/environments) depends on the approximation that the neural network provides which depends on the neural networks parameters and hyper-parameters.

For building a huge neural network : as a concept you can increase the depth of the neural network to get better results. However, the question is do you have enough data? in RL language, you need more different examples for the larger network to train. Theoretically, having infinite neural networks with infinite number of different experiments/environments will lead to the best results. however, we are not patient enough to wait for very long training time and most probably you don't have large number of different environments/experiments to teach the neural network. hence the standard way to do this is by starting with a small network (few neurons and 1 or 2 layers) and train it then expand it gradually step by step and stop when the network can't generalize well.

you can think of the size of the neural network as a water tank as long you have large amounts of water(experiments) you need larger tank to occupy it. A large network without sufficient number of learning experiments leads to worse results specially when the the agent face new unseen experiments.

:[1] https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf

:[2] https://www.semanticscholar.org/paper/Virtual-to-real-deep-reinforcement-learning%3A-of-for-Tai-Paolo/494af0cedf1abb2454d457d0a89e21b983233276

:[3] https://arxiv.org/pdf/1612.05533.pdf


Can points not being passed to the loss function influence it?

1.Technically they are the same.They will compute the same loss. Although there will be a subtle difference in performance. In the first case gradient will only be calculated for the subset but in the second case gradient will be calculated for the whole training set but only the gradient of subset will be used to compute cost and update.
2.No, the points won't be invisible to the training procedure. the gradient will be calculated for all of the points but they won't be used.


When does it make sense to choose gradient descent for SVM over liblinear?

As mentioned in the comment, if the question is when does it make sense to use coordinate descent over stochastic gradient descent, then, one advantage with coordinate descent is that, it updates only one parameter at a time. Thus, when the data has a very very large number of features, it might make sense to use Coordinate Descent over stochastic gradient descent as SGD will try to update all model parameters at the same time. Also, for huge amounts of data CD might have lower computational complexity as against SGD. Also, it makes sense to use Coordinate descent for optimization when the function is not differentiable. Although, there are ways to convert non-differentiable functions to differentiable ones, still Coordinate descent might be one of the ways to move forward.For example, coordinate descent is a good technique to optimize a loss with L1 penalty. It is considered state of the art for Lasso regression as well as for linear SVMs.


This problem is an instance of the more general assignment problem. In the assignment problem you have:

  The assigment problem has a number of agents and a number of tasks. Any agent can be assigned to perform any task, incurring some cost that may vary depending on the agent-task assignment. It is required to perform all tasks by assigning exactly one agent to each task and exactly one task to each agent in such a way that the total cost of the assignment is minimized.

In your description the agents are drivers and the tasks are "driving to the pickup location", the cost can be either the eta or the distance of the driver's location to the pickup location.

There are a number of algorithms for solving this problem, one of the most famous is the Hungarian algorithm also known as Kuhn-Munkres algorithm.  There is an implementation of the algorithm in the scipy library (this is in Python), under the name of linear_sum_assignment, the documentation is here.


Random Forest Regression. How to represent really long list of categories for processing

If the problem is that you don't know all the categories in advance and different records have different categories, then the hashing trick might help.


How to reach continue training in xgboost

In the fit method, parameter xgb_model can be specified to continue training an old model.

https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBRegressor.fit

Edit: Yes, as I understand it, the old trees are preserved in this method.  The new trees will fit the residuals of whatever data you pass: you could continue with only the new data, which will first be run through the existing trees and have their residuals fit by new trees; or continue with all the data, which will do the above but also use the old data when splitting (probably better, so that the model doesn't lose its fit to the old data, but maybe your use case cares about the new data substantially more?).  In some cases, it may be better to just retrain from scratch (if the existing trees do a poor job on the new data).


For which real world data sets does DBSCAN surpass K-means.?

I play around quite a bit with location data and have found examples both where k-means works fine and where k-means is a poor representation and DBSCAN is a great fit.

If you've ever gone hiking or mountain climbing on a day with fog or a low cloud cover, there are times where you get to the top of the peak and can only see the surrounding peaks poking up through the clouds.  I like to use this analogy when I think of DBSCAN.  The density filtering allows one to select a threshold where data is kept, and all of the remaining data is filtered out.

Take a look at this Seattle crime incident data.  Suppose I want to cluster the data by location to form pseudo neighborhoods i.e. neighborhoods that are roughly defined the the geography of where criminal events tend to occur. This is an example of k-means working just fine in location analytics:

Seattle crime data superimposing a primitive map with k-means clustering

If you know Seattle at all, you can see that the clusters tend to pick out neighborhoods and this works quite well in breaking up neighborhoods.  Now suppose I want to pick out the high crime areas of Seattle in order to identify the hot spots. No matter how I adjust k, the k-means clustering doesn't really provide any additional insight.  But the density filtering in DBSCAN does a wonderful job of identifying high crime areas:

Seattle crime data superimposing a primitive map with DBSCAN clustering

I think this gives the jest of some strengths that can be exploited using the two algorithms.  There's nothing special about the crime data.  For what its worth, I have a tutorial that goes through the analytics of identifying a user's home and work locations from their cell phone GPS pings. This is another case where DBSCAN was very useful, but the DBSCAN specific parts a buried a ways down in the tutorial.

Hope this helps!


Can we look just at the other features when we have a missing vaue?

Try a few options and pick the best.

Fill in the mean/median from known $c$'s in the $a,b,c$ model
Create a linear regression model to predict $c$ from $a,b$, fill the estimate $c$ from this model in the $a,b,c$ model
Train both models and use the applicable one

Personally, I think 2. will work best in most cases, but 1. can already be sufficient when $c$ is not correlated to $a$ or $b$.


How exactly do Gaussian Processes (square dist kernel) enforce smoothness? (Aka how are they computed to do so?)

It's by definition, when you fit a guassian process you specify the mean function m(x) and the covariance function (or kernel) k(x,x'). Often the mean function is 0 and the covariance is the radial basis function or squared expential kernel which is smooth (in fact infinitly differentiable).


I think you are looking for musical optical character recognition software. A google search pointed me to Wikipedia where links to two open source applications resides: Audiveris and OpenOMR. I can not vouch for either program.


Need to find out algorithim for entity recognition

For this kind of task if try to build your custom ner model it would be much better. For references you can check my kaggle notebook for custom model of ner which will give you head start in designing your own dataset & preparing dataset for modeling.

Kaggle Notebook Link


Is each form of word classification also considered to be '(named) entity recognition'?

Is this a fair assumption?

No: Named Entity Recognition (NER) is a specific task which consists in detecting named entities. The more general term for this kind of task in Machine Learning is sequence labelling, because it's not only about classifying words but annotating a sequence of instances in which order matters (e.g. words).
It's true that NER is certainly the most famous task of this kind, but there are other important ones, for instance Part-Of-Speech (POS) tagging.


How do I calculate a similarity matrix with a Student-t kernel?

You can use dt from the stats to get the density of a Student-t distribution. See the help page for extra information about this, and related, functions.

An example, showing the Student-t distrib

library(stats)

xs = seq(-5, 5, .1)
density = dt(xs, df=1)

plot(xs, density)


Predicting next element of a sequence given small amount of data

You might find the link helpful.
https://towardsdatascience.com/how-to-model-time-series-data-with-linear-regression-cd94d1d901c0


Correct,  all categorical features should be encoded into binary digits so ML algo have more predictive power as categorised features cannot have order or magnitude ( be careful also about
multicollinearity if you use regression)
Some ML framework such as Catboost automatically encode features for you if you specify the feature index. I also like using python patsy to create input matrices


"Each agent was evaluated every 250,000 training frames for 135,000 validation frames" What does this sentences stands for? in DQN nature paper?

0: After 250,000 steps in the training loop they start a validation environment to check the current performance without fitting the weights of the current neural networks.

2: One episode of a validation episode just takes at most 135,00 frames (steps) and were not truncated by time.

1: Some games are more difficult to learn than others and the DQN approach is more suitable to some games than to others.


I think you might be misunderstanding the phrase "extract" here. Think of it as "gets activated by" instead.
For example, the "nose filter" gets activated by inputs which look like a human nose (more precisely: it gets activated by activation maps of previous layers which correspond to a nose in the input image). And simply put, the following higher level feature maps in your example, which might encode a human face, then get activated if the previous layers contain activated feature maps for a human nose, mouth, eye etc. This article explains it well.
As NNs in general, this process is loosely inspired by how visual perception in humans and other animals works.


What is the purpose of standardization in machine learning?

Considering 3 points A,B & C with x,y co-ordinates (x in cm, y in
  grams) A(2,2000), B(8,9000) and C(10,20000), the ranking of the points
  as distance from origin for example (or any other point), will be the
  same whether the y values are in grams,pounds, tonnes or any other
  combinations of units for both x and y so where's the need to
  standardise.

This is true for the example you provided, but not for euclidean distance between points in general. Look at this example:

def euclidian_distance(a, b):
    return ((a[0] - b[0])**2 + (a[1] - b[1])**2)**0.5

a1 = 10   #10 grams
a2 = 10  #10 cm
b1 = 10   #10 gram
b2 = 100  #100 cm
c1 = 100  #100 gram
c2 = 10   #10 cm

# using (grams, cm)
A_g_cm = [a1, a2]
B_g_cm = [b1, b2]
C_g_cm = [c1, c2]

print('[g, cm] A-B:', euclidian_distance(A_g_cm, B_g_cm))
print('[g, cm] A-C:', euclidian_distance(A_g_cm, C_g_cm))

# using (kg, cm)
A_kg_cm = [a1/1000, a2]
B_kg_cm = [b1/1000, b2]
C_kg_cm = [c1/1000, c2]

print('[kg, cm] A-B:', euclidian_distance(A_kg_cm, B_kg_cm))
print('[kg, cm] A-C:', euclidian_distance(A_kg_cm, C_kg_cm))

# using (grams, m)
A_g_m = [a1, a2/100]
B_g_m = [b1, b2/100]
C_g_m = [c1, c2/100]

print('[g, m] A-B:', euclidian_distance(A_g_m, B_g_m))
print('[g, m] A-C:', euclidian_distance(A_g_m, C_g_m))

# using (kilo, m)
A_kg_m = [a1/1000, a2/100]
B_kg_m = [b1/1000, b2/100]
C_kg_m = [c1/1000, c2/100]

print('[kg, m] A-B:', euclidian_distance(A_kg_m, B_kg_m))
print('[kg, m] A-C:', euclidian_distance(A_kg_m, C_kg_m))

Output:

[g, cm] A-B: 90.0
[g, cm] A-C: 90.0
[kg, cm] A-B: 90.0
[kg, cm] A-C: 0.09000000000000001
[g, m] A-B: 0.9
[g, m] A-C: 90.0
[kg, m] A-B: 0.9
[kg, m] A-C: 0.09000000000000001

Here you can clearly see that the choice of units influence the ranking, thus the need for standardization.


How do we ensure that a model generalize well in the real world

You should try out the model that you built in real world by using test data that reflect real data.

Machine learning is an iterative process, you have to try it out and see the performance.

Your test data should be extended to reflect real life scenario that you expect it to work well.
You might want to change your objective function sometimes, your existing objective function might not have capture some real life factor. For example, if you see your model keep making mistakes on certain type of data, you might want to penalize it more.
Your model might not last forever, real life data might change over time and you have to tune your model to adapt to it. Your model might not have forseen some scenarios and might not know what should be the correct response to be.


How to set and check normal distribution on a data set?

It's hard to be sure of anything without having a glimpse at your data but what I assume from what you are saying is:

you have 2 classes which probably are  given as 1 columns

you have 29 features

you have 200 samples
Edit: Then while writing my answer I'm not too sure what you wanna do. If you have 2 classes it is likely that you are about to do some classification tasks so I don't really get what you want to modelize with a normal distribution.


How to best summarize and visualize the outcome and performance of a multi-class problems?

I would do a big confusion matrix and, by inspecting it, decide what classes to add to a smaller confusion matrix in order to clearly illustrate the main sources of confusion for the model.


When you simply use say plt.savefig() to jpg, matplotlib by default does not save it to a binary image as you require. JPG by default the definition is values ranging from 0-255.
Suggest using PIL instead of matplotlib then you can save as BW binary image.


Role derivative of sigmoid function in neural networks

The use of derivatives in neural networks is for the training process called backpropagation. This technique uses gradient descent in order to find an optimal set of model parameters in order to minimize a loss function. In your example you must use the derivative of a sigmoid because that is the activation that your individual neurons are using.

The loss function

The essence of machine learning is to optimize a cost function such that we can either minimize or maximize some target function. This is typically called the loss or cost funtion. We typically want to minimize this function. The cost function, $C$, associates some penalty based on the resulting errors when passing data through your model as a function of the model parameters.

Let's look at the example where we try to label whether an image contains a cat or a dog. If we have a perfect model, we can give the model a picture and it will tell us if it is a cat or a dog. However, no model is perfect and it will make mistakes.

When we train our model to be able to infer meaning from input data we want to minimize the amount of mistakes it makes. So we use a training set, this data contains a lot of pictures of dogs and cats and we have the ground truth label associated with that image. Each time we run a training iteration of the model we calculate the cost (the amount of mistakes) of the model. We will want to minimize this cost.

Many cost functions exist each serving their own purpose. A common cost function that is used is the quadratic cost which is defined as

$C = \frac{1}{N} \sum_{i=0}^{N}(\hat{y} - y)^2$.

This is the square of the difference between the predicted label and the ground truth label for the $N$ images that we trained over. We will want to minimize this in some way.

Minimizing a loss function

Indeed most of machine learning is simply a family of frameworks which are capable of determining a distribution by minimizing some cost function. The question we can ask is "how can we minimize a function"?

Let's minimize the following function

$y = x^2-4x+6$.

If we plot this we can see that there is a minimum at $x = 2$. To do this analytically we can take the derivative of this function as

$\frac{dy}{dx} = 2x - 4 = 0$

$x = 2$.

However, often times finding a global minimum analytically is not feasible. So instead we use some optimization techniques. Here as well many different ways exist such as : Newton-Raphson, grid search, etc. Among these is gradient descent. This is the technique used by neural networks.

Gradient Descent

Let's use a famously used analogy to understand this. Imagine a 2D minimization problem. This is equivalent of being on a mountainous hike in the wilderness. You want to get back down to the village which you know is at the lowest point. Even if you do not know the cardinal directions of the village. All you need to do is continuously take the steepest way down, and you will eventually get to the village. So we will descend down the surface based on the steepness of the slope.

Let's take our function

$y = x^2-4x+6$

we will determine the $x$ for which $y$ is minimized. Gradient descent algorithm first says we will pick a random value for $x$. Let us initialize at $x=8$. Then the algorithm will do the following iteratively until we reach convergence.

$x^{new} = x^{old} - \nu \frac{dy}{dx}$

where $\nu$ is the learning rate, we can set this to whatever value we will like. However there is a smart way to choose this. Too big and we will never reach our minimum value, and too small we will waste soooo much time before we get there. It is analogous to the size of the steps you want to take down the steep slope. Small steps and you will die on the mountain, you'll never get down. Too large of a step and you risk over shooting the village and ending up the other side of the mountain. The derivative is the means by which we travel down this slope towards our minimum.

$\frac{dy}{dx} = 2x - 4$

$\nu = 0.1$

Iteration 1:

$x^{new} = 8 - 0.1(2 * 8 - 4) = 6.8 $
$x^{new} = 6.8 - 0.1(2 * 6.8 - 4) = 5.84 $
$x^{new} = 5.84 - 0.1(2 * 5.84 - 4) = 5.07 $
$x^{new} = 5.07 - 0.1(2 * 5.07 - 4) = 4.45 $
$x^{new} = 4.45 - 0.1(2 * 4.45 - 4) = 3.96 $
$x^{new} = 3.96 - 0.1(2 * 3.96 - 4) = 3.57 $
$x^{new} = 3.57 - 0.1(2 * 3.57 - 4) = 3.25 $
$x^{new} = 3.25 - 0.1(2 * 3.25 - 4) = 3.00 $
$x^{new} = 3.00 - 0.1(2 * 3.00 - 4) = 2.80 $
$x^{new} = 2.80 - 0.1(2 * 2.80 - 4) = 2.64 $
$x^{new} = 2.64 - 0.1(2 * 2.64 - 4) = 2.51 $
$x^{new} = 2.51 - 0.1(2 * 2.51 - 4) = 2.41 $
$x^{new} = 2.41 - 0.1(2 * 2.41 - 4) = 2.32 $
$x^{new} = 2.32 - 0.1(2 * 2.32 - 4) = 2.26 $
$x^{new} = 2.26 - 0.1(2 * 2.26 - 4) = 2.21 $
$x^{new} = 2.21 - 0.1(2 * 2.21 - 4) = 2.16 $
$x^{new} = 2.16 - 0.1(2 * 2.16 - 4) = 2.13 $
$x^{new} = 2.13 - 0.1(2 * 2.13 - 4) = 2.10 $
$x^{new} = 2.10 - 0.1(2 * 2.10 - 4) = 2.08 $
$x^{new} = 2.08 - 0.1(2 * 2.08 - 4) = 2.06 $
$x^{new} = 2.06 - 0.1(2 * 2.06 - 4) = 2.05 $
$x^{new} = 2.05 - 0.1(2 * 2.05 - 4) = 2.04 $
$x^{new} = 2.04 - 0.1(2 * 2.04 - 4) = 2.03 $
$x^{new} = 2.03 - 0.1(2 * 2.03 - 4) = 2.02 $
$x^{new} = 2.02 - 0.1(2 * 2.02 - 4) = 2.02 $
$x^{new} = 2.02 - 0.1(2 * 2.02 - 4) = 2.01 $
$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.01 $
$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.01 $
$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $

And we see that the algorithm converges at $x = 2$! We have found the minimum.

Applied to neural networks

The first neural networks only had a single neuron which took in some inputs $x$ and then provide an output $\hat{y}$. A common function used is the sigmoid function

$\sigma(z) = \frac{1}{1+exp(z)}$

$\hat{y}(w^Tx) = \frac{1}{1+exp(w^Tx + b)}$

where $w$ is the associated weight for each input $x$ and we have a bias $b$. We then want to minimize our cost function

$C = \frac{1}{2N} \sum_{i=0}^{N}(\hat{y} - y)^2$.

How to train the neural network?

We will use gradient descent to train the weights based on the output of the sigmoid function and we will use some cost function $C$ and train on batches of data of size $N$.

$C = \frac{1}{2N} \sum_i^N (\hat{y} - y)^2$

$\hat{y}$ is the predicted class obtained from the sigmoid function and $y$ is the ground truth label. We will use gradient descent to minimize the cost function with respect to the weights $w$. To make life easier we will split the derivative as follows

$\frac{\partial C}{\partial w} = \frac{\partial C}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial w}$.

$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y$

and we have that $\hat{y} = \sigma(w^Tx)$ and the derivative of the sigmoid function is $\frac{\partial \sigma(z)}{\partial z} = \sigma(z)(1-\sigma(z))$ thus we have,

$\frac{\partial \hat{y}}{\partial w} = \frac{1}{1+exp(w^Tx + b)} (1 - \frac{1}{1+exp(w^Tx + b)})$.

So we can then update the weights through gradient descent as

$w^{new} = w^{old} - \eta \frac{\partial C}{\partial w}$

where $\eta$ is the learning rate.


Before talking about the solution, why don't you focus on the content instead? I think it would be more helpful to solve your problem, considering that most of the email addresses end with the sender's sign, Name Surname. Also, the probability of failing to obtain this information from an email address is much higher than the probability of failing to get it from the content. Especially, this is the case with company email addresses which might not contain the whole name in the email address (first letter of name and surname e.g. John Travolta - jtravolta@company.com), but it must contain the author's full name (at least the name) at the end. Furthermore, consider that plenty of email addresses will contain only name or surname or neither of them, but substitutive words like superboy122133@+++.com :D. But most of the email apps contain a default sign that includes name and surname. In addition, you can combine these two techniques. That is, combine the email address data with email content data so that, if it is infeasible or impracticable to obtain data from one of these, then you can use another one.
However, if let's say you have to do it with nothing but an email address I think using Machine Learning techniques would be overrating or overestimating the problem. Also, using non-machine learning techniques does not mean you are simplifying the solution, all these techniques give the best outcome when they are applied in the correct context. Let's imagine a simple situation: if you know or can easily infer that [tax] = 0.2 * [salary] + 20 $, why would you find (or fit) this equation using Machine Learning?
Unless you have data in the format of |email address, fullname|, you shouldn't start with using Machine Learning. (If you would have |email address, fullname| data, as an option, you would train a model to learn the general relationship between the email address and full name, thus you would identify similar email addresses).
However, in this current situation, one approach would be finding all possible patterns in the email addresses. Which can be

first letter of the name and surname jtravolta@+++.com
name, special characters and surname john_travolta@+++.com
name and numbers john1954@+++.com
...

Then these features that are extracted from email addresses using identified patterns can be compared with other emails either hashing or using string distance algorithms.
One alternative approach would be having a hashed dictionary of all available names and surnames, then you can cut pieces(substrings) from the email address then hash them to find the names and surnames from the address (Of course, vice versa would be highly inefficient). The email addresses that have the most similar, properties would be matched.
Another solution would be, using the above-mentioned patterns, you can generate a bunch of artificial email addresses. Considering that it is highly probable that there is not a dataset that includes the name and surname of people and their one or more email addresses, data augmentation is the first order of business. (I am not sure whether the data augmentation term fits this situation. If it does not then let's say data generation). So your input would be Name Surname (you can include middle name, number, etc.), and output would be randomly generated email addresses based on the pre-defined patterns. The number of emails that are generated for a single input should be randomly selected also, but be careful about the generation of the same email address more than once. E.g. input -> John Travolta -> output -> j_travolta12@+++.com, john.t.99@+++.com, john.travolta@+++.com (Lets suppose for this example we randomly choose 3 emails to be generated).
Then after you created, email addresses with all possible (almost) patterns you can get help from Machine Learning techniques. So the model might give you a probability with the relevant name and surname. (Also, you can configure the output so that it would give you top n name and surnames which have higher probability)
Another thing that is needed to be considered is the possibility of two different persons having the same name and surname. Lastly, independently from using which approach your solution cannot be perfect because, for example, it is not possible to understand whether the character 'j' stands for John or Jake in the email address. Thus, if you can integrate the email content into your solution, that will increase the performance drastically.
Update Accordingly: Check this answer which does not exactly answer your problem, but the context is the same.


Interpretation of tuning parameters (shrinkage and nrounds) in XGBoost

From my understanding (and I am no expert!)

Point 1

Let $ \hat f_k $ be a weak learner or one tree in the model, then we would make a prediction as the $$\hat y_i  = \sum_{i=0}^k \hat f_k(x) $$ with $ (x) $ being a particular feature The algorithm learns in an additive way so $$f_0(x) + f_1(x) + f_2(x) .... f_k(x) = \hat y_i $$ So adding a shinkage parameter $ \delta $ will penalise the model for each additional tree (or weak learner) you add to it, which looks like $ \delta f_k $. So having a low shinkage $ \delta $ would penalise the overall prediction score quite a lot.

Having a shrinkage score of 0.3 vs 0.01 with some $ (x) $ variable scores from each tree $f_k$;

Model 1: $ 0.3(2.1) + 0.3(2.9) + 0.3(3.4) + 0.3(3.5) = 3.57$

Model 2: $ 0.01(2.1) + 0.01(2.9) + 0.01(3.4) + 0.01(3.5) = 0.119 $

After four trees, so according to my understanding Model 2 is going to need many more trees to get to the same result as Model 1. However having a low number of rounds will likely miss important structural information in the model. I understand it this way.

Shrinkage extremely high and low, 1 and 0.01

Model 1x: $ 1(2.1) + 1(2.9) + 1(3.4) + 1(3.5) = 11.9 $

Model 2x: $ 0.01(2.1) + 0.01(2.9) + 0.01(3.4) + 0.01(3.5) = 0.119 $

The difference here being that Model 2x can capture 3 decimal places in its increments, whereas Model 1x goes up in increments of 0.1. So if we choose Model 1x then its possible to overshoot the optimal number of trees. If we were to increase the number of trees for Model 2x to 500 or 1000 then its possible that we would overfit the data, alternatively if we were to reduce the number of trees for Model 1x we would underfit the data. So this is where the cross validation and search for optimal parameters comes in.

So to answer point 1. Theoretically I understand yes, the numbers will converge, however this would lead to potential overfitting issues and a balance is necessary.

Point 2) nrounds to me, means the number of additional trees or weak learners added to the model.

At $ nround = 0 $ we have;

$$ \hat y_i^0 = 0$$

at $nround = 1$ we have;

$$ \hat y_i^1 = \hat y_t^0 + \delta f_1(x)$$

so $ \hat y_i $ adds the previous weak learners to its new prediction

then $ nround = 2 $ we have

$$ \hat y_i^2 = \hat y_i^1 + \delta f_2(x) $$

So at each $nround$ a new weak learner is added to the prediction.

Point 3) Here for me the mean evaluation metric is the mean score over k-fold cross validation. If you set k-folds to 10 then it would run the model 10 times and produce just the mean of these scores, the same with standard deviation scores.

Point 4) Theres many places to read about xgboost tuning, I have visited many of these websites countless times here

One really cool piece of code I am using from here. Although my code now has expanded this for most of the parameters of XGBoost and for an AUC evaluation metric not RMSE. I can post it if you are using AUC for evaluation.

I am still no expert but I am trying to answer some of the questions that you have myself, so I post my understanding of what I know, sure somebody will correct me :)

searchGridSubCol <- expand.grid(subsample = c(0.5, 0.75, 1),
                                colsample_bytree = c(0.6, 0.8, 1))
ntrees <- 100

#Build a xgb.DMatrix object
DMMatrixTrain <- xgb.DMatrix(data = yourMatrix, label = yourTarget)

rmseErrorsHyperparameters <- apply(searchGridSubCol, 1, function(parameterList){

    #Extract Parameters to test
    currentSubsampleRate <- parameterList[["subsample"]]
    currentColsampleRate <- parameterList[["colsample_bytree"]]

    xgboostModelCV <- xgb.cv(data =  DMMatrixTrain, nrounds = ntrees, nfold = 5, showsd = TRUE,
                           metrics = "rmse", verbose = TRUE, "eval_metric" = "rmse",
                           "objective" = "reg:linear", "max.depth" = 15, "eta" = 2/ntrees,
                           "subsample" = currentSubsampleRate, "colsample_bytree" = currentColsampleRate)

    xvalidationScores <- as.data.frame(xgboostModelCV)
    #Save rmse of the last iteration
    rmse <- tail(xvalidationScores$test.rmse.mean, 1)

    return(c(rmse, currentSubsampleRate, currentColsampleRate))

})


Let's start with your problem definition: "a good strategy make the relationships first and then count the occurrences".

That is, roughly, the basic strategy that market basket analysis algorithms use. However, algorithms like Apriori or FPGrowth are specially designed to analyze such datasets (at scale) and infer the inherent association rules between items across all baskets. My recommendation would be to use one of these to gather the relationships between items purchased, instead of reinventing them; especially because you'll face a lot of the hard problems these algorithms already solve (namely the large search space when generating combinations of basket items).

You can use any of several libraries or languages to do this, namely R, Python, etc. Doing this in Spark would be pretty simple using MLLib, your workflow would be something like: 1) choose an algorithm, e.g. FPGrowth; 2) prepare your data to fit the format required by FPGrowth (each transaction should be an array of basket items); 3) run FPGrowth and output its frequent itemsets.

There's a good example of this at Spark's website:

import org.apache.spark.mllib.fpm.FPGrowth
import org.apache.spark.rdd.RDD

val data = sc.textFile("data/mllib/sample_fpgrowth.txt")

//prepare the data to use with FPGrowth
val transactions: RDD[Array[String]] = data.map(s => s.trim.split(' '))

//create and run the model
val fpg = new FPGrowth()
  .setMinSupport(0.2)
  .setNumPartitions(10)
val model = fpg.run(transactions)

//output the frequent itemsets (items frequently bought together)
model.freqItemsets.collect().foreach { itemset =>
  println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)
}


When should the Keras functional API be used?

Sequential API is simpler to write and to read. Functional API on the other hand gives you more freedom in the implementation of the ANN architecture. More specifically, Functional API is what you need when the layers of your Networks do not form a simple concatenation or sequence (i.e. the first sends a signal to the second, that sends a signal to the third, and so on...). Sometimes more complex things are needed.

I'll make you a personal example: in my current DS project I am building a seq2seq Neural Network that has two Encoders: one is LSTM, the other is Convolutional. These two blocks process the same input signal, then their outputs are concatenated and fed to a Decoder that is LSTM + Dense. In this case, the whole model cannot be defined as a simple, linear concatenation of layers. You need to link different layers in a less "linear" way. This could be done exclusively with a functional API.

In computer vision, Residual Networks (ResNet) are the main example of the need for a non-sequential API, because the way the signal traverses the network is not simple and sequential (no pun intended) as in classical CNNs.

That said, coming to your question:

  I am not getting good accuracy everytime. Should I use the functional API? Will it do any good?

You can still reach SOTA results with the Sequential API. There are so many hyperparameters to tweak in a classical CNN to work on that for months (if not more). Could you please exaplain your problem, your data, what you tried so far, and what outcomes you got?


How to learn noSQL databases and how to know when SQL or noSQL is better

Check Martin Fowler's Personal website. He writes good and specially answers one of your questions in his book: "NoSQL Distilled"


How to find out which features were used frequently by RandomForestClassiferer in scikit-learn?

Feature importance it is a property of the random forrest classifier.

See an example here


Recommending movies with additional features using collaborative filtering

Here some resources that might be helpful:

Recommenderlab - a framework and open source software for developing and testing recommendation algorithms. Corresponding R package recommenderlab.

The following blog post illustrates the use of recommenderlab package (which IMHO can be generalized for any open source recommendation engine) for building movie recommendation application, based on collaborative filtering.

Research on recommender systems - a nice webpage with resources on the topic, maintained by Recommenderlab's lead developer Michael Hahsler.

Mortar Recommendation Engine - an open source customizable recommendation engine for Hadoop and Pig, written in Python and Java. Company, sponsoring the development of this project, Mortar Data, offers general commercial cloud platform for development and hosting data science software projects, including ones based on the Mortar Recommendation Engine (development and hosting of public projects are free): http://www.mortardata.com. Mortar Data provides help in form of public Q&A forum (https://answers.mortardata.com) as well as a comprehensive tutorial on building recommendation engine using open technologies (http://help.mortardata.com/data_apps/recommendation_engine).

"Introduction to Recommender Systems" - a relevant Coursera course (MOOC), which content and description provide additional resources on the topic.

PredictionIO - an open source machine learning server software, which allows building data science applications, including recommendation systems  (source code is available on GitHub). PredictionIO includes a built-in recommendation engine (http://docs.prediction.io/current/engines/itemrec/index.html) and supports a wide range of programming languages and frameworks via RESTful APIs as well as SDKs/plug-ins. PredictionIO maintains an Amazon Machine Image on AWS Marketplace for deploying applications on the AWS infrastructure.

Additional open source software projects, relevant to the topic (discovered via MLOSS website on machine learning open source software: http://www.mloss.org):

Jubatus
MyMediaLite
TBEEF
PREA
CofiRank

The following relevant R blog posts are also interesting:

"Simple tools for building a recommendation engine"
"Recommendation System in R"


Should I transform a multiple regression with outliers into ordinal regression?

Your final proposal (put an observations "on the edge" of intervals into both intervals) is unconventional, but probably worth trying. Note that ordinal logistic regression need more parameters than linear regression, so be careful adding too many categories because your data is small.

You can consider transforming the prediction target to reduce the distance between the extreme high values and the rest of the data.

In general, however, you will need to understand why those extreme high values are high. Are they incorrect entries (i.e. outliers)? Or is there some feature in the data that can be added to the model in order to better predict when a data point is higher than expected?

For example:

Let's say you are trying to predict student performance on a linear algebra midterm exam. Your data set consists of last year's exam scores. Maybe your model includes these features:

Student class year (freshman, sophomore, junior, senior)
Did student take honors calculus? (binary)
Average homework grade
Recitation attendance percent

Your model generally does well. Most students get in the 75-90 range, but some do unexpectedly well and score in the high 90s, almost perfect. Your model fails to predict these students' grades. So what went wrong? One possibility is that there is some other attribute shared by the high-scoring students that the other students might not have. You look back over the exams and realize that all the high-scoring students did well on a tough problem that you had covered in an office hour session. Now you add a new feature to the model:

Attended my office hours session on October 2nd.

and now your model correctly predicts the high-scoring outliers.


Binary Classification Comparing two time series of variable length

I think both LSTM and 1D-CNN could work, but it depends on your data first.

In your code you used a shared LSTM, which implies that you think your two input sequences are actually just different length sequences of a same variable? If that's the case, why not just use a LSTM to label, and then compare their label.

If that's not the case, you can then use separate input.

So the restriction for using 1D-CNN for sequence is that your sequence should have same length among samples (to my current knowledge). So if you have two types of sequences with fixed length $l_1$ and $l_2$ among samples, then you can try this approach.

If your have two types of sequences, both with variable length, then you will have to use one dynamic input LSTM for each of them, and then concat this two LSTM.


Using SMOTE for Synthetic Data generation to improve performance on unbalanced data

First of all, you have to split your data set into train/test splits before doing any over/under sampling. If you do any strategy based on your approaches, and then split data you will bias your model and that is wrong simply because you are introducing points on your future test set that does not exist and your scores estimations would be imperfect.

After splitting you data then, you will use only SMOTE on train set. If you use SMOTE from imblearn, it will automatically balance the classes for you. Also, you can use some parameter to change that if you dont want perfect balancing, or try different strategies.

https://imbalanced-learn.readthedocs.io/en/stable/over_sampling.html#smote-adasyn

So, basically, you would have something like this:

from sklearn.model_selection import train_test_split
from imblearn.over_sampling import SMOTE

X_train, X_test, y_train, y_test = train_test_split(X, y, split_size=0.3)
X_resample, y_resampled = SMOTE().fit_resample(X_train, y_train)

Then you continue fitting your model on X_resample, y_resample. Above, X is your features matrix and y is your target labels.


Sublime Text 2 with Pandas for Excel (Combining Data) & Data Visualization

Sublime Text is a text editor - it is independent of your Python setup. I would suggest following a tutorial on setting up Anaconda on your machine, then follow a second tutorial on making sublime text find the Anaconda Python executable.

Here is a tutorial for setting up Python with Anaconda on a Mac.
Here is something I found with a simple search (disclaimer: I don't know anything about sublime text!)

Once you have anaconda installed, you would need to run this command in terminal to install the packages you need:

conda install numpy pandas xlrd matplotlib


ValueError: Cannot feed value of shape (3,) for Tensor 'X:0', which has shape '(1, 3)'

trainX has shape (2500, 3), so when you iterate over trainX you get values with shape (3,).  To match the shape of your placeholder "X", you need them to have shape (1, 3).  This can be accomplished with numpy.reshape:

# do this after loading trainX
trainX = trainX.reshape((-1, 1, 3))
# new shape: (2500, 1, 3)


The problem you are describing is known as wake word detection or trigger word detection.

I'm sure you could use a CNN to classify a chunked Mel-spectrogram of your audio (see also librosa). As training labels you would simply use 0 for timestamps with no wake word (no "pizza") and 1 for timestamps with the wake word. Alternatively to classifying all timestamps of one chunk, you could also train for just the the center frame of each spectrogram chunk (makes things easier). In any case, you will have to make sure that your dataset is at least mildly balanced, i.e., you'll have to have enough wake word and non-wake word instances. One way to achieve this, is to overlay recordings of background noise with recordings of wake and non-wake words. There are some tutorials that detail how to do this, e.g. this YouTube video, this article or this GitHub repo. Note that all these approaches use RNNs for the task. However, it has been argued by Bia et al. that a temporal convolutional network (TCN) architecture (in essence a CNN skip connections and dilation) may work equally well or better for such tasks as the one you describe and is probably easier to train.

Hopefully this answer will give you some points to start from.


Dimensions of Transformer - dmodel and depth

d_model is the dimensionality of the representations used as input to the multi-head attention, which is the same as the dimensionality of the output. In the case of normal transformers, d_model is the same size as the embedding size (i.e. 512). This naming convention comes from the original Transformer paper.

depth is d_model divided by the number of attention heads (i.e. 512 / 8 = 64). This is the dimensionality used for the individual attention heads. In the tutorial you linked, you can find this as self.depth = d_model // self.num_heads. Each attention head projects the original representation into a smaller representation of size depth, then computes the attention, and then all the attention head results are concatenated together, so that the final dimensionality is again d_model. You can find more details on the individual computations in this other answer.

Note that the implementation of the multi-head attention in the tutorial is not a straightforward implementation from the original paper but it is equivalent: in the original paper, there are different matrices $W_i^Q, W_i^K, W_i^V$ for each attention head $i$, while in the implementation of the tutorial there are combined matrices $W^Q, W^K, W^V$ that compute the projection for all attention heads, which is then split into the separate heads by means of the function split_heads.


Rules/Methods to use for determining user groups with no review data

Start with some natural thresholds : >3* average, >4* good, >4,5*: excellent, 4,9+* perfect. Then you can correct your rating based on some averages, other metrics or even text (but that's hard). Honestly I am not sure it will work as ratings should be viewable and giving different status to customers with same average rating will get noticed.
Leave them blank, consumers like to know they are first to comment / know there is no reliable rating. No comment is an interesting information. This will push both customers and consumers to give a rating.


A deep learning approach that can calculate distance to obstacles

Actually, I don't think this is a problem that could be solved just from the image because it could be a small object that is really close or and big object really far. As far as I know, self driving cars use sensors for this. From what I found here, you would need the actual height of the object to calculate it from one image.

I strongly suspect you could also calculate the distance by having two images and then using the distance between the two cameras, and the Pythagorean theorem to calculate it. But that's just conjecture.


For training, close to equal distributed data will give you better results.

Type of data that you have, generally produced a biased model towards "neutral" class.

Are there any ways to deal with such problem?

I generally perform oversampling of the minority classes, such that for training(only),  have sufficient uniform count of data set.
 SMOTE, ADASYN are the few techniques of oversampling.


My neural network 'learns' to estimate each class equally

Some things you can try, to help debug your network:

Implemented a gradient check, to make sure you are calculating the gradients correctly.

Experiment with different learning rates, and different initializations of the initial weights.

Make sure to use randomization / symmetry breaking, or some suitable standard method, for initializing the weights.

Try starting with a single layer, and make sure your model can learn (you can solve some simple learning tasks) with a single-layer network.  Then try two layers.

Try a ReLU activation function.


What is the difference between a "cell" and a "layer" within neural networks?

In short
A cell is in essence a forward neural network consisting of neurons and so $cell_{RNN}\neq neuron_{NN}$.
Rationale
In the context of recurrent neural networks, a layer consists of cells e.g. LSTM cells. Although they seem to position similarly in terms of architecture (figure), there is a fundamental difference between an RNN_cell and a NN_neuron.

The number of RNN cells is generally equal to the sequence length (e.g. number of words in a sentence) or embedding output dimension (figure below). Conversely, the number of neurons in forward neural networks can generally be of any number.
Each RNN_cell acts much line a NN itself. Specifically in LSTMs, each cell consists of hidden state $W_{hidden}$ and cell state $W_{cell}$ matrices. These are used to store weights and perform transformations in the various gates (forget, input, output) of the LSTM cell.

In essence, the transformations that take place in each cell are matrix multiplications of $W_{hidden}$ and $W_{cell}$ much like those in a forward NN. And so, the size of $W_{hidden}$ and $W_{cell}$ cannot be but the number of neurons of that NN which essentially determine the memory of the LSTM cells and model itself.


No, it is not a confusion matrix.

Please have a look at the docs

This is how the confusion matrix is created in R.


A neural network that keeps learning as new training data becomes available

You can achieve that by training the network using batches of 100 examples. In this case, you train on a batch and update the neural network parameters, you predict on the second batch to estimate the accuracy and then you use the examples of this second batch to continue training the network and so on and so forth.

Batch training (a form of online learning) is implemented on every framework, see for instances here for Keras.


Create a single time series plot of multiple devices

You can create overlapping plots by looping over your dataset and plotting onto the same axes:
ax, fig = plt.subplots()
n = 30 # number of device
for i in (number+1 for number in range(n)):
    ax.plot(data[i], label='device_0' + i)

plt.show()

It might be best to first put your data into a pandas Dataframe.
In regard to the X-axis, you will need to convert the seconds to datetime, by adding an estimate of the starting date to each value in the dataset.


Does this data work with Convolutional Neural Network

What distinguishes CNNs from other deep learning neural networks is the use of a convolving filter which utilizes the structure of the data set.

CNNs are used for image classification because grayscale images are 2D arrays where the location of each pixel is relevant (if you rearrange all the pixels randomly, no one can tell what the image is). The convolving filter which is almost always a pixel of squares with weights is used to reduce the dimensionality of the image.

Likewise, 1D CNNs are used for time-series data because the location in time of each set of features is relevant (imagine a convolving filter moving across this 1D array in the same way the 2D filter moves over an image).

Does the pattern you mention (e.g. 1421143) have any relationship between different rows? Do the individual digits have a relationship with each other? If so then a CNN going in either axis (row-wise or column-wise, perhaps both) makes sense, but if, for example, scrambling all of the rows wouldn't make a difference in your analysis, then I am not sure if a CNN is the right direction over a shallow neural network given the small number of features.


Collapse a list to most common spellings

Credit to @Emre for setting me on this path with n-grams.

I discovered the library fuzzywuzzy which is well-built to solve this exact problem.  I'll give a brief synopsis, but a well-written tutorial and documentation made this pretty straight-forward.

I have a DataFrame where names are the index and there is a count column.
I loop through the index, finding high-similarity strings and then mapping the name to the name (including self) with the highest count.

names = df.index

#iterate through list find the top 10 (arbitrary) matching strings > 90 score, pop off the highest-volume order one
def find_top_match(name):
    my_matches = process.extract(name, names, limit=10, scorer=fuzz.token_set_ratio)
    my_matches = [t[0] for t in my_matches if t[1] >= 90]
    top_match = df.loc[my_matches].sort_values(by='count', ascending=False).index[0]
    return(top_match)

top_matches_set = []

for c in df.index:
    returned_match = find_top_match(c)
    top_matches_set.append(returned_match)

df['real_name'] = top_matches_set


Detection model - Training with class-instance limit-awareness

I can think of the following approach:

Let's say that you have two classes, A and B. Additionally, you now that for class A there is at maximum 5 instances (so 0, 1, 2, 3, 4 or 5) and for B 1 instance (0 or 1).

For this purpose, you can have 6 outputs for class A and 2 outputs for class B.
Between those 6 outputs for class A, only one should be active; same for B - only one of two should be active.

For example, if on some image there are 3 objects of class A and 0 for B, the outputs would be: [0, 0, 0, 1, 0, 0] for A, and [1, 0] for B (or something very close to 0s and 1s, right?)

With these outputs, you can also combine other outputs which are needed for detection.


When can embeddings be useful for small input spaces?

The advantages of embedding is converting categorical variables to numeric variables. Those numerical variables have semantic meaning. Thus any valid mathematical operation for numerical values has interpretable meaning.
One-hot encoding just tells the model that two entities are different. Embedding tells how different the entities are.


averaging multiple scores on small chunks of data or raw score on single collated data

I think this is 'mood' vs. 'character'. And as such, it depends on the question you ask. E.g. did I make my customer upset? Or is my customer loyal?

Though mood and character correlate, as someone with a certain character will be more likely to be in some mood, there is also variation.

I think you will measure 'mood' with the 'tone' analyzer, and 'character' with the 'personality' analyzer.

For the 'personality' you will need all the data you can get to get the 'average', for the 'mood', a single data point can be sufficient.


Opensource Speech Recognition Library that is secure and trained on large data

I'd recommend having a look at DeepSpeech.

You can find a maintained implementation of the project in here.

You may clone it, and train a model in order to have your very own model for speech recognition.


Depending upon how I download, I get two different files

This code works for sites where you don't need to be logged on. The Kaggle link only gives you the file when you are logged on to Kaggle. The file that is created with the code only contains html / javascript code of the Kaggle page.


How is Stochastic Gradient Descent used like Mini Batch gradient descent?

But in some cases they use Stochastic Gradient Descent and they define
  a batch size for training, I am confused about that?

If you defining a batch size, then you are performing mini-batch gradient descent.

Stochastic Gradient Descent ( SGD ) and Mini Batch Gradient Descent are often used interchangeably in many places. The idea is that if the batch size is 1, then its totally SGD.

If the batch size is equal to the number of samples in the dataset, then its Batch Gradient Descent.

  Also, what about Adam, AdaDelta, AdaGrad, are they all mini-batch
  gradient descent? or not?

Adam, Adadelta and AdaGrad are all different in which we update the parameters.  The main concept of GD is the calculation of gradient of the loss function which is fundamental to all the above optimizers.

The difference is where on how much samples from a dataset are the gradients of the loss function calculated.
After the gradients are calculated, the different optimizers mentioned use modified methods to update the parameters using the calculated gradients.

Finally,

  SGD, Mini batch/Batch GD are methods determining on how much samples are gradients calculated.
  AdaGrad, Adadelta etc. update the parameters differently based on
  calculated gradients.


SGDClassifier partial_fit() for online learning - is one step of gradient descent enough?

Monitor the metrics that you are attempting to optimize, and compute the change in those metrics after each epoch. Once those metrics stop improving according to certain criteria, then you can stop gradient descent. See this article for a way to keep track of the metric improvements.


As far as I know, Gaussian LDA and Gaussian Naive Bayes both assume the features follow normal distributions. However, GNB places a stronger assumption on feature independence, while Gaussian LDA does not require features to be independent.
See this.


Hyperparameter optimization, ensembling instead of selecting with CV criteria

I think you are looking for hyperopts, Optuna and Gpopy for Hyperparameters search without burning much of CPUs.


How hidden layer is made binary in Restricted Boltzmann Machine (RBM)?

As you correctly say, we calculate the probability of a hidden unit $h_j$
being one and then make it binary. That probability is given by
$$p(h_j=1) = \sigma\left(b_j + \sum_{i=1}^V w_{ij}v_i \right)$$
where $\sigma$ is the sigmoid function, $b_j$ is the bias of hidden unit $h_j$, $V$ is the number of visible units, $v_i$ is the (binary!) state of visible unit $i$, and $w_{ij}$ are the weights.

So, your MATLAB code for obtaining the probabilities hidden_probs is something like this (we write the sum implicitly by making a matrix multiplication):

hidden_probs = sigmoid(hidden_bias + data * weights)

Now, we have the probability $p(h_j=1)$ for each hidden unit $j \in [1,H]$. Now, this is only a probability. And we need a binary number, either 0 or 1. So the only thing we can do is pick a random sample from the probability distribution of $h_j$, which is a Bernoulli distribution.

As all hidden units are independent, we need to get one sample for each hidden unit independently. And also, in each training step, we need to draw new samples.

To draw these samples from the Bernoulli distribution, you can use the built-in functions of e.g. MATLAB (binornd) or Python (numpy.random.binomial). Note that these functions are to sample from a binomial distribution, but the Bernoulli distribution is just a special case of the binomial distribution with N=1. In MATLAB, that would be something like

hidden_states = binornd(1, hidden_probs)

which would create vector hidden_states which contains either 0 or 1, drawn randomly for each probability in hidden_probs.

As you probably have noticed, nobody does that! E.g. describes it in his Practical Guide to Training RBMs, as

  the hidden unit turns on if this probability is greater than a random number uniformly distributed between 0 and 1.

That is exactly what Hinton does in his RBM code: he gets a random number for each hidden unit using rand, i.e. randomly sampled from the uniform distribution between [0,1]. He then does the comparison:

hidden_states = hidden_probs > rand(1, H)

This is equivalent to using binornd, but is probably faster. For example to generate a random number that is 1 with p=0.9, you get a random number from [0,1]. Now, in 90% of the cases, this random number is smaller than 0.9, and in 10% of the cases it is larger than 0.9. So to get a random number that is 1 with p=0.9, you can call 0.9 > rand(1) - which is exactly what they do.

tl;dr: Pick a new random number from the range [0,1] for each hidden unit in each iteration. Compare it to your probability with hidden_probs > rand(1,H) to make it binary.


How to initialize word-embeddings for Out of Vocabulary Word?

You have a few options here. Of these, I think 1 will be the easiest to implement, as it's a standard language model with an alignment term added to the loss. I'd recommend 2a if you think you have the time, as I imagine its performance might be much better.

1 Use existing corpus to learn embeddings for new words

You can do this as you describe, though I would initialize the unknown embeddings with random noise instead of zeros. This will probably do fine, but it suffers from the problem that the model has no way of knowing that the SENNA embeddings are to be trusted more than the randomly initialized ones. So, it will generally take more (time, examples, etc.) to train this one well.

Another option is to try to capture the difference between known embeddings and randomly initialized ones. My suggestion would be to create your own embeddings [20k x emb_dim] and initialize it however you want. Then add a penalty to your model's loss for deviation from the SENNA embeddings. This is what's done in bilingual embedding models like this one. This will push the known embeddings close to SENNA and allow the unknown ones to freely vary. You could also reduce the coefficient for this alignment penalty as training progresses.

You could either learn these embeddings on a dedicated embedding task (like CBOW) or as part of your primary task.

2 Hybrid model for unknown embeddings

A second approach is to forgo attempting to learn embeddings for your unknown words at all. Typically, mosty word RNN models do this for low-frequency words through something like the <unk> tag. For you, that would be half your vocabulary. Fortunately, there are a lot of tools for filling in these out-of-vocabulary (OOV) words.

2a Subword embeddings

As @Emre pointed out, subword embeddings have been used to great effect to solve out-of-vocabulary problems. Here's an example of a model that uses subword embeddings to achieve an "open vocabulary" model. This means you would combine your pre-trained embeddings (which you would not need to make learnable) with a character-level model to learn when and how to produce OOV words.

2b <unk> replacement

Another approach is to, as a post-processing step, replace the <unk> symbols with words from your corpus. You can build your own language model (even a simple markov chain would be an improvement). Also, some modern models have more sophisticated methods for replacement that work quite well.


Tool for annotation of images for semantic segmentation

Diffgram is really great for this! I used it for a construction monitoring project. It's Open Source. From their site:
Semantic Segmentation Tools:

Auto Bordering: Automatically detects edges to create 100% coverage
masks. Simple select the intersecting shape.
Combo Shapes: Create shapes that are partially curves and partially straight lines.
Points to Full Polygon: Draw a box (or a few points), get a polygon.
Correct Pre-Label Segmentation Map: Load an existing segmentation prediction (pixels.) Get an editable polygon for correction.

Semantic Segmentation Data Annotation


How can I monitor the usage of NVLink connections?

nvidia-smi nvlink

This command shows various information about nvlink including usage. If nvlink connections are utilized, usage should go up during training.

nvidia-smi nvlink -h

Shows available performance counters on present cards.

"NVLink Usage Counters" section in this tutorial shows how to see if data is being transferred across nvlink.

https://blog.exxactcorp.com/exploring-nvidia-nvlink-nvidia-smi-commands/


Where is the rotated angle actually located in fitEllipse method?

I've implemented the following simple code:
import cv2
import numpy as np

nr_im = 9876
font = cv2.FONT_HERSHEY_SIMPLEX
fontScale = 1
colorText = (0, 0, 255)
thickness = 2

img = cv2.imread('testing/' + str(nr_im) + '.jpg')
original = img.copy()
blured_img = cv2.GaussianBlur(img,(17,17),5)

image = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
lower = np.array([0, 0, 140], dtype="uint8")
upper = np.array([0, 0, 255], dtype="uint8")
mask = cv2.inRange(image, lower, upper)

# Morphological Closing: Get rid of the noise inside the object
mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (25, 25)))

# Find contours
cnts, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)

print(len(cnts))

cntsElps = []
for num_cnt, cnt in enumerate(cnts):
    genElipse = cv2.fitEllipse(cnt)
    cntsElps.append(genElipse)
    cv2.ellipse(original,genElipse,(0,255,0),2)
    cv2.putText(original, str(num_cnt+1), (int(genElipse[0][0]),int(genElipse[0][1])), font, fontScale, colorText, thickness, cv2.LINE_AA)
    print("Ellipse nb: " + str(num_cnt+1) + " has angle: " + str(genElipse[2]) + "\n")

cv2.imwrite('testing/' + str(nr_im) + '_' + 'trash2' + '.png', original)

And I used this image as example:

I've got the following image result:

And the rotation angle for each ellipse was:

Ellipse nb: 1 has angle: 55.63788986206055
Ellipse nb: 2 has angle: 108.58539581298828
Ellipse nb: 3 has angle: 170.23861694335938
Ellipse nb: 4 has angle: 73.59089660644531

So, my conclusion is that an angle between horizontal axis and major side of rectangle(=major ellipse axis) is the rotation angle in fitEllipse() method.


Best way to build a wave classification system

I don't work in this field, however, recently I have read a magazine article (in German) about this. The article explained how  one of the most well-known companies who does this, Shazam, works.

Building a spectrogram, hashing, ???, indexing.

Video, 8:50: How Shazam Developed Their Signature, Awesome Algorithm | Shazam Founder Chris Barton
Whitepaper, from 2003, 7 pages, very technical
github repo from an open-source implementation: https://github.com/itspoma/audio-fingerprint-identifying-python See the READme file for more links