What is the interpretation of the expectation notation in the GAN formulation?

The functions $$ L_a(x) = \log(D(x))\;\;\;\;\&\;\;\;\; L_b(z) = \log(1-D(G(z))) $$
are defined for one value (either $x$ or $z$).

We don't want to optimize $L_a$ or $L_b$ for one input, we want to minimize it for all possible inputs $x$ and $z$. Suppose $x\in X\equiv\mathbb{R}^n$ (e.g., by "unfolding" it as an image) and $z\in Z\equiv \mathbb{R}^m$. Then we can write minimizing over all possible inputs as minimizing $\int_{\mathbb{R}^n} L_a(x)\,dx $ and $\int_{\mathbb{R}^m} L_b(z)\, dz$. But there's something wrong with this! In fact, we don't care about, say, pure red images or electron microscope micrographs (if $x$ were images), we only care about "real" photographs, i.e., the "natural images". So instead of optimizing over all $X$ and $Z$, we prefer to optimize over the ones that "matter". We do this by placing a probability distribution on $X$ and $Z$, say $P_x$ and $P_z$, such that sampling from these distributions gives us the $x$ and $z$ values that we care about. Low probability is given to unnatural images $x$; high probability is given to e.g. images of dogs and cats.
For GANs, we often choose $P_z$ in advance, as a uniform or normal distribution.
(Some people prefer to say that the natural images form a manifold in high-dimensional space; this $P_x$ can be thought of as inducing that structure.)

But how to optimize over these "likely" images. The obvious thing to do is optimize the expected value over these densities. In other words, we want to optimize the values of $L_a$ and $L_b$ that we expect to get if we compute them on samples drawn from $P_x$ and $P_z$. This is a weighted average over all of data space, where the weight is given by the probability of seeing that data point:
$$
\mathbb{E}_{x\sim P_x}[L_a(x)] = \int_X L_a(x) p_x(x)\, dx
\;\;\;\;\&\;\;\;\;
\mathbb{E}_{z\sim P_z}[L_b(z)] = \int_Z L_b(z) p_z(z)\, dz
$$
where this integral (expectation) is approximated by a sum or average over a minibatch in practice.

In summary, we add the expectations because we care about how we do across all "important" $x$ and $z$ values, not just one.

The explanation in this post might also be of use.


What if MNIST dataset had another feature

Convolutional layers are useful for images because they take into consideration the neighborhood of pixels. However, for labels like gender and handedness a convolutional layer may not be particularly useful.

However, after the convolutional layers you usually tend to place some densely connected layers. It is there that you may want to add these additional features. When you reshape the 2D matrices which results from the convolutions, you can concatenate the additional features, then feed this new vector to your Dense layer.


Multi-class Confusion Matrix is very well established in literature; you could find it easily on your own. Anyhow, Scikit-learn can do it easily like:

from sklearn.metrics import confusion_matrix

y_true = ['Cat', 'Dog', 'Rabbit', 'Cat', 'Cat', 'Rabbit']
y_pred = ['Dog', 'Dog', 'Rabbit', 'Dog', 'Dog', 'Rabbit']

classes=['Cat', 'Dog', 'Rabbit']

confusion_matrix(y_true, y_pred, labels=['Cat', 'Dog', 'Rabbit'])

array([[0, 3, 0],
       [0, 1, 0],
       [0, 0, 2]])

You can even plot it nicely using the below function:

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    import itertools
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    plt.tight_layout()

like this:

cnf_matrix = confusion_matrix(y_true, y_pred,labels=['Cat', 'Dog', 'Rabbit'])
np.set_printoptions(precision=2)

# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=['Cat', 'Dog', 'Rabbit'],
                      title='Confusion matrix, without normalization')

More examples here and here.


Stochastic gradient descent and different approaches

I think you're actually asking two separate questions: 1) What is stochastic gradient descent (SGD)? and 2) how do I find the matrices $P$ and $Q$ in the matrix factorization of $M$?

For question 1, I don't think either of your explanations is correct. (Explanation 2 sounds to me like normal (batch) gradient descent, although I found your description a little confusing.)

Regardless, both the Wikipedia article and Chapter 5 of the Deep Learning book provides good explanations of SGD. In many machine learning problems, the loss function, $L$ can be decomposed as the sum of losses for individual training examples:

\begin{equation}
L(x, y; \theta) = \sum\limits_{i=1}^{m} L(x^{(i)}, y^{(i)}; \theta)
\end{equation}

Therefore, in normal gradient descent, the number of terms in the gradient scales with the number of training examples, $m$. Computing this term can be very costly as $m$ grows large. In stochastic gradient descent, we instead approximate the gradient by using a "mini-batch" of samples to compute it. So, instead of

\begin{equation}
\nabla_{\theta} L(x, y; \theta) = \sum\limits_{i=1}^{m} \nabla_{\theta} L(x^{(i)}, y^{(i)}; \theta)
\end{equation}

we use

\begin{equation}
\nabla_{\theta} L(x, y; \theta) \approx \sum\limits_{i=1}^{m_{mini}} \nabla_{\theta} L(x^{(i)}, y^{(i)}; \theta)
\end{equation}

where $m_{mini} \ll m$. The key point is that, when running SGD, we randomly shuffle the training examples and iterate through different minibatches during training. So, SGD is not equivalent to normal (batch) gradient descent for a smaller training set. In SGD, we train on all of the training examples, but it just takes us $m/m_{mini}$ iterations to get there. For instance, if $m_{mini} = 1$, and we took $m$ SGD steps, each step would correspond to using a single training example to approximate the gradient, and all $m$ steps would involve using a different training example.

For question 2, solving for the parameters in matrix factorization is usually done via alternating least squares (ALS) or SGD. Here's an excellent blog post explaining how both techniques work in the context of matrix factorization. The post is too long to summarize, but the author carefully and slowly derives all of the relevant equations from scratch, so I recommend you read it!


Decision tree where identical set of features results in different outcomes

The task in hand can be modeled as a classification problem.
Meaning based on some given features we aim to predict the 'right' class of the outcome.
Once trained, most classification models, decisions tree included, will predict the most probable class given a set of input features.
Learning what is the most probable class, is usually done in the 'fitting' step of model training.
In your example the set of features results in 2 possible classes(hard, none) meaning that there is a 50% chance being either.
The classifier is just as wrong as its right in predicting the 'none' label.

** Some decisions tree implementations include an option to predict the probability for each class, and predict a class only if a specific confidence threshold is defined.


By looking at your code snippet, I realize you are training your CNN from scratch.

Use Transfer Learning Instead. Training a new model (choice of model architecture i.e. how deep your model should be, hyperparameters etc.) is very difficult if not impossible with only 1500 images. You can achieve great results quickly by using  an already-trained model (aka Transfer Learning). If you are not quite familiar with the subject, read this article Transfer learning from pre-trained models, or this one First steps with Transfer Learning for custom image classification with Keras. There are codes included that helps to get started faster. One of the recent advances in Transfer Learning is efficientnet, you may want to jump using that one! But I would guess boats would be easy even with earlier models.


As a starting point, you can think about unsuvervized image classification as a type of image clustering. You can - for instance - use VGG16 weights, extract image pseudo-features, and run some clustering on this feature set.
Here is some "starter code" (Python):
# Imports
from keras.preprocessing import image
from keras.applications.vgg16 import VGG16
from keras.applications.vgg16 import preprocess_input
import numpy as np
from sklearn.cluster import KMeans
import os, shutil, glob, os.path
from PIL import Image as pil_image
image.LOAD_TRUNCATED_IMAGES = True
model = VGG16(weights='imagenet', include_top=False)

# Variables
imdir = 'C:/indir/' # DIR containing images
targetdir = "C:/outdir/" # DIR to copy clustered images to
number_clusters = 10

# Loop over files and get features
filelist = glob.glob(os.path.join(imdir, '*.jpg'))
filelist.sort()
featurelist = []
for i, imagepath in enumerate(filelist):
    try:
        print("    Status: %s / %s" %(i, len(filelist)), end="\r")
        img = image.load_img(imagepath, target_size=(224, 224))
        img_data = image.img_to_array(img)
        img_data = np.expand_dims(img_data, axis=0)
        img_data = preprocess_input(img_data)
        features = np.array(model.predict(img_data))
        featurelist.append(features.flatten())
    except:
        continue

# Clustering
kmeans = KMeans(n_clusters=number_clusters, random_state=0).fit(np.array(featurelist))

# Copy images renamed by cluster
# Check if target dir exists
try:
    os.makedirs(targetdir)
except OSError:
    pass
# Copy with cluster name
print("\n")
for i, m in enumerate(kmeans.labels_):
    try:
        print("    Copy: %s / %s" %(i, len(kmeans.labels_)), end="\r")
        shutil.copy(filelist[i], targetdir + str(m) + "_" + str(i) + ".jpg")
    except:
        continue


Maybe look at this paper and references therein
Variational Encoding of Complex Dynamics


Why does my GridSearchCV always break up?

First, you are fitting $5 \cdot 3\cdot2\cdot2\cdot2\cdot5=600$ models and n_estimator=500 is quite big.
Of course, this depends on your dataset and in your computing power.

My first guess will be that you have not enough RAM memory on your laptop(if you are running it there) and that is why it is collapsing.

If the error is this one, I recommend sampling your data to 1/10 or less (depending on your data) and searching for the best hyperparameter there and then using your whole data for the final model.


K fold cross validation reduces accuracy

Chance plays a big role when the data is split. For example maybe the training set contains a particular combination of features, maybe it doesn't; maybe the test set contains a large proportion of regular "easy" instances, maybe it doesn't. As a consequence the performance varies depending on the split.

Let's imagine that the performance of your classifier would vary between 0.80 and 0.90:

  In one approch I just split the dataset into training set and test set

With this approach you throw the dice only once: maybe you're lucky and the performance will be close to 0.9, or you're not and it will be close to 0.8.

  while with the other approch I use k fold cross validation.

With this approach you throw the dice k times, and the performance is the average across these $k$ runs. It's more accurate than the previous one, because by averaging over several runs the performance is more likely to be close to the mean, i.e. the most common case.

Conclusion: k-fold cross-validation isn't meant to increase performance, it's meant to provide a more accurate measure of the performance.


Xgboost multiple class predictive performance beats one versus rest

Multiclass models in XGBoost consist of n_classes separate forests, one for each one-vs-rest binary problem.  At each iteration, an extra tree is added to each forest.  But it isn't actually a one-vs-rest approach (as I thought in the first version of this answer), because these trees are built to minimize a single loss function, the cross-entropy of the softmax probabilities.
https://discuss.xgboost.ai/t/multiclassification-training-process/29
https://github.com/dmlc/xgboost/issues/806
https://github.com/dmlc/xgboost/issues/3655

In general, the one-vs-rest models are very good at identifying the single class, whereas the multiclass model has to balance performance on all of them.  More specifically, I think that the softmax may be responsible for the phenomenon you're displaying.  (I'm still thinking about it, but I thought I should post the above for now.)

Suppose one of your documents is reasonably likely to be in either of two topics: the probability scores given by the forests are 0.9, 0.85, then <0.1 in all the rest.  In your topic-1 model, you make a fairly confident judgement that this document is of topic 1 (score of 0.9).  But in the multiclass ensemble, you see things as much more uncertain; maybe the model applies softmax, so that the probability of topic 1 is only ~0.5.

More extreme, suppose the individual topic model scores are all 0.9.  Now the multiclass ensemble applies softmax and produces equal 1/17 probabilities for each topic!

In the other direction, suppose one of your documents is judged unlikely to fit any of the topics: all the individual topic model probability scores are 0.01.  In the multiclass ensemble, that gets scaled up to 1/17 (OK, 17 topics makes this a harder sell).

Hrm, except how likely is it to get the 0.9 and 0.85, since a training sample in one of these two topics will be pushed toward 0 by the other model... ?  Especially when your scores are fairly high, so it's not like the models have huge blind spots.
(This part still causes a problem with the correct understanding of how XGBoost works; the log-loss of the softmax probabilities still penalizes being confident about belonging to two different classes...)


Most non-technical people often use Excel as a database replacement. I think that's wrong but tolerable. However, someone who is supposedly experienced in data analysis simply can not use Excel as his main tool (excluding the obvious task of looking at the data for the first time). That's because Excel was never intended for that kind of analysis and as a consequence of this, it is incredibly easy to make mistakes in Excel (that's not to say that it is not incredibly easy to make another type of mistakes when using other tools, but Excel aggravates the situation even more.)

To summarize what Excel doesn't have and is a must for any analysis:

Reproducibility. A data analysis needs to be reproducible.
Version control. Good for collaboration and also good for reproducibility. Instead of using xls, use csv (still very complex and has lots of edge cases, but csv parsers are fairly good nowadays.)
Testing. If you don't have tests, your code is broken. If your code is broken, your analysis is worse than useless.
Maintainability.
Accuracy. Numerical accuracy, accurate date parsing, among others are really lacking in Excel.

More resources:

European Spreadsheet Risks Interest Group - Horror Stories

You shouldn’t use a spreadsheet for important work (I mean it)

Microsoft's Excel Might Be The Most Dangerous Software On The Planet

Destroy Your Data Using Excel With This One Weird Trick!

Excel spreadsheets are hard to get right


Predict the corresponding value in one column using a list of values found in another column

Not a very clever solution though. But I managed to do some trick to make it work. I am not fully satisfied by the result but the algorithm is able exactly predict the Item based on given Composition column.

from io import StringIO
import json
col = ['Item', 'Composition']
df = dfmin[col]
df['Item'] = df['Item'].apply(lambda x: ''.join(str(x).strip('[]') if      isinstance(x, list) else x))
df['Composition'] = df['Composition'].apply(lambda x: ''.join(str(x).strip('[]') if isinstance(x, list) else x))
df['Composition'] = df['Composition'].apply(lambda x: x.replace(',',' '))
df['Composition'] = df['Composition'].apply(lambda x: x.replace(' ',''))
df['Composition'] = df['Composition'].apply(lambda x: x.replace("'",''))
df['Composition'] = df['Composition'].apply(lambda x: "".join(x.rstrip()))
df.head()

Doing this removed all spaces in the text in Composition column and it ended up becoming one long string set.

df['Indexer'] = df['Composition'].factorize()[0]
#df['Indexer']
Indexer_dfmin = df[['Item',    'Indexer']].drop_duplicates().sort_values('Indexer')
df_to_Indexer = dict(Indexer_dfmin.values)
Indexer_to_df = dict(Indexer_dfmin[['Indexer', 'Item']].values)
#df['Composition'] = df['Composition'].str.split(",")
df.head()

I factorized the Composition column using pd.factorize().

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(sublinear_tf=True, norm='l2', encoding='latin-1', ngram_range=(1,8), analyzer='word',lowercase=True)
features = tfidf.fit_transform(dftest.Composition.values.astype('U')).toarray()
labels = dftest.Indexer
features.shape

Using sklearn RigdeClassifierCV I was able to model the data and predict the Item name using the string of the Composition column.

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.linear_model import RidgeClassifierCV
X_train, X_test, y_train, y_test = train_test_split(df['Composition'].values.astype('U'),df['Item'].values.astype('U'), random_state=42)
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
clf = RidgeClassifierCV(alphas=[1e-3, 1e-2, 1e-1, 1]).fit(X_train_tfidf, y_train)

Issues with this approach:
1) The model is not learning the words in the Composition column and building a vocabulary.
2) So, if I use the words with space from the Composition column the model will not be able to predict properly.
3) The accuracy can go near to 100% percent which means the model is overfit.

That is why I say this is not a clever solution. I would like to get to know how to fix the mistakes and issues in my model.
Appreciate some expert advice on this. Please feel free to comment. Thanks in advance.


Assigning new items to existing similarity based clustering

I suggest you think about this in terms of "data set" and "training set" (technically, it is also recommended to have a separate test set). Once you have your clusters defined on the training set, your can start using them to classify any amount of new data without recalculating, by simply measuring similarity to cluster centroids, for example.

(This doesn't prevent you from deciding to enlarge your training set and data set later, just try to not do that selectively to avoiding overfitting.)


Basically, we want to cluster similar comments and assign a name/entity to it. I suggest you to use Doc2Vec to convert the comments to fixed-sized vectors. Each of your comments will then be a n-dimensional vector. Comments with similar words/phrases will lie in the close proximity of one another.

  Now, using K-Means Clustering, we can form clusters of vectors that
  represent comments having a similar meaning. Once the clusters are
  formed, assign a name to each of them.

For a given sample ( comment ), the model will first transform the given comment into a vector. The model will then check for a cluster that is nearest to the given sample's vector. The output will be the name of the nearest cluster.


Finding out which values lead Random tree to a decision

If your dataset was separable into a series of neat decisions, then a classification and regression tree (CART), would give you the type of solution you're looking for. The price you pay for random forest is that by reducing the variance through generating many random trees, you're also reducing the interpretability of the model significantly. Solutions that can give you local solutions are LIME or you can calculate SHAP values or feature importances, but that gives you importances in the context of the model and may not be useful for the type of decision you're looking to make.


What is industry standard for workflow when training models?

There are online models and there are offline models. Online models make sense when you need to have your model constantly be updated. Otherwise you can save money by updating your model as frequently as necessary; nightly, weekly, etc. The machines used to train models are resources that cost money, after all, and you do not want to reserve them longer than necessary.

One trick to save time is to use the last run's model parameters as initial values for the next iteration.


Choosing a model for input: categorised, weighted sequence, output: binary variable

One option is a model comparison approach. Start with simpler models then progressively trying more complex models. Along the way, checking if the additional complexity is resulting in increased predictive ability.

If you take a simplifying representation with a bag-of-words model, you can fit a Naive Bayes classifier.

If you make the Markov assumption, you'll only need to look at the last element in the sequence. Then it can be modeled as a simple Probabilistic Graphical Model (PGM).

Then start relaxing the Markov assumption by looking back at progressively more time steps.

Recurrent Neural Network (RNN) might work if you have thousands of labeled data points and the relationship is very complex.

Long Short Term Memory (LSTM) does not make much sense because the sequences are so short. LSTMs work better RNNs when the relevant information could possible be many time steps back.

My hypothesis is that a Naive Bayes classifier will be a good-enough model because there are only 4 features and the sequences are so short they can be ignored.


Chinese word segmentation using neural networks

So, the main question is about how to represent the Chinese characters in your Chinese word segmentation task.
Since effectively these characters are non-ordinal categorical variables, we would represent these as one hot encodings (https://medium.com/@michaeldelsole/what-is-one-hot-encoding-and-how-to-do-it-f0ae272f1179) of dimensionality n = number of unique Chinese characters in your dataset.
Sometimes n can be very large. So in this instance, we typically have an Embedding layer, which can reduce data sparsity and collapse the number of dimensions. (https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding#:~:text=Arguments%20%20%20%20input_dim%20%20%20,the%20embed%20...%20%202%20more%20rows%20)


How to combine human-labelled data with user behavior data?

That is a common scenario in a learning to rank problem. One heuristic is to separately model explicit (human-labeled) and implicit (user-behavior) features. Then combine the separate feature groups with a learned weight for their final relative contribution. Improving Web Search Ranking by Incorporating User Behavior Information by Agichtein et al goes into greater detail.
RankNet takes this approach using a neural network.


Aggregating multiple encoded categorical values

If individual categories are important in your analysis, you could split the category column into multiple columns based on the amount of different category values then pivot your data set to have multiple row entries per category.

Visually

recordid | category | value
-------- | -------- | -----
   1     |   a      |  5
   2     |   a,b,c  |  10

would become

recordid | category |  value
-------- | -------- | ------
   1     |   a      |   5
   2     |   a      |   10
   2     |   b      |   10
   2     |   c      |   10

You could then make further aggregations or transformations as necessary.

Otherwise you could consider "a" and "abc" and "abcd" a new category, i.e. an item with categories "abc" is different from an item with just the category "a"


MCMC for finding Bayesian Neural Network

There are in fact a bunch of papers on this topic. I'd recommend this one and this one. Not sure what you want to do with your Bayesian Neural Network, but maybe this one is useful too.


Among Naive Bayes assumptions the main one is that features are conditionally independent. For our problem we would have:

$$P(Play|Outlook,Person) \propto P(Play)P(Outlook|Play)P(Name|Play)$$

To address question is Harry going to play on a sunny day?, you have to compute the following:

$$P(Yes|Sunny,Harry) = P(Yes)P(Sunny|Yes)P(Harry|Yes)$$
$$P(No|Sunny,Harry) = P(No)P(Sunny|No)P(Harry|No)$$

and choose the probability with bigger value.

That is what theory says. To address your question I will rephrase the main assumption of Naive Bayes. The assumptions that features are independent given the output means basically that the information given by joint distribution can be obtained by product of marginals. In plain English: assume you can find if Harry plays on sunny days if you only know how much Harry plays in general and how much anybody plays on sunny days. As you can see, you simply you would not use the fact that Harry plays on sunny days even if you would have had that record in your data. Simply because Naive Bayes assumes there is no useful information in the interaction between the features, and this is the precise meaning of conditional independence, which Naive Bayes relies upon.

That said if you would want to use the interaction of features than you would have either to use a different model, or simply add a new combined feature like a concatenation of factors of names and outlook.

As a conclusion when you do not include names in your input features you will have a general wisdom classifier like everybody plays no matter outlook, since most of the instances have play=yes. If you include the name in your input variables you allow to alter that general wisdom with something specific to player. So your classifier wisdom would look like players prefer in general to play, no matter outlook, but Marry like less to play less on Rainy.

There is however a potential problem with Naive Bayes on your data set. This problem is related with the potential big number of levels for variable Name. In order to approximate the probability there is a general thing that happens: more data, better estimates. This probably would happen with variable Outlook, since there are two levels and adding more data would probably not increase number of levels. So the estimates for Outlook would be probably better with more data. However for name you will not have the same situation. Adding more instances would be possible perhaps only by adding more names. Which means that on average the number of instances for each name would be relatively stable. And if you would have a single instance, like it is the case for Harry, you do not have enough data to estimate $P(Harry|No)$.

As it happens this problem can be alleviated using smoothing. Perhaps Laplace smoothing (or a more general for like Lindstone) is very helpful. The reason is that estimates based on maximum likelihood have big problems with cases like that.

I hope it answers at least partially your question.


If you have $n$ time series of rainfall measurements every hour ($x$=time, $y$=amount of rain), and compute the distance matrix between each pair of time series based on Dynamic Time Warping, and then plot the clusters, what do the X and Y axes on the cluster plot represent?

They remain the same, $x$=time, $y$=amount of rain - dynamic time warping (DTW) shifts the times so the curves of the amount of rain align their peaks and valleys.

A different example might be easier to understand. In the paper by Berndt and Clifford (see below) the population of the Snowshoe Hare is plotted and subjected to DTW. You can see as the population of hares increased so did the ability to hunt Lynx. As the population of Lynx increased the population of hares declined. That affected the hunting of Lynx, which again allowed the hare population to increase. This creates a double-top peak in the data, using DTW these can be aligned.

  In the k-medoids or k-means plot, is X still time, and Y still the amount of rain?

Having aligned the time/population (similar to time/rainfall) the locations on a map can be plotted with a k-medoids or k-means plot.

K-medoids (also: Partitioning Around Medoids, PAM) uses the medoid instead of the mean, and this way minimizes the sum of distances for arbitrary distance functions. K-means clustering aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster.

The time variable would be plotted on the map for a period of time, for example a month, season, or year; so while the time is not 'lost' it's also not $x$ or $y$. Instead a color on the map could represent the amount at a location for the entire duration of the period of time.

See how when k-medoids or k-means is applied "time" does not sit on $x$ or $y$, in this example location is on the $x$ and $y$:

References:

"Alignment of curves by dynamic time warping", by Theo Gasser and Kongming Wang.

"Using Dynamic Time Warping to Find Patterns in Time Series", by Donald J. Berndt and James Clifford.

"Clustering analysis applied to NDVI/NOAA multitemporal images to improve the monitoring process of sugarcane crops", by Luciana Alvim Santos Romani, Renata Ribeiro do Valle Gonçalves, Bruno Ferraz Amaral and Agma Traina.


unable to plot contour/surf graph of cost vs parameters for multiple linear regression.In python

Try using a trisurf plot! It is very simple to get a nice surface plot.

Have a look at this Matplotlib 3d documentation that includes many examples.

Here is one of the examples, where I additionally print the size of the three arrays used to plot. We see that they are all single vectors of the same length, therefore your data should just work fine using the code below:

from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
import numpy as np

n_radii = 8
n_angles = 36

# Make radii and angles spaces (radius r=0 omitted to eliminate duplication).
radii = np.linspace(0.125, 1.0, n_radii)
angles = np.linspace(0, 2*np.pi, n_angles, endpoint=False)

# Repeat all angles for each radius.
angles = np.repeat(angles[..., np.newaxis], n_radii, axis=1)

# Convert polar (radii, angles) coords to cartesian (x, y) coords.
# (0, 0) is manually added at this stage,  so there will be no duplicate
# points in the (x, y) plane.
x = np.append(0, (radii*np.cos(angles)).flatten())
y = np.append(0, (radii*np.sin(angles)).flatten())

# Compute z to make the pringle surface.
z = np.sin(-x*y)

# Inspect the array shapes
print(x.shape)                    # gives: (289,)
print(y.shape)                    # gives: (289,)
print(z.shape)                    # gives: (289,)

# Create the plot and show it

fig = plt.figure()
ax = fig.gca(projection='3d')

ax.plot_trisurf(x, y, z, linewidth=0.2, antialiased=True)

plt.show()


Can Reinforcement Learning work for Dutch auctions?

Whether or not a Dutch Auction can be framed as a reinforcement learning problem depends on whether there is meaningful state that can be changed by the auctioneer's actions.

If the bidders are behaving in a simplified rational manner* - e.g. they are working to a simple mechanic of buying the item when it reaches a price they have decided to pay - then the behaviour of the auctioneer will not make much difference, it is very formulaic as it is already part of a price optimising strategy. There is no meaningful state change between actions - the bidder valuations and attention stay fixed no matter what the auctioneer does. In an extreme, automated case, the optimal action would be to set a very high initial price and then reduce it in many small increments so as to trigger the highest purchase possible from bidders. This is not done in real auctions because it is a waste of everyone's time (and it would be weird, socially). However, it shows you a problem with the framing of the problem. In a basic rational setting, with nothing to gain or lose apart from money and value of the item, there is little room for altering strategy.

The problem becomes far more interesting when bidders are not purely rational optimisers themselves (or have some other unknown values related to the auction such as personal time they are willing to invest in the process, or rivalry with other bidders), and when some information is available about their past behaviour, or at least in general about human behaviour. In that case the problem can modelled as a Partially Observable Markov Decision Process (POMDP), and you can concern yourself with rewards that might vary due to your actions - e.g. risk of making bidders skittish with "odd" behaviour as the auctioneer. There are limits to this due to the anonymous bidders. If you kept bidder identity between items (if a single auction consisted if multiple item sales with the same bidders) then you might gain some information on each sale in order to optimise better your actions within that auction.

Another possible way to have stateful behaviour is if item ownership has value, such that market forces of supply/demand come into effect (once a bidder has one item maybe they value the next one less, or maybe more if multiple items have higher value to the bidder when kept together). Again this is only going to be relevant to the auctioneer if bidder identities remain constant for a multi-item auction, and even then only makes a difference if there is non-rational pricing behaviour in the bidders.

The main problem you will have with training RL auctioneer to work with non-rational bidders, is modelling those bidders. You could maybe bootstrap with data from real auctions with human auctioneers - this can still be done in a Q learning framework. Q-Learning can learn about optimal policy from observing non-optimal behaviour within limits, optimal actions still need to be observed, but don't have to all be in the same episodes. However, in general to improve from that model and apply reinforcement learning, you have to you put it in control. When it explored unusual actions in order to learn, that could make it a liability if real money was at stake - to defend against this you may need to keep action choice limited.

* Staying with "purely rational" approaches, you could get more sophisticated. You could maybe model the bidders as trying to predict the auctioneer and other bidder's next action, and extract the best expected value, in which case you have framed the problem more like adversarial game theory. This could make the auctioneer's actions have more impact. The most interesting part of this is occurring in the bidder though, not the auctioneer, as the bidder would have to incorporate a meaningful value to owning the item and be smart enough to second-guess other bidder's valuations.

The auctioneer's goal would then be to find the best price that the highest bidder was willing to pay, and behave so that the highest bidder predicted that a rival would bid on the next lower increment. This would be very similar to the non-rational approach in practice, but more tractable to do in simulation - you would need a good simulated model of the bidders before you could train an auctioneer, and the bidders would probably need to be RL or similar too, with both auctioneer and bidder models evolving together.

Intuitively, and assuming the game theory model was stable here, I would expect the auctioneer, with no other information to go on than the number of bidders, would have an optimal strategy where selling price decreased in increments inversely sized to some fractional power of the number of bidders, as more rivals means that bidders will predict rivals willing to pay values closer to their own evaluation. A partially stochastic policy might also be favourable to the auctioneer, as being unpredictable means that bidders cannot rely on guessing the next price reduction accurately (although they might counter that by choosing whether to bid stochastically).


Which scoring for GridSearchCV is best, when imbalanced multiclass dataset?

There is lots of metrics to measure performance of classifiers. The fundamental ones are based on the idea of:

true positive (TP) — sample’s label is positive and it is classified as one
true negative (TN) — sample’s label is negative and it is classified as one
false positive (FP) — sample’s label is neg., but it is classified as positive
false negative (FN) — sample’s label is pos., but it classified as negative

From what I have seen in white papers, F1-score is the most used metric that consider in imbalanced classification scenarios. But I also see ROC-AUC as a frequent used metric. As I mentioned, there is lots of metrics, but I strongly recommend you to keep these most used to provide to the others some standard sense of performance.


Is it correct to say that a high variance feature will be important to my model?

High feature variance does not imply any sort of relationship to the target variable you're interested in modeling. Suppose you are looking at a population and have two variables for each person, their height in meters and annual income in dollars. The variance of height will be a small number (heights don't vary by much more than 1 meter), but the variance of income will be a large number (incomes can vary by thousands). If you want to predict the person's arm span, their height will be a much better predictor than their income, despite the fact that it has a numerically smaller variance.
A zero variance feature does not have any predictive power, but even numerically small fluctuations can be very important depending on your goal.


Model localization: one big model vs two small models

When and how to choose 1 vs 2? Data exploration, understanding the problem, and testing the models. Make sure you use metrics, a cut-off level and test data that is appropriate for your business problem.

Take for example, if there is a fundamental difference between US and CA customers. Then allowing the model to focus on the signal of each country, without dealing with the noise of the other country, may produce a better model for each. By doing data exploration (plot purchases by country over time and some other important features) you might convince yourself that it is worth the effort to test multiple models.

This is the art vs science of modeling.


Should I take random elements for mini-batch gradient descent?

It should be enough to shuffle the elements at the beginning of the training and then to read them sequentially. This really achieves the same objective as taking random elements every time, which is to break any sort of predefined structure that may exist in your original dataset (e.g. all positives in the beginning, sequential images, etc).
While it would work to fetch random elements every time, this operation is typically not optimal performance-wise. Datasets are usually large and are not saved in your memory with fast random access, but rather in your slow HDD. This means sequential reads are pretty much the only option you have for good performance.
Caffe for example uses LevelDB, which does not support efficient random seeking. See this, which confirms that the dataset is trained with images always in the same order.


Decision Tree Induction using Information Gain and Entropy

Let's us start by defining what are you trying to achieve in Decision tree.

So we want a tree which correctly classify the data?For that out of number of features available I want to select those features which gives me the best information about my classes i.e. How well a feature split my data? which shall help me classify better.

Now entropy is such techniques which helps me understand how "Pure" my subset it, i.e. if i select feature_1 to split upon how well it shall split my data that most of the class labels are from same class.

So Entropy is an impurity measure. If all instances are from same class then there is 0 impurity hence E = 0, if there are same number of instance from both classes then impurity is highest E=1.

Now, we need to select which attribute is the best to split upon first and then recursively which ones are later on ?

Here comes the Information gain.

Information gain just tells me how much useful/good an attribute is from rest of the attributes. For that we compare the entropy of the "Parent Node" before splitting to the impurity of "child Nodes" after splitting. Larger the difference , better the attribute test condition.

Higher gain = purer class

  So, the initial entropy should equal 1.

1 is the highest entropy it means if I have 4 instance, 2 says +ve and 2 says-Ve hence its highly impure. It could very well be 0 if classes are like 4+ and 0 -ve. so on as per the calculations.

  We gain information if we decrease the initial entropy

this initial entropy is the entropy of the data set before splitting or entropy of the parents node.It depends on the data.

  The more info_gain is larger than 0, the more the feature makes order
  in system.

No, Its not compare with 0 but the gains of all the attribute with each other,

  gain(w_1) = .4, g(W_2) = 1.2

Then highest gain is of W_2 hence DT will use W_2 for the split.

Imagine if you have a dataset with 3 attributes A1, A2, A3, then which attribute you should test first?

So, compute the entropy of complete set, i.e. E(complete)

same way E(A1), E(A2), E(A3),

Now

gain(A1) = E(complete)-E(A1) = .2, gain(A2) = E(complete)-E(A2) = .5,gain(A3) = E(complete)-E(A3) =1

highest gain = 1, hence we shall split on A3.

If there is same gain for attributes then we take the order in consideration.


Issue with IPython/Jupyter on Spark (Unrecognized alias)

IPython has now moved to version 4.0, which means that if you are using it, it will be reading its configuration from ~/.jupyter, not ~/.ipython. You have to create a new configuration file with

jupyter notebook --generate-config

and then edit the resulting ~/.jupyter/jupyter_notebook_config.py file according to your needs.

More installation instructions here.


What are the ways to identify a good attribute test while constructing a decision tree?

You might have mixed up CSP (Constrained Satisfaction Problem) search trees and decision trees:

CSP search trees

'Minimum remaining values', 'Degree Heuristic' and 'Least Constraining Value' are used to solve CSPs and not in decision trees (i.e. we are talking symbolic AI here and not a sub-symbolic AI method like decision trees).

However, as far as I know the 'least constraining value' approach is not used to select a variable but after having selected one to choose the order to inspect its values.

So it could go rather like this:

1. Choose a variable applying 'minimum remaining values'
2. If step 1 fails select a variable applying the 'degree heuristic'
3. For the selected variable select its values ordered according to 'least constraining value'

Besides the ones you mentioned you could go the really simple route: just pick 'the next' variable or pick randomly.

Also see section 6.3.1 in 'Artificial Intelligence: A Modern Approach' by Russel and Norvig.

Decision trees

In contrast decision trees are a machine learning method. Here you usually choose the best splits for the tree using a greedy algorithm providing the biggest ad-hoc gain (see below list for how to define 'gain'). Typical criteria for the greedy split are:

Classification

Information gain (based on entropy)
Gini score
Classification error (usually not used when building the tree)

Regression

Squared error

Also see sections 9.2.2 and 9.2.3 in 'The Elements of statistical learning' by Hastie, Tibshirani and Friedman.


Technique to make NN (specifically autoencoders) more robust by artifcally making input data more sparse

I've finally found it. The term I was looking for was 'masking corruption' and is mentioned in 'Practical Recommendations for Gradient-Based Training of Deep Architectures' by Yoshua Bengio.


How to define discrete action space with continuous values in OpenAI Gym?

Here is a sample environment which demonstrates this. It relies on the environment to successfully filter out the correct continuous control element

import gym
from gym.spaces import Dict, Discrete, Box, Tuple
import numpy as np

class SampleGym(gym.Env):
    def __init__(self, config={}):
        self.config = config
        self.action_space = Tuple((Discrete(2), Box(-10, 10, (2,))))
        self.observation_space = Box(-10, 10, (2, 2))
        self.p_done = config.get("p_done", 0.1)

    def reset(self):
        return self.observation_space.sample()

    def step(self, action):
        chosen_action = action[0]
        cnt_control = action[1][chosen_action]

        if chosen_action == 0:
            reward = cnt_control
        else:
            reward = -cnt_control - 1

        print(f"Action, {chosen_action} continuous ctrl {cnt_control}")
        return (
            self.observation_space.sample(),
            reward,
            bool(np.random.choice([True, False], p=[self.p_done, 1.0 - self.p_done])),
            {},
        )

if __name__ == "__main__":
    env = SampleGym()
    env.reset()
    env.step((1, [-1, 2.1]))  # should say use action 1 with 2.1
    env.step((0, [-1.1, 2.1]))  # should say use action 0 with -1.1
```


How to predict number of orders for coming weeks using previous week's data?

If you treat this problem as a time-series one, you will gain more information for predictions, because it includes several time related components, and you can customize your orders predictions based on week day(week-end, Black Fridays)

If you treat this problem as linear regression, you will only be able to predict for the one upcoming week and not several weeks in advance, as they will only depend on the latest record you gave as the predictor.

analytics vidhya has great tutorials on time-series problems.


Training on a subset of data for few epochs and then proceeding to the next subset for few epochs and so on?

Training until convergence on a subset of data and starting again on another subset is not a good idea.

Gradients of loss will have high variance over your batches and so optimizing over it will not be useful (you basically start at a far new point on your loss for the next subset).

Instead, the good way to go in batch-training methods is to iteratively make training steps on differents subsets of your data until a global convergence.
Your may hear about an epoch, which consists of training steps on subsets until the overall data has been seen.
You see how convergence is over the epochs and not over one subset.


What algorithm to use for extracting information from bank statements

Am currently working on something in this domain.

The rough process I am currently following is -

Extract data from PDFs (ubiquitous version of Bank Statements nowadays) into more usable formats. Currently,converting them to TXT files first as an intermediate step.
Generically, bank statements (from a specific bank) tend to be structured in the same format. Hence, while converting the TXT to CSV, you can structure the algo such that it knows what to pick up - based on a rough analysis of the TXT file.

Use 2 different data frames to store the statements content - one df keeping track of the transactional data (storing 3 values - Date, Description and Amount) and the other to keep track of the metadata (storing 2 values - Key and Value)
You can determine the rough structure of the TXT file by going through a few TXTs. e.g. the transactions generally begin after a specific element's occurrence in every bank statement. Ignore the page footers / headers that recur in case of multi-page transactions. Also, another observation would be that all data in a specific column of the transactions (the transactions are read in as a table by my program [link given below]) appear together.
Doing the above enables the transaction and meta data to a relatively structured and processable format

Perform analysis on the Description column or the entire transactional df you have thus generated.

You can use the following code to convert a given PDF to a txt file -

from __future__ import unicode_literals
import os
import sys
reload(sys)
sys.setdefaultencoding('utf8')
from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter
from pdfminer.pdfpage import PDFPage
from pdfminer.converter import XMLConverter, HTMLConverter, TextConverter
from pdfminer.layout import LAParams
from cStringIO import StringIO

def pdfparser(data):
    dest = data[:-3]+"txt"
    print "\n\n\n\n\n",dest
    fp = file(data, 'rb')
    rsrcmgr = PDFResourceManager()
    retstr = StringIO()
    codec = 'utf-8'
    laparams = LAParams()
    device = TextConverter(rsrcmgr, retstr, codec=codec, laparams=laparams)
    # Create a PDF interpreter object.
    interpreter = PDFPageInterpreter(rsrcmgr, device)
    # Process each page contained in the document.
    for page in PDFPage.get_pages(fp):
        interpreter.process_page(page)
        data =  retstr.getvalue()
        data = unicode(data,'utf-8', errors='ignore')
    #print data
    # write data to a file
    print data
    with open(dest, "w") as f:
        f.write(data)

# set the working directory
path = "F:/banking/payments/"
os.chdir(path)
fls=os.listdir(path)

for x in fls:
    if x[-3:]=="pdf":
        pdfparser(path+x)


How to improve machine learning model using 2+ datasets

You may wish to combine the two models using an approach called Ensemble Methods.
However, in your case I'm surprised that using the combined data in a single model is so much worse if as you say the distribution of the features is similar.
So two potential outcomes:

if the distributions of the data are the same, the it could be the individual models are overfitted and given the larger data set this is exposed.

if the distributions are different then each model is being tuned for specific features and hence a single general model is hard to achieve.

To get a better feel of what to expect, I would suggest doing some distribution plots for each feature to see if they are as you say similar.
Another simple test is to take Model A and use it to predict everything in Dataset B and vice-versa. If the actual accuracy is around 70% then everything is working as expected.


Why does my Keras Conv Net only return 1?

It is only outputting 1 because softmax makes no sense when given a single input. Softmax computed with only one input $x$ is equivalent to $\frac{e^x}{e^x} = 1$.

In other words, the line

model.add(Dense(1, activation='softmax'))

is wrong. Use sigmoid instead:

model.add(Dense(1, activation='sigmoid'))

or read more about one-hot encoding your expected output and the helper function https://keras.io/utils/#to_categorical.


The difference between data science and algorithm development

A data scientist has many general skills, such as

data collection
software developer
data engineering and cleansing
algorithm development, selection and refining **
statistical analysis
data visualization
etc.

A deep learning algorithm developer would be more specialized, concentrating more on that one aspect of the data science field, and less on the other skills.
Typically an algorithm developer will be part of a data science team, consisting of members having different skills, but overall the team will possess all of them.


How neural networks handle large variations in the input?

They can handle large variations of inputs because the neurons have weights and those weights get optimized as part of learning a good model. So even though the model might take in a value that ranges from 0 to 100,000, if it isn't relevant to predicting the outcome, it will have a very small effect.


Which is better: GPT or RelGAN for text generation?

According to [Caccia et al., 2018], in general textual GANs are no rival for LMs regarding several quality measures. These are the conclusions of the paper:

  This research demonstrates that well-adjusted language models are a remarkably strong baseline
  and that temperature sweeping can provide a very clear characterization of model performance. A
  well-adjusted language model outperforms the considered GAN variants as evaluated on both local,
  and more surprisingly, global metrics of quality and diversity. Our temperature sweeping framework
  shares characteristics with a Receiver Operating Curve. Analogously, if one needed a single scalar to
  compare NLG models, one could compute area under the curve and seek the model with the smallest
  value (lower is better for our considered metrics).

  GAN-based generative models have been proven effective on real-valued data, however, but there
  exist many difficult pernicious issues of moving to discrete data. These issues must be overcome
  before they will improve over the strong MLE baselines. On the datasets and tasks considered,
  potential issues caused by exposure bias were less than the issues of training GANs in discrete data.
  GAN training may prove fruitful eventually, but this research lays forth clear boundaries that it must
  first surpass.

This way, OpenAI's GPT and GPT-2 may be considered superior in text generation quality to current textual GANs.


df_2['bigram_scored'] = df_2['bigram_finder'].apply(lambda x: x.score_ngrams(bigram_measures.raw_freq))


Convert Numpy array with 'n' and 'y' into integer array of 0 and 1

It is quite trivial

(X=='y').astype(int)

Should do the trick. It simply converts your array to True or False according to your requirements and then astype will impose the required datatype. By default int will give you 1 for True and 0 for False.


How to download images and bounding boxes from imageNet?

But the downloaded images and bounding boxes don't have matching  ames. So it is impossible to tell which xml file is for which image.

I don't think that happens, image and annotation may have same name. But Bbox and image is odd.

Still if you meant image and annotation that would be odd.


Is it ideally correct to benchmark neo4j as graph processing platform?

No

As Emre has rightly pointed out, the Chief Scientist of the company himself has written a blog post claiming the same.

However, Neo4j does a lot of computations which graph processing tools can do. In fact, it does graph traversals much faster than giraph, due to the Hadoop overhead and also as it stores the adjacent nodes in a doubly linked list.

So, it's not rare to confuse Neo4j with a graph processing platform (thus the claim in the paper) due to it's overlap with the features of a processing platform like Pregel and Giraph.


feature selection using genetic algorithm in Python?

Feature selection is a combinatorial optimization problem. And genetic algorithms is an optimization technique.

So there really isn't anything special, you just need to formulate your problem as an optimization one, and understand how do genetic algorithms optimize. There are enough tutorials on this.

Whether it's better or worse you already know the answer. It depends. On the dataset, constraints etc. What I can tell you from experience is that

You can not expect it to blow your mind but they do work pretty well
They are a great ensembler, meaning results are pretty different (yet accurate) from tree-based methods, NN etc...

Finally regarding implementation, here is completely (maybe too much) automated library based on genetic programming. (notice the word programming here referring to optimization not writing code) Also, it covers feature selection.


How implement a feed forward network with arbitrary node connections in keras?

A possible solution can be found at https://stackoverflow.com/questions/48111217/how-implement-a-feed-forward-network-with-arbitrary-node-connections-in-keras?noredirect=1#comment83198128_48111217

"The suggestion is simply work with many little layers and make as many skip connections as you want. Many parallel layers with 1 node represent well a single layer with many nodes. So you can, for instance, create lots of Dense(1) layers and treat them as nodes. Then you connect them in any way you like."


The book quotes in your question are the outline of a proof that shows single output sigmoid represents the same class probabilities as a two dimensional softmax.

  What does it mean by "impose a requirement that one element of $z$ be fixed." I understand you can get the $n^{th}$ one if you know n-1 but "fixed"?

It doesn't mean the value has to be fixed. It means that we can choose any $z_n$ value, and have the other's adjusted to represent any valid target probabilities. So for argument's sake, we can pick any fixed value.

  How is it equivalent to having $z_1=0$?

It is an equivalence between $\sigma(z)$ and $softmax(z)_1$ with $z_1 = 0$:

$\sigma(z) = \frac{1}{1 + e^{-z}}$

For softmax, this is the function for the first element from a 2-dimensional output:

$softmax(z)_1 = \frac{e^{z_1}}{\sum_{i=1}^{2}e^{z_i}} = \frac{e^{z_1}}{e^{z_1} + e^{z_2}}$

But we have set $z_1 = 0$, and $e^0 = 1$ so:

$softmax(z)_1 = \frac{1}{1 + e^{z_2}}$

This is the same function as $\sigma(-z_2)$, so the two functions have identical form (if you plot them, they overlap), although the parameters drive the specific value differently.

Note that as per step 1, we have not lost anything about the general behaviour by setting $z_1 = 0$, we just did it to simplify the equation. In fact it is possible to make a very similar argument without fixing any $z$ values, and you end up with a more generic linear combination of $z_1$ and $z_2$ in the softmax function being equivalent to just $z$ in $\sigma(z)$.


How to decide to use K-fold validation or not?

The main criterion is that you need enough data in your training set to get a good model fit - which is both a function of the data quantity and the model complexity.

When you don't have much training data, you want as much data in your training set as possible - in this case, you can do "leave one out" cross-validation.  This is like k-fold, except it's N-fold, where N is the number of observations you have. This means you hold one observation out, build a model using everything else, validate using that one sample, and repeat N times.

When you have more data with a simple model, you can use a much more loose validation strategy. For example, if you have a million records and are doing linear regression with one predictor, you can get away with using a few thousand samples to build your model, since you'll get really good parameter estimates from thousands of samples (or fewer).

Those are the extremes. The suggestions you've seen live somewhere in-between.  When you pick high K, you have to build lots of models (this takes time), but each model is built with more data (good for complex models).  When you pick low K, you have to build fewer models, but each model is built with less data.

Personally, I pick a low value for K when I'm doing model exploration, but a higher value when I'm ready for validation of a good candidate model.  Actually, in this case you should have three sets - a training set, a validation set (that you use to find a good set of parameters), and a second validation set that you've held out the ENTIRE time (since, in a way, the other validation sets are helping you 'train' the model parameters).


Is there any formal explanation for the sensitivity of AdaBoost to outliers?

Consider a supervised problem with target values y either being -1 or 1. Let's say we are fitting function f(x) using our AdaBoost classifier. Intuitively we want positive margins, where a margin is defined as $y_{i}f(x_{i})$. A positive margin means we have correctly classified that sample since the sign of both f and y are the same. A negative margin means we have misclassified the sample.

The loss function for Adaboost is defined as an exponential:

$L(y, f(x)) = exp(-yf(x))$

This exponential then ends up more strongly penalizing points with a large negative margin (i.e., outliers) than other classifiers. Here is a plot of the loss function vs. the margin for several classifiers:

So, to summarize: Adaboost is susceptible to overfitting outliers due to the form of its loss function.

Source: Hastie, et. al, The Elements of Statistical Learning, 2017, pp. 343-347


Is there an inherent recency bias in deep learning?

Your question is very interesting, however I feel you are overlooking a key point in your reasoning:
You usually take a model checkpoint at the point that it performs best on the validation set. This means that the instance of the model you keep is inherently the most robust and generalizable version of the model that you have evaluated, thus suffering the least from recency bias.
Suppose though, you don't checkpoint the model but stop it at a point arbitrarily. Naturally, you'd think that the samples in the final batch influenced the current state of the model much more than the first batches of the epoch. In practice this would show up as regular overfitting, however, instead of recency bias.
Some ways to deal with this:

relatively small learning rate
equivalently regularization as parameter norm penalties (i.e. L1, L2, ...)
ensembling
other more specialized techniques such as SGDA


remove seasonality from weekly time series data

I can't replicate (without having the data), but from what I see, you are trying to apply the decompose function to the entire object - it's probably expecting a single vector, but receives a data-frame-like object and attempts to apply the decomposition to both columns. either use a different time series package (like zoo) which does not store timestamps as a column, or apply decompose explicitly to salestsDec[,2].


Are there real world applications where deep fully connected networks are better suited than ConvNets

Yes, there are plenty use case where a fully connected network is more appropriate.

ConvNet works because the weights are shared in its kernels. It makes a lot intuitive sense to share the weights in image recognition task, because a flower at the top left of a picture has the same local pattern as the flower at the right bottom of the picture. What we want is to learn of pattern that is local.

However, this logic no longer applies when you consider program such as the famous iris  classification problem. The idea of "local" simply doesn't exist for such data set. each of the features (i.e, petal width, petal length etc) are all "standalone" concept. In those case, it simply doesn't make sense to use ConvNet.

BTW, you can use ConvNet on financial data as well. As financial problem usually involves a time dimension, where the concepts of "local" does exist.


Here is a suggestion:

Code your analysis in such a way that it can be run on sub-samples.
Code a complementary routine which can sample, either randomly, or by time, or by region, or ...  This may be domain-specific. This is where your knowledge enters.
Combine the two and see if the results are stable across subsamples.


Explanation for MLP classification probability

It depends a little on the specific model.

If your model is using a softmax output layer, then the values are usually interpreted as mutually-exclusive class probabilities and should sum to 1.

If your model is using a sigmoid output layer, then the values for a classification problem can be interpreted as the individual, non-exclusive probabilities. In that case, it is possible for an example to not be in either of your classes (an output close to [0,0]), or to definitely be in both classes (an output close to [1,1]) - if that is not the case in the problem that you have trained the NN to model, then you might want to consider altering the model.

So, if your classification should be mutually-exclusive, then the person asking you the question has a valid point. The probabilities should sum to 1, and you may need to fix the model. In that case, you should use a softmax output layer, and would not have this problem because the probabilities would always sum to 1.

Whether it is worth fixing the model, depends on your understanding of mutually-exclusive vs non-exclusive classes in the problem, whether the current model meets its goals (it is not a big issue in practice, if the model is not a strict theoretical match to the problem, but has a perfectly good accuracy) and how much effort it would take to alter your code and re-train the model.

Your statement:

  the probability stands for confidence concept in mathematics.

is technically correct, but not useful as an answer to your colleague's query. For classification, the terms confidence and probability are almost interchangeable, so you cannot really use one word to explain the other. If someone asked why one of the numbers was larger than another in your model, you would not answer "that is because x is greater than y" - essentially this is what you have done here. So your statement is not wrong as such, but neither does it explain the result.


What is the difference between all the different types of learning within machine learning?

Supervised Learning

In general, supervised learning refers to a situation in which you have some $X$ that is related to $y$, so that you can model how both are related (on average). In this case you can build a statistical model:

$$y = \beta X + u.$$

The model learns $\beta$, so that you are able to predict $y$ by: $\hat{y}=\hat{\beta}X$.

Examples are regression or classification tasks (e.g. using linear models, neural nets, boosting etc).

Unsupervised Learning

When you have only $X$ and no response $y$, you face an unsupervised problem. In this case, the focus is to identify patterns in $X$, e.g. by clustering. Examples for unsupervised learning are k-nearest neighbors (KNN) or principle components (PCA).

The book Introduction to Statistical Learning provides a very good overview. Alternatively, the book Elements of Statistical Learning provides a more comprehensive overview.

Reinforcement Learning

Reinforcement learning is a little different. Here you try to find an optimal response to some problem, based on past experiance. Examples are self driving cars or a game of tic tac toe. So for every state of the environment (e.g. the state of the tic tac toe game), RF aims at finding an optimal response (best next move). Googles DeepMind and AlphaGo are RF-based. The book by Sutton and Barto is the standard literature here.


Yes, if your 0.85 AUC is good enough for your use case this is a good enough model. The performance on the training set indicates how well your model knows the training set. This we don't really care about, it's just what the model tries to optimize. The performance on the test set is an indication on how well your model generalizes. This is what we care about, and your model gets to around 0.85 as an estimate for your generalization. Differences between training and testing are the norm and in this case it could be that you might get a better performance by adding stronger regularization but if 0.85 is good enough, go for it!


How to train on extended data set correctly

Transfer learning is what you want to need.

Checkout this note from CS231n, it provides some general advices on transfer learning and model fine-tuning.

In the meanwhile, this blog from keras shows how to use pre-trained VGG16 network to classify dogs&cats.


This splits the string and filters the substrings by length, no regex:

string <- "blabla a test this that b á"
paste(Filter(function(x) nchar(x) > 1,
             unlist(strsplit(string, " "))),
      collapse=" ")
# [1] "blabla test this that"

(does not work with multiple space characters)


significance test and sample size estimation for classifiers

I think there are two primary possibilities for answers, so I'll try to address both of them:
(Recommended) If you have multiple independent trials per classifier:
This is a lot simpler than if you have only one trial per classifer (and much better). If you have multiple $F_1$ scores per classifier, then you can simply conduct a paired t-test or a Wilcoxon signed rank test. You'd probably go with Wilcoxon if you don't have many trials to approximate a normal distribution via the central limit theorem or if for whatever reason your $F_1$ scores are not normally distributed.
If you only have one trial per classifier:
In this situation, we are going to find a way to express $F_1$ in terms of $F_1\pm \delta_{F_1}$ to see if the two scores overlap given some confidence level.
Step 1: Decomposition
We can first decompose the metric we are trying to calculate:
$$F_1\equiv2\frac{precision\cdot recall}{precision+recall}\equiv2\frac{PPV\cdot TPR}{PPV+TPR}$$
We are looking to find the uncertainty in the $F_1$ score, so we first need to find the uncertainties in the values composing it. To do this, we can consider the precision and recall metrics in terms of Bernoulli trials by partitioning the instances into $P$ and $N$, positives and negatives. Let's also assume $P+N\gg1$.

Edit to answer the question about determining sample sizes:
First, choose a maximum width $w$ for your confidence interval in terms of percentage points (i.e. choose a margin of error MoE and double it to account for both sides of the mean). Then, using our notion of Bernoulli trials, we can determine your minimum sample size based on the z-score for your confidence interval as well as the width (MoE):
$$n\ge \frac{4z^2(p(1-p))}{w^2} \because z\sqrt{\frac{p(1-p)}{n}}\le MoE$$
The safest estimate would be to choose $p=0.5$ which would maximize $n$ for a given $z$ and $MoE$. Realistically though, this would be an overestimate for the required $n$ because you probably don't have a random classifier.
As an example, if you wanted an MoE of 10% for a CI of 95%, you'd need:
$n\ge 97$ assuming worst case $p=0.5$.

When we consider Bernoulli trials, we can talk in terms of success/failure. So the number of successful $P$, which would be $TP$, can be considered a percentage of the total number of $P$ instances, which is simply the true positive rate $TPR$. The same goes for $TN$ and $N$, being $TNR$. In other words:
$$TPR=\frac{TP}{P}=\frac{TP}{TP+FN}\enspace \enspace, \enspace\enspace TNR=\frac{TN}{N}=\frac{TN}{TN+FP}$$
Step 2: Binomial Proportion Confidence Intervals
Now, we can develop binomial proportion confidence intervals per $P$ and $N$. There are many formulas to do this, but I am going to stick with probably the most basic one. That way, we make sure that our result assumes the form $TPR\pm \delta_{TPR}$.
Now let's pick a confidence level:

If you want a $99\%$, then $z_{0.01/2}\approx2.576$
If you want $95\%$, then $z_{0.05/2}\approx 1.96$ confidence level

We can calculate the individual confidence intervals:
$$TPR\pm z\sqrt{\frac{TPR(1-TPR)}{P}}\enspace \enspace, \enspace\enspace TNR\pm z\sqrt{\frac{TNR(1-TNR)}{N}}$$
To get $PPV$, we can first define $TP=P(TPR)$ and $FP=N(1-TNR)$. Then:
$$ PPV=\frac{TP}{TP+FP}=\frac{P(TPR)}{P(TPR)+N(1-TNR)} $$
Step 3: Propagate PPV Uncertainties
We want to find $PPV\pm \delta_{PPV}$, so we need to propagate the uncertainties from $TPR$ and $TNR$. Because $P$ and $N$ are determined by the dataset rather than your model (and they are exact "measurements"), they are without uncertainties:
$$ PPV\pm \frac{P(TPR\pm \delta_{TPR})}{P(TPR\pm \delta_{TPR})+N-N(TNR\pm \delta_{TNR})}=\frac{X}{Y} $$
Let's first find the individual components $X$ and $Y$, keeping in mind that when uncertain values add, their uncertainties add as well:
$$ X=P(TPR)\pm P(\delta_{TPR}) $$
$$ Y=(N+P(TPR)+N(TNR))\pm (P(\delta_{TPR})+N(\delta_{TNR})) $$
$$ = \frac{P(TPR)\pm P(\delta_{TPR})}{P(TPR\pm \delta_{TPR})+N-N(TNR\pm \delta_{TNR})}$$
Now we can divide $X$ by $Y$. To do so, we need to keep in mind that the resulting uncertainty is the sum of their fractional uncertainties, like so:
$$ PPV=\frac{X}{Y}\enspace \enspace , \enspace \enspace \frac{\delta_{PPV}}{|PPV|}=\frac{\delta_X}{|X|}+\frac{\delta_Y}{|Y|}\Rightarrow \delta{PPV}=\frac{X}{Y}\left(\frac{\delta_X}{|X|}+\frac{\delta_Y}{|Y|}\right) $$
For simplicity, I'll write $PPV$ because we know that $$PPV\pm \delta_{PPV}\equiv \frac{X}{Y}\pm\frac{X}{Y}\left(\frac{\delta_X}{|X|}+\frac{\delta_Y}{|Y|}\right)$$
(Finally) Step 4: Propagate TPR and PPV Uncertainties
Just as we did with $PPV$, we need to propagate the uncertainties for the $F_1$ score calculation. Fortunately, it's the same process as the uncertainties for multiplication combine in the same manner as division:
$$ F_1 = 2\frac{PPV\cdot TPR}{PPV+TPR} = 2\frac{A}{B}$$
$$ A=(PPV\cdot TPR)\enspace \enspace, \enspace \enspace B=(PPV+TPR) $$
$$ \frac{\delta_A}{|A|}=\frac{\delta_{PPV}}{|PPV|} + \frac{\delta_{TPR}}{|TPR|}\Rightarrow \delta_A=|A|\left(\frac{\delta_{PPV}}{|PPV|} + \frac{\delta_{TPR}}{|TPR|}\right) $$
$$ \delta_B = \delta_{PPV} + \delta_{TPR} $$
Then we also have to combine the uncertainties for the division of $A$ by $B$:
$$ \delta_{\frac{A}{B}} = \left|\frac{A}{B}\right|\left(\frac{\delta_A}{|A|}+\frac{\delta_B}{|B|}\right) $$
This yields our final answer because we know that:
$$ F_1 \pm \delta_{F_1} \equiv 2\left(\frac{A}{B} \pm \delta_\frac{A}{B}\right) $$
Now you can see whether your $F_1$ scores overlap given some confidence interval! You could also probably see how much they overlap in terms of a ratio, but it'd probably be more understandable if you just stuck with a binary yes/no overlap.


Does save_best_only in Keras prevents overfitting?

It really only tracks the value of the metric you selected, there is no tolerance option. In the relevant documentation, the definition is given:

  save_best_only: if save_best_only=True, the latest best model
  according to the quantity monitored will not be overwritten.

In your case, combing the callback with the EarlyStopping callback would be the best option. This callback does offer some kind of tolerance, i.e. it requires the improvement of the metric to be at least as good as min_delta to be considered an improvement.

You could combine both callbacks with something like this:

from keras.callbacks.callbacks import EarlyStopping

early_stop = EarlyStopping(monitor='val_loss', min_delta=0.05, patience=3, verbose=1, mode='auto')
checkpoints = ModelCheckpoint('model_weights.h5', monitor='val_acc', save_best_only=True)

my_callbacks = [early_stop, checkpoints]
...
model.fit(..., callbacks=my_callbacks)


CNN always predicts either 0 or 1 for binary classification

Sigmoid functions might have saturation problems. The values that it receives are probably too far from zero, and the sigmoid is returning 'extreme' results (i.e.: 0 or 1). I suggest you to keep your signal zero centered by putting BatchNormalization() layers between each of your layers (certainly between Dense() ones, but also between conv layers if you prefer).

Additionally, sigmoid functions are thought to perform not very well in general. The main configuration for classifiers is to put one output node for each target category (in this case, 2) and use softmax activation. Your loss at this point would be BinaryCrossentropy(). This requires one-hot encoding of your target variable.

I would also try other regularization tricks, such as dropout and l1-l2 regularization, or different initializations. There no rule of thumb here, you have to play with these hyperparameters and figure out what's the best configuration for your task.


Algorithms for aggregating duplicate identities based on non-numerical data?

I haven't yet successfully solved my record linkage problem, but I wanted to share some of the stuff I've found in the process in case it's of use to anyone else. This is a work in progress based here on GitHub.

Record Linkage Resources

(also known as deduplication, data matching, entity resolution)

Background

Documents

Wikipedia pages on Record Linkage and Data Deduplication
Overview slideshttps://www.umiacs.umd.edu/~getoor/Tutorials/ER_VLDB2012.pdf
[Dedupe]'s explanation of how their software works: https://dedupe.io/developers/library/en/latest/How-it-works.html

Talks

Peter Christen, Record Linkage lectures at ADRC-Scotland: 1 2 3 4 (2015)
Mike Mull, The Art and Science of Data Matching: https://www.youtube.com/watch?v=Y-nYEOgq3YE (2015)
Rhydwyn Mcguire, Join for real life: https://www.youtube.com/watch?v=cEcVIjyHfiQ (2013)
Andrew Rowe, Big Data Deduplication and Data Matching using Python: https://www.youtube.com/watch?v=Z6mlvrYEYnk (2013)

Books

Peter Christen, Data Matching: http://www.springer.com/us/book/9783642311635
Thomas N. Herzog, Fritz J. Scheuren and William E. Winkler, Data Quality and Record Linkage Techniques: http://www.springer.com/us/book/9780387695020

Free software

(last updated, github stars as of Nov 2017)

Python

Dedupe: https://github.com/dedupeio/dedupe (2017, 1,413)
RLTK: https://github.com/usc-isi-i2/rltk/ (2017, 13)
Febrl: https://sourceforge.net/projects/febrl/ (2013)

Java

Duke: https://github.com/larsga/Duke (2016, 469)
JedAI: https://github.com/scify/JedAIToolkit (2017, 12)
LSHDB: https://github.com/dimkar121/LSHDB (2017, 9)
FRIL: http://fril.sourceforge.net/ (2011) (Windows binary available)

R

RecordLinkage: https://r-forge.r-project.org/projects/recordlinkage/ https://cran.r-project.org/web/packages/RecordLinkage/index.html
fastLink: https://github.com/kosukeimai/fastLink

Other

OpenRecLink: http://reclink.sourceforge.net/ (2016)

C++ with GUI

Registry Plus™ Link Plus: https://www.cdc.gov/cancer/npcr/tools/registryplus/lp_tech_info.htm (2007)

Commercial software and solutions

Data Ladder DataMatch: https://dataladder.com/
Dedupe: https://dedupe.io/ (freemium frontend for Dedupe Python library)
LinkageWiz: http://www.linkagewiz.net/
WinPure Clean and Match: http://www.winpure.com/cleanmatch.html
Reifier: http://nubetech.co/

For SAS

(free but requires SAS) The Link King: http://www.the-link-king.com/

Data Cleaning

Name Parsers

Python

probablepeople: https://github.com/datamade/probablepeople (2017, 204)
Name Parser: https://github.com/derek73/python-nameparser (2017, 232)

JavaScript

parse-full-name: https://github.com/dschnelldavis/parse-full-name (2017, 18)

Papers

US Census Bureau Data Linkage research: https://www.census.gov/srd/csrm/RecordLinkage.html
Learnable Similarity Functions and their Application to Record Linkage and Clustering (via)
[https://www.microsoft.com/en-us/research/publication/improving-entity-resolution-with-global-constraints/ (via)
https://arxiv.org/abs/1312.4645 (via)
http://people.cs.umass.edu/~mwick/MikeWeb/Publications_files/wick09entity.pdf (via)
http://homes.cs.washington.edu/~pedrod/papers/mrdm04.pdf (via)
https://www.researchgate.net/publication/318874102_Clink_-_A_Novel_Record_Linkage_Methodology_based_on_Graph_Interactions

Organizations

US Census Burea - Center for Statistical Research and Methodolgy - Record Linkage: https://www.census.gov/srd/csrm/RecordLinkage.html
Stanford Entity Resolution Framework: http://infolab.stanford.edu/serf/
ANU Data Mining and Matching Group https://dmm.anu.edu.au/ https://web.archive.org/web/20160515215747/datamining.anu.edu.au/projects/linkage.html (Archive.org link)

Misc

DuDe (framework for comparing record linkage results): https://hpi.de/naumann/projects/data-quality-and-cleansing/dude-duplicate-detection.html
Datasets to use for evaluating deduplication software: https://hpi.de/naumann/projects/repeatability/datasets.html
https://www2.vrdc.cornell.edu/news/3/20050420-Record%20Linkage%20Software.pdf


Should I expect major performance improvements by scaling my features?

It's impossible to know whether the performance will improve without knowing what algorithm you are using. Even then the only way to tell is to try both.

That being said, I can't think of any scenario that standardisation would hurt the performance.


Improve the loss reduction in a neural network model

Generally, the decrease in loss tends to be smaller, the longer you train your model. You can think about this in a way, that the model first makes good progress in learning, but later any further improvement becomes harder (thus slower). At some point the model stops to learn. This comes from the logic of gradient decent, a numerical optimization process which is behind most ML models. If the model has learned what it is able to learn, loss does not decrease any more.

What can you do about this? You can try to make your model „better“ in terms of learning capacity. You can increase the capacity of the model (more neurons) or add more layers. You can also adjust the learning rate during learning by „callbacks“ (ReduceLROnPlateau). In this case, you lower the LR automatically if learning progress becomes small. By doing so you can try to make the model to learn more detailed patterns. See callbacks for Keras: https://keras.io/callbacks/.

Here is a nice blogpost about how to train NN: http://karpathy.github.io/2019/04/25/recipe/


Sound Classification for Multiple Classes for English Letters

If in each file the letters are spoken separately, with silence in between, and always in the same order (A,B...,Z) then one can try to automate finding each section and its label.
Use a Voice Activity Detection (VAD) module to detect each spoken character. Then assign A to the first voiced area, B to the next etc. If you have very clean and uniform audio, then it may work to use volume instead of voice activity detector.
Note that this process is a bit vulnerable to missed sections, because then all the subsequent labels will be shifted. It is also vulnerable to speakers saying characters out of order, or repeated.
An alternative would be to use a Speech Recognition system. That should be able to handle arbitrary order of utterances.


What is an appropriate approach to sampling for probability of default using a classification model?

You should have a look at this Kaggle competition on fraud detection. In this kernel the winning team explains their approach.

This could be a time series approach (same as yours) but in this case:

  We are not predicting fraudulent transactions. According to the competition host Lynn here. Once a client (credit card) has fraud, their entire account is converted to isFraud=1. Therefore we are predicting fraudulent clients (credit cards).

For this team that won the challenge the solution was:

  We did a CV GroupKFold using month as the group.

It seems that perhaps you could use the same validation strategy.

https://www.kaggle.com/c/ieee-fraud-detection/discussion/111284


Recurring events - finding in a time series

I'd suggest to use ARIMA (autoregressive iterate moving average) as a way to detect the regularities in your time series. Take a look to the following link, in which you will be introduced to ARIMA by means of series of blog posts:

http://ucanalytics.com/blogs/forecasting-time-series-analysis-manufacturing-case-study-example-part-1/


Mapping one embedding to another using Deep Learning

You should take into account a few factors:

some_value is key here. You should probably choose a value greater than e1_dim and e2_dim. This may be a problem due to the large memory needs for high embedding dimensionalities.
if e2_dim is bounded to $(-1,+1)$, you may want to set a tanh activation in the last layer (i.e. after Dense(e2_dim)). With that in place, you may want to replace the activation of the hidden layer with a ReLU, to avoid squashing the gradients unnecessarily.
If your embeddings are directional, or they are in the unit ball, then you should be fine with cosine_proximity. Maybe a better option would be to use the von Mises-Fischer distance, but I think it's not straightforward. Otherwise, you may try World-Mover's Distance.

That being said, a lot of embedded space mapping works out there assume that the embedded spaces are approximately isomorphic and just go ahead with a linear transformation. You may want to have a look at the relevant bilingual embedding mapping literature; you may start with the work by Artetxe et al., 2016. I don't know if the mentioned assumption is met to any degree in your scenario (skipgram $\leftrightarrow$ node2vec), but you may assess this with the VF2 algorithm, like Artetxe et al. did.


The classical solution is to try a one-class SVM applied only to the elements ranked above the others - you can read some papers about it looking by SVM rank or SVM ranking that will show you why that works.


Identifying and Accounting for trend/seasonality in Predictor Variables

Using time as an explanatory variable may be a good start. Then, depending on your model you may want to add interaction effects between time and your other variables. This will take into account change over time.

However, it will be a bit difficult to link evolution of the model to the evolution of some explanatory variable. If you know what variable evolve over time, you may just want to add time differentiated variables ($X_n - X_{n-1}$). This will give you a clearer view on what evolution is impacting the model output. Depending on your problem you might want to consider different time lags.


Using historical label as a feature in my ML model?

Correct me if I'm wrong, but it sounds like you are using an EMA to generate the price that you will be using to label as either 1,0, or -1. If so, that is not a problem to do so with historical data, as EMA calculations are only affected by data previous to whatever record the EMA is being based off of. Just make sure that your historical data is calculated as if it was the present moment for every record.


Interesting problem. You certainly could use time series methods for this, but you should consider whether you think there is face-validity to song plays following a temporal pattern. From your plot above, I don't see any structure in the variation, but perhaps there is something that an algorithm can pick up. A good starting point would be R's auto.arima function for autoregressive integrated moving average (ARIMA). The ARIMA method is more fully described elsewhere (e.g., here), but an interesting extension to this approach might be using exogenous predictors in your model. For example, does the presence or absence of rain affect the likelihood of some songs being played? My guess would be yes (I love listening to post-rock bands on rainy days), but perhaps that's not true of everyone. This extension to ARIMA is referred to as ARIMAX in the literature. A really nice breakdown of how to handle covariates in your type of analysis can be found on Rob Hyndman's blog.

A second thought is that, with sufficient computational resources, there's an interesting hidden markov model (HMM) problem here. You could potentially model the transition probabilities from one song to another. Presumably there's a high likelihood of transitioning from track 1 to track 2 of a given record, but sometimes people might switch to a different record after listening to a favorite track on an album. The extent to which this is true for different tracks could be a part of this analysis, and you could attempt to model the hidden underlying states that motivate a listener to stay within a record, or to move to a new one.


As per my knowledge, it doesn't matter for a decision tree model whether the features are ordinal or categorical. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. Decision trees describe patterns by using a list of attributes.

For a more detailed explanation, I am providing here some links which you will find helpful for such queries.
http://www.ibm.com/support/knowledgecenter/SS3RA7_17.0.0/clementine/nodes_treebuilding.html

http://scikit-learn.org/stable/modules/tree.html

http://www.ryerson.ca/~rmichon/mkt700/SPSS/Creating%20Decision%20Trees.htm


Columns with no (or nearly no) differences between rows worth keeping?

The features you described are known as low variance features and in general you should remove those features. The rationale behind this, is that low variance feature contain less information. See this for a succinct explanation and some code in python. Be sure to normalize your features. One of the advantages is to speed the training phase.


This is a good question to ask as it pertains to important neural network concepts.

"Why don't we have several separate layers, each of which does a
specific extraction of some features and once all the needed has been
extracted"

The answer is two-fold.
First, the layers depend on each other. The reason deeper layers can extract higher-level features is that they build upon the lower features extracted in the earlier layers. The layers may be different in architecture but they could simply be the same layer duplicated multiple times and initialized with the same weights.
Second, it is important to understand that the layers are not "told" what type of features to extract. Realizing that each layers represented various level of features was very important in better understanding the behaviour of neural networks and to confirm that no, it isn't all black magic. And it is important to understand that is a learning artefact which may not always occur depending on the network, data, and training strategy involved.


Improving accuracy on highly imbalanced dataset

I'm not very sure what you mean by "60% accuracy using AUC". Accuracy and AUC are two different metrics... I'm going to answer as if you're referring to classification accuracy, since that's in your title and the first sentence of your post.

First of all, don't use accuracy to evaluate performance on imbalanced data!

Your dataset has an imbalance ratio of 6843/159730 which is around 1/23. This means that if you make a dummy classifier that just predicts the majority class you'd get an accuracy of 96%. There are better options for imbalanced data such as the f1 score or any macro-averaged metric (you can read this post more information).

Secondly, I'm not sure what you're doing it but, just in any case, you shouldn't evaluate on the oversampled dataset.

As for ideas for improving performance, I don't have many because you are doing most things right. Tree-based algorithms (e.g. XGBoost) are good for dealing with imbalanced data. You are already oversampling the data, which helps a lot. Some other ideas are:

Try different oversamplers, undersamplers or perhaps a combination of over and under-sampling techniques.
Search to optimize the hyperparameters of your XGBoost. I can't tell by the information you gave, but maybe you're overfitting.
Try different algorithms (catboost, lightgbm, etc.), or maybe ensembles of those models (stacked models, etc.).


Brownian motion in models for virus spread?

I read this article when it came out. I don't think the point is about whether or not people behave like particles in a fluid medium, but about how quickly people can spread the coronavirus when randomly interacting with one another in an open space (in my opinion, this model would be a reasonably good representation for a crowded area like a mall lobby or a beach where people are moving around more or less randomly).

However, even in other settings when people aren't moving around randomly, if anything, people are more likely to interact with each other in restricted spaces like sidewalks where there is less choice of where to position yourself spatially.


As the gensim tool cites the very famous paper by Mikolov - "Distributed Representations of Words and Phrases..." using which it is implemented. In the paper if you look at the section "4 Learning Phrases" they give a nice explanation of how n-grams are calculated (Equation 6).

So, if want to count bigrams this formula is straight-forward; score(wi, wj) is the score between any two words occuring together. But when counting trigrams, 'wi' will be a bigram and 'wj' will be a word. And same follows for any number of grams.


Stuck on building a customer support chatbot from scratch using reddit dataset

Welcome to the site! I'm going to make some assumptions here to clarify your question and attempt to give you some guidance: it sounds like (1) you are working with a general chatbot framework and (2) that framework has been trained on a dataset that contains text from reddit.

So, what you need to do is now take that framework and train it again on your own custom dataset (relating to colleges?). I'm assuming that the training can be done based on using a chatbot framework that has this flexibility.

At this point it would beneficial if you posted (1) which chatbot framework you are using and (2) tell us about your data. Do you have a lot of it? Is it in question and answer format?


Is a Neural Network with 20 times the number of input neurons (on hidden layers) guaranteed to overfit? When is this not so?

Figuring out the correct size for hidden layers in NN is a bit of an art, that said I have never used hidden layers with more than twice the number of neurones as the input layer.

As an example the MNIST problem has an input size of 784, I cannot imagine how the network will not overfit if the hidden layer is of size 784*20...

So answering to your question, I think overfit will pretty much occur on those occasions, if unsure, just try it in the MNIST dataser


Any classifier that naturally provides feature importance can be applied here. For example, take a look at Decision Trees and Random Forrest.

For other algorithms that do not provide such information (eg KNN and Neural Nets), you may take a (forward/backward/bidirectional) feature selection. For example, chose a subset of features, train a model and evaluate it with any preferred metrics (better to use bootstrapping or cross-validation). Choose another subset if features and try again. After several trials, you can find the subset with the best prediction power. This is the set of the mist contributing features.

Note that different techniques/measures may give you different results.


How to treat the datasets with unreliable labels

You have a common issue in which your learning data is noisy. Of course if you can do anything to make your data better/use more data this is the way to go. But I assume manual cleaning at large is not possible.

Reasons for the noise?

There are different approaches in machine learning to deal with noisy data. It depends a lot on where your noise is coming from. It helps to understand why your labels are incorrect. In your case it sounds like there is a structural mistake where you systematically misclassify under certain conditions.

ML methods to solve the noisy data issue

Besides simple rules (if applicable) there are more complex ways to approach the issue. There are many papers (e.g. this, this and this) that suggest different variants of the following:

Manually create a small subset of cleaned labels
Train the network on clean and noisy labels
Use an additional network (or layer) to either create a mapping from noisy to clean labels or to learn about the noise distribution

This way you try to make the network recognizing its own mistake. Some of these models are rather complex, others are quite complex. There is even a paper that trains the network to also classify the noise and than make predictions based on features + noise classification.

I am not sure if there are any best practises yet which architecture works best in which case. You may need to try out some alternatives.


Data preprocessing : Aggregation, feature creation, or else?

You are looking for names to attribute to the two items listed? For (1) I would just call it a transformation as it is a straight mapping with no change in distribution of the feature. When working with data, make sure you make copies of your data transformation and do not alter the original data set.

For (2), since it is a single number per group, where group here is the full data set I would call it an aggregation. Likewise if you did a similar calculation per user. If however, you computed a new value from existing features for each record, this would be feature generation or creation.


How does C have effects on bias and variance of a Support Vector Machine?

The C being a regularized parameter, controls how much you want to punish your model for each misclassified point for a given curve.
If you put large value to C it will try to reduce errors but at the same time it may happen that it would not perform better on test dataset hence cause overfitting.
To get to know more about effect of C in svm. Refer this.


Training multi-label classifier with low quality training set

You almost solved the problem in the last paragraph. Expressed more formally, your cost function could be

$$\frac{1}{N} \sum_{i,j} c_{i,j} y_{i,j} \log x_{i,j}$$

where $i$ runs over items/documents, and $j$ runs over classes, $x$ is your prediction, $y$ is the binary label (1 if item $i$ has class $j$), and $0 < c < 1$ is your confidence. This is a simple modification of the cross entropy. When the confidence $c$ is low, the value of the prediction matters less.