What is the “learning” step in Gaussian Naive Bayes classification?

I'm assuming you're asking about the intuition behind Naive Bayes (NB). For the sake of clarity I'm considering only categorical features. Gaussian NB is simply an application of NB to numerical features (assumed to be normally distributed).

During training every $p(f_i|C_k)$ is calculated by counting how often the feature value $f_i$ is associated with class $C_k$ among all the other possible features values associated with $C_k$:

This is done by counting how often $f_i$ and $C_k$ appear together across all the instances. That's how NB generalizes: the fact that a feature appears with a particular class is just an example, but the fact it appears proportionally more often with class A than class B forms a pattern.
The probability $p(f_i|C_k)$ represents how important $f_i$ is within class $C_k$.

When predicting the class for a new instance:

NB "weighs all the pros and cons" for this instance to be predicted as $C_k$ by combining all the $p(f_i|C_k)$ corresponding to this instance (in the sense that some of the probabilities $p(f_i|C_k)$ are low and some are high, so their product reflects the combination of "pros" and "cons" indications).
But even if $p(f_i|A) > p(f_i|B)$, it doesn't imply that $f_i$ is a strong indication of class $A$, because maybe this is due to class $A$ being less frequent than class $B$. This is taken into account in the prior $p(C_k)$, which gives more importance to a frequent class than a rare one (this is the basis of the Bayes theorem).

These last two points show how NB uses the "knowledge" of the trained model to make a prediction about any unknown instance.


How to optimize for time correlated hidden function - the magical candy machine

Thought #1

From the short description you give, and assuming you have some data (or can synthesise it), I would have a go at training a Hidden Markov Model. Have a look here for an awesome visual primer. Intuitively, HMMs do what you describe.

There are some observables:

how much candy you put in the machine,
how much comes out, and
which timestep we are in

There are certain states (here just discrete states):

amount of dispensed candy goes up (strictly increasing)
amount of dispensed candy goes down (strictly decreasing)

Lastly, there is a transition: between these state, given the inputs. These are the observables, as far as you are concerned.

With HMMs, we are however assuming that there is more than meets the eye; there are some hiddens states (latent states), which are unobservable to use. The model keeps track of these and uses them in conjunction with the observables list above to decide how to act - it has the full mapping function. Below is a schematic diagram of such a model:

This just shows several states, along with the probabilities of transitioning to the other states, or stating in the same state. Check out the source for a detailed description of the diagram. One can already imagine sketching this drawing for your problem.

For more on the theory of HMMs, I recommend this text based walkthrough by Jeff Bilmes, and this YouTube series by mathematicalmonk (chapter 14 of the playlist). If you go this route, you might consider trying out the Pomegranate library (in Python).

The final logic of deciding what to do, what action to take, is something you could either hard-code or use a model to take it for you. For example, a hard-coded approach would be something like:

if current_state == 'normal':
    take_lots_of_candy()
elif current_state == 'unsure':
    take_some_candy()
elif current_State == 'dangerous':
    pause_taking_candy()
else:
    catch_other_states()

For a model-based approach, you could expand further on the ideas outlines below.

Thought #2

Of course, you are in essence just mapping some input to some outputs. Almost any machine learning algorithm will have a good go at it and - depending on the complexity of the internal workings of your magical machine - will be better or worse. Algorithms such as boosting (e.g. via gradient descent) generally require you to put some kind of information into the model e.g. by specifying a regression equation, mapping inputs to outputs. A good implementation is the mboost package in R. There is a great tutorial to help decide what might be important.

Thought #3

Coming back to the point of just mapping inputs to outputs: a simple Multi-Layer Perceptron (MLP) from Scikit-Learn - a.k.a. feed-forward neural network - should theoretically offer the flexibility to approximate any function you might throw at it (with a few assumptions).
You could also implement this using one of the various deep learning frameworks, should your MLP just not cut the mustard.

Final thoughts

I would lastly like to highlight that your final model will only likely be as good as the data you feed it, and that while it may work well on your synthesised training data, there are no guarantees of generalisation, without putting more thought into the specific problem and specific distributions from which your data are sampled.

If you go the way of Markov models, you could perhaps then use their state predictions as inputs to further models; thus creating an ensemble.

If you want to get really fancy and more towards current state-of-the-art models in sequence analysis, you could venture into the world of Recurrent Neural Networks, e.g. utilising LSTM cells, which maintain an internal state representation, including new information and selectively discarding old information. This might be a good way to take all your points into consideration in one large model. The catch here would be that you generally require vast amounts of data to train such a model.


StandardScaler's mean and standard deviation for real-life data?

It has to be saved somewhere i.e. Database after the training is done.
The saved values should be used on new data and all these steps should work in a loop i.e. when you re-train again the values will be updated and saved again.
e.g. If we see the Keras pre-trained models, it provides the necessary pre-processing function. We can directly use that
from keras.applications.resnet50 import preprocess_input
train_datagen = ImageDataGenerator( preprocessing_function=preprocess_input)


np.nan_to_num() does the trick, as @Oxbowerce pointed out.
import numpy as np
np.nan_to_num(np.log(1-y_pred))


tflite_convert a Keras h5 model which has a custom loss function results in a ValueError, even if I add it in the Keras losses import

I tried the code you posted the following way:

from keras import Model
from keras.applications import VGG19
import keras.backend as K

class Vgg19Loss:
    def __init__(self):
        pass

    @staticmethod
    def define_loss_model(high_resolution_shape):
        model_vgg19 = VGG19(False, 'imagenet', input_shape=high_resolution_shape)
        model_vgg19.trainable = False
        for l in model_vgg19.layers:
            l.trainable = False
        loss_model = Model(model_vgg19.input, model_vgg19.get_layer('block5_conv4').output)
        loss_model.trainable = False
        return loss_model

def build_vgg19_loss_network(ground_truth_image, predicted_image):
    loss_model = Vgg19Loss.define_loss_model(high_resolution_shape) # where is this variable coming from?
    return K.mean(K.square(loss_model(ground_truth_image) - loss_model(predicted_image)))

import keras.losses
keras.losses.build_vgg19_loss_network = build_vgg19_loss_network

print(keras.losses.build_vgg19_loss_network)  # <function build_vgg19_loss_network at 0x7f05e8e1cbf8>

I get no error messages and the function is assigned to the losses module. That means the problematic lines are probably not part of what you posted. It would be nice to know which line of code raises the error that you quoted.

However, I'm not sure where this high_resolution_shape argument on line 22 in your build_vgg_19_network function is coming from. If this is a global constant, it should be written in all uppercase letters separated by underscores to prevent confusion. If it is not defined it will throw a NameError sooner or later.

If I execute keras.losses.build_vgg19_loss_network(None, None) after running the code above, I get the following error message:

  NameError: name 'high_resolution_shape' is not defined

Edit: If this error happens only during TFLite conversion, it does so because custom objects are not yet supported by the TFLiteConverter in tensorflow 1.x. However, there is a commit on the tensorflow Github repo that addresses this issue and adds support for custom objects (see also the related pull request). It should be part of the official tensorflow v2.0.0-beta1.


Datasource for regression model prediction : Machine Learning

There are multiple datasets available in the internet but you need to pay for most of them.
One of the best datasets for starting (and it is free) is the Quantquote Free dataset.

You can download it from here.

This is the description (borrowed from their website):

  This collection of daily resolution data goes back to 1998 for all symbols currently active in the S&P500. It is updated quarterly, the last update was 07/31/2013.


Predicting the Evolution of a Vector with a Neural Network (RNN?)

Ahoy, all! Figured out the answer.

Firstly, this is a time-series forecasting problem. You're looking to forecast (predict the temporal-evolution) of these 2D profiles. In a classification problem (most popular application of neural networks), the target value vector can be a one-hot encoded vector, where the "1" value is the correct classification; the other values (classes) are "0". You give this to your network and it will "learn" patterns in the data that correspond to each particular class. It will do this by minimizing the cost function until the predicted class (the one closest to 1) matches the correct class, and the other predictions are very close to zero. The network is essentially getting closer and closer to the actual target values.

This EXACT idea can be applied to this forecasting problem, where you feed in a vector of target values, but this time the vector isn't one-hot encoded (or, as in other cases, a probability distribution of the classes). It is just the known (target) values at each point along the profile. Yes, RNNs are good, but MLPs can also do the job. Specifically, for RNNs, you can use LSTM units and cells. No need for seqtoseq or NMT architectures for this particular problem.

One important difference is that forecasting problems do not use softmax functions. Instead, they use linear activation functions such as ReLU, and MSE/MAD for the loss.

Setting up this type of problem in your model (in the pre-processing step) may be slightly different than in classification problems. This tutorial here is very good overall, but also has invaluable pre-processing code that you can use in many forecasting applications (particularly the series_to_supervised() function.

Hope this helps!


Is splitting the data set into train and validation applicable in unsupervised learning?

For an unsupervised technique, if you have some metric of "goodness of fit", it makes sense to have a train-test split. In your case, it seems as though you'd want to split the data so that you can see if your SVM classifies the test data as outliers.

However, your goal is unclear. It might help if you clarify what you mean.


Can mini-batch gradient descent outperform batch gradient descent?

First of all though SGD gives frequent updates which help improve performance of the model but it can result in noisy result and frequent updates is computationally expensive for large dataset.
Batch gradient has less update frequency which result is stable error gradient but as it stores whole training data the learning process is slow(when data is large).

Mini-batch gradient descent seeks to find a balance between the
robustness of stochastic gradient descent and the efficiency of batch
gradient descent.

Mini batch GD allows to not have all training data in memory and can be performed in distributed manner.
For more info you can check chapter 8 of Deep Learning book by Ian Goodfellow.


You're asking a complex question that is dependant on what you aim to find.

If there are missing values, are they in columns relevant to the result you're looking for? If not then they needn't be filled at all, so the order does not matter. That being said, if a column with missing values is pertinent to your desired result, then it's often prudent to fill the missing dataset before analysing it further.

Again, outliers need to be addressed in relation to what you're trying to ascertain. In many cases, presence of the outliers can highlight a result in itself. Which may not answer what you're asking, but instead make you question the accuracy of the data itself, or make you determine to include them or exclude them.

You need to be more specific in what it is you're trying to ascertain. Then approach each of the preprocessing steps in the order that is relevant to you. This changes in many different situations, so it's difficult to give a definitive answer.

The tutorial at kaggle to resolve the Titanic Data Science Solutions lays out a good methodology for approaching such problem, and reinforces the concept of sometimes reorganising workflow tasks.

  Workflow stages

  The competition solution workflow goes through seven stages described in the Data Science Solutions book.

  Question or problem definition.
  Acquire training and testing data.
  Wrangle, prepare, cleanse the data.
  Analyze, identify patterns, and explore the data.
  Model, predict and solve the problem.
  Visualize, report, and present the problem solving steps and final solution.
  Supply or submit the results.

  The workflow indicates general sequence of how each stage may follow the other. However there are use cases with exceptions.

  We may combine mulitple workflow stages. We may analyze by visualizing data.
  Perform a stage earlier than indicated. We may analyze data before and after wrangling.
  Perform a stage multiple times in our workflow. Visualize stage may be used multiple times.
  Drop a stage altogether. We may not need supply stage to productize or service enable our dataset for a competition.

  Kaggle - Titanic Data Science Solutions


What are the tools to speed up the running time of machine learning algorithms?

It all depends on the model you are running and your chosen sample sizing.

The sample sizing is obvious, so I won't go into that (or at least it should be obvious to you vis-a-vis how R handles data). But the chosen model can have an effect. For example, neural networks are one of the rare problems in computing that you actually can throw more hardware at to improve. That's going to be a while different experience than, say, gradient boosting or something along those lines.


Yes this is a good strategy. The only thing you need to figure out is the threshold value for negative to neutral, and neutral to positive.
What you could do is decide on some precision value (say 95%) that you want to achieve when classifying a positive article as positive and a negative article as negative. Based on this desired precision you can calculate the threshold you would need to set by evaluating your trained model on a holdout set (if the positive threshold is too low maybe you won't be able to acheive 95% precision, or if it's too high it might be too restrictive and label most headlines as neutral).
The design decision for these thresholds depends on the "business problem" you are trying to solve or the ideal client/use of your tool.


I think I found the answer. The embedding layer in keras is nothing more than a set of vectors for distinct words. The keras embedding layer initializes the word embedding with some random values (the default values from a uniform distribution)and then updates the values when train the whole network. Therefore, there is no need to compose the model in details, because the backpropagation do it perfectly :)


Overfitting in Huggingface's TFBertForSequenceClassification

From my experience, it is better to build your own classifier using a BERT model and adding 2-3 layers to the model for classification purpose. As the builtin sentiment classifier use only a single layer. But for better generalization your model should be deeper with proper regularization. As you have around 13 class you should use deeper model with a good number of training examples for each class.


What is the right way to keep track of the different things we try?

You can maintain multiple versions of booster_params.pickle either:
1) through a version control system
or
2) manually, e.g. booster_params_v1.pickle, ...v2, etc. and a separate file where you would describe each version


How to create a big data frame in Python

You can use pandas.Dataframe.sparse.from_spmatrix.  It will create a Dataframe populated by pd.arrays.SparseArray from a scipy sparse matrix.
Pandas used to have explicit sparse dataframes, but in more modern versions there is no such concept.  Only normal pd.Dataframe populated by sparse data.


When you read the two csv files and store the data in two dataframes, you could then combine it into one dataframe, do the dropna and then split it back. I will give an example using pandas
import pandas as pd
df1 = pd.read_csv('test_data.csv')
df2 = pd.read_csv('submission_data.csv')
df3 = pd.concat([df1, df2], axis=1) # this will combine the two dfs.
reduced_data = df3.dropna(thresh=math.ceil(test_data.shape[1]/2))
predictions = reduced_data.loc[:,['predictions']]
reduced_data.drop(columns=['predictions'], inplace=True)
#instead of 'predictions', use whatever column name you have for the predictions in submission_data.csv file.
Hope this helps.


Is too much or very few training sample of a specific feature hamper the neural network model?

Theoretically speaking, there aren't any disadvantages to having too much or too few data. It will only reflect in the overall performance of your model. Based on the Sherlock paper, it seems that it's a choice they made for their preprocessing. This is their explanation:

Certain types occur more frequently in the VizNet corpus than
others. For example, description and city are more common
than collection and continent. To address this heterogeneity,
we limited the number of columns to at most 15K per class and
excluded the 10% types containing less than 1K columns

They did this to reduce the overall imbalance of their dataset.


Are LDA clusters identical across different runs?

No, as there is randomness in the method implementation, for example here (in LdaModel of the gensim library). Hence, it can affect your final result in each run. Therefore, if you want to keep the result reproducible, you can set the random_state property of the model to a constant seed (see the documentation for more details).


What is the current best state of the art algorithm for graph embedding of directed weighted graphs for binary classification?

I'm no expert myself, but recently (i.e. this is true to 2019), I've heard (at a meetup from an expert) that Node2Vec is the SOTA.

Here's a link to a post on Medium explaining it - basically, Node2Vec generates random walks on the graph (with hyper-parameters relating to walk length, etc), and embeds nodes in walks the same way that Word2Vec embeds words in a sentence.

Note that since random walks are generated on the graph, intuitively you don't have to use edge weights, and can use multiple edges.

P.S. There's an older (2014) algorithm for this, called "DeepWalk", which I don't know much about but is supposed to be similar, simpler, and not as performative. I'm just adding this in case you want to search the term.


How to make it possible for a neural network to tune its own hyper parameters?

I am curious about what would happen to hyperparameters when they would be set by a neural network itself

In general this is not possible as many hyper-parameters are discrete, so they are not differentiable with respect to any objective. For example, this applies to layer sizes, number of layers, choices of transfer functions. This prevents using any form of gradient descent to tune them directly as learnable parameters.

In fact the separation between parameters and hyperparameters is exactly that hyperparameters are not learnable by the model type. This applies to other ML models, not just neural networks.

  or by creating a neural network that encapsulates and influences the hyperparameters of the network it encapsulates.

This is more feasible. You could use one neural network to try and predict the results from another. Then prefer to run tests on target networks that look like they will do well. However, using a "meta" neural network like this has some major drawbacks:

Neural networks require a lot of training data. Getting enough samples to make good predictions would require that you train your primary neural network (a time-consuming process) many times
Neural networks are bad at extrapolating to data outside of areas already experienced, so not so great at making creative predictions of new parameters to try
Neural networks have a lot of hyper-parameters to tune. Would you need a "meta meta" neural network to predict the performance of your "meta" network?

  either it has never been done before or the idea is just really dumb

This is a real issue that comes up repeatedly. In general the search for best hyper-parameters is a chore. It is an active area of research and experimentation to find efficient ways of automating it, or avoiding it by making some hyperparameters less important or not necessary.

The reason you are not finding neural networks that tune neural networks is due to the issues listed above. So the main areas of research focus on different approaches, that can work with limited data and don't have so many hyperparameters themselves. Or models that are robust to large ranges of hyperparameters, so precise tuning is not a big deal.

Here are a few pointers to help with automated searches:

You could use a variety of hyperparameter optimisation schemes, including random search, grid search, genetic algorithms, simple gradient methods etc.
Random searches, perhaps constrained by previous experience or second-hand knowledge from similar problems, can be reasonably effective.
The quality of any search is limited by the quality and amount of cross-validation data. There is not much point tuning the cv loss value to the point that you care about changes that are much less than the standard error in its estimate.
Response to hyperparameters is typically non-linear over the search space, which makes things harder.

Outside of automation, expert analysis is often a good starting point, especially if you want to assess success of regularisation. Typically you can look at learning curves for training and cross-validation data, and based on that you can make a reasonable guess as to whether to increase or reduce regularisation hyperparameters and/or learning rate, even from observing results from a single training run.

There have likely been attempts to automate some parts of reading learning curves, since sometimes it is relatively easy to detect over-fitting and under-fitting scenarios. However, I could not find any examples when searching just now.


How to fetch text from pdf to further proceed with question answer based model from the same document?

You can use pypdf2 to extract text from pdf.

import PyPDF2

with open('sample.pdf','rb') as pdf_file, open('sample_output.txt', 'w') as text_file:
    read_pdf = PyPDF2.PdfFileReader(pdf_file)
    number_of_pages = read_pdf.getNumPages()
    for page_number in range(number_of_pages):   # use xrange in Py2
        page = read_pdf.getPage(page_number)
        print('Page No - ' + str(1 + read_pdf.getPageNumber(page)))
        page_content = page.extractText()
        text_file.write(page_content)


Recursive Feature Elimination (RFE) with Logistic Regression and little correlation between the features and the target (SKLearn)

Do you think I implemented the code in the right way?

Code is correct, but he is minimal possible.

  The features (X) have low correlation with the labels (Y).

It's could be the biggest problem, features have to have correlation with labels.

  Do you have any suggestion to enhance the accuracy?

Make transformation of labels(X). Preparing of labels its one most important parts in Machine Learning. Please read data transformation or feature extraction.
After this try different algorithms like xgboost, adaboost, random forest etc.
Perhaps you should use regularization L1 or L2.
Try early stopping.
After this step use cross-validation.
Choose suitable metrics for yours problem.


How to print accuracy in each fold of Validation Dataset? and assign fold number to each row in the dataframe?

You're mixing up GridSearchCV and cross_val_score; you should only need to run one of them.

GridSearchCV will search through your hyperparameter space, for each combination using cross-validation and producing a score.  You can access these scores through the attribute cv_results_.

cross_val_score has no hyperparameter search; it just scores using cross-validation.  The output is a list of the individual fold scores.

If you've already used GridSearchCV, there's probably no reason to use cross_val_score.  (After hyperparameter searching, you've seen and used all the data in that set, so the scores in cv_results_ are biased as would be the scores out of cross_val_score; if you need an unbiased estimate of performance, you'll need another test set (or nested cross-validation in the first place).)

If you want to keep track of which samples go in which fold, I think you need to use a cross-validation generator or iterable for the cv parameter instead of an integer.  Then you can use that generator/iterable to also tell you which samples are in which fold.


Is there a way to set a different activation function for each hidden unit in one layer in keras?

If I get the point, you can use a similar code like the following:

from keras.layers import merge, Convolution2D, MaxPooling2D, Input

input = Input(shape=(256, 256, 3))

seq1 = Dense(1, activation = 'relu')(input)
seq2 = Dense(1, activation = 'sigmoid')(input)
seq3 = Dense(1, activation = 'tanh')(input)

acum = merge([seq1, seq2, seq3], mode='concat', concat_axis=1)

Depending on your task, specify concat_axis.


Is it ok to train the model only on the interested part of the data?

I think it depends on your understanding of the data set.

How similar are car $A$ with car $B$ and car $C$?

Is car $A$ an electric car and car $B$ and car $C$ running on gas? Is one of them a self driving car and the others are not?

If you are training them together, I think there is an implicit assumption that their behavior is similar and you want to take advantage of that, in particular, perhaps you do not have sufficient data from car $A$ and you are hoping that you can use data from car $B$ and car $C$ to help you.

However, if car $B$ and car $C$ are very distinct and you are trying to predict accident or car failure, adding it to them might not help that much. If the design of cars are very distinct, they might not cause accident or car failure due to the same features.


Compare Classification Performance in Datasets drawn from Different Populations

I understood it wrong ,here is the paper which discuss using multiple data set for the same classifier-

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.141.3142&rep=rep1&type=pdf

They conclude-
"
We theoretically and empirically analyzed three families of statistical tests that can be used for
comparing two or more classifiers over multiple data sets: parametric tests (the paired t-test and
ANOVA), non-parametric tests (the Wilcoxon and the Friedman test) and the non-parametric test
that assumes no commensurability of the results (sign test). In the theoretical part, we specifically
discussed the possible violations of the tests’ assumptions by a typical machine learning data. Based
on the well known statistical properties of the tests and our knowledge of the machine learning data,
we concluded that the non-parametric tests should be preferred over the parametric ones."

I recently learned about 5x2cv paired t test procedure to compare the performance of two models.

Please refer below-

http://rasbt.github.io/mlxtend/user_guide/evaluate/paired_ttest_5x2cv/

It Implements the 5x2cv paired t test proposed by Dieterrich (1998) to compare the performance of two models.


Choosing the size of Character Embedding for Language Generation models

There is a theoretical lower bound for embedding dimension

I would urge you to read this paper, but the gist of it is dimension could be chosen based on corpus statistics

GLOVE paper discussed embedding, check page 7 for graphs. What I want to say with this reference is that you can treat it as hyperparameter and find your optimal value.

EDIT: Here is my personal/borrowed from google rule of thumb. Embedding vector dimension should be the 4th root of the number of categories is start with that, and then I play around it. Read this toward the end when they explain their embedding. Why COULD (it must not) make sence: What is BOW rather than one hot encoding of your n-grams.

Does it make sence to make it larger? it depends. On one hand you are right if we make it too big we loose the distributed representation property of the word embedding matrix, on the other hand it works in praxis.


Testing Multi-Arm Bandits on Historical Data

The problem you have is that the users were originally shown A or B under a different policy to that which the optimiser is learning. You probably don't know the probabilities for A or B from that policy (if you do, that would allow for a bit more subtlety).

You will need to reject samples that do not match what the optimiser does at any point. This may leave you with far less historical data to mine, and the possibility of getting a biased result.

  to do this we sample uniformly at random from the set of data where users were shown the output to obtain this reward.

If I understand this correctly, you are planning to:

Pick an event (including user data) for input context
Generate the choice of A or B using your learning agent
Find another user that was presented the same output in the historical data, by sampling randomly from "all users shown A" or "all users shown B"
Count the reward generated in the sampled historical record

Sadly this will not work, because in step 3 you divorce the context from the agent's decision. So you will not be measuring the agent's likely reward, but the hybrid reward of an agent that uses a different user population and gains rewards according to the historic policies (which you may not even know).

A less biased approach might be:

Pick an event (including user data) randomly from historic data
Generate the choice of A or B using your learning agent
If the agent's choice and historic choice for that event match, then count the reward, else discard the event from the test

You may get sampling bias issues here if the agent's policy is different to historic policy for large number of users. The problem is that you are systematically removing users where old policies said one thing and new policies another, due to lack of data about the reward. So unless the original policy was completely random, the samples you actually train/evaluate on will not be chosen with correct ratios across the user population as it varies. Importance sampling could help mitigate that, but only if you know the probabilities of making A|B selection in both historic and learning agent policies.

Another possibility is to train a supervised model that predicts reward given user data and action from your historic data, and use that to drive a simulator of the online environment. If you are also training your agent during this test, you may need to simulate variance - e.g. turn probability of a click-through event into either a click or no click event by sampling $x < p(click)$ - because that is the data you need your online algorithm to learn from. This is clearly limited by the accuracy of the supervised model from the training data, but could reduce variance, because you will get some kind of indication of reward for events that did not happen historically (with some bounds on accuracy that you can estimate). That is due to function approximation in the supervised model - it is pretty similar idea to bootstrapping the Critic evaluation in an Actor-Critic reinforcement learning model.


How to interpret the rec curve for a regression task?

The title of the graph says "REC curve for various models" but this curve is for a single model. It shows as Y the percentage of "correct" predictions by the model depending on what "correct" means, which is given as X: for example if correct means an error less than 10 ha (on X axis), then the model is correct around 55% of the time (on Y axis). In other words the graph splits the instances between correct/incorrect ones based on whether the error value (difference between true and predicted value) is lower than a threshold X.
In particular this graph shows that:

this model predicts very few values with an error less than 8
this model very often (around 50%) predicts values which are 8-9 ha off the true value.
there are couple more values that the model predicts with an error around 15.
more than 40% of the predicted values are not visible on this graph, which means that their error is higher than 20.

I'd suggest plotting the true vs. predicted value, it's easier to interpret (although it can be harder to see where are the large groups of instances).


I found some sites providing open data.

You can find crime data by year, by city in here.
Data contains incident number, crime type, date, time, location.
This site also provides crime data.

I hope this help.


What initial steps should I use to make sense of large data sets, and what tools should I use?

I will try to answer your questions, but before I'd like to note that using term "large dataset" is misleading, as "large" is a relative concept. You have to provide more details. If you're dealing with bid data, then this fact will most likely affect selection of preferred tools, approaches and algorithms for your data analysis. I hope that the following thoughts of mine on data analysis address your sub-questions. Please note that the numbering of my points does not match the numbering of your sub-questions. However, I believe that it better reflects general data analysis workflow, at least, how I understand it.

Firstly, I think that you need to have at least some kind of conceptual model in mind (or, better, on paper). This model should guide you in your exploratory data analysis (EDA). A presence of a dependent variable (DV) in the model means that in your machine learning (ML) phase later in the analysis you will deal with so called supervised ML, as opposed to unsupervised ML in the absence of an identified DV.

Secondly, EDA is a crucial part. IMHO, EDA should include multiple iterations of producing descriptive statistics and data visualization, as you refine your understanding about the data. Not only this phase will give you valuable insights about your datasets, but it will feed your next important phase - data cleaning and transformation. Just throwing your raw data into a statistical software package won't give much - for any valid statistical analysis, data should be clean, correct and consistent. This is often the most time- and effort-consuming, but absolutely necessary part. For more details on this topic, read this nice paper (by Hadley Wickham) and this (by Edwin de Jonge and Mark van der Loo).

Now, as you're hopefully done with EDA as well as data cleaning and transformation, your ready to start some more statistically-involved phases. One of such phases is exploratory factor analysis (EFA), which will allow you to extract the underlying structure of your data. For datasets with large number of variables, the positive side effect of EFA is dimensionality reduction. And, while in that sense EFA is similar to principal components analysis (PCA) and other dimensionality reduction approaches, I think that EFA is more important as it allows to refine your conceptual model of the phenomena that your data "describe", thus making sense of your datasets. Of course, in addition to EFA, you can/should perform regression analysis as well as apply machine learning techniques, based on your findings in previous phases.

Finally, a note on software tools. In my opinion, current state of statistical software packages is at such point that practically any major software packages have comparable offerings feature-wise. If you study or work in an organization that have certain policies and preferences in term of software tools, then you are constrained by them. However, if that is not the case, I would heartily recommend open source statistical software, based on your comfort with its specific programming language, learning curve and your career perspectives. My current platform of choice is R Project, which offers mature, powerful, flexible, extensive and open statistical software, along with amazing ecosystem of packages, experts and enthusiasts. Other nice choices include Python, Julia and specific open source software for processing big data, such as Hadoop, Spark, NoSQL databases, WEKA. For more examples of open source software for data mining, which include general and specific statistical and ML software, see this section of a Wikipedia page.
UPDATE: Forgot to mention Rattle, which is also a very popular open source R-oriented GUI software for data mining.


Solving Bayes Theorem equation -> I can't calculate proper result

Bayes theorem: $P(A|B) = \frac{P(B|A) P(A)}{P(B)}$

In our case: $P(disease|test_+) = \frac{P(test_+|disease) P(disease)}{P(test_+)}$

Given:

$P(test_+|disease) = 0.85 \\
P(test_-|healthy) = 0.9 \\
P(disease) = 0.02$

Since a test can be positive while the patient has a disease and it also can be positive while the patient is healthy, the overall probability for a positive test is calculated as follows:

$P(test_+) = P(test_+|disease) P(disease) + P(test_+|healthy) P(healthy) \\
P(test_+|healthy) = 1 - P(test_-|healthy) \\
P(healthy) = 1 - P(disease)$

So:

$
\begin{align}
P(disease|test_+)
&= \frac{P(test_+|disease) P(disease)}{P(test_+|disease) P(disease) + P(test_+|healthy) P(healthy)} \\
&= \frac{P(test_+|disease) P(disease)}{P(test_+|disease) P(disease) + (1 - P(test_-|healthy)) (1 - P(disease))} \\
&= \frac{0.85 \times 0.02}{0.85 \times 0.02 + (1 - 0.9) (1 - 0.02)} \\
&= \frac{0.017}{0.017 + 0.1 \times 0.98} = \frac{0.017}{0.115} \approx 0.148
\end{align}
$

So, the probability that you have the disease given that you were tested positively is around 14,8% in this case.


ARMA/ARIMA on energy forecasts timeseries: strange prediction

I'm not an expert on time series, but I have a general advice: may I suggest you to try other packages (and various parameters) to see, if there are any differences in results.

Also, unless you have to use Python, I'd recommend to take a look at the R's extensive ecosystem for time series analysis: see http://www.statmethods.net/advstats/timeseries.html and http://cran.r-project.org/web/views/TimeSeries.html.

In particular, you may want to check the standard stats package (including functions arima() and arima0), as well as some other packages: FitARMA (http://cran.r-project.org/web/packages/FitARMA), forecast (http://cran.r-project.org/web/packages/forecast) and education-focused fArma (cran.r-project.org/web/packages/fArma), to mention just a few. I hope this is helpful.


You can use any base learner for boosting (Adaboost requires sample weighting though). Keep in mind however that the original idea is to use weak learners with strong bias and reducing that bias through boosting.

If it is a classification problem, usually logarithmic loss is used to calculating the residual/gradient for boosting.

For Python, there is a nice AdaBoost wrapper in scikit-learn (AdaBoostClassifier) which can take for example a Random Forest as base learner.


SAS procedure IF THEN ELSE loop did not recognize the Roman letters or numbers as value

SAS initializes the length and type of the variable, I.e. creates and assigns a position for it in the program data vector, at the first reference to the variable it sees.

If (condition) then output='I'; defines it as length 1.

Workarounds

Your discovered solution of having the same lengths.
The first reference to the variable being LENGTH output $2;
Assigning the longest length first.
The first reference to the variable being If 0 then output='zz';

You can confirm the issue, which also pops up when concatenating datasets, by looking at variable properties with proc contents or proc datasets.

From the documentation:

  Note:   When you create character variables, SAS determines the length of the variable from its first occurrence in the DATA step. Therefore, you must allow for the longest possible value in the first statement that mentions the variable. If you do not assign the longest value the first time the variable is assigned, then data can be truncated.

http://support.sas.com/documentation/cdl/en/basess/58133/HTML/default/viewer.htm#a001336069.htm

Sample code:

data a;
    input firstname $ lastname $ salary;
    cards;
Billy Bob 20000
Bob Graham 50000
Graham Alice 30000
Alice Graham 40000
;

proc print data=a; run;

*Did not work;
DATA test;
set a;
if salary < 35000 then grade = 'I';
else grade = 'II';
run; proc print data = test; run;

*Worked;
DATA test;
length grade $ 2;
set a;
if salary<35000 then grade = 'I';
else grade = 'II';
run; proc print data = test; run;

*Worked;
DATA test;
set a;
if salary>=35000 then grade = 'II';
else grade = 'I';
run; proc print data = test; run;

*Worked;
DATA test;
if 0 then grade='ZZ';
set a;
if salary<35000 then grade = 'I';
else grade = 'II';
run; proc print data = test; run;

Also, as I stated, you can run PROC CONTENTS DATA=TEST; RUN; after each data step to confirm the data length of each column.


Intuition in Backpropagation (gradient descent)

in what sense applying the transpose is like "moving the error backwards"?

It isn't. Or at least you shouldn't think too hard about the analogy.

What you are actually doing is calculating the gradient - or partial derivative - of the error/cost term with respect to the activations and weights in the network. This is done by repeatedly applying the chain rule to terms that depend on each other in the network, until you have the gradient for all the variables that you control in the network (usually you are interested in changing the weights between neurons, but you can alter other things if they are under your control - this is exactly how deep dreaming works too, instead of treating the weights as a variable, you treat the input image as one).

The analogy is useful in that it explains the progression and goal of the repeated calculations across layers. It is also correct in that values of the gradient in one layer depend critically on values in "higher" layers. This can lead to intuitions about managing problems with gradient values in deep networks.

There is no relation between taking the matrix transpose and "moving the error backwards". Actually you are taking sums over terms connected by the weights in the NN model, and that happens to be the same operation as the matrix multiplication given. The matrix multiplication is a concise way of writing the relationship (and is useful when looking at optimisations), but is not fundamental to understanding neural networks.

The statement "moving the error backwards" is in my opinion a common cause of problems when developers who have not studied the theory carefully try to implement it. I have seen quite a few attempts at implementing back propagation that cargo cult in error terms and multiplications without comprehension. It does not help here that some combinations of cost functions and activation functions in the output layer have been deliberately chosen to have very simple derivatives, so that it looks like the neural network is copying error values around with almost magical multiplications by activation value etc. Actually deriving those simple terms takes a reasonably competent knowledge of basic calculus.


Adding variables to Random Forest decreases performance

Even if your variables are linearly uncorrelated, they may still be non linearly correlated. Just using the correlation coefficient to say the data is unrelated is not enough for non linear regression.

Still, adding variables to a model may not improve the end result if the variables cannot explain anything new from the target data. This seems to be such a case.


Generally, I'd say its the same way as finding your "best" feature list, pre-processing methods and algorithm - you'd find most answers starting with "give it a try and compare", "check which combination gives you a better cross-validation?" etc..

More specifically, when there is no farther documentation (and its an open source project) I usually take a look at the code for more intuitions.

In this case concat, dot, average and the like where strait forward implemented as it sounds (concatenating date, averaging, etc..), but it does imply about the amount of processing needed: some of the merging functions adds to the input needed to be processed (concat, add) and some reduces it like an aggregation function (average, min, max)

Beyond that the resulted tensor object will perform a bit different for different datasets. Please give it a try and let us know what merging combination worked best for you, and the dataset type :)


Micro Average vs Macro average Performance in a Multiclass classification setting

Micro- and macro-averages (for whatever metric) will compute slightly different things, and thus their interpretation differs. A macro-average will compute the metric independently for each class and then take the average (hence treating all classes equally), whereas a micro-average will aggregate the contributions of all classes to compute the average metric. In a multi-class classification setup, micro-average is preferable if you suspect there might be class imbalance (i.e you may have many more examples of one class than of other classes).

To illustrate why, take for example precision $Pr=\frac{TP}{(TP+FP)}$. Let's imagine you have a One-vs-All (there is only one correct class output per example) multi-class classification system with four classes and the following numbers when tested:

Class A: 1 TP and 1 FP
Class B: 10 TP and 90 FP
Class C: 1 TP and 1 FP
Class D: 1 TP and 1 FP

You can see easily that $Pr_A = Pr_C = Pr_D = 0.5$, whereas $Pr_B=0.1$.

A macro-average will then compute: $Pr=\frac{0.5+0.1+0.5+0.5}{4}=0.4$
A micro-average will compute: $Pr=\frac{1+10+1+1}{2+100+2+2}=0.123$

These are quite different values for precision. Intuitively, in the macro-average the "good" precision (0.5) of classes A, C and D is contributing to maintain a "decent" overall precision (0.4). While this is technically true (across classes, the average precision is 0.4), it is a bit misleading, since a large number of examples are not properly classified. These examples predominantly correspond to class B, so they only contribute 1/4 towards the average in spite of constituting 94.3% of your test data. The micro-average will adequately capture this class imbalance, and bring the overall precision average down to 0.123 (more in line with the precision of the dominating class B (0.1)).

For computational reasons, it may sometimes be more convenient to compute class averages and then macro-average them. If class imbalance is known to be an issue, there are several ways around it. One is to report not only the macro-average, but also its standard deviation (for 3 or more classes). Another is to compute a weighted macro-average, in which each class contribution to the average is weighted by the relative number of examples available for it. In the above scenario, we obtain:

$Pr_{macro-mean}={0.25·0.5+0.25·0.1+0.25·0.5+0.25·0.5}=0.4$
$Pr_{macro-stdev}=0.173$

$Pr_{macro-weighted}={0.0189·0.5+0.943·0.1+0.0189·0.5+0.0189·0.5}={0.009+0.094+0.009+0.009}=0.123$

The large standard deviation (0.173) already tells us that the 0.4 average does not stem from a uniform precision among classes, but it might be just easier to compute the weighted macro-average, which in essence is another way of computing the micro-average.


How to improve loss and avoid overfitting

I am not familiar with the software you are using but keep in mind: You EXPECT accuracy to drop if you reduce over fitting. It is not a bad thing. Over-fitting is essentially "fake accuracy".

Some good approaches in general to avoid over-fitting though: Use cross-validation, normalize your features, increase size of data-set and dont just increase your data-set by copying data.


here you go:
http://www.rexamples.com/14/Sample()

sample(x, size, replace = FALSE, prob = NULL)


In most cases, I would go for NumPy.
Implement a Python function f(t) that calculates the $t$-th summand.
Then run

import numpy as np
result = np.array([f(t) for t in range(1,m+1)]).sum()

This will be very fast, unless $m$ is so large that [f(t) for t in range(1,m+1)] does not fit into memory. In this case, I would follow your approach and use a for-loop:

result = 0
for t in range(1, m+1):
    result += f(t)


Is it acceptable to make new independent variables out of old ones?

It is completely acceptable if you make out new categories out of old ones or add a different representation of them like in the case of (North East, Midwest, South, Etc.) from state this process is called feature engineering. All you need to pay attention to is that you are not loosing any information in any way when you are doing so because you want to feed your model with as much information as possible.

Sometimes it is useful to keep both the new variable you made out of other variables like for say you have a dataset containing infect_people_from_covid19 in region and population_of_region you can create a variable as ration of people infected but when building a model and feeding data you have to decide weather adding all 3 of them will be a good idea or you just need the summary variable you made.


Installing Prototype Widgets in Orange running on Mac

The easiest way to install Orange's addons is through the application itself. Open Orange, in the menu click Options -> Addons. In the popup window mark Orange3-Prototypes and click OK. Note, that doing so you will get the latest version that is published on PyPi.

If you would want to install the bleeding edge version directly from github — assuming that you installed Orange by downloading a bundle from it's website and not for example by using Anaconda — do the following. Open a terminal and paste this command:

/Applications/Orange3.app/Contents/MacOS/pip install git+https://github.com/biolab/orange3-prototypes.git

Doing so you will trigger the pip that came with the bundle and instruct it to install the addon directly from github.

Further, to install opencv, which is required by some widgets (e.g. webcamecapture) run

/Applications/Orange3.app/Contents/MacOS/pip install opencv-python

Note, however, that this is unofficial version of opencv but it seems easier to install.


Correlating company entities between different data sources

First of all I would clean up INC, LLC, BV etcetera from both the sources. After this there are a few options. Since Levenshtein is a metric you can use metric trees to search your space more efficiently (about O(n*log(m))). This will still be very slow so there are approximations available, for example the cosine similarity on bi-grams of the names. You can do this using matrix multiplication which is both very efficient and easily distributable. Instead of taking the highest similarity you could take the top-n and do further analysis on these, for example the real Levenshtein distance. The fact that you have additional information could be useful, you could add this to your similarity function in some way but this will be guess work. Most of these ideas I got from a PyData meetup that was recorded, a speaker from ING (a big bank) discusses the exact problem you have albeit on a bigger set with less additional information:

https://www.youtube.com/watch?v=4ohTsblxOJs


Not sure if I can get the direction of your "why", but here's a try:

If you were to use some sort of interpolation, you would have an explicit model of this function in the mathematical sense. This would be a white box, as you actually have the formula/algorithm to do the prediction. If you use a conventional decision tree or linear regression you are still in the same white-box category in the sense that your trained model can be expressed by a manageable set of formulas or rules.

However, in typical machine learning scenarios there are at least two factors that limit the transparency of this box:

We care more of the result, and less of the explanation. Thus, we might use different approaches. As long as they fit in the same "interface" as you define it with your function, we don't really care which one is being used. This allows for A/B testing of different approaches and putting in place the one that currently yields best results. After some time, based on new training data or on new advances in machine learning, we can decide to use another algorithm, but the general setup remains the same.
Contemporary algorithms are extremely complex, both because the dimensions of the input and output are much higher than our intuition can cope with, and (inherently) because they use many more iterations. Examples of these iterations could be the layers of deep networks, but also the range of algorithms used in ensemble methods like random forests.


Feature selection for tracking user activity within an application

Well, this may not answer the question thoroughly, but since you're dealing with information retrieval, it may be of some use. This page mantains a set of features and associated correlations with page-ranking methods of search engines. As a disclaimer from the webpage itself:

  Note that these factors are not "proof" of what search engines use to rank websites, but simply show the characteristics of web pages that tend to rank higher.

The list pointed may give you some insights on which features would be nice to select. For example, considering the second most correlated feature, # of google +1's, it may be possible to add some probability of a user making use of such service if he/she accesses many pages with high # of google +1 (infer "user context"). Thus, you could try to "guess" some other relations that may shed light on interesting features for your tracking app.


What does it mean for the training data to be generated by a probability distribution over datasets

There is a common assumption that data that is being modeled is independent and identically distributed (i.i.d.) samples from a probability distribution. There is the same underlying probability distribution for both the training and test datasets. And each sample is independent of the other samples.

Examples of these assumptions being violated:

Data is generated by a completely random process such as random walk.
The training and test dataset come from different probability distributions, either completely different probability distributions or the same probability distribution with different parameters.
Samples are not independent. An example of dependent samples is dealing cards from a single deck, the probability of later cards is dependent on previously dealt cards.

The model fitting process only has access to the data samples, not the underlying probability distribution. Parametric modeling fitting makes a guess about the functional form of that probability distribution (e.g., Bernoulli or Gaussian) and then, estimates the associated parameters.


Are there any control-flow/conditional statements in AI/ML models?

Yes, there are control flow constructions in neural networks, for instance:

Variable-length sequence classification with recurrent units: in text classification tasks with recurrent units (vanilla RNNs, LSTMs, GRUs), both at training and inference time, you basically iterate over the time dimension of the input batch, passing the next token and the previous steps' state to the recurrent unit. This is a for loop.

Variable-length sequence generation: in text-generation encoder-decoder networks, at inference time the decoder generates one discrete token at a time until a special "end-of-sequence" token is generated. This is pretty much a while construction. The decoder can be based on any type of neural net units: LSTMs, CNNs, self-attention (Transformer), etc.

As you can imagine, this kind of control flow construction is typical of variable-size input/outputs. You won't see this kind of construction in most image processing architectures.
Control-flow stuff is more difficult to implement in declarative deep learning frameworks, like Theano or Tensorflow 1.x. In Tensorflow 1.x, there was actually a specific primitive to implemente while loops: tf.while_loop. With imperative frameworks like Pytorch or Tensorflow 2.x, control-flow constructions are far easier to implement, because you can integrate normal logic between the computations.


Recommendation - item based vs user based

It depends on what data you are using to calculate similarity between your items.

If you are using data from a user interaction with an item - like viewing a web page or buying an item - then each time there is a user interaction with an item, the similarity between it and other items will change. If you are doing this calculation offline, then your model won't take that change into account until the next time the data set is updated and your recommendations are recalculated.

However, if you are using item meta-data to calculate similarity, then user behavior won't make any difference. For example, if you took the number of shared tags between two items, or the number of shared inbound links, etc.

I have always used user browsing data to build content recommendation engines, but it is certainly possible to calculate similarity in many different ways. Have fun!


Predicting churn - deal with missing dates in time series and improve modelling result

It would be interested to deal with it as a sequence classification problem. For instance, you could use HMM (Hidden Markov Model) or equivalent to classify the sequences. The data format would be:

ID:  sequence      label
id1: 39,0,6,...,88  1
id2: 300, 2         1
id3: 1              1
id4: 100            0

Some suggestions:

Create more samples to balance also your dataset (e.g id1   39, 0    0)
Possibly bin the variables (e.g. to the decimal 6 -> 10, 39 -> 40)


What are some (non image) datasets where deep networks are required for good performance?

Welcome to the community!

In text-related tasks, DL have shown amazing breakthrough e.g. in NLU for reading comprehension or question answering. If you have heard about Google's BERT, that was one of big steps which, to some extent, pushed the boundaries of SOTA in NLP/NLU last year and currently, a heavy wave is going on doing research on different NLP/NLU tasks mostly based on either the idea coming from BERT or directly from its pretrained models.

The quora QA dataset, SQuAD dataset, etc. are great starting points to play with bert and its successors and see how great the idea was.

Hope it helps!


Custom objective function in xgboost for Regression

Looking at the documentation example here, a xgboost custom loss function needs to return the gradient and second-order gradient. Your function does not return those values for the stated goal.


Reframing action recognition as a reinforcement learning problem

Given the significant advancements in reinforcement learning

Worth noting that many of the recent advancements are due to improvements in neural networks used as function approximators, and understanding how to integrate them with reinforcement learning (RL) to help solve RL challenges involving vision or other complex non-linear mapping from state to best action.

So at least some of current improvements in RL were due to researchers asking the opposite question "Given the significant advancements in neural networks . . ."

  I wanted to know whether it is possible to recast problems such as action recogniton, object tracking, or image classification into reinforcement learning problems.

Generally, at the top level, the answer is Yes, but it offers no benefit, and could perform a lot worse.

That is because in typical supervised learning scenarios there is nothing to match the concept of actions that modify state. In the example classifiers you can have the equivalent of a state (the input to be classified), an action (the choice of category) and the reward (whether of not the choice matches the label). But taking an action does not lead to another state, rewards are not sparse or cumulative across multiple actions within a specific environment or "episode". There are no time steps.

RL algorithms are generic MDP solvers - they can learn about relationships between state, action and likely next state, and optimise long-term goals over may time steps when given a current state. That also makes them less efficient learners when those relationships are not valid or important in the problem you want to solve. If you trained e.g. Q-learning on a typical image classification dataset, and added time steps, it would spend a lot of time/resources establishing that its choice of action had no influence on which images it was subsequently presented with, or how easy it was to get a reward from later images as opposed to earlier ones depending on how the state varied. If you really did allow the choice of action to determine the next image, then you would be training the RL to do something else other than classification.

You could frame the classifiers as contextual bandits, which is perhaps closer match. However, that still throws away knowledge that you have about the classification problem, replacing it with a generic reward system. For instance, a contextual bandit solver would deliberately guess wrong classes to check whether sometimes there was a small chance of a high reward for doing that.

If you were very careful about how you represented actions and rewards, and set other hyper-parameters, then you might be able to re-create a similar gradient setup to normal supervised learning, and only lose a little bit of efficiency through using RL or contextual bandit framing of your problem. However, you would still have added some non-necessary complexity.

If you search, you may find some ways to combine RL with supervised learning, for instance in this paper, the authors propose using RL to refine a generative RNN. However, these currently seem niche and are not aimed to improve upon or replace supervised learning.

Finally, in theory you could allow RL to control a video camera pan/zoom as part of activity recognition (or any other video or multi-image classification task). That would be a full RL problem, because the agent's actions really would be influencing the later states and hopefully improving the accuracy of recognition. For learning efficiency, you would likely want to combine this initially with a network that had already been trained to recognise actions on a supervised data set. You would need to experiment with how much the recognition part was trained compared to the RL part (as it will start to collect data outside of your normal datasets). And of course setup and training of the combined system could be a major project. You may be able to simulate it in a game engine perhaps in the early stages.


Find inputs that give highest variance in output space

Very interesting question! I didn't get if the out put of one input vector is just one image or a set of images but this does not change the solution much.
I propose you directly learn the relation between input and the variance of output images. For simplicity, I assume you target the pixle-wise variance within an output image. Then your training data would become $\{x_i, Var(y_i)\}_{i=1}^N$ instead of $\{x_i, y_i\}_{i=1}^N$!
Then your problem can be solved as:

Classification: You threshold the "high variance" and discritize your output into high variance and low variance classes
Regression: Even better! you directly learn the value of variance and have more flexibility on interpreting the results at the end (e.g. the thresholding mentioned above can be applied afterwards).

I hope I understood your question right, I was clear and it could help!
Good Luck!


DeprecationWarning: The 'categorical_features' keyword is deprecated in version 0.20

You can do this to get rid of the deprecation messages

from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

ct = ColumnTransformer(
    [('one_hot_encoder', OneHotEncoder(), [0])],    # The column numbers to be transformed (here is [0] but can be [0, 1, 3])
    remainder='passthrough'                         # Leave the rest of the columns untouched
)

x = np.array(ct.fit_transform(x), dtype=np.float)


How to implement multi class classifier for a set of sentences?

I was thinking if Apriori might be more suited for your purpose.

For your consideration:
1) Tokenize the training sentences into bag of words:
Review Sentence | Upside | Calm | Swimming | Cat

2) Tag the correct consequents for bag of words.

3) Apriori should produce 1 rule for ambiance and 1 rule for Entertainment.

Hope this helps.


How can I prove bottleneck layer of my CNN auto encoder contain useful information?

Yes to both of your questions. Your autoencoder can overfit and this will cause your bottleneck to store useless information (besides any useful information it already stores).

Some ways to prevent this is:

Find a larger dataset, or augment the current.
Add noise to the input (see de-noising autoencoders).
Regularization (e.g. early stopping, sparsity constraints)


Neural Network - Adjust number of hidden layers and neurons

Sorry for not knowing exact number for the first question, but my point is to start from the most simple model, rather than any random parameter. Then use a grid search for the parameter by validation.

As for the second question, the error of a model comes from two parts, variance and bias. Bias comes from lack of complexity of your model. For example, there will always be a large bias when using linear model to fit the distance v.s. the shooting angle of a canon. This can be overcome by using a more complex model,  adding layers to the NN for example.

And variances comes from being too sensitive to the data, even to the noise in the data. When there is a high variance due to using a too complex model, it is called over-fitting. You may take a look at this page(over-fitting on wiki). And using more layers/neurons is using a more complex model, which means the model may suffer from high variance. This can be overcome by lower the model complexity(which may increase the bias) or adding training data.

Basically if you got constant number of data, then you have to find somewhere where both bias and variance is not too high.


Keras: matching logistic regression performance with sequential neural network?

Some question you might want to think about:

Is your dataset big enough? What kind of data is it? Time series? Should you be shuffling? What learning rate are you using? Can you change it and see the effect on the learning curves: plot(history) output?

With regards to your model, you usually leave the last connected layers joined only by a linear activation (i.e. don't use an activation function, just an identity matrix). You have used the sigmoid all the way through, which is fine, but not for final layer! I have corrected this by not including such a non-linearity, rather the softmax activation, which will squash all values into the range of [0, 1], so they can be interpreted as probabilities.

I have increased the number of layers and neurons in the initial layers, and swapped in the preferred non-linearity: the Rectified Linear Unit (ReLU). I would recommend having a quick read of this intro from Stanford's CS231n course, which covers some of the best practices. Also, have a look here for a sample of a larger network performing classification, to see how R Keras is best used.

Give the following code a test, and also plot the history to get more intuition as to how the training is progressing: how many epochs might be necessary, whether you are over- or underfitting, etc.

model <- keras_model_sequential() %>%
    layer_dense(units = 200, activation = "relu", input_shape = 39) %>%
    layer_dense(units = 100, activation = "relu") %>%
    layer_dense(units = 100, activation = "relu") %>%
    layer_dense(units = 50, activation = "relu") %>%
    layer_dense(units = 2, activation = "softmax")

model %>% compile(
              optimizer = "rmsprop",
              loss = "categorical_crossentropy",
              metrics = c("accuracy")
          )

history <- model %>% keras::fit(
                                x_train,
                                y_train,
                                epochs = 500,
                                batch_size = 16,
                                validation_split = 0.7, shuffle = T
                            )

plot(history)                           # Are we overfitting?

Disclaimer: If you don't have much data, this model might be even worse that your original one - or you will massively overfit and get 100% training accuracy and terrible validation/test accuracy.

It could well be the case that your data is just better modelled with a simpler model, or that you do not have enough data to get a neural network to hone in on a nice optimum in its optimisation.


When use any sampling technique ( specifically synthetic) you divide your data first and then apply synthetic sampling on the training data only. After you train you use the testing set ( which contains only original samples) to evaluate.
The risk if you use your strategy is to have the original sample in training ( testing) and the synthetic sample ( that was created based on this original sample) in the testing ( training) set.


How to export one cell of a jupyter notebook?

See this stack question

You can use %%capture Jupyter notebook' magic command to catch output of cell and then paste it to your text file with

with open('output.txt', 'w') as out:
   out.write(cap.stdout)

if you want to cell code to specific file for example code.txt you can use magic function %%writefile

%%writefile example.txt
ab = 'This is code'
a = 5
print(a+ 2)

Also if you want to append to file you must use -a parameter


How do I use a custom stopwords filter in the Java Weka API?

First of all you have to prepare a text file for your custom stopwords. Then you can use the following code:

import weka.filters.unsupervised.attribute.StringToWordVector;

StringToWordVector filter = new StringToWordVector(10000);
filter.setStopwords(new File(".../stopwords.txt"));

Hope it will help you.


Why RMSProp converges faster than Momentum?

The basic intuition is that you should not have the same learning rate for different dimensions. For instance, you can have a high slope in one direction but not for another. Consequently, you should not have the same speed for the two directions. Momentum adds acceleration. Suppose gradient is your instant velocity and the average is your average velocity. Momentum is actually viscosity or somehow friction. Suppose that you are near your optimal points, your gradients become zero and you have low average which means your speed changes slowly. They have both alpha term but what is going to be used is the running average, just a kind of average which is simple to be calculated. Take a look at here and here for making an analogy.


PicklingError in pyspark (PicklingError: Can't pickle : attribute lookup Person on __main__ failed)

Possible answer from here

The problem is that you're trying to pickle an object from the module where it's defined. If you move the class into a separate file and import it into your script, then it should work.

That solution isn't viable for me in an iPython notebook though. So here I some additional information from here

Python's pickle actually does not serializes classes: it does
serialize instances, and put in the serialization a reference to each
instance's class - and that reference is based on the class being
bound to a name in a well defined module. So, instances of classes
that don't have a module name, but rather live as attribute in other
classes, or data inside lists and dictionaries, typically will not
work.
One straight forward thing one can try to do is try to use dill
instead of pickle. It is a third party package that works like
"pickle" but has extensions to actually serialize arbitrary dynamic
classes.
While using dill may help other people reaching here, it is not your
case, because in order to use dill, you'd have to monkey patch the
underlying RPC mechanism PySpark is using to make use of dill instead
of pickle, and that might not be trivial nor consistent enough for
production use.
If the problem is really about dynamically created classes being
unpickable, what you can do is to create extra meta-classes, for the
dynamic classes themselves, instead of using "type", and on these
metaclasses, create proper getstate and setstate (or other
helper methods as it is on pickle documentation) - that might enable
these classes to be pickled by ordinary Pickle. That is, a separate
metaclass with Pickler helper methods to be used instead of type(...,
(object, ), ...) in your code.
However, "unpickable object" is not the error you are getting - it is
an attribute lookup error, which suggests the structure you are
building is not good enough for Pickle to introspect into it and get
all the members from one of your instances - it is not related (yet)
to the unpickleability of the class object. Since your dynamic classes
live as attributes on the class (which is not itself pickled) and not
of the instance, it is very well possible that pickle does not care
about it. Check the docs on pickle above, and maybe all you need there
is proper helper-method to pickle on you class, nothing different on
the the metaclass for all that you have there to work properly.


Attitude to text mining and preparing tokens, irrelevant words, low accuracy

The question is very broad, so the general answer is: it depends on the specifics of your data and the problem you're trying to solve. The general idea would be to analyze what's happening:

Is it actually possible to solve the problem given the indications in the data? Would a human expert do better than 50-60% accuracy given only the information in the data? If yes which clues would they use, and are these potential clues directly available as features to the learning algorithm?
Ratio between size of data and size of features: if there are not enough instances, the algorithm doesn't have enough to generalize from. If there are too many features, the algorithm is likely to overfit (i.e. consider things which happen by chance in the data as significant patterns). And of course if there are not enough features the algorithm doesn't have the indications necessary to make accurate predictions.

Common things to try:

remove features (here words) which appear very rarely, because they can't help and can cause overfitting.
if the problem seems to be that words are not informative enough, try with bigrams or even trigrams... but be careful to avoid overfitting, the number of features is likely to increase a lot.
use weights (e.g. TFIDF) to help the learning algorithm focus on the relevant features.
depending on the task, more sophisticated methods could be relevant: topic modeling, disambiguation techniques, syntax...


Sequences of time series data with only 1 output classification

With only one output, there is no task for a supervised learning algorithm for classification. Your algorithm can only predict your only one class, you do not give any other situations of other classes for your algorithm to work on; a classification algorithm will need at least two classes. I have strong experience with time-series learning problems, but it seems that your problem has nothing to do with whether your data is time-series.

However, depending on your task, you can implement unsupervised learning which does not require any labels. You can seperate your data into clusters which have similar properties as time-windows. Then you can predict with new (or test) samples that which cluster should it be from, what it is alike.


Why a GAN trained on same data and same parameters may produce different results?

The loss function error surfaces of both discriminator and generator are complex. There are many different optima on both surfaces that can yield usable solutions. There is a element of randomness where on the loss function error surfaces a training run with stop.


You could define a window (distance between red and blue line) and slide it from top to bottom over the y-axis and then count the local minima and maxima in that slice. One way with scipy is argrelextrema(x, np.less) for minima and argrelextrema(x, np.greater) for maxima which gives you the indices of the extrema. With len(argrelextrema(x, np.less)) you would get the number of minima.

Each slice would be a level and the results of counting the minima and maxima the corresponding importance. The size of the window is up to you. If your data is very granular you might want to consider smoothing it first (e.g. with numpy.convolve()


I would take a look at t-digest algorithm. It's been merged into mahout and also a part of some other libraries for big data streaming. You can get more about this algorithm particularly and big data anomaly detection in general in next resources:

Practical machine learning anomaly detection book.
Webinar: Anomaly Detection When You Don't Know What You Need to Find
Anomaly Detection in Elasticsearch.
Beating Billion Dollar Fraud Using Anomaly Detection: A Signal Processing Approach using Argyle Data on the Hortonworks Data Platform with Accumulo


How can you build a model based on non-independent imbalance data?

So in essence (if I understand correctly), you want to predict a customer "not being active" within a week versus "being active"?! So you have a binary problem. The choice of method depends on your data. But a good start might be, for instance, catboost, a gradient boosting tool. I have good experiance with it.

https://catboost.ai/docs/concepts/python-usages-examples.html

With the CB classifier, you can specify class_weights, which might help to cope with unbalanced classes.

https://catboost.ai/docs/concepts/python-reference_catboostclassifier.html#python-reference_catboostclassifier

See also: https://stackoverflow.com/a/45708903/9524424

Here is a Python tutorial: https://github.com/catboost/tutorials/blob/master/python_tutorial.ipynb

Regarding the "non-independent" aspect: Do you mean that the SAME customer can fall in both classes at different points in time? Maybe this is not too much of a problem as long as something changes in the X variables (the ones used for prediction). Time certainly changes, right? So you might include time, e.g. as a factor (or dummy) variable. Often this works well with gradient boosting. Maybe (just an idea) you could also add a customer ID in addition to time, in order to try to control for each individual customer or (similar) groups of customers.

Alternative methods/tools would be lightgbm, logistic regression (with regulation), or even neuronal networks (e.g. Keras/tensorflow based). Time can also be included as count or dummy in logistic regression. In neuronal networks, dealing with time often is a bit complicated. To say more about it, one ultimately need to see the data in detail (number of X variables/columns, number of observations etc).


Topic Segmentation - should it be done in Raw, TfIdf or Semantic Space?

Clustering method can applied directly to TfIdf matrix (which will be generally sparse) or to documents in the derived semantic space like LSI.  Since LSI embedding is a dimensionality/noise reduction step, it is a good idea to cluster on documents embedded in semantic space. Clustering methods will generally use Euclidian distance and will probably perform better on dense matrix. If the clustering algorithm implementation that you use doesn't support sparse data out of the box (most implementations don't) then definitely use a transformation like LSI first and then apply clustering.
LSI method itself is a matrix decomposition operation on the original TfIdf matrix. So applying matrix decomposition on top of LSI is not needed.


How do I choose a laptop if I'm interested in learning and applying data science?

This is a bit off-topic and very subjective. It is subjective as it depends on what you want to achieve. I am answering because I have been in this journey for the last 2 years and I can confidently comment

  You certainly DO NOT need a high-performing laptop to start off with data science/machine learning.

based on the following reasons (at least not for the first year or two):

Building the foundations of machine learning, statistics and programming takes longer time that you may think (your objective is solely learning), till you get to do heavy deep learning optimization on images etc.
Even if you get to heavy tasks earlier, there is increasing availability of free clouds like Kaggle, Google Colab etc. that you can run heavy jobs. Yes, runtime on those clouds are at present limited (e.g. as of now it is 1-hr GPU computing at Kaggle), but this will change really soon.

Thus, you are good with a laptop with 1TB hard-disk, 16GB RAM and Core i7, surely depending on your budget lower specification like mine (MacBook Pro 256GB, 8GB RAM, Core i5) also works just fine, unless you have more money to spare.

P.S.: I admit not being able to work on some image-based ML competitions in Kaggle due to limited computing power, but that time (~ a year ago) free clouds were less available.

By the way, Welcome Onboard!


Using a combination of gradient boosting with LSTM for classification?

I'm not aware of any streamlined way of doing this without writing the proper code to connect the two models.  It should be fairly straight forward to run XGBoost, grab the feature importance and use this as a way to identify the features to use for the LSTM.  This part should be not so hard, just have a list of features with some criterion of whether to keep or drop it.  Once you have the list of features you want to use for the LSTM, just filter the whole dataset, and build out the LSTM model using that set.


What makes columnar databases suitable for data science?

A column-oriented database (=columnar data-store) stores the data of a table column by column on the disk, while a row-oriented database stores the data of a table row by row.

There are two main advantages of using a column-oriented database in comparison
with a row-oriented database. The first advantage relates to the amount of data one’s
need to read in case we perform an operation on just a few features. Consider a simple
query:

SELECT correlation(feature2, feature5)
FROM records

A traditional executor would read the entire table (i.e. all the features):

Instead, using our column-based approach we just have to read the columns
which are interested in:

The second advantage, which is also very important for large databases, is that column-based storage allows better compression, since the data in
one specific column is indeed homogeneous than across all the columns.

The main drawback of a column-oriented approach is that manipulating (lookup, update or delete) an entire given row is inefficient. However the situation should
occur rarely in databases for analytics (“warehousing”),
which means most operations are read-only, rarely read many attributes in the same
table and writes are only appends.

Some RDMS offer a column-oriented storage engine option. For example, PostgreSQL
has natively no option to store tables in a column-based fashion, but Greenplum has
created a closed-source one (DBMS2, 2009). Interestingly, Greenplum is also behind
the open-source library for scalable in-database analytics, MADlib (Hellerstein et al.,
2012), which is no coincidence. More recently, CitusDB, a startup working on high speed, analytic database, released their own open-source columnar store extension for
PostgreSQL, CSTORE (Miller, 2014). Google’s system for large scale machine learning
Sibyl also uses column-oriented data format (Chandra et al., 2010). This trend
reflects the growing interest around column-oriented storage for large-scale analytics.
Stonebraker et al. (2005) further discuss the advantages of column-oriented DBMS.

Two concrete use cases: How are most datasets for large-scale machine learning stored?

(most of the answer comes from Appendix C of: BeatDB: An end-to-end approach to unveil saliencies from massive signal data sets. Franck Dernoncourt, S.M, thesis, MIT Dept of EECS)


The evaluation metric used for classification .score is accuracy.
Your model has higher accuracy and lower precision. Looking at your confusion matrix, you have class imbalances which can lead to that result.
You should not change your evaluation metric to get better results. It is better to pick the evaluation that makes the most sense for the problem. Then improve the model through collecting more data, engineering better features, changing algorithms, or tuning hyperparameters.


Categories with the same mean in target encoding

I assume tree-based models in this answer
Bayensian Mean Encoding can help
The main problem is that the model will be unable to split between the merged categories. So you implicitly accept that there is no interaction between the two merged categories and other variables. (succession of splits will be the same for cat1 and cat2)
If you want to enable the tree to split anywhere (because you suspect interactions), you have to create a space between the values of cat1 and cat2 for the tree to split. Bayesian Mean Encoding can help you. It takes the frequency of the categories into accounts in the calculation of the target means.
The formula is the following :
\begin{equation}
\mu = \frac{n * \bar{x} + m*w}{n+m}
\end{equation}
where :

$\mu$ is the category mean
$n$ is the frequency of this category
$\bar{x}$ is the target mean for this category
$m$ is the weight you want to give to the overall mean
$w$ is the overall mean

nb: with small $m$ you will have a result very similar to simple target encoding while moving slightly the means (enough to enable the split). The main goal of Bayesian Mean is to limit overfitting, which is also a problem when doing target encoding.
There is no point of doing that
The idea behind target encoding is to make the assumption that the category has no interaction with other variables. So you convert the categorical space (high dimensional) into a simple continuous space, where only the value counts. If you think that the algorithm might confuse two categories and that this is bad, it means target encoding is not suited. If different categories should follow a different decision path, then it means there are interactions between the categorical variable and the others.
If you want every category to have different decisional paths why don't you keep categories? Because implicitly, the tree will compute mean of the splits (explained in this video: https://www.youtube.com/watch?v=g9c66TUylZ4)


How to find the weights for weighted least squares regression?

No, the weight does not always has to be equal to the inverse of the square of variance. There is no universally accepted scheme. Different weighting schemes are used, such as $1/x$, $1/x^2$, $1/\sqrt{x}$, where $x$ are your data. The weighting scheme depends on many things: the distribution of your data, their meaning, and in which regions you want to fit them more closely (whether small data values should have lower error), etc.
This is a paper that compares different weighting schemes for data in chemistry: [1]. The introduction in that paper has references to many other papers discussing different weighting schemes.
[1] Ram B. Jain, Comparison of three weighting schemes in weighted regression analysis for use in a chemistry laboratory, Clinica Chimica Acta,
Volume 411, Issues 3–4, 2010, Pages 270-279, ISSN 0009-8981, https://doi.org/10.1016/j.cca.2009.11.021.


Nesterov Accelerated gradient is in correct order?

The standard momentum would have these steps:

straight away, recompute the new momentum:
$$\mu_{t+1} = \mu_{t} \cdot (decayScalar) + (learnRate)\cdot \nabla$$

adjust the weights by this new momentum
$$\theta_{t+1} := \theta_{t} - \mu_{t+1}$$

Nesterov momentum has this:

Make a big jump: correct the weights by any $\mu$ we have so far in our posession:
$$\theta_{t+1} := \theta_{t} - \mu\cdot (decayScalar)$$

Compute the gradient $\nabla$ from the new weights $\theta_{t+1}$

Correct these weights by this gradient (right now without any momentum):
$$\theta_{t+2} := \theta_{t+1} -  (learnRate)\cdot \nabla$$

At the very end, re-compute the momentum as follows:
$$\mu := \theta_{t+2} - \theta_{t}$$

So, the momentum is updated at the very end. It becomes a vector from "weights before the big jump", pointing towards the "weights after the correction by the fresh gradient".
reference: Geoffrey Hinton Lecture 6C Corsera

Re-arranging:
To avoid sticking the gradient computation in the middle of our optimizer function (steps 2 and 3), we can instead re-arrange things as follows:

compute gradient for the weights we have thus far.

correct such  weights by the gradient (right now without any momentum), as follows:
$$\theta_{t+1}:=\theta_t - (learnRate)\cdot \nabla$$

update the momentum:
$$\mu := \theta_{t+1} - \theta_{cached}$$
$$\theta_{cached} := \theta_{t+1}$$

big jump
$$\theta_{t+2} := \theta_{t+1} - \mu\cdot (decayScalar)$$

Notice, this way steps 2,3,4 are all inside of our optimizer. We can compute the gradient, outside of our optimizer (during step 1) making our code much more readable :)
    size_t _numApplyCalled = 0;

    //Nesterov Accelerated Gradient.
    //Placed at the end of a backprop, should be followed by a usual forward-propagation
    // https://datascience.stackexchange.com/a/26395/43077
    //
    void apply( float *toChange,  float *newGrad,  size_t count ){

        float learnRate = get(OptimizerVar::LEARN_RATE);//scalar
        float momentumCoeff = get(OptimizerVar::MOMENTUM_1);//scalar

        const bool isFirstEver_apply = _numApplyCalled == 0;

        for (int i=0; i<_arraySize; ++i){
                //correction by gradient alone:
                toChange[i]  -= newGrad[i]*learnRate;

                // determine momentum:
                if (isFirstEver_apply){//nothing was cached yet.
                    _momentumVals[i] = 0.0f;
                }
                else {
                    _momentumVals[i] = toChange[i] - _weightsCached[i];
                }
                //caching, AFTER momentum calc, but BEFORE the jump:
                _weightsCached[i] = toChange[i];

                //jump:
                toChange[i] -= _momentumVals[i] * momentumCoeff;
        }//end for

        ++_numApplyCalled;//increments by 1
    }


Is it possible to create a predictive model for a dataset that consists of only positive occurrences of the dependent variable?

My dataset would only contain data about earthquake occurrences and no
data about non-earthquake occurrences as that would basically be any
other period of time which is not kept in the dataset.

It would be nice if you could specify the features used for the prediction.

In that case, I assume a decision tree or logistic regression would
not work as we don't have a dichotomous dependent variable (as the
occurrence only gets into the dataset if an earthquake occurred).

Here you are right!. So it's better to resort to anomaly detection algorithms for the same. Here are some examples

Unsupervised anomaly detection
Outlier detection
Anomaly detection in python


How to measure the correlation of different algorithms

For regression tasks correlation will be simply the correlation between the predicted values, for binary classification it will be correlation between predicted probabilities. In multiclass classification you can find correlation between predicted factor variables using the hetcor package in R


PCA and maintaining relationship with target variable

The short answer is that the y_original and x_reduced are still connected to each other, so it is safe to train your data using y_original and x_reduced.  While x_reduced is on a different scale, as you mentioned via eigenvectors, it still is representative of the data that was attached to that observation, just in a different format.  You lose a lot of interpretability as far as what the actual numbers mean which is why it may seem confusing, but it's just a transformed representation of the x_original that (hopefully) contains enough of the x_original variability to make it useful.


Can we use boosting algorithms like Adaboost and gradient boosting with only one classifier

Boosting typically only use one algorithm as it's base learner (almost exclusively decision trees). However, you could use a mixed set of algorithms as your base learners.

Something like this:

Boosting round 0: Add decision tree
Boosting round 1: Add neural network
Boosting round 2: Add KNN
Boosting round 3: Add decision tree
...

The reason you only see boosting using the same algorithm is probably just because it works better. I speculate that the diversity that comes from using several algorithms shine more when they are trained in parallel and combined. In boosting the base learners are trained in a sequence.


Does MS PowerBI directly connect with Azure's ML workbench?

Yes, they can connect natively. You can manage data and then put it all in different services like it is showed next. As you can see you can use SQL database, blob storate and also PowerBI.

Here you can find a tutorial on how to do streaming analytics with Azure and PowerBI.


What is the difference between outlier detection and anomaly detection?

(I actually wanted to write this as an answer to the Cross Validated question: Difference between Anomaly and Outlier, but the question is protected - I think answering it here should be fine, despite the lower visibility)

People occasionally argue that there is no difference between an outlier and an anomaly by citing Charu Aggarwal, author of the Book "Outlier Analysis" - particularly, this statement:

  Outliers are also referred to as abnormalities, discordants, deviants, or anomalies in the data mining and statistics literature.

(Source: "Outlier Analysis" (Springer), Charu Aggarwal, 2017, http://charuaggarwal.net/outlierbook.pdf )

However, this statement  does not imply that outliers and anomalies are the same thing - analogously to saying that "Dogs are sometimes referred to as animals" does not mean that they are the same thing.

It's hard to give a formal definition of the terms. The Wikipedia page about outliers refers to the Wikipedia page about anomaly detection and vice versa, and they both contain lots of  possible definitions and interpretations of the terms. Things are becoming worse due to the domain-specific definitions and colloquialities, where it seems to be sufficient when two people of the same field roughly know what the other one is talking about...

However, Varun Chandola tries to give a more precise meaning to the term "anomaly" in his anomaly detection survey. Particularly, he classifies anomalies into three categories:

Point anomalies: An individual data instance can be considered as
anomalous with respect to the rest of data
Contextual Anomalies: If a data instance is anomalous in a specific context (but not otherwise)
Collective Anomalies: If a collection of related data instances is anomalous with respect to the entire data set

(Summarized from "Anomaly Detection - A Survey", Varun Chandola et al, ACM Computing Surveys 2009, http://cucis.ece.northwestern.edu/projects/DMS/publications/AnomalyDetection.pdf )

Here, the term "point anomaly" seems to be closest to what I'd consider as a possible definition of the word "outlier". And this is in line with the statement by Aggarwal: An outlier is an anomaly. But not every anomaly is an outlier.

(The latter may depend on the definition of the word outlier. Of course, one can define it on a meta-level, and say that an outlier is whatever a certain outlier detection algorithm (or model) detects as such. But most definitions that I encountered so far are based on some sort of "distance", "dissimilarity", or "difference" from a "majority" of other data elements. That sounds reasonable...)

An example: There may be several data points:

14.5, 14.2, 14.4, 14.4, 14.4, 14.4, 14.4, 14.4, 14.4, 14.3, 14.2, 14.6

One can compute the mean and standard deviation and will have a hard time arguing why one of these points should be an "outlier".

For a sequence of data points like this

14.5, 14.2, 14.4, 14.4, -64564.4, 14.4, 14.4, 14.4, 14.4, 14.3, 14.2, 14.6

spotting "the outlier" should be easy.

However, assuming that the first sequence describes, for example, average daily outside temperatures, the fact that the exact same average temperature of 14.4 degrees was measured for a whole week could certainly be considered as an "anomaly".

(Probably a "collective anomaly" according to the definitions above, but I won't argue about that...)

Although I'm on thin ice when arguing about the precise or intuitive meaning of certain terms (because I'm neither a data science expert nor a native English speaker), this would mean that "anomaly" is a much broader term than "outlier". But maybe the data science community is just in the process of sorting out proper definitions of these terms.

Update:

Maybe my gut feeling about the literal meaning of certain words is wrong. But for me, the word "outlier" seems to say "lying somewhere out of (or far away from) something (based on some distance measure)". In that sense, the 14.4s in the first example are not "outliers" per se. But of course, things become tricky very quickly here: One could imagine a model for the data that contains the number of consecutive days with equal temperatures (as in a run length encoding). Computing this model for the given data would yield

1 * 14.5
1 * 14.2
7 * 14.4
1 * 14.3
1 * 14.2
1 * 14.6

where the value 7 does have large distance (difference) to the other values in the model. So the "collective anomaly" of 7 consecutive days with equal temperatures has been turned into a "point anomaly" by this transformation.