In Neural network, if one node is deleted, where should other nodes be connected?

Whether you are actually removing the nodes or simply deactivating them, more or less the same concept would apply. The idea is called Dropout and is employed to inhibit the tendency to overfit. Suppose you're removing a node from the $i^{th}$ layer which has $n_i$ nodes. Removing it will not only involve removing the connections from the previous layers (${n_{i-1}}$ connections) but also removing the connections from the removed node to the next layer ($n_{i+1}$ connections) which would  essentially remove $n_{i-1}+n_{i+1}$ connections.
It would be much better to simply set the weights and biases to $0$. Here's the math behind the idea and you can find one possible implementation here.


Strange out of memory while loading lots of pictures before input CNN for deep learning

You should load images in batches. 480K of 224*224 sized images is too large for any machine let alone a 64GB one.

You can use image_generators from keras or tf.keras

https://www.tensorflow.org/tutorials/load_data/images#load_using_keraspreprocessing

https://keras.io/preprocessing/image/


Survival analysis to estimate kanban tasks completion times

It's a matter of defining exactly which problem you want to solve, and there might be many variants:

If the goal is really to estimate "time completion", then imho you should use only completed tasks, since the other tasks haven't been "completed". Note that in this case you're counting time actually spent on the task.
If the goal is to estimate "time of solving the task", whether by completing it or cancelling it, then you're counting the duration between the time the task was initialized and the time it was either completed or cancelled. Note that in this case the duration may include time spent on other tasks.

In both cases above, I don't see any proper way to include tasks which are still pending. My idea for these cases would be to calculate a different statistic, something like "rate of completed tasks after X days" for instance.


How are Lagrangian multipliers zero except for support vectors in dual representation of SVM?

In optimization, we have something called complementary slackness condition, it is part of the KKT conditions.

Every constraint, $g_i(x^*)\le 0$ in the primal corresponds to a dual variable $\mu_i$ (Lagrange multiplier).  The condition state that $$g_i(x^*)\mu_i=0$$

For points that are not support vectors, we have $g_i(x^*)<0$, hence we must have $\mu_i=0$.


I think a number of clustering algorithms that normally use a metric, do not actually rely on the metric properties (other than commutativity, but I think you'd have that here).  For example, DBSCAN uses epsilon-neighborhoods around a point; there is nothing in there that specifically says the triangle inequality matters.  So you can probably use DBSCAN, although you may have to do some kind of nonstandard spatial index to do efficient lookups in your case.  Your version of epsilon-neighborhood will likely be sim > 1/epsilon rather than the other way around.  Same story with k-means and related algorithms.
Can you construct a metric from your similarity?  One possibility: dist(ei, ej) = min( sim(ei, ek) + sim(ek, ej) ) for all k ...  Alternately, can you provide an upper  bound such that sim(ei, ej) < sim(ei, ek) + sim(ek, ej) + d, for all k and some positive constant d?  Intuitively, large sim values means closer together: is 1/sim metric-like?  What about 1/(sim + constant)?  What about min( 1/sim(ei, ek) + 1/sim(ek, ej) ) for all k? (that last is guaranteed to be a metric, btw)
An alternate construction of a metric is to do an embedding.  As a first step, you can try to map your points ei -> xi, such that xi minimize sum( abs( sim(ei, ej) - f( dist(xi, xj) ) ), for some suitable function f and metric dist.  The function f converts distance in the embedding to a similarity-like value; you'd have to experiment a bit, but 1/dist or exp^-dist are good starting points.  You'd also have to experiment on the best dimension for xi.  From there, you can use conventional clustering on xi.  The idea here is that you can almost (in a best fit sense) convert your distances in the embedding to similarity values, so they would cluster correctly.
On the use of predefined parameters, all algorithms have some tuning.  DBSCAN can find the number of clusters, but you still need to give it some parameters.  In general, tuning requires multiple runs of the algorithm with different values for the tunable parameters, together with some function that evaluates goodness-of-clustering (either calculated separately, provided by the clustering algorithm itself, or just eyeballed :)  If the character of your data doesn't change, you can tune once and then use those fixed parameters; if it changes then you have to tune for each run.  You can find that out by tuning for each run and then comparing how well the parameters from one run work on another, compared to the parameters specifically tuned for that.


How to classify whether text answer is relevant to an initial text question

Both message and answer are your input, so your feature vector should contain information about both.

Here's a simple structure of a possible solution using scikit-learn:

import numpy as np
from sklearn.svm import SVC
from sklearn.feature_extraction import DictVectorizer

dataset = (("Hey @foobar, have you been hacked?",
            "@barfoo it seems so, thx for suggesting",
            True), # True for relevant, False for not relevant
           ("Hey @foobar, have you been hacked?",
            "Lose 20 pounds quickly! http://blabla.com",
            False))

def extractMessageFeatures(message):
    # here comes your real feature extraction algorithm
    return { 'message_predicted_spam': False,
             'message_contains_valid_username': True }

def extractAnswerFeatures(answer):
    # here comes your real feature extraction algorithm
    return { 'answer_predicted_spam': False,
             'answer_contains_valid_username': True }

def extractFeatures(data):
    features = []
    for instance in data:
        instanceFeatures = extractMessageFeatures(data[0])
        instanceFeatures.update(extractAnswerFeatures(data[1]))
        features.append(instanceFeatures)
    return features

def trainClassifier(data):
    features = extractFeatures(data)
    vec = DictVectorizer()
    featureVector = vec.fit_transform(features)

    print vec.get_feature_names()
    print featureVector.toarray()

    svc = SVC()
    svc.fit(featureVector, np.array([i[2] for i in data]))
    return svc

clf = trainClassifier(dataset)

# now, you can clf.predict(...)

Now, the hardest part is to decide which features to extract from both messages and answers. It's up to you.

One of the simplest solutions would be to use n-gram features.

Other approach would be to use some spam detection to decide whether answer is spam or not and treat this information as a feature.

You can also use Twitter-specific information (for example, whether users are mentioning each other in their tweets, using the same hashtags etc.).

You can combine these features in whatever fashion you like, of course.

Except of creating feature extraction functionality you need a labeled dataset of messages, answers and relevant/non-relevant labels.

But once you have both (feature extraction functionality and a proper dataset), you're good to go with a task which clearly matches standard machine learning approaches.


In this discussion, it is described how a neural network that distinguishes odd and even numbers can be constructed.
Your question can be rephrased more generally: Is there a function that cannot be learned by a machine learning algorithm. This question was discussed here and here. Both discussions refer to the Universal Approximation Theorem that basically states that any computable function on a given finite range can be approximated by a neural network. So, what is left is uncomputable functions or undecidable problems. These cannot be learned by a machine-learning algorithm.


Model with two targets(classification, regression)?

You can also consider a multi-output model using the Keras functional API (see "Deep Learning with R", Ch. 7, p. 224). Here are the respective Keras docs. There are also some tutorials online. I'm not sure if a NN is suitable for your problem, but this would be an option to consider. The advantage is that your model can also take correlation between the features and the two outputs into consideration while learning (which may give better results).


Efficient solution of fmincg without providing gradient?

If you do not have a gradient available, but the problem is convex, you can use the Nelder-Mead simplex method. It is available in most optimization packages, for example in scipy.optimize.


Historical exchange rate data via World Bank API?

I don't think so, since:

USD is usually used as a reference.
Providing exchange rates for all possible pairs would be to much data. If there are 100 countries, there would be 100! (factorial) different pairs.
Exchange rates for all different pairs can be calculated manually by using the USD rates of two pairs.


I don't see how embeddings can help. Embeddings serve to lower the dimensionality of your data. I think you just need to introduce a single feature that is 1 if there were people interested and 0 if there were none. Then if you have sufficient data, the convnet should be able to understand the relation between the time-series and the feature you added.

It's a bit hard without having any idea about what your data looks like, but you might also be able to ignore the missing data, and train a model that predicts a continuous time series. Then the model may flag an anomaly when there is data missing, but you only actually declare an anomaly if the data is not missing and the anomaly is real.


Why do we need XGBoost and Random Forest?

It's easier to start with your second question and then go to the first.

Bagging

Random Forest is a bagging algorithm. It reduces variance.

Say that you have very unreliable models, such as Decision Trees. (Why unreliable? Because if you change your data a little bit, the decision tree created can be very different.) In such a case, you can build a robust model (reduce variance) through bagging -- bagging is when you create different models by resampling your data to make the resulting model more robust.

Random forest is what we call to bagging applied to decision trees, but it's no different than other bagging algorithm.

Why would you want to do this? It depends on the problem. But usually, it is highly desirable for the model to be stable.

Boosting

Boosting reduces variance, and also reduces bias. It reduces variance because you are using multiple models (bagging). It reduces bias by training the subsequent model by telling him what errors the previous models made (the boosting part).

There are two main algorithms:

Adaboost: this is the original algorithm; you tell subsequent models to punish more heavily observations mistaken by the previous models
Gradient boosting: you train each subsequent model using the residuals (the difference between the predicted and true values)

In these ensembles, your base learner must be weak. If it overfits the data, there won't be any residuals or errors for the subsequent models to build upon. Why are these good models? Well, most competitions in websites like Kaggle have been won using gradient boosting trees. Data science is an empirical science, "because it works" is good enough. Anyhow, do notice that boosting models can overfit (albeit empirically it's not very common).

Another reason why gradient boosting, in particular, is also pretty cool: because it makes it very easy to use different loss functions, even when the derivative is not convex. For instance, when using probabilistic forecast, you can use stuff such as the pinball function as your loss function; something which is much harder with neural networks (because the derivative is always constant).

[Interesting historical note: Boosting was originally a theoretical invention motivated by the question "can we build a stronger model using weaker models"]

Notice: People sometimes confuse random forest and gradient boosting trees, just because both use decision trees, but they are two very different families of ensembles.


Azure Cloud SQL - Querying large number of rows with Python

You have several phases of the data retrieval process: connection time, download time, and database procesing time.

If you store your data in a csv file in a blob storage then the processing time will be faster (essentially zero). So every day you could save the data from the database to a csv file and then access the file when you need it.

Azure serverless will reduce the connection time and download time (if your internet connection is slow), but will not reduce the processing time of the database.


You can look at the different evaluations as evaluation of the different stages of the model.

by evaluating the model's performance on the training dataset, you could assess how and what the model has learned from the data structure. This evaluation is mainly relevant for your research stage and the reporting of your results.
by evaluating the model's performance on the current/test dataset, you would see how the model is behaving and performing on current data the real predictions. This may help you to asses that

These two may differ in cases where the current data is behaving differently than the data that the model been trained on, and to help you to understand that the model needs to be updated and retrained on the data that it is performing poorly on.
If you're not monitoring your current data feed (distribution, range of values, the ratio of missings), I would suggest you do both for assessing if your ongoing observed performance is not what you are expecting.


What's the best way to use binned data in a tree-based model?

For a complete decision tree, either of your proposed models would be able to represent the same set of concepts since a decision tree can be decomposed into a disjunction of conjunctions. I don't think the fact that the bins have different sizes is really an issue because placing any significance on the exact location of splits occurring "inside" a bin is probably not a good idea anyway. From a performance perspective, what probably matters most is how your data are distributed (i.e., does your concept split the data between a single pair of adjacent bins or is it more complicated).

Since you have the "random-forest" tag associated with your question, I'm inclined to agree with you that keeping them ordered is probably better because if you are generating many trees, the ordered representation will likely result in more terse/smaller trees.


How should I format input and output for text generation with LSTMs

You can use word embedding, to encode words as vectors of real numbers. Then all calculations, such as comparison of words (to find similarity), are performed in that high-dimensional space.

  "What would be the best way to format my input (and my output) for
  this problem?"

I cannot tell which is the best approach (depends on your problem) but this one is very common. It's also very easy to implement in Keras, using the Embedding layer.


Differences between pseudocolor plot and heatmap?

A heatmap is effectively a pseudocolor plot with labelled rows and columns. So, basically heatmap is a pseudocolor plot based on a pandas DataFrame rather than a matrix. Basic difference is in the plotting methods available but you can convert the matrix to dataframe and vice-versa.
Reference :

Pseudoplot
Heatmap


The LSTM has a set of 2 matrices: U and W for each of the (3) gates. The (.) in the diagram indicates multiplication of these matrices with the input $x$ and output $h$.

U has dimensions $n \times m$
W has dimensions $n \times n$
there is a different set of these matrices for each of the three gates(like $U_{forget}$ for the forget gate etc.)
there is another set of these matrices for updating the cell state S
on top of the mentioned matrices, you need to count the biases (not in the picture)

Hence total # parameters = $4(nm+n^{2} + n)$


Concrete example to elaborate on the relationship between `linear function` and `aﬃne function` in the context of machine learning

There's a great answer on the Mathematics stack exchange about the distinction between linear and affine functions.

  A linear function fixes the origin, whereas an affine function need not do so. An affine function is the composition of a linear function with a translation, so while the linear part fixes the origin, the translation can map it somewhere else.

In section 5.1.4, the book introduces linear regression:
$$\hat{y} = \pmb{\omega}^{T} \pmb{x} + b$$

In this example, the $\hat{y} = \pmb{\omega}^{T} \pmb{x}$ part is a linear function.  It has to pass through the origin.  The addition of a vertical translation $b$ causes $\hat{y}$ to become an affine transformation, since it no longer has to pass through the origin.

I think the authors point out this distinction because it's technically incorrect from a purely mathematical point-of-view to describe $$\hat{y} = \pmb{\omega}^{T} \pmb{x} + b$$ as a linear function of the parameters $\pmb{x}$.  It's actually an affine function, so "linear regression" may appear to be a misnomer if you have a strong background in linear algebra.

For what it's worth, I don't think this is very helpful information to include in the main text of an introductory chapter on machine learning.  If you still don't understand, then just keep reading! Don't let this block you from progressing through the book.


Syntax error in function but everything seems correct

If you use Python earlier than 3.6 f-strings are not available, and that code has an incorrect syntax. In that case you can use the format method of the strings instead like that (2 lines are modified):
def  saveSlice(img, fname, path):
     img  = np.uint8(img * 255)
     fout = os.path.join(path, '{}.png'.format(fname))
     cv2.imwrite(fout, img)
     print('[+] Slice saved: {}'.format(fout), end='\r')


Train a RNN with strings using positive and negative samples

I think Sequence Classification with LSTM Recurrent Neural Networks in Python with Keras is one of the best tutorial available for this task. It does sentiment analysis on IMDB Dataset and is the simplest to understand. Also, it uses Keras framework which I use right now because of it's simplicity and it's online support and documentation available. The RNN of keras has multiple options which you can explore based on your requirement. Also, keras make it real easy to use an optimizer from a wide variety of options.


This is definitely possible. When you are reducing the threshold, you will never decrease the recall (you can only flag more of the positive examples as positive). Precision is looking at all the examples that you flag positively, and of those the fraction that are truly positive. This means when you are reducing the threshold, you might not add any true positives but only false positives, thereby lowering your precision. Let's take a look at a fabricated example, where P is positive and N is negative. The samples are ranked by score/probability. Everything before the threshold is flagged as positive:

PPNPNNPNNN

If we put the threshold between items 2 and 3, we get a precision of $1$ and a recall of $0.5$:

PP - NPNNPNNN

When we put it between items 3 and 4, we keep the same recall but our precision drops to $0.6667$:

PPN - PNNPNNN


How to generate list with out for loop in python

ndf = df.loc[ np.repeat(df.index.values, df['Number']) ]
values = ndf['Capacity'].tolist()
# [5, 5, 30, 60, 90]

Repeat the rows you wish you duplicate using np.repeat, and then, takes the selected column (here, capacity) and turn it into a list using .tolist().


Plotting three lines on the same plot (with four hr frequency)

First, in order to plot three different lines with a 4 hour interval you will need to resample your data. (This is how you get your frequency conversion). You can read about that more here.

pd.resample()

Then as long as all three of your "lines" are contained in the same data frame, you should have no problem plotting them by just calling

three_lines = your data frame

plt.plot(three_lines)

Although one note - just looking at your data, you might not get the desired result given the scale difference in your data. Said a different way, the values < 1 might be minimally visible when plotted on the same figure as values ranging around 100.

Hope this helps. Please drop a comment if you need any further clarification.


How to analyze CNN model summary and improve it?

What is the meaning of: Non-trainable params: 454? Should this ideally be 0? If so, how can this be made 0.

I guess it is due to using batch normalisation. Try to ascertain by investigating the math of that. If you see the paper, you can easily find out that the bias terms and $\beta$, if I remember, both get added. Consequently, one can be ignored and won't be trained because it is not needed.

  Dense layer should be added at the end (in the part you mentioned in your comment) or can it be added in previous layers also?

Not really, dense layers should be employed after conv layers. What they do is classifying the extracted features obtained by conv layers. About conv layers, they are employed for reducing the number of parameters and finding local patterns.

There is no consensus on how to change the number of filters in convolutional layers, at least as far as I know. But there is a point here. In the following lines of your code, you've employed a kind of pooling layer just before dense layer. If the number of activations coming from conv layer is many, you can use it but consider that by doing so, you ignore important features. I suggest you not doing that, especially, for the last conv layer. Also, try to increase the number of neurons in a dense layer or add extra layers for better accuracy.

conv2d_4 (Conv2D)            (None, 16, 16, 64)        73792
_________________________________________________________________
global_average_pooling2d_1 ( (None, 64)                0
_________________________________________________________________
dense_1 (Dense)              (None, 13)                845
=================================================================

About using batch norm, it is used due to a kind of problem which is called Covariat Shift. It simply tries to keep the distribution of the outputs of different layers in order to facilitate the learning process.

Based on your questions, I highly recommend you watching professor Andrew Ng's course about ConvNets in Coursera.


Can you use clustering to pick out signals in noisy data?

Have you looked at DBSCAN? It is a density-based spatial clustering of data with noise that can define non-linear clusters (unlike k-means).

It doesn't require knowing the number of clusters. However, it does require two parameters (minimum cluster size and neighborhood size) that measure density. But you may be able to estimate them in your particular domain.


Image Embeddings - Negative Sampling and Imbalanced Class Issues

I don't think we are creating a number of negative examples for every positive example. Negative sampling is done to efficiently compute the softmax.

Word2vec tries to maximize the similarity for words in similar contexts. For e.g. given drink the NN should predict water with maximum probability.

Suppose you have V (typically > 1 million) words in your training data, so the last layer of your model has V neurons. For every word in your training data, you would have to compute output from all the V neurons (to compute softmax). It is computationally very expensive. Negative sampling is one way to address this problem. Instead of computing the all the V outputs, we just sample few words and approximate the softmax.

Negative sampling can be used to speed up neural networks where the number of output neurons is very high. Hierarchical softmax is another technique that's used for training word2vec.


Multiple search/replace on pandas series

Create a definition for string replacements:

def change_string(x):
    return x.replace('|', '_').replace("elementary","elem").replace('old_value', 'new_value)

Then use map command to perform operation on entire column:

# You could create a new column to check whether the output is as expected
# If it is as expected, you could modify the original column
school_perf_report["SCHLNM_MODIFIED"] = school_perf_report["SCHLNM"].map(lambda x: change_string(x))

Hope this helps.


How can I explain the cause of different performances for two different LSTM models and improve the performance?

LSTM especially works for time-series data. I'm assuming these features are not dependent temporally, so it's not ideal to use LSTM or any RNN variant.

Remove the LSTM layers, reduce the number of neurons, use regularizer, dropout.
Also, using ReLU in the last layer doesn't make any sense. If it's a regression problem, just use linear activation instead.

To reduce the error, do a hyperparameter tuning with a randomized search or grid search.


Rate of convergence - comparison of supervised ML methods

This is a pretty involved question since this is an active area of research. The first statement is that often, the architecture is important (or number of parameters) before we can say something to the effect of we require $O(n^{k} log(\frac{1}{\delta^i}))$ for $i, k \ge 1$ samples to converge to a local optima. Guaranteeing accuracy is also depending on the data, so it is likely specific to what the data is itself. So you can break your question down into the analysis of stochastic gradient descent and the analysis of it in the context of neural networks. Unfortunately, neither of these deal with specific datasets, so your question about wanting accuracy $\ge n$ is still not possible with such a claim. To the best of my knowledge I am not familiar with claims made along that direction, however, with the former (analysis of SGD/specific neural architectures), there are some claims and papers that I can link below.

On the convergence rate of training recurrent neural networks. While I haven't read this paper -- the results appear to be specific to recurrent neural networks architectures and they give an analysis of a regression type error for certain parameterized forms of RNNs. Their results are also specific to the training error. (https://arxiv.org/abs/1810.12065)
On the rate of convergence of fully connected very deep neural network regression estimates -- This paper deals with a specific loss function not often found in classification ($\ell_2$ norm between the predictions and targets) but might serve as a useful reference. (https://arxiv.org/abs/1908.11133)
Optimization Methods for Large Scale ML -- a pretty comprehensive document on techniques and guarantees for optimization methods, but probably lacking specific results related to neural nets. Also a great reference point for earlier work on the subject. (https://arxiv.org/abs/1606.04838)

Note that this isn't supposed to be a comprehensive list by any means, in fact, this probably just scratches the surface of it.
There are also papers of a different flavour that use Langevin dynamics as a way to analyze the descent trajectory of SGD and provide bounds for it. Here's one https://arxiv.org/abs/1707.06618 but there are obviously several more in the reference of the paper.


why is more layers better in deep learning?

You can think of Neural Networks (however deep) as an approximation of an ideal function. The more layers/nodes are available, the more the Neural Network successfully approximate that ideal function. There is a theorem which states it.

Let's say you have to recognize human faces in pictures: there is at least one (ideal and purely hypothetical) mathematical function that is able to do the job. Than, a Neural Network can approximate that mathematical function in a more or less precise way.

Now: if you have the collection of every possible picture which can exist in the world since ever and forever (a so-called complete set), you could choose whatever number of layers and nodes, in whatever configuration, and you will never fall in an overfit situation. Instead, the more layers/nodes are available, the better your network will perform.
But in this limited real world, you can get just a tiny subset of that main set.

So you have to choose the optimal number of layers and nodes, given a configuration, to avoid to fall in the situation of overfit. The more data samples you have, the more you can add up layers and nodes to the configuration, with the result of having better performances, i.e. a Neural Network which better approximate the (ideal and purely hypothetical) mathematical function introduced above.


Question 1) Is this a good idea in general?

Solving this problem as a supervised learning regression problem is a fantastic idea and is the type of solution that will greatly benefit your company since it will translate to other similar and dissimilar problems much more easily than deterministic methods.

However using a neural network to solve this supervised learning regression problem is probably a very bad idea. Neural networks can add great value to certain problems, are among the very best algorithms for complex problems where computing power is not an issue, and have fascinating implications for the future of machine learning and artificial intelligence. But... they can be very tricky to train, require a lot of computing power, require a lot of data, and perform poorly in many cases.

I suggest you scope the problem using linear regression and then try a support vector regressor (SVM, SVR) or naive Bayes regressor. The analogue methods in SVR work very well with limited data and provide surprisingly accurate results.

Question 2) If yes, what type of NN would be most suited for such a problem?

If you must use a neural network then try playing with the problem.  Start with a feed forward neural network. Note that it will likely underperform an SVR.  Then think about moving toward a convolution neural network. Again, this is probably a very bad way to go. Try linear regression followed by other methods first.

Moving forward

The documentation for either Scikit-Learn in python, H20 in Java, or Weka provide very shallow learning curves to jump on the merry-go-round and take a spin or two. Please make sure that you thoroughly understand cross validation and scoring metrics as this is essential to continued, adequate progress.

Hope this helps!


Is there a clustering algorithm that works with only pairwise distances as input?

Yes, this is a classic Graph-Based Clustering problem in which each location is a node and you have the distances between them. Forgetting about concept of graphs and graph-based algorithm which might be complicating, I directly jump to your answer.
The most well-known algorithm is Spectral Clustering. There are tones of tutorial out there and it is well implemented in all programming languages including Java.
I briefly explain but do not Panic if you are not familiar with Math terms. They are pretty intuitive and easy. You just need to follow a good tutorial.

Calculate the Similarity Matrix (often referred to as Affinity Matrix in literature). As you have distance matrix you have several ways to do it. Simplest starts from saying $sim = \frac{1}{dist}$ till more sophisticated ones which use Gaussian Kernels for calculating similarity.
Calculate the Laplacian Matrix out of similarity matrix.
Calculate eigenvectors and eigenvalues in order to embedd your data points into their eigen-space (like exactly what we do in PCA)
Use a simple clustering algorithm like K-Means to cluster points in that space


Get data from intermediate layers in a Pytorch model

You can save the activations for intermediate layers into a variable within your forward method and then return them in addition to the output value.


How to improve the accuracy of my MLP (Current benchmark 77%)

1. Parameters optimization

As you have not mentioned the strategy, which you use for parameters choosing, I'd suggest to attend to it. For example in this article https://towardsdatascience.com/hyperparameter-optimization-with-keras-b82e6364ca53 you can find, how parameters can be chosen.

The other powerful approach is Bayesian Optimization Methods https://towardsdatascience.com/automated-machine-learning-hyperparameter-tuning-in-python-dfda59b72f8a It allows to move in parameters values space, taking into account previous results of optimization.

Note: it's crucial to consider parameters as a whole, but not just separately (if value $a$ of the parameter $A$ is not the best choice, it doesn't mean, that it won't be the best option in combination with value $b$ of the parameter $B$)

2. Deal with outliers

It would be reasonable to check outliers in the data (objects which appear to be inconsistent with the remaining instances). For this aim you can use box plot approach. It might be the reason, why it's impossible to get particular model performance.

3. Working with feature space

It also might be useful to provide feature analysis.


The NER model performance on a particular text depends on which data it was trained with originally, and naturally the standard models (like en_core_web_sm) are trained with English data which doesn't contain a lot of names from non-US/UK origin (same for other kinds of entities like organizations or locations).
Better performance can be achieved by training your own model with your own labelled data, but that requires you (or somebody) to annotate a reasonably large sample of data manually.


How do I make an interactive PCA scatterplot in Python?

There is an awesome library called MPLD3 that generates interactive D3 plots.

This code produces an HTML interactive plot of the popular iris dataset that is compatible with Jupyter Notebook. When the paintbrush is selected, it allows you to select a subset of data to be highlighted among all of the plots. When the cross-arrow is selected, it allows you to to mouseover the data point and see information about the original data. This functionality is very useful when doing exploratory data analysis.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sb
import mpld3
from mpld3 import plugins
%matplotlib inline

iris = sb.load_dataset('iris')
from sklearn.preprocessing import StandardScaler
X = pd.get_dummies(iris)
X_scal = StandardScaler().fit_transform(X)

dim = 3
from sklearn.decomposition import PCA
pca = PCA(n_components = dim)
Y_sklearn = pca.fit_transform(X_scal)

# Define some CSS to control our custom labels
css = """
table
{
  border-collapse: collapse;
}
th
{
  color: #ffffff;
  background-color: #000000;
}
td
{
  background-color: #cccccc;
}
table, th, td
{
  font-family:Arial, Helvetica, sans-serif;
  border: 1px solid black;
  text-align: right;
}
"""

fig, ax = plt.subplots(dim,dim, figsize=(6,6))
fig.subplots_adjust(hspace=.4, wspace=.4)
tooltip = [None]*dim

N = 200
index = np.random.choice(range(Y_sklearn.shape[0]),size=N)

for m in range(dim):
    for n in range(m+1):
        ax[m,n].grid(True, alpha=0.3)
        scatter = ax[m,n].scatter(Y_sklearn[index,m],Y_sklearn[index,n],alpha=.05)

        labels = []
        for i in index:
            label = X.ix[[i], :].T.astype(int)
            label.columns = ['Row {0}'.format(X.index[i])]
            labels.append(str(label.to_html()))

        ax[m,n].set_xlabel('Component ' + str(m) )
        ax[m,n].set_ylabel('Component ' + str(n) )
        #ax[m,n].set_title('HTML tooltips', size=20)

        tooltip[m] = plugins.PointHTMLTooltip(scatter, labels,
                                           voffset=20, hoffset=20, css=css)
        plugins.connect(fig, tooltip[m])

plugins.connect(fig, plugins.LinkedBrush(scatter))
test = mpld3.fig_to_html(fig=fig)

with open("Output.html", "w") as text_file:
    text_file.write(test)

See it in action on my blog.

Update [July 9, 2016]:
I just found out that Plot.ly has an offline mode and is now open source. It has a lot of the bells and whistles prepackaged, but MPLD3 may still be appropriate in some cases.


Performance measure: Why is it called recall and sensitivity?

I think the term "sensitivity" comes from the world of medical tests. A very sensitive test will test positive for most or all people who take the test and really have a disease, as well as for many people who don't. This corresponds to high recall, which means the query retrieves most or all of the relevant documents, as well as many that may not be relevant.


Conv1D specify output chanels in tensorflow 2.1

Refer to this: https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv1D

The parameter filter determines the output channel.


Multiclass classification problem with multiple targets to be predicted

You might want to visit the definition of multi-class and multi-label problem. In this case looks like are aiming to solve a multi-label problem. Refer to the following problem, this might help you better understand your situation.  https://stats.stackexchange.com/a/11866/286349


variables selection in regression models

Lasso stands for ´least absolute shrinkage and selection operator´. It has a penalty that is the absolute value and makes a lot of variables converge to cero. There is a ton of blogs that explain really well Lasso on the internet, have a look!

Elastic Net is a combination of Ridge and Lasso. So it will also reduce the variables a lot.

Ridge is a quadratic penalty so the convergence is mellower. There are also some posts about why this happens Why will ridge regression not shrink some coefficients to zero like lasso?

In Multiple Linear regression, there is no penalty, so coefficients have not a special reason to be zero. They will only be cero when there is no relation with the target variable. Imagine that you add a feature that is just random noise. You will end up with a cero coefficient for your linear model.

This should give you an intuition of why Lasso shrinks most of the features, Enet a little bit less, Ridge not so many and Linear Regression only some or none.


Distinguish randomly generated texts from reasonable for human texts

You could train a character-level language model, e.g. an LSTM, on the real short texts, and use the perplexity as the signal to know whether a piece of text is real or not.
In order to find an appropriate perplexity threshold, you can have a look at the distribution of perplexities over a validation holdout dataset.
UPDATE: There are multiple implementations of language models. For "classical" options, you can go for KenLM or if you have GPUs to train the model, you can use fairseq. Just remember to prepare your text to have character-level tokens (normally you just need to have a space between every letter).


What ML/DL techniques power Youtube/Netflix search systems?

At Netflix, the machine learning algorithms used are more complex since the data is really vast and increases day to day. Take a look at these:

Netflix Recommendations: Beyond the 5 stars (Part 1)

Machine Learning Is Everywhere: Netflix, Personalized Medicine, and Fraud Prevention | Udacity


Standard PCA only sees your data as real values and hence doesn't really provide the best low rank approximation. There exist versions of PCA designed for boolean data types. Personally I think the best paper covering the whole broad topic of generalised PCA is Generalized Low Rank Models. In section 5.2 they specifically cover Boolean PCA.


You are saying "For various group of products" and this is your answer.

Forecast each group of product independently, then split the group total on individual products. The reason for this strategy are that it is easier and more accurate to forecast big numbers than rare cases. An other reason is that by grouping similar product, you have chances to capture niceties like new products cannibalizing old products.


The automate the extraction of a title is an example of extractive summarization. It is typically solved with a combination of supervised and sem-supervised learning. More recently Deep Learning models with attention have been giving state-of-the-art results.

You'll need labeled to train your own model or used established packages such as symy.


There are two issues here.
First, you are in a regression setting, where accuracy is meaningless - see What function defines accuracy in Keras when the loss is mean squared error (MSE)?
Although the linked thread is not directly applicable here (since you use your own custom calc_accu function), the essence of the argument remains: assessing K.equal(K.round(y_true), K.round(y_pred)) is not the correct way to measure the performance of a regression model. In such settings often the loss (MAE, MSE, RMSE) and the metric are the same; what you do here (accuracy aside), i.e having an MSE loss and an RMSE metric is also valid.
Second, there is no way to tell if an RMSE of ~ 0.5 is "good" or "bad" in itself, even more since here you scale your output y before feeding it to the model, hence the calculated RMSE is on these scaled values, and not in the true, unscaled ones. What you have to do is calculate the "true" RMSE in the original unscaled values, and then decide if it is good enough for your purpose or not. Essentially what you have to do is:

Get predictions for your (scaled) test data
Transform back your predictions to the initial scale of the original test data
Get the RMSE between these invert-transformed predictions and the original data

To do so (and a good practice in general), it is necessary to define different scalers for your x and y, i.e.:
from sklearn.preprocessing import StandardScaler

# feature scaling
sc_X = StandardScaler()
X_train = sc_X.fit_transform(X_train)
X_test = sc_X.transform(X_test)

# outcome scaling:
sc_y = StandardScaler()
y_train = sc_y.fit_transform(y_train)
y_test = sc_y.transform(y_test)

Now, assuming the data have been scaled as I show above, here is how you should calculate your RMSE on the original (unscaled) test data:
from sklearn.metrics import mean_squared_error

y_pred = model.predict(X_test) # get model predictions (scaled inputs here)
y_pred_orig = sc_y.inverse_transform(y_pred) # unscale the predictions
y_test_orig = sc_y.inverse_transform(y_test) # unscale the true test outcomes

RMSE_orig = mean_squared_error(y_pred_orig, y_test_orig, squared=False)

It is this RMSE_orig that you should decide if it is satisfactory for your problem; plotting the original unscaled predictions y_pred_orig against the original (unscaled) outcomes y_test_pred might also be useful and informative.
See own answer in SO thread How to interpret MSE in Keras Regressor for a similar case.

Other than that, it is speculative to say anything more since we do not have your data; nevertheless, a general rule is that we don't use dropout by default - only when we suspect that our model might overfit the data. So, I suggest experimenting with all dropout layers removed, and possibly putting them back in only gradually, and only in the case you observe overfitting.


How to use a set of pre-defined classifiers in Adaboost?

One possible solution is using a Stacking classifier as follows:
from sklearn.ensemble import StackingClassifier
# Assumed you already import  all the models you are using

dt = DecisionTreeClassifier(max_depth=DT_max_depth, random_state=0)
rf = RandomForestClassifier(n_estimators=RF_n_est, random_state=0)
xgb = XGBClassifier(n_estimators=XGB_n_est, random_state=0)
knn = KNeighborsClassifier(n_neighbors=KNN_n_neigh)
svm1 = svm.SVC(kernel='linear')
svn2 = svm.SVC(kernel='rbf')
lr = LogisticRegression(random_state=0,penalty = LR_n_est, solver= 'saga')

estimators = [
    ('rf', rf),
    ('svm1', svm1), ('svn2', svn2), ('xgb', xgb), ('knn', knn), ('lr', lr), ('dt', dt)
]

Option 1:
stacker = AdaBoostClassifier()
model = StackingClassifier(
    estimators=estimators, final_estimator= stacker
)

Option 2:
base_model = StackingClassifier(
    estimators=estimators
)
model = AdaBoostClassifier(base_estimator = base_model)

Option 2 will be without question the most expensive* of both and as far as I understand, option 2 fits better what you are looking for.

Strategy:


What are your favorite sources for news about Machine Learning and Data Science?

I like following certain tags on Cross Validated, as well as the r tag StackOverflow.
arXiv.org is also pretty cool, it has several subsections devoted to stats and ML.


Document classification: tf-idf prior to or after feature filtering?

As you've described it, Step 4 is where you want to use TF-IDF. Essentially, TD-IDF will count each term in each document, and assign a score given the relative frequency across the collection of documents.

There's one big step missing from your process, however: annotating a training set. Before you train your classifier, you'll need to manually annotate a sample of your data with the labels you want to be able to apply automatically using the classifier.

To make all of this easier, you might want to consider using the Stanford Classifier. It will perform the feature extraction and build the classifier model (supporting several different machine learning algorithms), but you'll still need to annotate the training data by hand.


How to retrive the results saved in model of gensim?

The question is not really clear.

If the question is how can you save and retrieve the cosine-similarity of each word with every other word, then you can save them in an array and use pickle for saving and loading them again.

Or, If the question is how you can retrieve the gensim model, then the function is gensim.models.word2vec.Word2Vec.load()


The real first question is why are people more productive with DataFrame abstractions than pure SQL abstractions.
TLDR; SQL is not geared around the (human) development and debugging process, DataFrames are.
The main reason is that DataFrame abstractions allow you to construct SQL statements whilst avoiding verbose and illegible nesting. The pattern of writing nested routines, commenting them out to check them, and then uncommenting them is replaced by single lines of transformation. You can naturally run things line by line in a repl (even in Spark) and view the results.
Consider the example, of adding a new transformed (string mangled column) to a table, then grouping by it and doing some aggregations. The SQL gets pretty ugly. Pandas can solve this but is missing some things when it comes to truly big data or in particular partitions (perhaps improved recently).
DataFrames should be viewed as a high-level API to SQL routines, even if with pandas they are not at all rendered to some SQL planner.

You can probably have many technical discussions around this, but I'm considering the user perspective below.
One simple reason why you may see a lot more questions around Pandas data manipulation as opposed to SQL is that to use SQL, by definition, means using a database, and a lot of use-cases these days quite simply require bits of data for 'one-and-done' tasks (from .csv, web api, etc.). In these cases loading, storing, manipulating and extracting from a database is not viable.
However, considering cases where the use-case may justify using either Pandas or SQL, you're certainly not wrong.  If you want to do many, repetitive data manipulation tasks and persist the outputs, I'd always recommend trying to go via SQL first.  From what I've seen the reason why many users, even in these cases, don't go via SQL is two-fold.
Firstly, the major advantage pandas has over SQL is that it's part of the wider Python universe, which means in one fell swoop I can load, clean, manipulate, and visualize my data (I can even execute SQL through Pandas...).  The other is, quite simply, that all too many users don't know the extent of SQL's capabilities.  Every beginner learns the 'extraction syntax' of SQL (SELECT, FROM, WHERE, etc.) as a means to get your data from a DB to the next place.  Some may pick up some of the more advance grouping and iteration syntax.  But after that there tends to be a pretty significant gulf in knowledge, until you get to the experts (DBA, Data Engineers, etc.).
tl;dr: It's often down to the use-case, convenience, or a gap in knowledge around the extent of SQL's capabilities.


Classification: Dealing with unknown class balance in the prediction dataset

This phenomenon is called Covariate Shift and it can affect features,target, you name it.

This happens when the test data distribution is different from the training data. Or your problem is a time-series one where time is the biggest independent variable.

First case : No model can adapt to the target shifting in its behavior. You have to either wait until you collect enough data from the current times, and retrain a fairly satisfying model, or do online learning which means that every datapoint that you predict for , is fed into the model to be trained on. That way; if changes in the target occur, your model will get updated with every new datapoint.

Second case : Lets say your target variable has a tendency to go up every 2 years, and then goes down. This is an assumption i'm making. Your task is to detect that shift in time that affects your target and model it. You create a model that takes into account that every 2 years, your target goes down. and so on. It's what we call a time-series problem where time is the biggest contributor to the target's changes.

If the second case doesn't make sense to you, and doesn't relate to your problem, then look at first case.


How to specify steps_per_epoch and validation steps on infinite dataset?

This can be explained only if you specify the total number of samples in your entire data set. The 200 Gb file size might be because of a large number of features for each sample in the dataset. The is how steps_per_epoch and  validation_steps work.
The training generator will yield steps_per_epoch batches.
When the epoch ends, the validation generator will yield validation_steps batches.
so usually the ball-park value for a decent number of batches in training is 75.  and that of validation is 25. if youre having a train to validation split of 3:1 for a 1000 sample data set. so that each of train and validation set batches will have 10 samples each.


Is MLlib compulsory to work with distributed data?

Yes. There are a million ways to engineer a solution.

You can use pandas for data wrangling and distribute this task via parallel python.

I personally use python for distributed - data acquisition, data wrangling, and algorithmic computation. This way i get the capability of Multi-machine, multi CPU, & multi GPU.

If you want to use python all they way, i recommend recording all the different options in terms of libraries, then order them by most active in development releases, or by whatever criterion you value the most.


How do you perform multilabel classification that is also a multiclass problem?

From the description this is not a multilabel problem because:

Each of the three "classes" (columns) must have a label. In a multilabel problem every class is optional.
Every "class" (column) appears to have a specific purpose subdivided into 4 labels. In a regular multilabel problem the labels are exchangeable, e.g. a document can have topics "sports" and "society" but not "politics", all of these labels have no order and no specific role distinct from the others.

It seems that you simply have three regular independent multiclass problems:

problem 1 = predict "class 1"
problem 2 = predict "class 2"
problem 3 = predict "class 3"

Note: the word "class" for the columns is confusing because these don't correspond to the regular concept of class.


Why word2vec performs much worst than both CountVectorizer and TfidfVectorizer? [Text classification]

My guess is that the dataset is very small. Word2Vec won't be able to capture word relationship in the embedding space with limited information. Try to train word2vec on a very large corpus to get a very good word vector before training your classifier might help.


I think you should replace this:
classifier.add(Dense(2))

because you have 2 class: cat & dog.


Simple Binary Classification Example in Python

The problem you are describing is a type of sequence prediction.

I think this is too complicated for any numpy or pandas library to solve. Maybe sklearn has something close, but I think a neural network would give you the most accurate predictions.

Before deciding, some of the points to consider are:

Is there a limit on max length for the string pattern you are
looking for?
Do you only want one prediction, based on one pattern that is
considered most likely, or do you want to consider multiple
patterns?
Others

Examples:
001001001 -> next digit should be 001
01001010010 -> there are 2 subpatterns and another larger pattern. next digiti could be 50% chance of 0 (01 pattern) and 50% chance of 1 (001 pattern)

I think the best approach is to let an LSTM find any patterns and predict the next digit based on the model it built. The model will contain all the information mentioned above without you needing to define them.

Depending on how simple or complicated you want to make this, you could start with a simple LSTM model and go from there.

https://machinelearningmastery.com/sequence-prediction/

or

https://machinelearningmastery.com/models-sequence-prediction-recurrent-neural-networks/


1) Store the data. For the moment I have plain text files (each file
  has rows of a fixed number of columns - most of the data are
  categorical). Would it make sense to save those in some database (SQL)
  or hdf5? Any hints on which is preferable?

Yes, it would make sense to store in a local database, rather than using large csv/text files. As you say that the data is derived from a NoSQL source, I assume unstructured data. So, using a SQL/relational store is out of question. As you say you are using Python, I would suggest you use TinyDB, which is both light-weight and easy to handle.

  2) Which plotting library would you propose for the graphs? I have
  seen about bookeh and matplotlib supports interactive widgets but I
  don't know what people normally use.

Matplotlib would be good enough. Actually, this question is more opinion-based than anything else. There are a lot of visualization libraries you can use, like Bokeh, Seaborn, etc.

  3) Could I export the analysis results in an IPython notebook and then
  in html programmatically?

Yes, you can do the analytics directly in an Ipython notebook(Jupyter), which also supports Markdown and HTML cells.

In addition, you can also use widgets and interactive visualization with Jupyter Ipy notebooks and Matplotlib.  Tutorials for the same


Is there any single disadvantage to use GPU in deep learning?

Transportation of data to and from the GPU. Data goes into GPU through the bus, allocated into the memory, processed and sends back the results through the same channel. Whether it is noticeable or not it takes time.

Unlike CPU which features sequential execution, GPU features parallel executions and its blazingly fast.

One main disadvantage of GPUs is that it's not built for deep learning instead its originally designed to implement graphics pipelines. As deep learning also align the same kind of computation(matrix multiplications), GPUs were used for deep learning.

More at Are there any disadvantages of using GPU in deep learning?

GPU's are very fast at applying the same instruction to multiple data points (SIMD). However, if you add branching (an if), the two branches will be serialized (first the if on all data that takes it, then the else on all data that doesn't - so still quite parallel, but not entirely). Also, if you have very few data points, the overhead of uploading to the GPU and downloading the result will likely dominate the overall execution time - it might be cheaper to just execute on the CPU instead.


NLP algorithm: sentiment with specific guidelines

This seems like a straight forward classification problem: "Increased", "Decreased" (I assume "Neither Increased nor Decreased" can be a third category as well). Something simple like a Support Vector Machine (SVM) could work.

That being said, only 400-500 example cases aren't a lot so you might run into overfitting especially if you try to run whatever model you land on later down the road.


How to update weights in a neural network using gradient descent with mini-batches?

Let us say that the output of one neural network given it's parameters is $$f(x;w)$$
Let us define the loss function as the squared L2 loss (in this case).
$$L(X,y;w) = \frac{1}{2n}\sum_{i=0}^{n}[f(X_i;w)-y_i]^2$$
In this case the batchsize will be denoted as $n$. Essentially what this means is that we iterate over a finite subset of samples with the size of the subset being equal to your batch-size, and use the gradient normalized under this batch. We do this until we have exhausted every data-point in the dataset. Then the epoch is over. The gradient in this case:
$$\frac{\partial L(X,y;w)}{\partial w} = \frac{1}{n}\sum_{i=0}^{n}[f(X_i;w)-y_i]\frac{\partial f(X_i;w)}{\partial w}$$
Using batch gradient descent normalizes your gradient, so the updates are not as sporadic as if you have used stochastic gradient descent.


You begin by asking about image normalisation, but then refer to other techniques, which I believe all fall under "image augmentation". So I will answer the more general question:

  how can I perform image augmentation to improve my model?

I would generally say that the more augmentation you can apply, the better. A caveat to that statement is that the augmentations must make sense for your target case. I will explain at the end what I mean by that.

Let's list some ways to augment an image, along with some common values:

Augmentations

1. Flipping

Starting with perhaps the easiest to implement, you can (easily) flip images:

horizontally: makes an image of an arrow pointing left point in the opposite direction - to the right, and vice-versa
vertically: makes an arrow pointing down point up, and vice-versa.

It is more difficult, due the general rectangular shape of images, but you could flip an image diagonally, but I haven't seen anyone do that really. It is getting close to the idea of rotation, I suppose.

2. Rotation

In this case, we simply rotate the image about its center by a random number of degrees, usually within a range of [-5, +5] degrees. Even tiny small angles, which we can maybe hardly perceive, this is a simple trick that can add a lot of rubustness to a model.

3. Crop

We take "chunks" of an image, either at random or using a pre-defined pattern. This helps the model perhaps focus on certain areas of images and not be overwhelmed by perhaps unimportant features. It also simply creates more data.

You might also incorporate this augmentation method as a means to resize your input images to fit into a pre-trained model, which took input images of a different size to your data.

4. Normalisation

This is more of a numerical optimisation trick that one which can really be interpreted from a visual perspective. Many algorithms will be more stable whilst working with smaller numbers, as values are less likely to explode. Numbers closer together (in the sense of continuous space with a linear scale) are also more liekly to produce a smoother optimisation path.

One way of performing normalisation is:

first computing a mean and standard deviation of the images, then
(from all images) we subtract the mean and divide by the standard deviation.

This will produce images whose pixel values that lie over some range, but have a mean value of zero. Have a look here for some other methods and more discussion. You can normalise e.g. over whole images or just over the separate colour channels (RedGreenBlue).

One thing to keep in mind, is that the values used for optimisation must come from the training dataset itself, and must not be computed over the entire dataset including the validation/test datasets. This is because that information should not be passed to the model in any way - it is kind of cheating.

5. Translations

SImply move the image up, down, left or right by some amount, such as 10 pixels, or again within a range of [-5, +5] % of the image size. This will produce pixels along either one or two axes, which must be filled by another colour because the picture has moved out of the frame. It is common to use black or white for these "empty" parts, but you could also use the average pixel value or even crop the shifted image to remove them.

6. Rescaling

This is a less obvious augmentation step, but can add value because the model will be able to extract different features (more or less general) from larger and smaller images. A small image without much detail would only allow the model to learn higher level, or hazier features and not be able to focus on specific details that are available in high resolution images.

Methods such as progressive resizing, commonly used for training GAN architectures, start with smaller images and slowly work up to larger images. This can be done to keep training as stable as possible (in GANs it can increase robustness against mode collapse), but also coincides with the notion that early layers in a network learn high-level features, and layers deeper into e.g. a convolutional network learn representations that are more detailed. Why do the early layers require high resolution images? Let's give them low resolution images, train quicker and hopefully generalise slightly better!

7. Be creative!!

I once made a model for a self-driving car. I knew that the track where the model would be tested had many trees and quite a few high walls, which oth created shadows over the road. These shadows looked a lot like straight lines representing the edges of the road! So I added shadows to my images, at random intensities and random angles with random sizes. The car learnt that a change in brightness over a straight line did not necessarily mean the edge of the road and hence stopped freaking out when it reached one!

Think about the artefacts of your training data and the situations your model will face in the validation data, and try to incorporate them. This is really like feature engineering, incorporated into a model via augmentation... and it can help a lot!

8. Examples

Just to add a nice picture, here is a great display of how interesting variations can be made from a single image, taken from the Keras tutorial, linked below. The original image is top left and the remaing 7 are the output of combinations of a set of random augmentations:

You can have a look at an article like this, which explains augmentation quite well, with examples.

Another great example here, part of a series of articles, goes through augmentation possibilities in some detail.

Implementations

1. Keras: ImageDataGenerator

This is a single class, where you can say which augmentation steps to apply and (if relevant) how much. It allows very easy integration of augmentation into a model.

Check out the official Keras tutorial for some great examples with images.

2. PyTorch: Transforms

You can build a small pipeline of transformations, gathering them together into a Compose object, to have fine-grained control over each augmentation step, its parameters and with which probability it is applied.

Have a look at this tutorial for an example of combining Rescale and RandomCrop.

3. Augmentor: Augmentation library:

This standalone library allows you build up great augmentation pipelines, which then hand off the images into a model. There are a great number of possibilities implemented, such as random warping/distortion, which looks really funky:

It also look as easy to use as the Keras solution shown above!

Going back to the start

Now going back to my opening statement, the reason we want to apply as many augmentations as possible, is because it is synthetically creating more data for our model to learn from. We are trying to teach the model a distribution of possible data inputs and make it learn how to produce the correct output. We usually do this on a training set and then ask for its predictions on a validation set, which contains unseen data. The more data that the model has seen from the distribution/function that generates the data, the less surprised and better it will perform on the "unseen" data. We want our model to be as robust as possible to make kinds of alterations to data, and exposing it to more of that helps.

The caveat, that we only use relevant augmentations, is necessary because as soon as we start using augmentations that skew the input distribution somehow (e.g. showing the model images that make no sense1).

So use as much augmentation as possible, within the constraints of reality for your problem; as well as the time it takes to train the model  ;-)

1 - Imagine training a model to detect cars on a road. If we perform a vertical flip, images shown to the model will contain cars that are essentially driving upside down; "on a grey the sky". This really doesn't make sense and will never occur in a realistic validation set or in the real world.


You are probably thinking in terms of regular SQL but spark sql is a bit different.

You'll need to group by field before performing your aggregation. Thus the following, you can write your query as followed :

df.groupBy('name').agg({'number': 'mean'}).show

You can also write it using a SQL dialect by registering the DataFrame as a temp table and then query on it use the SQLContext or HiveContext :

df.registerTempTable("people")
df2 = sqlContext.sql("select name, avg(number) as average from people group by name")

I hope that this answers your question.


Time series data combined with multiple feature. Predicting difference from the mean

With random forest there is no need to modify your output variable (i.e difference from the mean) when performing a regression.

It may be that the features you have are not relevant, or that you don't have enough data. Also try different model parameterisations (num trees, num layers etc).

Try to do some feature engineering to create intelligent features that might help the classifier.


Which model for a content suggestion in which only some of the items are currently available

It is pretty common in recommendation systems to have business constraints to filter against before a final recommendation is made (e.g. checking whether each recommendation is in stock).
One way to manage this is to produce a superset of recommendations (e.g. 100 if you need to produce 10 recommendations), apply your filters (which could drop the number of recommendations down considerably), and then return the top-10 recommendations from the filtered set. So, you may not need to produce the full 1000 item recommendation set each time, but you most likely will need to generate a list which is initially large and filter down.
A potentially helpful link:
https://buildingrecommenders.wordpress.com/2015/11/10/the-components-of-a-recommender-system/


How to run tensorflow model twice before computing the loss

So, as in comment:

Put as many inputs in the batch as you want, and the run prediction on that batch. That way, you could use all outputs to calculate the loss function.

Keras

In Keras framework, if you want to calculate the mean_absolute_error as you state in your question, you could use predefined mean_absolute_error loss. Also, to have two inputs as you state in your question, specify the batch size of the model to 2.


In particular, I want to select as features the instructions, so the push, mov, jmp,..etc and create a feature vector that contains the number of times a feature appears.

Instead of reading each instruction entirely for a sample program, you can read the first word of each instruction to get the opcode and then create a feature vector out of these opcodes.

Suppose,

sample_program[i] =  ['push r12', 'push rbp', 'xor edx edx', ...]

i.e. a list of instructions, then define opcodes[i] as

opcodes[i] = [instruction.split[0] for instruction in sample_program[i]]
#['push', 'push', 'xor']

Now you can create feature vectors out of collections of such opcodes.


From the wikipedia page:

  Previously there has not been any theoretically sound justification for using a logarithmic reduction factor2 other than the fact that it produces a smooth reduction. But Wang et al. (2013)[3] give theoretical guarantee for using the logarithmic reduction factor in NDCG. The authors show that for every pair of substantially different ranking functions, the NDCG can decide which one is better in a consistent manner.

The work showed that the log function is able to converge to consistent results for different ranking functions. It distinguishes the methods nicely even at high numbers of documents (at thelimit).

For example, here is a graph of the ranking measures using NDCG@k, where only the top k entries in the ranked list are considered:


Strange Pearson Correlation Coefficient Given DataFrame

If two variables are independent, then their correlation will be zero. However, you cannot say the opposite. Zero correlation doesn't necessarily imply independence.

It's hard to answer your question without any plot or the raw data, but you need to seek information for non-linear dependency.

This is a great example from Wikipedia page to understand that Pearson cannot prove dependency


How to teach neural network a policy for a board game using reinforcement learning?

You need to use some function approximation scheme. In addition, experience replay would be useful for two reasons: (1) you want to keep past memories (2) you need to decorrelate the way to teach your network.

Have a look at Deepmind's DQN on ATARI games. What you are describing is basically what they have solved. The paper is in their website: http://deepmind.com/dqn.html

Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, et al. Human-level control through deep reinforcement learning. Nature. 2015 Feb 25;518(7540):529–33.

With respect to the network architecture, it will definitely require some experimentation. For an alternative, you can also have a look at HyperNEAT. They evolve the network topology:

Hausknecht M, Khandelwal P, Miikkulainen R, Stone P. HyperNEAT-GGP: A HyperNEAT-based Atari general game player. In: Proceedings of the 14th annual conference on Genetic and evolutionary computation p. 217–24. Available from: http://dl.acm.org/citation.cfm?id=2330195

For more strategic games. Maybe you can have a look at "Giraffe: Using Deep Reinforcement Learning to Play Chess" http://arxiv.org/abs/1509.01549


Using Information from the rest of a Sequence to Predict the Label for any one Item

The fastest way to train a model to predict each item's label is using Conditional Random Fields (CRT) like in this example. h/t @erwin


Finding cause and effect correlation in a dataset

Identifying correlation in data is possible with standard statistical tools, and it can be done programmatically. Correlation requires a representative sample, so in a case like the single graph that you show it wouldn't work: you'd probably need at least 20-30 such graphs in order to have a chance to confidently show correlation.
Identifying causation (cause and effect) is much harder. To the best of my knowledge it cannot be done from raw data without any additional knowledge. This is simply because with any dataset, there's no way to know if some external variable plays a causative role in the observations.
The way causation is proved is through a specific experimental setting, i.e. it's the design of the experiment which allows collecting data susceptible to prove causation. A/B testing is a method used for this kind of experiment. Example: suppose we want to test whether new drug A has an effect on outcome B. Design the experiment to observe the response variable B on a representative sample of patients. Randomly split the sample into two subsets: the first one is the control group, they are given a placebo. The second is given drug A. At the end of the experiment one can prove causation if and only if the second group shows a significantly different outcome compared to the control group.


The recurrent part of a network allows you, generally speaking, to model long and short term dependencies. So your model can have some sense of state.

This is typically advantageous if you're using timeseries. For example, if you have data from a heart rate monitor and like to like to classify between resting, stress and recovering. If your datapoint says your heart rate is at 130, it depends on whether you are recovering from high loads or something else.

Edit:
I forgotten your second question.

  It seems to me that recurrent just means doing a convolution, adding the result to the original input, and convolving again. This gets repeated for x number of recurrent steps. What advantage does this process actually give?

I could think off a few possible answers. By convoluting the recurrent part you kind of filtering it. So you get a cleaner signal and errors won't stack as much. Vanilla rnn suffer from exploding vanishing gradients, so this could be his approach to overcome it.
Furthermore, you are embedding your features within the rcnn, which can lead, as he stated, to more paths to exploit. Which makes it less prone to overfitting, thus more generalizable.


Differences between Test & Score results and calculated by Confusion Matrix (accuracy, sensitivity and specificity)

Test on train data uses the whole data set for training and then for testing. This method practically always gives wrong results.

So, yes, in this case, the model is learned and tested on the same data set. In most cases, the model learned this way will be overfitted and will perform poorly on unseen data (e.g., your testing dataset).

  Test on test data: the above methods use the data from Data signal only. To input another data set with testing examples (for instance from another file or some data selected in another widget), we select Separate Test Data signal in the communication channel and select Test on test data.

So, the answer to the second question is also "yes:" The model is trained on the data labeled Training in you diagram and then tested on the Testing data. The "quality" of the data and the way it was split in the training and testing set will influence the performance of the model.

Orange documentation on "Test & Score":


Is it possible to use NEAT networks for solving video games?

Implementing the NEAT algorithm for playing video games is quite realistic, especially considering that it is not novel, so there are extensive resources and tutorials to help you.  In fact, there are publications that will walk you through exactly what you want to do, such as this paper on using the NEAT algorithm for playing Pong.

Of course designing a self-driving car would be significantly more advanced than playing the 2D games you mentioned, but it appears that much of the work for developing AI in GTA V exists.

The person you mentioned, Sentdex, has a github page for developing GTA V code.  However, note that:

  We read frames directly from the desktop, rather than working with the game's code itself. This means it works with more games than just GTA V, and it will basically learn (well, attempt to learn...) whatever you put in front of it based on the frames as input and key presses as output.

This is a very primitive approach.

I am by no means an expert on self-driving cars, but after some Googling, deep drive appears to be the premier simulated self-driving car development platform.  If you are going to develop the GTA network, I suggest checking out it out.

As for comparing to an implementation of AlexNet:

Again, I am not an expert on self-driving cars, but I would expect an ensemble of networks to be used for this type of task, so I imagine an object detection network or semantic segmentation network to be used in conjunction with a network designed with the NEAT algorithm.

For instance, you can use a fully convolutional AlexNet model to detect objects such as pedestrians/cars/sidewalks/etc and feed that data into another network that also takes other inputs such as your current speed, wheel angle, road conditions etc that combines all this information and translates it into actions such as applying the brakes, turning etc.

AlexNet is a Convolutional Neural Network (CNN), which is primarily used in the field of vision processing.  The NEAT algorithm typically generates Multilayer Perceptrons (MLP), which are not used individually for machine vision challenges; they are different tools for different tasks.  It would be like asking, "which is better: a hammer or a screwdriver".  The answer depends on the task.

TL;DR:

Yes, using NEAT for video game AI development is a good introductory project for some of the more advanced topics in machine learning and deep learning.


What are some options for an offline chatbot on Android?

EasyNLU project might interest you: https://github.com/kolloldas/EasyNLU

  EasyNLU is a Natural Language Understanding (NLU) library written in Java for mobile apps. Being grammar based, it is a good fit for domains that are narrow but require tight control.


how can we feed both data time series and non time series data together in machine learning classification model

You need two backbones: for time series, use LSTM/GRU and for non-time series, use 1D CNN or Linear layers. Once you get the final embeddings from both of them, concatenate the embeddings and finally feed to the classification layer.


It is typically called a class imbalance issue, where the occurrence of a label happens so infrequently that makes predictions unreliable.

For instance, if I know Vancouver, Canada rains 85% of the time in winter, I would simply predict that it is raining when it is winter + vancouver. You don't want your algorithm to favour one label over another because one label predominates.

One common strategy would be resampling. If you have enough data, downsampling could make more sense as (oversampling requires the creation of synthetic data (e.g., SMOTE, etc.)). so that the algorithm can properly learn the difference between the two classes and wouldn't favour one over another. 50-50 split between the two classes would probably be a good starting point, but it also depends on what is available. You still want your negative labels to be representative, and even an 80-20 split would be a vast improvement already.

Another common solution is to increase the penalty of incorrect predictions. The way to think about it, false positive = admin spending time investigating a false alarm; false negative = rouge activities went undetected. For different business, one cost could be more severe than another so you could potentially say getting a false negative is 1000x more costly to business, etc.

Not knowing your data, those are probably the first two things (separately or together) to try. Most ML packages could handle either strategy rather easily so that's why I think those are reasonable things to try.

There are many thorough articles and tutorials with additional strategies available. Try the keywords anomaly detection and class imbalance and it seems to give me some pretty good results.


How do I use a model after it's fitted to predict the class of a single string?

[[0.41100174 0.5889983 ]] what this means is the probability of class 0 is 0.411 and probability of class 1 is 0.588. Since probability of class 1 is greater than probability of class 0, it belongs to class 1.

a = [[0.41100174 0.5889983 ]]

np.argmax(a)

Output : 1

np.argmax will get you the class.


Rationale behind most published works in medical imaging trying to reduce false positives

TL;DR: diseases are rare, so the absolute number of false positives is a lot more than that of false negatives.

Let's assume that our system has the same false positive and false negative rate of 1% (pretty good!), and that we're detecting the presence of new cancers this year: 439.2 / 100,000 people, or 0.5% of the population. [source]

No cancer, no detection: 99.5% x 99% = 98.5% (98.505%)
No cancer, detection: 99.5% x 1% = 1.0% (0.995%)
Cancer, detection: 0.5% x 99% = 0.5% (0.495%)
Cancer, no detection: 0.5% x 1% = 0.005%

So we can see that we have a problem: for everyone who has cancer, two people who didn't have cancer wind up with invasive surgery, chemotherapy or radiotherapy.

For every person who fails to have a present cancer detected, two hundred people receive actively harmful treatment they didn't need and can't really afford.


Using BERT for co-reference resolving, what's the loss function?

NER is approached as a sequence-labeling problem and the end of the network, there is a categorical distribution estimated by softmax that is trained using cross-entropy loss.
The paper you are specifically asking about claims to be based on the End-to-end Neural Coreference Resolution paper that does something more tricky. They explicitly consider all spans to the entities spans and compute the probability for that. Nevertheless, when they get the probabilities they still do in principle the cross-entropy loss (cf. their code).


Firstly, I am assuming that you are using Anaconda Navigator, and are trying to use Orange Canvas (the GUI method)

If the package has been installed correctly (from within Anaconda, using the Environments tab) you will see the following when opening Anaconda Navigator:

So, there is not a Windows desktop or start-menu icon; instead launch Orange from Anaconda Navigator.

If you have instead only installed the biolabas Orange python library, 'Orange3', you will only be able to access the functions via a python programming shell by calling:

  import Orange3


How to use fminunc in Matlab on only some specified parameters of a function

What we have here is a function fun(X, y, X0) and fminunc(fun,X0). You know X, and y, lets call them X* and y*, so we could replace fun(...) by fun*(X0)

% function to be minized
fun = @(X, y, X0) (dot(X,y) + X0)^2
X_star = [1,2,3]
y_star = [4,5,6]
$ same function but with X and y filled in
fun_star = @(X0) fun(X_star, y_star, X0)
X0_star = 0
fminunc(fun_star, X0)


Creating the optimal set of utterances to train a natural language processing engine

Great question, I've been checking back to see if anyone responds but nothing yet, so I'll do my best.

It seems like me, you have also found out that simply adding more and more training data does not increase the model’s accuracy.

You need to make sure your model contains balanced, representative samples for each intent. Simply picking utterances at random means they may not be represent all of the types of language a user may use.

For example, let’s say you had two intents - “buy_food” for buying some food,  and “list_menu”  for finding out what is available for order. If you were to heavily populate buy_food with lots of utterances that begin with “I want to”, the NLP engine will associate the phrase “I want to” with buying food, even though someone may quite rightly say “I want to see what’s on the menu”. To overcome this, make sure you balance out each intent’s utterances and don’t focus on phrases that aren’t specific to the intent. In this case, you could counterbalance by having some utterances in your “list_menu” intent such as “I want to see what’s on the menu”.

As a rule, for each intent I would try and include up to 5 or 6 ways to say something, and then for each of these, include 4 or 5 variations of this, leaving you with about 25 samples for an intent in total.

There is also the question around how you plan your intents. For example, do you have a “buy_food” intent and a “buy_drinks” intent, or should you simply have a “buy” intent, and then have the thing they want to buy as an entity? (think of your intent as a method, and your entity as a parameter) My suggestion would be to keep your intents broader and have the specifics handled by entities. This will help avoid confusion between intents.

I recommend checking out a tool called QBOX. It connects to LUIS/Watson/DialogFlow and lets you score and visualise intents so you can see any possible confusion between them. This should help you to plan your intent structure and visualise the confusion between your intents, and know if you're causing regressions as you expand your model.
(Disclaimer: this is a product I am working on, and built precisely because of the type of problem you're having, so I've not linked to it, but if you Google the name along with the term 'chatbot', you should find it!)


Is it possible to make a label automatically in supervised learning(Machine Learning)?

The problem you ask is probably Semi-supervised learning.

In a Semi-supervised learning problem, you have a small amount of labeled data with a large amount of unlabeled data. The goal of this problem is to build a model that has higher accuracy than using labeled data alone.

On the other hand, if you have no labeled data at the beginning, then you might have to use some automatic system (e.x: sensors) to determine the label of your data. In this paper about robotic grasping, Sergey Levine automatically detects an attempt of grasp as success or failure based on the openness of the grippers (aka fingers) of the robotic arm and the difference of images before and after the attempt. He called this type of problem Self-supervised learning.

Hope this help.


Calculate this by finding the threshold at which the false positive rate is 1%. Your model outputs a probability, not a category. You get a category by seeing on which side of some threshold, probably 0.5, the probability lies. Test some other thresholds. You can use the ROC curve for this.
Once you have that threshold, calculate the accuracy.
However, be cautious about accuracy (or any threshold-based scoring rule, such as F1 score) as a scoring rule.
https://stats.stackexchange.com/questions/312780/why-is-accuracy-not-the-best-measure-for-assessing-classification-models/312787#312787
https://stats.stackexchange.com/questions/464636/proper-scoring-rule-when-there-is-a-decision-to-make-e-g-spam-vs-ham-email
https://stats.stackexchange.com/questions/494023/proper-scoring-rule-in-optical-character-recognition
(Frank Harrell’s blog that gets linked is a good read, too.)


How can I plot a line for time series data with categorical intervals in R

Would something like this work? I simply add an extra column that indicates the row number (which is later used as the x-axis) to make sure all values are displayed as a new point instead of plotting on top of each other for the same day. I then specifiy the custom x ticks and labels by selecting the first row for each day and get the row number (which specifies where the ticks and labels have to be drawn) and the day name (which specifies what the labels should display).
library(readr)
library(ggplot2)

df <- read_csv("testData.csv") %>%
  mutate(
    date = as.Date(date, "%d-%m-%Y"),
    day = weekdays(date),
    row = row_number()
)
ticks <- df %>% group_by(day) %>% filter(row_number() == 1) %>% select(row)

ggplot() +
  geom_line(data = df, aes(x = row, y = value, group = 1), color = "red") +
  ggtitle("Some Measure over Time") +
  xlab("Time") +
  ylab("Some Measure in %") +
  scale_x_continuous(breaks=pull(ticks, row), labels=pull(ticks, day))


In Sutton & Barto, vectors are considered column vectors by default. So if you have this kind of product:
$$\mathbf{a}\mathbf{b}^T$$
where $\mathbf{a}$ and $\mathbf{b}$ are $d$ dimensional vectors, it does not calculate the scalar product. Instead it treats both vectors as matrices and calculates a matrix product, which will be a $d \times d$ matrix because you are multiplying a $d \times 1$ matrix by a $1 \times d$ matrix.
Worthing noting that the scalar product can also be calculated as a $1 \times 1$ matrix if follow the same matrix multiplication rules but with the first vector transposed instead:
$$\mathbf{a}^T\mathbf{b}$$
which leads to multiplying a $1 \times d$ matrix by a $d \times 1$ matrix. This is why the value function approximation can be written as $\mathbf{w}^T\mathbf{x}_t$ (there is a small liberty taken of assuming a $1 \times 1$ matrix is the same as a scalar value in terms of notation).


Algorithm for deriving multiple clusters

The fact is that you could use any of the algorithms you mentioned, and in general any algorithm that requires to set the number of clusters as a parameter (or any other parameter that indirectly sets the final number of clusters, like the threshold in a hierarchical clustering algorithm.)

The solution to your problem is model selection. Model selection methods evaluate different clustering solutions, and select the one that optimizes a given criterion.

For instance, in the case of K-means, you could find a solution for a range of k values, and keep the one that maximizes any cluster validation measure (see the Wikipedia entry for cluster analysis to read about some examples of cluster validation measures).

There are automatic and more complex approaches (one specific example is "Automatic Cluster Number Selection Using a Split and Merge K-Means Approach" by Muhr, M. and Granitzer, M., but this is just an example). These methods use cluster validation measures to automatically split or merge clusters, but the idea is basically the same.


How to render environment in Tensorforce?

In case you use
https://github.com/tensorforce/tensorforce/blob/master/examples/act_observe_interface.py ,
the following modification works.
Import gym:
import gym
from gym import wrappers

then define a Tensorforce environment with a Gym environment:
    env = gym.make('CartPole-v1')
    env = wrappers.Monitor(env, 'tmp', force=True)

    environment = Environment.create(environment=env, max_episode_timesteps=500)
    # environment = Environment.create(environment='/Users/rondelion/git/tensorforce/benchmarks/configs/cartpole.json')

and render the Gym environment:
            ....
            num_updates += agent.observe(terminal=terminal, reward=reward)
            env.render()
            sum_rewards += reward
            ....


How to enable GPU on GradientBoostingClassifier?

You are looking for this.
 Gradient Boosting, Decision Trees and XGBoost with CUDA


NLP Emotion Detection - Model fails to learn to recognize negations

In general this is a difficult problem, it's about the problem of Natural Language Understanding which is far from being solved.
The advanced option requires a full syntactic parsing of the sentence, ideally followed by some kind of semantic representation of the sentence, for example by extracting relations. As far as I know this is rarely used because these steps will often cause as many errors as they solve.
Some more reasonable heuristics can be considered, for instance detecting specific negation words and either including this information as feature or modifying the original features accordingly (e.g. when a negation is detected replace the token "happy" with "not(happy)" in the features).
Note that it's unlikely to be perfect anyway, due to the usual obstacles: hedging ("I would assume that it was quite good"), sarcasm ("Sure, I was very happy with the terrible service"), metaphors ("I was feeling like a fish in water"), etc.


Ideal strategy for multi variable regression attempting to maximize the target

Interesting question. As you know, indeed, it can be seen as a pure optimization problem. For that you need to come up with mathematical modeling of dependent variable based on independent ones. I might have an easier ML-based suggestion:

Create a new target out of all 3 variabales in a way you want them to behave. For instance, you said bid and cost low but sales high, so you might want to turn them simply to:

$y_{new} = \frac{bid+cost}{sales + \epsilon}$

$\epsilon$ is simply a factor to avoid dividing by zero. Please note that there seem to be a strong $TotalCost = CostPerSale\times Sales$ relation. Have it in mind in case you observe bad/strange results.

As seen in $y_{new}$, smaller it is, more interesting it is to you. You apply a regression to the new data (it will be a uni-response regression now as we combined all targets) and if prediction is less than a threshold (defined by you through experiments) you consider it as good. Or you can just report the predictions and the speak for themself; smaller the better.

PS1: That is the simplest formulation i came up with. Indeed, there is an optimization problem to be solved to find a proper formula. Here we assumed the summ of bid and cost is the minimum. One may say their product should be in the enumerator for example. You need to design that formula.
PS2 Strongly recommend to inspect your variables carefully, have an insight about probable correlations between them, remove non-informative features if any, etc.

Good Luck!


What quantile is used for the initial DummyRegressor for Gradient Boosting Regressor in scikit-learn?

Yes, a GBM with Huber loss initializes with the median.  The relevant bit of code is the method init_estimator of the loss class, in the file _gb_losses.py.  For HuberLossFunction:
def init_estimator(self):
    return DummyRegressor(strategy='quantile', quantile=.5)

(source)


How duplicated items can be deleted from dataframe in pandas

The best way would be to use drop_duplicates(). If you have a larger DataFrame and only want those two columns checked, set subset equal to the combined columns you want checked.

df = df.drop_duplicates()

or

df = df.drop_duplicates(subset=['userid', 'itemid'])


ANN Classifier for extracted discrete image features

If you have already extracted a feature vector $X$ from your images. Then indeed you can use an artificial neural network (ANN) to classify these given you have labelled instances.
You can do this in python using multiple different libraries such as scikit-learn, Keras, or tensorflow quite easily. Scikit-learn is the easiest to implement but the least flexible. Keras allows for more complex ANNs, and tensorflow allows for every kind of custom structure you can imagine.
Scikit-learn
This is the easiest way to implement a single-layer ANN.
from sklearn.neural_network import MLPClassifier
X = [[0., 0.], [1., 1.]]
y = [0, 1]
clf = MLPClassifier(solver='adam', alpha=1e-5,
                    hidden_layer_sizes=(32,), random_state=1)
clf.fit(X, y)
clf.score(X, y)

https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier
Keras
First we build a model we want to use. This is an example of a simple single-layered neural network. $k$ is the number of features per instance and $num_classes$ is the number of output classes you are expecting.
from __future__ import print_function
import keras
from keras.models import Sequential
from keras.layers import Dense, Flatten
from keras.models import model_from_json
from keras import backend as K

input_shape  = (k,)

model = Sequential()
model.add(Dense(32, activation='tanh',
             input_shape=input_shape))
model.add(Dense(num_classes, activation='linear'))

model.compile(loss=keras.losses.mean_squared_error,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

You can see the summary of the model by using
model.summary()

Then we can train this model using
batch_size = 128
epochs = 10
model.fit(x_train, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test_binary))

You will need to tune all the hyper-parameters in this code sample to better suit your data.
https://keras.io/api/models/sequential/


When do we say that the dataset is not classifiable?

Take a sample element from one class and a sample element from the other class. Is it possible for these two elements to have the exact same feature vector? If that can ever happen then the two classes are not completely seperable using your current feature vectors (since the classification decision is based entirely in the feature vector for a given element).

On the other hand, if *every" element in one class has a corresponding element in the other class such that the two elements have the same feature vectors, then the two classes are indistinguishable using your current feature vectors.

Furthermore, if that condition holds for only some of your elements and not others, then you are somewhere in between, and you can use that as a basis to measure how well you can hope a classifier to perform using your current feature set.

All of these evaluations can be used to argue to varying degrees that you need to extract more features.


How would you describe the trade-off between model interpretability and model prediction power in layman's terms?

Interesting question. I think that you can illustrate this by thinking about different use cases. The one example I've heard that I like is around lending decisions for loan applications. That's an algorithm but, because of regulations, it can't be strictly "black box". The decision has to be, effectively, interpretable because the bank has to give you a reason for decline on the loan. So, there's certainly better algos out there for loans that can give a binary result, but do you want a bank to just tell you yes or no?


Data Preprocessing, how separate background from image to detect animals?

You need to do some Background Subtraction on the images. If you have the Background image without the animal, you can simply subtract it from the current image to get just the animal.
Once you have just the animal, you can apply SIFT or CNNs or whatever.
This is called frame differencing.

If you don't have the background image, you can try methods like this provided by opencv

Basically what you are looking for is background subtraction/foreground detection.

Hope this helps.

image source: http://docs.opencv.org/3.3.0/d1/dc5/tutorial_background_subtraction.html