SPC vs Autoencoders in anomaly detection

Control charts are widely used still (whether people know it or not). However it is limited to cases where it is possible to define meaningful and stable targets and bounds for a metric. The classical example is in manufacturing, when the intent is to produce items which has a certain property at a certain value. Like each cereal box produced should be 510 grams, 500 grams of net contents and 10 grams for the box. Being outside for example +-5% of this would be considered an anomaly.

However when there are no such clear metric, or targets, more advanced tools can be useful. For example consider you run a web service which serves webpages to users. Different time of day have different usage patterns. Different users make different kinds of request. Different kinds of requests have different impact on the system resources and thus the ability for your system to serve other (unrelated) requests. System performance will vary quite a lot, but a lot of it due to factors outside your control. And often it is not individual metrics which are the clue to "something" is wrong, but combinations of them.
This makes it hard define single-value metrics, and to set control-chart targets and limits. But an autoencoder can learn time-based patterns, the user-dependent patterns, request-type dependent patterns, and their relationships. And give an overall score for how anomalous the overall situation is.


Punctuation such as , . () ? ! ... ? are all included in pretrained word vectors such as Glove. However, it is common to find phrases, abbreviations, and misspellings that are not contained within pretrained vectors. What to do with those phrases is highly dependent on the goals of the analysis and context of the text.


Adaptive regression parameter estimation in R

Here is some notes for an R package 'earth' for adaptive regression. This may be useful.

http://cran.r-project.org/web/packages/earth/vignettes/earth-notes.pdf


The dimensions of the low dimensional space have no meaning. Note that the t-SNE loss function is solely based on the distances between points ($y_i$ and $y_j$) and probability distributions over those distances ($p_{ij}$ and $q_{ij}$):

$$
\frac{\delta C}{\delta y_i}=4 \sum_j(p_{ij} - q_{ij})(y_i-y_j)(1+||y_i -y_j||^2)^{-1}
$$

Thus there is no projection from the whole high-dimensional space to the low-dimensional space, t-SNE only finds a mapping from a specific set of high-dimensional points to a specific set of low dimensional points. Because there is no function from one space to the other there is also no inherent meaning of the axes.

Things you can imagine to illustrate this:

Rotating or translating the high-dimensional or low-dimensional space does not influences distances between the points. Hence, t-SNE does not care about rotation or translation in both spaces. Thus there is not absolute interpretation of the axes.
The t-Student distribution has fat tails. This causes the low-dimensional representation to be invariant to changes in points that are far away in the high-dimensional space. This also causes that points that are far away in the high-dimensional space can be either reasonably far away, far away or really far away in the low dimensional space. In this sense it stretches certain parts of the low-dimensional axes (in any arbitrary direction).

That being said, t-SNE is primarily a visualization technique and its dimension reduction effectiveness for other purpose is not obvious (probably not suitable for clustering, feature extraction or feature selection).

Also: the paper.


Impulse Response Function - Negative Shocks on R

Here is a simple example that should work:

library(vars)
data("Canada")
var.2c=VAR(Canada,p=2,type="const")
# 1 SD impulse reponse function
irf.rw.e=irf(var.2c,impulse="rw",response=c("e"))
# gamma is the number of standard deviations for the irf
gamma=-0.25
irf.rw.e_gamma = irf.rw.e
n=length(irf.rw.e_gamma$irf$rw)
for(i in 1:n){
    irf.rw.e_gamma$irf$rw[i]   = irf.rw.e_gamma$irf$rw[i]*gamma
    irf.rw.e_gamma$Lower$rw[i] = irf.rw.e_gamma$Lower$rw[i]*gamma
    irf.rw.e_gamma$Upper$rw[i] = irf.rw.e_gamma$Upper$rw[i]*gamma
}
plot(irf.rw.e)
plot(irf.rw.e_gamma)

Source


LSTM multivariate time series prediction is performancing very poor

Several things you can do here:

try normalizing the time series
try using another loss: loss='mse'
maybe do a 80-20 split (make sure that your test data comes chronologically after the training data to avoid forward looking bias)


Try using a different seed for each loop. You can do it like this.
my_seeds <- c(1:10) # These are 10 seeds, 1, 2, 3...10. Change to whatever.
for (i in 1:10){
          set.seed(my_seeds[i])
          ##########################
          ##########################
          boot.sample = sample(n, 1000, replace = TRUE)
          bootSample = dataset[boot.sample, ]
          bootSample

          split = sample.split(bootSample$blueWins, SplitRatio= 0.80)
          training = subset(bootSample, split == TRUE,  replace=TRUE)
          test = subset(bootSample, split == FALSE,  replace=TRUE)
          print(training)
    }


What is batch size is required for training 30000 images in keras cnn?

Changing the batch size might change things, but I would doubt an increase in acciracy of 16% by change to batch size alone, given you have 30,000 images.

I would recommend playing with the learning rate in conjunction with batch size or at least allowing your model to train for more epochs. You can use the following points to guide your trials.

Changing the batch size to something smaller will generally:

Make the learning curve more volatile, because each loss update is the average of a smaller set of samples, meaning the variation between batches will likely increase
Make learning faster because the model will get bigger (but maybe more biased) updates for weights in each step
Decrease overall model accuracy - assuming the model with larger batches were to be allowed to train for much longer, given the first two points above

There has been a lot of research published on the effects of batch size, so maybe have a quick look through some papers (like this one) for the general outcomes.

Also have a look at the answer to this question about the effects of batch size on DNNs.


How do we decide between XGBoost, RandomForest and Decision tree?

Decision Tree is very useful if you want to be able to explain where your result comes from you can often print the tree and see how your model came to this answer.
Random Forest can also provide such information, but you'll have to browse all trees and make some "stats" into them, which is not as easy. But Random Forest often give better results than Decision Tree (except on easy and small datasets).
Finally, XGBoost could give a better result than Random Forest, if well-tuned, but you can't explain it easily. If you don't mind about results-explanation, I'd suggest you to try both XGBoost and RandomForest, with a bit on tuning, to see which one is best fitting on your dataset.


"Results do not have equal lengths" using ldply in R package plyr

Here's a slightly modified version using some newer "tidyverse" packages:

library(rvest)
library(purrr) # flatten/map/safely
library(dplyr) # progress bar

# just in case there isn't a valid page
safe_read <- safely(read_html)

fetch_current_players <- function(letter){

  URL <- sprintf("http://www.baseball-reference.com/players/%s/", letter)
  pg <- safe_read(URL)

  if (is.null(pg$result)) return(NULL)

  player_data <- html_nodes(pg$result, "b a")

  html_text(player_data)

}

pb <- progress_estimated(length(letters))
player_list <- flatten_chr(map(letters, function(x) {
  pb$tick()$print()
  fetch_current_players(x)
}))


nnet in caret. Bootstrapping or cross-validation?

There is a mistake in your train function.  It should be trControl
not trainControl.

By default the train function doesn't do any pre-processing
(preProcess = NULL)

Have trace = FALSE in the train function to stop printing the
trace to the screen.

You can find available model for caret at https://topepo.github.io/caret/available-models.html

Looks like nnet uses sigmoidal activation functions for the hidden layer; see here https://stats.stackexchange.com/questions/78252/whats-the-activation-function-used-in-the-nodes-of-hidden-layer-from-nnet-libra

Additionally, you might want to set linout parameter TRUE for regression and FALSE for classification.


Extracting list of locations from text using R

I use the function grepl() for this.

text<-c("China","Japan","perspective","United Kingdom","formatting","clear",
"India","Sudan","United States of America","Bagel","Mongolian")

#Make a logical vector
arg<-grepl("China",text)|grepl("Japan",text)|grepl("UnitedKingdom",text)|
grepl("Mongolian",text)

#Pick out from text
text[arg]


IterativeImputer - Returning -0 and other wierd results

Despite the fact that I'm not sure why it's that, it's seems that's normal. Please see the example from sklearn documentation: https://scikit-learn.org/stable/modules/impute.html#multivariate-feature-imputation
import numpy as np
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer
imp = IterativeImputer(max_iter=10, random_state=0)
imp.fit([[1, 2], [3, 6], [4, 8], [np.nan, 3], [7, np.nan]])

X_test = [[np.nan, 2], [6, np.nan], [np.nan, 6]]
# the model learns that the second feature is double the first
print(np.round(imp.transform(X_test)))

They use np.round at the end to get results:
array([[ 1.,  2.],
       [ 6., 12.],
       [ 3.,  6.]])

Running this code without rounding gives results similar to yours:
array([[ 1.00007297,  2.        ],
       [ 6.        , 12.00002754],
       [ 2.99996145,  6.        ]])

Try to add np.round before casting to pd.DataFrame to see if it helps.


Machine learning solution approach to match loan repayments

I don't think machine learning is the right tool.  It's difficult (at least for me) to formulate this as a learning problem.

You could consider genetic algorithms as an alternative approach.  Based on your description of how you do the matching by hand, it seems like the following assumptions hold:

You're looking for an assignment of payments to loans such that the difference between (principle + interest) and $\sum debits$ is minimized
You favor solutions where the time between loan and repayment is minimized.

If the above is true, you may want to consider genetic algorithms as an approach.   For GAs, you need to specify a representation, an objective function, and mutation/crossover operations.  I think the following would work:

Individuals in the population (i.e. candidate solutions) are represented as vectors of length $N$, where $N$ is the total number of debits.  Each entry in this vector assigns the debit to a credit.  For example, the individual $[1, 1, 2, 3, 2, 3]$ means that the first two debits are associated with the first credit, the third debit is associated with the second credit, the fourth debit is associated with the third credit, and so on.

The objective function that you wish to minimize is described by the two bullet points above.  I think it should be pretty easy to programmatically compute this function.  For each credit, compute the magnitude of the difference between (principle + interest) and $\sum debits$.  Sum these differences, then add a penalty for temporal separation between credit and repayment.  One easy-ish way to implement the penalty is to add the edit distance between a solution $V$ and its sorted counterpart $sort(V)$.  You'll want to play with the weight of the time penalty such that some delay between credit and debits is allowed, but you still favor quick repayments.

You'll need to pick suitable mutation and crossover operations.  For this representation, the traditional two-point crossover would likely work well, and uniform mutation would probably work fine.

As an aside, I sure hope this isn't a real-world approach to loan repayment matching.  You will wind up with some angry customers when a debit is inevitably assigned to the incorrect credit.


How to visualize trending over geographical area?

Overly simple answer: "animate" this in PowerPoint by having one slide per time slice then advancing the slides quickly with the keyboard or set them to auto-advance.

If you have a little more flexibility outside PPT, there are free tools (e.g. Microsoft Power BI) where you could do a shaded map and have a time slider to make it more interactive.


pandas apply function with multiple condition?

Alternatively, you can use loc:

import pandas as pd

df = pd.DataFrame({"age": [-100, 300, 400, 500, 600, 700]})
df["age"].loc[(df["age"] < 500) & (df["age"] >= 0)] = 0

Now your df looks like this:

    age
0   -100
1   0
2   0
3   500
4   600
5   700


Identifying trend and seasonality of time series data

Another idea could be Fourier Transformation, which takes a time serie as an input (time domain), and converts it into frequency domain.

Consider this example:

When you transform the time series data from time domain into frequency domain, you can observe the repeated patterns (=seasonality). In this case, there is peak at 12 (day/night rhytm), at 24 (day) or 168 (week).

I can imagine you can perform the FFT and then extract the peaks based on certain threshold

The Git repo also contains a nice introduction into FFT.

Source: https://github.com/balzer82/FFT-Python


Right ML mode and metric to minimize FN and FP on imbalanced dataset

If you use class weights, I don't think it would make much of a difference which model you use (regarding the imbalance). If you are familiar with python, I would suggest this sklearn function that could help you out in computing the class weights. However, I have found that oversampling (or undersampling) often works better than class weights. Consider using SMOTE for this.
While there are metrics and loss functions better suited at handling class imbalance (GMS, quadratic kappa, etc.) I don't think you would have an issue with using any loss
if you considered oversampling your data.


Predicting positive/negative experience with very few labels and labels from only one class

This is not a data science solution, but wouldn't the fact that the user watched the video in the first place (took the time to find and locate the video then started the video) imply that the user have some interest in that particular video over users that never interacted with the video? I would consider the fact that the entry exists as a positive response, the lack of a "like" as a neutral response, and (maybe?) the lack of watching it as a negative response. It, of course, depends heavily on the source and assumptions of your dataset.

Nonetheless, I agree that clustering the videos with likes is a good starting point in this situation, and that would definitely be my first gut instinct thing to do.


Oversampling only balances the training set, what about the testing set?

Use a stratified split (as pointed out by louic in the comments). It will distribute your classes evenly across all folds. It can be done using sklearn's StratifiedKFold.


mAP scores on tensorboard (Tensorflow Object Detection API) are all 0 even though the loss value is low

I recommend several checks to make sure you get reasonable mAP@IoU scores for object detection API:

Try varying the Intersection over Union (IoU) threshold, e.g 0.2-0.5 and see if you get an increase in average precision. You would have to modify matching_iou_threshold parameter in object_detection/utils/object_detection_evaluation.py
Try different evaluator classes (the default one is EVAL_DEFAULT_METRIC = 'pascal_voc_detection_metrics'). If you are training on Open Image Dataset it makes sense to use open_images_V2_detection_metrics
Check your eval config file and increase the number of examples used in the evaluation set, e.g.

eval_config: {
  num_examples: 20000
  num_visualizations: 16
  min_score_threshold: 0.2
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  max_evals: 1
}
Train the object detector for more iterations
Check current mAP against reported metrics (e.g. COCO mAP@IoU=0.5):


Training with a subset of data: relationship between subset size and training metric?

Do I understand correctly from your test RMSE, that the error is lower as you increase the size of your fraction used in training?

If you do 5-fold cross validation using a fraction of 0.01, that implies here that you are only a total fraction of only 0.05 of your data? So 0.04 for training and 0.01 for testing.

It should not come as a surprise then, that the performance improves with increased amounts of data used for training. Your RMSE values increase with fraction size - I would normally expect such a correlation (assuming your data is fairly coherent).

EDIT:

To answer your questions,

I would argue it is a well-known relationship, even in the simplest of regression problems. Have a read here (including comments on accepted answer) for more discussion on minimum required data points for valid models. One reason highlighted is that very few data points makes any estimate of variance less meaningful
I don't think there is a name given to this relationship - at least I don't know of one (and neither do people in the link above). I would just describe it as being inherent to problem that the dataset is trying to represent. For example, if we have a high-dimensional complex parameter space, and then only a handful of data-points sampled from that space, it is very unlikely that we can fully describe the original space. The more samples we have, the closer we can approximate it. This is extremely evident in data-intensive modelling techniques, such as image-based tasks (i.e. convolutional neural networks, with millions of parameters.) Other methodologies, such as Bayesian optimisation using Gaussian processes. These are computationally more intense, but perhaps offer higher data efficiency.
There might be some convergence theories, stating how approximations to ground truth reach a threshold given a certain amount of data (that kind of framework would make sense to me) - but alas, I have not seen any formal analysis in this direction. Regarding number of samples, in the meantime, I just follow the mantra: the more the merrier!


Learning rate in the Perceptron Proof and Convergence

(My answer is with regard to the well known variant of the single-layered perceptron, very similar to the first version described in wikipedia, except that for convenience, here the classes are $1$ and $-1$.)

Let's denote:

$d$ is the dimension of a feature vector, including the dummy component for the bias (which is the constant $1$).
$x^r\in\mathbb R^d$ and $y^r\in\{-1,1\}$ are the feature vector (including the dummy component) and class of the $r$ example in the training set, respectively.
$\eta _1,\eta _2>0$ are training steps, and let there be two perceptrons, each trained with one of these training steps, while the iteration over the examples in the training of both is in the same order.
$w_0\in\mathbb R^d$ is the initial weights vector (including a bias) in each training.
for $i\in\{1,2\}$: let $w_k^i\in\mathbb R^d$ be the weights vector after $k$ mistakes by the perceptron trained with training step $\eta _i$.
for $i\in\{1,2\}$: with regard to the $k$-th mistake by the perceptron trained with training step $\eta _i$, let $j_k^i$ be the number of the example that was misclassified.

If $w_0=\bar 0$, then we can prove by induction that for every mistake number $k$, it holds that $j_k^1=j_k^2$ and also $w_k^1=\frac{\eta_1}{\eta_2}w_k^2$:

base ($k=1$):
$w_0^1=w_0=w_0^2$, so both perceptrons would predict the same class for each example until and including the first mistake, so $j_1^1=j_1^2$.
Now: $$w_1^2=w_0+\eta_2y^{j_1^2}x^{j_1^2}=\eta_2y^{j_1^2}x^{j_1^2}=\eta_2y^{j_1^1}x^{j_1^1}$$ and thus:$$w_1^1=w_0+\eta_1y^{j_1^1}x^{j_1^1}=\eta_1y^{j_1^1}x^{j_1^1}=\frac{\eta_1}{\eta_2}\eta_2y^{j_1^1}x^{j_1^1}=\frac{\eta_1}{\eta_2}w_1^2$$
step (let $k$ be a mistake number such that the claim holds for $k$, and for every natural smaller than $k$):
After the $k$-th mistake, perceptron $1$ would predict that an example $r$ is positive iff $(w_k^1)^Tx^r>0$. According to the induction hypothesis, this is true iff $(\frac{\eta_1}{\eta_2}w_k^2)^Tx^r>0$, but we can divide both sides by $\frac{\eta_1}{\eta_2}$, as it is a strictly positive scalar, and thus we get: $$(w_k^1)^Tx^r>0\leftrightarrow (w_k^2)^Tx^r>0$$ i.e. after the $k$-th mistake, both perceptrons would predict the same class for each example. According to the induction hypothesis, $j_k^1=j_k^2$, and so both perceptrons would iterate over the same examples in the same order after the $k$-th mistake. Therefore, $j_{k+1}^1=j_{k+1}^2$.
Now: $$w_{k+1}^2=w_k^2+\eta_2y^{j_{k+1}^2}x^{j_{k+1}^2}=w_k^2+\eta_2y^{j_{k+1}^1}x^{j_{k+1}^1}$$ and thus (according to the induction hypothesis):$$w_{k+1}^1=w_k^1+\eta_1y^{j_{k+1}^1}x^{j_{k+1}^1}=\frac{\eta_1}{\eta_2}w_k^2+\eta_1y^{j_{k+1}^1}x^{j_{k+1}^1}=\frac{\eta_1}{\eta_2}(w_k^2+\eta_2y^{j_{k+1}^1}x^{j_{k+1}^1})=\frac{\eta_1}{\eta_2}w_{k+1}^2$$

We showed that the perceptrons do exactly the same mistakes, so it must be that the amount of mistakes until convergence is the same in both. (You could also deduce from this proof that the hyperplanes defined by $w_k^1$ and $w_k^2$ are equal, for any mistake number $k$.)
Thus, the learning rate doesn't matter in case $w_0=\bar 0$.

In case $w_0\not=\bar 0$, you could prove (in a very similar manner to the proof above) that in case $\frac{w_0^1}{\eta_1}=\frac{w_0^2}{\eta_2}$, both perceptrons would do exactly the same mistakes (assuming that $\eta _1,\eta _2>0$, and the iteration over the examples in the training of both is in the same order).
Thus, for any $w_0^1\in\mathbb R^d$ and $\eta_1>0$, you could instead use $w_0^2=\frac{w_0^1}{\eta_1}$ and $\eta_2=1$, and the learning would be the same.
In other words, even in case $w_0\not=\bar 0$, the learning rate doesn't matter, except for the fact that it determines where in $\mathbb R^d$ the perceptron starts looking for an appropriate $w$.

Finally, I wrote a perceptron for $d=3$ with an animation that shows the hyperplane defined by the current $w$.
I think that visualizing the way it learns from different examples and with different parameters might be illuminating.

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation
import itertools
import time

EXAMPLES1 = np.array([[9, 0], [8, 0]], dtype='float')
EXAMPLES2 = np.array([[5, 0], [6, 0]], dtype='float') # closer to the origin
EXAMPLES_CLASSES1_2 = np.array([1, -1])

EXAMPLES3 = np.array([[20, 26], [16, -11], [21, 2], [18, 25],
                    [26, -19], [11, 6], [20, -10], [17, 17]], dtype='float')
EXAMPLES4 = EXAMPLES3 - 11 * np.ones(EXAMPLES3.shape) # closer to the origin
EXAMPLES5 = np.array([[20, 26], [16, 27], [21, 24], [15, 25],
                    [22, 23], [14, 27], [20, 20], [13, 24]], dtype='float')
EXAMPLES_CLASSES3_4_5 = np.array([1, -1, 1, -1, 1, -1, 1, -1])

W0_0 = np.zeros(3)
W0_1 = np.array([1., 0, -2])
LEARNING_RATE_1 = 0.00156
W0_2 = np.array([1., 0, -2]) / 0.00156
LEARNING_RATE_2 = 1

EXAMPLES = EXAMPLES1
EXAMPLES_CLASSES = EXAMPLES_CLASSES1_2
LEARNING_RATE = LEARNING_RATE_2
W0 = W0_2
INTERVAL = 3e2
INTERVAL = 1
INTERVAL = 3e1

N = len(EXAMPLES)
assert(EXAMPLES.shape[1] == 2)
assert(N == len(EXAMPLES_CLASSES))
assert(LEARNING_RATE > 0)
assert(len(W0) == 3)

fig, ax = plt.subplots()
XS = EXAMPLES[:, 0]
YS = EXAMPLES[:, 1]
max_abs_x = max(max(XS), abs(min(XS)), 1)
max_abs_y = max(max(YS), abs(min(YS)), 1)
bottom_limit = -2 * max_abs_y
top_limit = 2 * max_abs_y
left_limit = -2 * max_abs_x
right_limit = 2 * max_abs_x
plt.xlim(left_limit, right_limit)
plt.ylim(bottom_limit, top_limit)
x_range_size = right_limit - left_limit
y_range_size = top_limit - bottom_limit

# plot the examples
for x, y in zip(EXAMPLES, EXAMPLES_CLASSES):
    if y == 1:
        plt.plot(x[0], x[1], marker='o', markersize=4, color="green")
    else:
        plt.plot(x[0], x[1], marker='o', markersize=4, color="red")

# add a dummy column of ones for the bias.
EXAMPLES = np.hstack((EXAMPLES, np.ones((N, 1))))

hyperplane, = plt.plot([], [], color='blue')
hyperplane.set_visible(False)
k_text = plt.text(ax.get_xlim()[0] + x_range_size / 70,
                  ax.get_ylim()[1] - y_range_size / 25, 'mistakes: 0')
next_example_idx = 0
k = 0
w = W0

def get_hyperplane(w):
    w_x, w_y, bias = w
    if w_x == w_y == 0:
        print(f'No hyperplane is defined by w')
        return None
    if w_y == 0:
        x1 = x2 = -bias / w_x
        y1 = bottom_limit
        y2 = top_limit
        if not left_limit <= x1 <= right_limit:
            print(f'The hyperplane is out of range of our graph')
            return None
        else:
            print(f'The hyperplane defined by w is `x = {x1}`')
    else:
        # find the hyperplane representation as y = m * x + b.
        m = -w_x / w_y # m = -1 / (w_y / w_x)
        b = -bias / w_y
        # b = -bias / np.linalg.norm(np.array([w_x, w_y]))
        print(f'The hyperplane defined by w is `y = {m} * x + {b}`')

        if m == 0:
            y1 = y2 = m * left_limit + b
            x1 = left_limit
            x2 = bottom_limit
        else:
            potential_points_on_limits = {
                (left_limit, m * left_limit + b),
                (right_limit, m * right_limit + b),
                ((bottom_limit - b) / m, bottom_limit),
                ((top_limit - b) / m, top_limit)}
            for x, y in set(potential_points_on_limits):
                if not left_limit <= x <= right_limit:
                    potential_points_on_limits.remove((x, y))
                elif not bottom_limit <= y <= top_limit:
                    potential_points_on_limits.remove((x, y))
            if len(potential_points_on_limits) < 2:
                print(f'The hyperplane is out of range of our graph')
                return None
            x1, y1 = potential_points_on_limits.pop()
            x2, y2 = potential_points_on_limits.pop()

    return (x1, x2), (y1, y2)

def update(frame):
    global k, w, next_example_idx

    for j in itertools.chain(range(next_example_idx, N),
                             range(next_example_idx)):
        x = EXAMPLES[j]
        y = EXAMPLES_CLASSES[j]
        perdicted_y = 1 if np.dot(w, x) > 0 else -1
        if perdicted_y != y:
            print(f'\nMisclassification of example {x}')
            w += LEARNING_RATE * y * x
            k += 1
            k_text.set_text(f'mistakes: {k}')
            print(f'This is mistake number {k}')
            print(f'Update w to {w}')

            hyperplane_xs_and_ys = get_hyperplane(w)
            if hyperplane_xs_and_ys:
                hyperplane.set_data(hyperplane_xs_and_ys)
                hyperplane.set_visible(True)
            else:
                hyperplane.set_visible(False)

            next_example_idx = j + 1
            return hyperplane, k_text

    # The perceptron successfully predicted all of the examples.
    time.sleep(3)
    exit()

ani = FuncAnimation(fig, update, blit=True, repeat=False, interval=INTERVAL)
plt.show()


Why is 10 considered the default value for k-fold cross-validation?

The statement:

  We most often use k=10 because evidence shows it's the best value for k. Smaller values don't give as good estimates, and larger values don't provide much better results either.

Is just categorically false. The reason people default to K=10 is because they don't know how changing K effects their estimates of the generalization error and they (like you) heard somewhere along the line that K=10 was good.

To understand what makes a good value of K (and whether or not K=10 is in fact better than say K=9 or K=11) you need to understand what changing this value has on your estimate.

As K decrease the bias in your estimate increases. This is because with lower values of K you are training on less data. For example K = 2 trains on only half the data, thus you will have a pessimistic bias in your estimate since you've decreased the amount of data available for your model to learn from. K = 3 trains on two thirds of your data, more data available to train on, better performance.

It used to be thought that there was a bias/variance trade-off in that a decrease in K would cause a decrease in variance (to go along with your increased bias) and while this is partially true it does not always hold. Lower values of K will have lower variance due to the fact that your training sets are less correlated. Think of the extreme example where K = N (leave one out). All of the training sets will look extremely similar, meaning that the estimate you receive is highly dependent on the sample you have to train on. If you were to draw many samples from the population and estimate error using leave one out you would have large variance in your estimates because of the variance between samples. This was the original reasoning for believing there existed a bias/variance trade-off with the choice of K.

However, this post outlines that this is not the case and that there is no universal truth as to what happens to the variance as K increases or decreases. Some studies show the variance increases with K, some show it decreases with K.

https://stats.stackexchange.com/questions/61783/bias-and-variance-in-leave-one-out-vs-k-fold-cross-validation/357749#357749

The other thing to consider is computational complexity. If you are dealing with datasets with millions of records it may be infeasible to use a very large value of K, especially if you are doing nested and/or repeated cross validation. So many people make their choice of K based simply on time to compute.


Dealing with unbalanced error rate in confusion matrix

Welcome to the site!

I think Ensemble Method is very tricky. when one of the model doesn't work well then the accuracy of the Ensemble also goes down.

For instance let us consider that you are using RandomForest(RF) and Rpart for classification and RF accuracy is 90% and Rpart accuracy is 60%. If you ensemble these 2 models then the Ensemble accuracy goes down.

Coming to your scenario, you need to be very careful at the time of stacking, you need to select the models that are performing moderately and then stack them to improve the accuracy.

How is the distribution of 0/1's, if they are imbalanced then you need to balance to improve the accuracy of the model. To handle imbalance data situation we use packages like SMOTE,ROSE etc.

Feature Engineering like adding external factors or adding new features, might help you to improve your models accuracy.

Do let me know if you have any additional questions.


Carlification of the MSE loss sum symbol

I think it depends on what you're doing.

If you want to find the MSE of just one training example, then you do number 2. sum all in one example.

If you want to see if it's stable or converging to stability, you could calculate MSE on all examples independently, and then average them to give you a sense of overall behavior


Creating new columns by iterating over rows in pandas dataframe

There's a much more pythonic solution in pandas...

This takes less than a second on 10 Million rows on my laptop:

for x in X11.E.unique():
    X11[x]=(X11.E==x).astype(int)
X11

Here are the details laid out:

Simple small dataframe -

import numpy as np
import pandas as pd

X11 = pd.DataFrame(np.random.randn(6,4), columns=list('ABCD'))
X11['E'] = [25223, 112233,25223,14333,14333,112233]
X11

Binarization method -

for x in X11.E.unique():
    X11[x]=(X11.E==x).astype(int)
X11

Dataframe with 10 Million rows -

pd.set_option("display.max_rows",20)
X12 = pd.DataFrame(np.random.randn(10000000,4), columns=list('ABCD'))
foo = [25223, 112233,25223,14333,14333,112233]
bar=[]
import random
for x in range(10000000):
    bar.append(random.choice(foo))
X12['E'] = bar
X12

Timed binarization (aka one-hot encoding) on 10 million row dataframe -

import time
start = time.clock()

for x in X12.E.unique():
    X12[x]=(X12.E==x).astype(int)
elapsed = (time.clock() - start)

print "This is the time that this took in seconds: ",elapsed

X12

Hope this helps!


How to implement clipping the reward in DQN in keras

Since you're using keras-rl, you could use its class Processor. And simply write a new processor and assign to your agent. The new processor will be something like:

class MyProcessor(Processor):
    def process_reward(self, reward):
        """Processes the reward as obtained from the environment for use in an agent and
        returns it.

        # Arguments
            reward (float): A reward as obtained by the environment

        # Returns
            Reward obtained by the environment processed
        """
        # Change min and max according to your needs. I supposed that your threshold was 1.
        min = -1
        max = 1
        return float(np.clip(reward, min, max))

Note that there isn't a real need to cast the reward as a float and in some agent implementation it could also fail (but it's wrong the agent implementation if it does). I've done it because in Gym the reward is defined to be a float. As previously said, usually nothing goes wrong if it is an int, a numpy.float64, or something else but easily castable as a float.


Can I force DecisionTreeClassifier to use integer conditions when the variable is integer?

Just for visualization you could use the precision parameter of plot_tree():

precision: int, optional (default=3)
Number of digits of precision for floating point in the values of impurity, threshold and value attributes of each node.

(see here)
However, since it also applies precision to impurity you might want to turn that off. Here is an example:
tree.plot_tree(clf) gives

tree.plot_tree(clf, precision=0, impurity=false) gives

(If gini importance was shown, all gini values would be 0 here)
Since it uses $\leq$ and all split thresholds are between the closest $X$ values the split criteria should still be correct. Even if your tree does not use integers. But I'd always double check and compare both plots.


How can I normalize my output data for neural network?

Normally, you shouldn't make normalization on your target data. If you want to increase your accuracy of model, you can try different neural network architectures or activation functions and monitor your train/val/test scores.

If you insist to normalize target data, then you may want to convert your predictions and then calculate the accuracy.

Steps may be like below:

normalize data => fit model => take predictions => denormalize predictions => calculate accuracy


Instead of transforming, and calculating the whole cosine similarity matrix every time you receive a new data point, store your existent data, and whenever you get a new sample calculate the cosine similarity of a new sample against the existent data. Then append and store it.
That is, you need a two function:

First function - Calculate the distance matrix among N existent documents, then store it to the [N x N matrix], also you need to store transformed data, count_matrix.

Second function - When you get a new data point, calculate the cosine similarity of a new datapoint to all existent datapoints in the count_matrix, and append the new [1 x N+1] list to the existent cosine similarity matrix. Also, update the count_matrix. Afterward, store both.

Simply, instead of creating [N+1 x N+1] matrix, store the data and when you have new data, calculate [1 x N+1] and combine it with the existent matrix and store it.


Computing number of batches in one epoch

The double-slash in python stands for “floor” division (rounds down to nearest whole number), so if the result is not an integer, it will always miss the last batch, which is smaller than the batch size.

For example:

Given a dataset of 10,000 samples and batch size of 15:

# Original calculation:
# (10000 + 15 - 1) // 15 = floor(667.6) = 667

# Your calculation:
# 10000 // 15 = floor(666.667) = 666

Your formula calculates number of full batches, theirs total number of batches.


Difference between Time series clustering and Time series Segmentation

Actually there is no fixed terminology and these two terms sometimes used in the same meaning and sometimes different. I would suggest following the terminology bellow for yourself, then you can differentiate methods according to this:

Time-Series Segmentation means partitioning an individual time series to similar segments i.e. clustering within an individual time-series (e.g. i have a video in which someone is reading a book for a while, then starts walking and then starts cycling. now I want to segment these three actions).

Suggestion: State-Space reconstruction, moving Autocorrelation, moving DTW, Fourier Analysis, Visibility Graphs or any other method which can measure the similarity of a time-series with itself.

Time-Series Clustering (or this) means finding similar time-series within a dataset of time-series (e.g. i have 10 brain signals, 5 from healthy subjects 5 from patients without knowing who is patient and who is healthy. Now I want to cluster this dataset into two clusters)

Suggestion: Build a similarity matrix between time-series using e.g. DTW and then apply Spectral Clustering (just improvised. If you search literature there should be more mature solutions)

Hope it helped :)


How to determine if a string is 'food' or 'drink'?

Menu is generally a list of items served in restaurant. Items broadly divided in to two sections:

Food: Where all food items are listed
Drinks: Where all drink items are listed.

If all restaurant menus that you have, listed items in similar way, I'm bit confused how you will use ML in this case.

But if restaurant menu includes recipe/ ingredients of the food/drink items, then yes I can apply machine learning in order to identify the category of an item.
In such a case we can use NLP or simple text classifier(by dividing the recipe in some key word tokens and segregate between food and drink)model for prediction.


Is there a particular order in which to do feature selection and sampling?

In my opinion, the methodological correct way to do it would be to first randomly select the observations (e.g., with stratified random sampling to sustain the class balances as in the original data) and then do any model building (feature selection is a part of model building) based on the sected examples.

You should asses the quality of your model with remaining observations that have not been used for either selecting features or training a model.

Hope this helps.


Best way for data preparation to have accurate prediction

The simplification of the data may make the model more stable, but it will also remove its ability to use more specific input criteria. For example, the way you move from Approach A to Approach B you are aggregating specific products into product categories. This means that if your model is successful in Approach A, it will be able to predict based on specific products. On the other hand, if your model training succeeds in Approach B it will only be able to predict on product category (and you will have to convert products into its categories before supplying it to the model).

So to answer your questions, the number of data samples you have determine how much you have to aggregate and simplify your data. The data itself in its most detailed form could also fail to train the model properly, in which case the approach taken in Approach B is the best next step.


How CNN applies backpropagation to update its weights and biases?

the 3 main layers for CNN are convolutional layer, ReLU layer and pooling layer

Not necessarily in fact. Conv layers are the only layers that you absolutely need in order to implement a CNN. All the other are not strictly necessary. For example, I trained a CNN on Fashion MNIST dataset using ELU activation (you can check my Notebook here). Moreover, if the size of the input data is not too big and you have enough computing power, you can get away with Pooling layers and use only the Convolutional ones. (As computing power increasese, Pool layers will be progressively abandoned since they loose too much information).

Coming to your question:

  how CNN updates its weights and biases using backpropagation. I understand that backpropagation uses partial derivatives. But I do not see how they are used in CNN.

Backpropagation works this way: it calculates how much each weight is contributing to the final Loss value (this computation is done finding the first partial derivative with respect to each weight). This trick works in the same way for "normal", Dense layers as for Convolutional layers. The only difference is that Conv layers are not fully connected, i.e. not all the nodes of one layer are connected with all the nodes of the following layer. The difference is that only the nodes that fall under each sliding filter are connected. That is a trick to process large amount of data (such as pixels).

Conv layers have a different architecture from Dense layers, but their training works exactly the same as for Dense layers.

Pool layers, on the other side, do not contain any learnable parameter. They are just mechanical operations that don't require any training.

EDIT: This is a great explanation of how backprop works.


Can Hadoop be beneficial when data is in database tables and not in a file system

'SQL' on Hadoop is very much a thing, though I use quotes since it's probably more accurate to say it's SQL-like. Some options for bringing SQL-like capabilities to Hadoop include Hue, Hive/bee (Heading towards Stinger? So punny Apache), Impala, SparkSQL (probably not a great solution for a bank given the possibility of concurrency issues), among others (Seems like everyone has their own version of it these days)

To be honest though, if you're asking if it could be helpful, you probably don't need Hadoop (sorry in advance of that comes off harshly, it's not intended to). Lots an lots of places think they need Hadoop, but very few actually need Hadoop. There are of business that are down on the tech because they transitioned to it when the need didn't really exist. If you truly do need Hadoop or another distributed system it'd be almost impossible to determine which setup would be beneficial to your org without an intimate understanding of your data, and your specific business means.


How to define a multi-dimensional neural network with keras

If you want to use a simple neural network which takes in a 50x2 matrix you can do this y flattening your matrix to a 100x1 vector. This will make no difference in the way that the neurons are processed. Each neuron will take all the inputs and make a decision on its weight accordingly.

If you want some neighborhood feature selection then you can use a convolutional neural network like

model = Sequential()
model.add(Conv2D(32, kernel_size=(2, 2),
                 activation='relu',
                 input_shape=input_shape,
                 padding='same'))
model.add(Conv2D(64, (2, 2), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(1, activation='softmax'))

model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

This will do a feature mixing within a 2x2 window across your matrix which will thus mix your two values before feeding them to the deeper part of the network.

Doing this is also a good idea as it serves as a means to reduce the complexity of your model which may lead to better results.


Questions on ensemble technique in machine learning

Instead, model 2 may have a better overall performance on all the data
  points, but it has worse performance on the very set of points where
  model 1 is better. The idea is to combine these two models where they
  perform the best. This is why creating out-of-sample predictions have
  a higher chance of capturing distinct regions where each model
  performs the best.

It's not about training on all the data or not. Both models trained on all the data. But each of them is better than the other at different points. If I and my older brother are tying to guess the exact year of a song, I will do better in 90s songs and he in 80s songs - it's not a perfect analogy but you get the point - imagine my brain just can't process 80s songs, and his can't process 90s songs. The best is to deploy us both knowing we each have learnt different regions of the input space better.

  Simply, for a given input data point, all we need to do is to pass it
  through the M base-learners and get M number of predictions, and send
  those M predictions through the meta-learner as inputs

k-fold is still just one learner. But you're training multiple times to chose parameters that minimize error in the left-out fold. This is like training only me on all the songs showing me k-1 folds of data, and I calibrate my internal model the best I can... but I'll still never be very good at those 80s songs. I'm just one base learner whose functional form (my brain) isn't fit for those songs. If we could bring the second learner along, that would improve things.


Is it possible to generate syllogisms using an NLP algorithm?

That would probably be related to textual entailment and also relation extraction.

I'm not aware of any specific work but I would check in the biomedical domain, because there are resources such as SemRep and I wouldn't be surprised if people tried to use it for similar purposes.


Use of Nash-Equilibrium in big data environments

This question isn't terribly clear.  Data analysis and strategic modeling (game theory) are different tasks.  Nash equilibrium is a way of understanding the incentives they have by assuming a set of players with assumed utility function and making deductive inferences about what they ought to do to maximize those utility functions given their interaction.  Data analysis is an inductive process.

There are a number of ways game theory and data analysis might interact, here are the easy top two:

Someone might use data to infer players' utility functions (I'm sure this exists in econometrics-land somewhere; also, political scientists have a technique called "ideal point estimation," to infer political preferences from voting behavior---which you can easily google to learn more);
Someone might use game theory to generate behavioral predictions which are testable by data.

Thinking about the specific kinds of cases you mention, the obvious application would be in the stock market one.  Suppose you have a ML model that can reliably predict the market behavior of other people at time T from a given feature set.  Then the consumer of the ML model might have an optimal purchase at T-1, and finding that optimal purchase is going to be strategic.

But combining the two approaches might just break the ML. This is really interesting to think about... musing out loud...

Consider the simple case of a two-player market in one stock. Player 1 wants to buy at T-1 if player 2 will be buying at T (because the price will go up); player 1 wants to sell at T-1 if player 2 will sell at T (because the price will go down).  The naive approach for player 1 is "use my ML model to predict what player 2 will do, then do it first at T-1." But, of course, P1's behavior at T-1 is itself observable by P2, and changes P2's behavior (the price has gone up); moreover, by definition P1's behavior at T-1 can't be a feature of the ML model used to predict P2's behavior at T, because it's behavior that is chosen on the basis of the ML prediction.  All sorts of fun puzzles begin here, but none of them look real good...


Computing Item-to-Item Similarity Using Cosine

Filtering users will create bias in your training data. This may be good or bad, depending on your data and goals. The best way to find out, for your specific system, is to try and test both methods, optimizing for whatever metric you choose is best.

Honestly, I think you should consider all users. Say you have a system for recommending movies. In this system, i1 represents a children's movie, that is rated by users u1 through u5, with u5 being a parent and u1, u2, u3 and u4 children. Similarly, i2 represents a terror movie, that is rated by users u5 through u9, all adults. In such a system, u5 being the only common ground between i1 and i2, if this user rated both positively and you only consider them, then you will have a high similarity between the items; but if you consider all users that rated either movie, you will have a very low similarity.

Or in other words, I think intuitively it's best to consider a missing rating as a false/negative rating. Excluding terms where both ratings are zero makes no difference in the cosine similarity anyway. But again, you should run and test it.


Stopwords for programming languages (for, while, print,...)

Stop words are common words in a language and they are usually removed when there appearance is not indicative to the analysis goal.

Suppose our goal is to text mine and find out if a given text is about sports or politics. If we use a bag of words, "not" is probably a stop word we should remove since it is probably not indicative of either categories.

On the other hand, if we would like to differ between "This article will not discuss politics" and "this article will discuss politics", we must not remove "not".

The examples you gave look like programming languages reserved words. In most languages you can find this documentation and skip the learning phase. Looking at the frequent tokens will probably also give you the reserved words (and probably plenty of English stop words that will appear in the remarks).

However, before utilising such a list, be sure that neglecting them will serve your goal. In case that typing them doesn't differ from typing other words, you might better leave them. This way you will have more data the represent better the user behaviour.


Titanic Kaggle Data: Why am I getting lower accuracy on Kaggle submissions than on held-out data?

It could be because of the percentage of the different class.
Imagine your data is 30% survived and 70% died but in Kaggle's test data this ratio may change, i.e 50%-50%. So your model could not predict kaggle's survived part as well as your test data.
you may impute missing by the mean. if you use test data for calculating mean it could be a cheating for your model.


Dividing data between test, learn and predict

Supervised Learning:

Do you have a saved time history of the data?  For a supervised learning set you need some churned="No" cases and some churned="Yes" cases, but it sounds like you only have churned="Yes" and the unknown cases e.g. current customers who may or may not churn.  With some time history you can go back in time and definitively label the current customers as churn="No".

Then it is very easy to split up the data.  And no, you probably don't want to predict on any data that you trained on since you can only train on it if you already know the solution so it will be a waste of time and throw off any metrics you might use to assess accuracy (precision/recall/F1) in the future.

Unsupervised Learning:

If you don't have saved time history of the data then this is an unsupervised learning set for which you have churned="yes" and churned="maybe".  You could then employ anomaly or outlier detection on this set.

novelty detection:
    The training data is not polluted by outliers, and we are interested in detecting anomalies in new observations.

outlier detection:
    The training data contains outliers, and we need to fit the central mode of the training data, ignoring the deviant observations.

You can do either one but novelty is more powerful.  This is kind of a flip around as the novelty here is Churned="No" since all of your data is the confirmed Churn="Yes" cases.

Hope this helps!


Why doesn't neural networks use the concept of degree of freedom?

Actually neural networks do take this information into account.

When we do a hard classification with NN (i.e. only one class for the output), we (almost always) use a softmax output layer, then taking the max of the softmax layer to get the class and its probability.

Then why do we still have 10 outputs before the softmax and why 10 after? Well, you need to have the 10 inputs to the softmax to indicate the weight the network gives to each class, as this is still not scaled back to a probability. Then to ease the usage of the output layer, we want to keep having 10 outputs to be able to use the values without reconstructing the missing value (which would need to be computed anyway due to the way softmax works).


determining batch size and epoch size of neural network

Depends on many things. Batch size is how many instances will be accepted as input by your model, in order to make a complete forward pass and propagate the appropriate gradients backwards. Having a batch size too small could lead to gradients that derived from a batch which is not representative of the data distribution, thus these gradients could be harmful. Having a large batch size in a huge dataset could lead to memory issues. Having said that, use 128.


A SVM has 3 very important components: the support vectors, the separating hyperplane and the margin.

When a missclassification occurs, it is because a given point is on the
wrong side of the separating hyperplane, and that's called a
classification error.
Whenever a point is inside the margin, that counts as a margin error.
The total error of a SVM, is the sum of the classification error and
the margin error.

Now, to understand the C parameter, you've got to know that there are 2 types of SVMs:

Hard margin: these try to maximize the margin without introducing any
kind of errors.
Soft margin: which also share the same objective, but they allow for some classification and margin errors to occur. The number of errors allowed is controlled by the C parameter (which is often called the penalty parameter): if C is small, the SVM allows for some errors and therefore reaches better generalization; but as C increases, the SVM penalizes these errors more and more, eventually reaching the point that it allows no errors at all.

As you can see bellow on the left, a very large C restricts the model by not allowing any errors, showing very bad results when outliers are present. On the right, by using a small penalty parameter, we let that negative outlier to be in the positive area, therefore maximizing the margin value and reaching a more respectful separating hyperplane.

Although, observe that if you have a soft margin SVM with a very large C value, it will behave just like a hard margin SVM.


What does it mean that an hypotesis is consistent?

I'm not especially familiar with this but from the example provided we can deduce that:

An hypothesis is a partial assignment of values to the features. That is, by "applying the hypothesis" we obtain a subset of instances for which the features satisfy the hypothesis.
An hypothesis is consistent with the data if the target variable (called "concept" apparently, here EnjoySport in the example) has the same value for any instance in the subset obtained by applying it.

First case: $h_1=<?,?,?,Strong,?,?>$. All 4 instances in the data satisfy $h_1$, so the subset satisfying $h_1$ is the whole data. However the concept EnjoySport can have two values for this subset, so $h_1$ is not consistent.
Second case: $h_2=<Sunny,Warm,?,Strong,?,?>$. This hypothesis is more precise than $h_1$: the subset of instances which satisfy $h_2$ is $\{1,2,4\}$. The concept EnjoySport always have value Yes for every instance in this subset, so $h_2$ is consistent with the data.
Intuitively, the idea is that an hypothesis is consistent with the data if knowing the values specified by the hypothesis gives a 100% certainty about the value of the target variable.


Proceeding with various methods for news recommendation

Recommender Systems are a huge topic of its own right and goes without saying, with a lot of research going on.

This book does a deep-dive into recommender systems and may not be something you want, but it's helpful as a reference. It seems like you were unsure what those terms mean. The Berekely AI Course covers most of these topics and their lectures are available for free.

You might be able to make better choices after going through some of the above materials.


Are calibrated probabilities always more reliable?

The results seem quite reasonable to me, but I cannot be sure based on your given information. In your table of results, you have shown 10 instances where the uncalibrated probability for the mixed class is between 0.97 and 0.99. It means that on average one would expect only about 2% of misclassifications on these instances. In the small sample that you are showing, the misclassification rate is 10% (1 out of 10), since one of the instances has the negative label. Maybe you have chosen these 10 instances randomly, and actually in most other samples of size 10 there would be 2 or 3 misclassified instances? If that is true, then 70% or 80% probability would indeed be better calibrated.

Looking at the misclassified instance, however, I am not sure whether the negative label is even correct for it? Maybe it should be mixed as well? So maybe there are labelling errors also in the data? Calibration is taking the labels as the ground truth, so if your model's predictions are even better than the labels, then the confidence is still reduced, because calibration is trusting the labels.

The above was my short answer, but below I will try to interpret your figures in more detail.

In your task, you have 3 classes: positive, negative, and mixed. In an attempt to make it easier to follow my answer, let me first give names to your figures:

Figure 1a - confidence histogram of uncalibrated predictions;
Figure 1b - confidence-reliability diagram of uncalibrated predictions;
Figures 2a, 2b, 2c - classwise-reliability diagrams of uncalibrated predictions for classes Neg, Pos, Mixed;
Figures 3a, 3b, 3c - classwise-reliability diagrams of Dirichlet-calibrated predictions for classes Neg, Pos, Mixed;
Figure 4a - confidence histogram of Dirichlet-calibrated predictions;
Figure 4b - confidence-reliability diagram of Dirichlet-calibrated predictions.

Fig. 1a shows that the vast majority of predictions have confidence above 0.9. Fig. 1b shows that in the highest-confidence bin (0.9 and above) there is a tiny bit of under-confidence, as the blue bar (actual accuracy within this bin) seems to be higher than the predicted (above the diagonal) - this is indicated by the red gap color mixing with the blue actual color to obtain dark red. Since the gap is small, the predictions seem quite well confidence-calibrated.

However, looking at the classwise-reliability diagrams it can be seen that the negative class is quite well-calibrated (Fig. 2a), whereas the positive class predictions are under-confident (Fig. 2b), and the mixed class predictions are under-confident (Fig. 2c). Figures 3a, 3b, 3c show that after applying the Dirichlet calibration, the probabilities are better classwise-calibrated.

The confidence-reliability diagram in Fig. 4b shows better calibration in the lower confidence bins, compared to Fig. 1b. However, most of the predictions are still in the highest confidence bin with confidence above 0.9 (Fig. 1a). In that bin, there is perhaps a tiny bit more under-confidence than before calibration. A possible explanation for this is that the reduction of loss in lower bins is more than the tiny increase of loss in the highest bin.

Based on the diagrams and table of results alone I cannot be sure whether Dirichlet calibration has been applied correctly, but it is possible that everything is correct. There seems to be improvement in classwise calibration and probably in confidence calibration also. I can try to interpret the results further, if you could give more information:
 - the classwise histograms of class probability to go with figures 2 and 3;
 - the learned parameter values for Dirichlet calibration;
 - size of the dataset on which you calibrate and on which you show the reliability diagrams, and perhaps the class distribution as well.

Here are some more thoughts based on the confusion matrices and numeric results for evaluation measures that were added to the question after my above answer.

I don't think that the original model was already calibrated, given your reported numeric evaluation information and the classwise calibration plots. Dirichlet calibration has certainly improved calibration.

It seems that all the measures are improving applying after Dirichlet calibration, with the single exception being the accuracy on the mixed class. I can think of two reasons for this, which probably together result in this effect.

First, the classes are imbalanced, and thus 1% of increase in accuracy for the majority class is for the classifier and for the calibrator worth more than 1% decrease in accuracy for the minority class. You haven't specified your exact class distribution, but for example, if there is more than 3.5 times difference in size of positive and negative classes vs the mixed class, then increase of 1% both in positive and negative class accuracy is worth more than 7% (3.5%+3.5%) decrease in the mixed class accuracy. Well, Dirichlet calibration is optimising log-loss rather than accuracy, but the point remains roughly the same.

Second, there are misclassified instances in the validation set. This forces the calibrator to lower the confidence, as discussed in my earlier answer above.

Maybe there are further factors relating to different distributions of confidences for different classes. This could be checked if you would provide the histograms of predicted class probabilities for each class separately (maybe both before and after calibration).

If you are particularly interested in having a better accuracy in the mixed class, then you need to upweight the mixed class during calibration. Ideally, calibration should not push towards the overall class distribution, but to some extent this can happen due to the limitations of parametric modelling. I suggest you to try oversampling the mixed class to be of the same size as the positive and negative classes during calibration.


Desicision tree classification with a "false" attribute

Maybe I'm not interpreting the scenario correctly, but task 1 seems impossible and task 2 easier than you've made it sound.

I'm reading it as a sample for each patient, with one feature for each student, possible values yes/no/dunno.

In that case, at any split of the decision tree the imposter and the chosen real student will split the data in the same fashion.  If either of them is chosen as the split criteria, then it's random which of the two is used, but if we remove the imposter first, then the real student will be used; but there is not difference in the way the samples are split.

As for finding the imposter, all you can do AFAICT is look for pairs who give exactly opposite responses.  But if there are multiple such pairs, I don't see how to pick one, and even when you have a pair of opposites, I don't see any way to determine which is real and which is imposter.  The 60% figure seems to just indicate that nobody answers "I don't know" too often.  I guess we can make an additional assumption that the students are better-than-random, so that the imposter is worse?


Object Recognition for classification, is it being used in industry?

As far as I am aware, object recognition is not extensively used in industry yet. Google's image search, for instance, is based on exploiting the text on the web pages rather than the images themselves. I have seen several start-up companies that market prototypes based on object recognition, but these products are either prototypes or not widely used.

Regarding state-of-the-art techniques, you should take a look to recent literature on deep learning and deep neural networks applied to object recognition, along with more specific techniques like convolutional networks.


What is the efficient way to use apply method in column of pandas Dataframe for large dataset?

You can try pandarallel it works very efficiently for parallel processing. You can find more information about it here.

You should not use this if your apply function is a lambda function. Now assuming you're trying to apply it on a DataFrame called df:

from pandarallel import pandarallel

pandarallel.initialize(nb_workers=n) #n is the number of worker used for parallelization, you can leave it blank and it will use all the cores

def foo(x):
   return #what ever you're trying to compute

df.parallel_apply(foo, axis=1) #if you're applying to multiple columns
df[column].parallel_apply(foo) # if its just one column

Another option you can try is using the python multiprocessing library, here you will break your dataframe into smaller chunks and run them together.

import numpy as np
from multiprocessing import cpu_count, Parallel

cores = cpu_count() #Gets number of CPU cores on your machine
partitions = cores #Define number of partitions

def parallelize(df, func):
    df_split = np.array_split(df, partitions)
    pool = Pool(cores)
    df = pd.concat(pool.map(func, df_split))
    pool.close()
    pool.join()
    return df

Now you can run this parallelize function on your df:

df = parallelize(df, foo)

The more number of cores you have the faster this will be!


Replacing missing values with mean of feature calculated from previously replaced values

So, is it alright to replace missing values with the mean calculated from values that include some that have once replaced other missing values?

There's no law against it :)

But by replacing missing values everywhere in many different ways you're probably damaging the reliability of your dataset at some point, and that could in turn mess up your experiment.

You don't give any detail about the task or the data so I assume you chose this approach on purpose, but just in case let me remind you of other options:

you could discard features for which most values are missing, unless they happen to be very important for the task
you could discard instances which have many/some missing values, especially if you have a very large dataset anyway
you could use a ML method able to deal with missing values by itself


AWS machine learning prediction schema problems

I ran into the same problem today, tried googling for people having the same issue and found your question.

I solved my problem by creating the data source first and then running the prediction from there. So, instead of selecting the following option,

Batch Predictions > Create new batch prediction > ML model for batch prediction > My data is in S3, and I need to create a datasource

which fails, I first did:

Datasources > Create a new datasource...

Next, I ran the batch prediction from an existing datasource successfully.


predict future value in every one hour using (t+60 minutes) LSTM neural network in python

One option is to :

Break data into constant-frequency observations (E.g.: Assume that g,p,c,out are same for all time periods between two observations). With this, you will get samples every N minutes.
Training data will then be a set of [Last M observations till T, Observation for T + 60]

This Train and test data can be fed into a network.


If there are 20 independent variables and 1 dependent variable, a linear regression model can be viewed as a 20-dimensional hyperplane in 21-dimensional space. The hyperplane is not "vertical" with respect to any independent variable. So if you pick a value for each of the 20 independent variables and draw a vertical line at the point consisting of those values, it will cross the hyperplane once at the height that the model outputs when you plug in the values of the independent variables.


Changing categorical data to binary data is not reflected on the dataset

You need to transform your independent variables into numeric values. Normally for binary variables, we use the 0-1-encoding. Introduce a new variable called is_female. If the observation is a male person then give the variable the value $0$ and if the observation is a female person give the observation the value $1$.

If you have categorical variables like a city with three categories you will need to create additional variables. Imagine we have the possible values ["New York", "London", "Moskau"]. Then you create three variables is_new_york, is_london, is_moskau.
If we have an observation from New York this will result in is_new_york=1, is_london=0, is_moskau=0. If we have an observation from London then the values will be is_new_york=0, is_london=1, is_moskau=0 and if we have an observation from Moskau then the values will be is_new_york=0, is_london=0, is_moskau=1. This type of encoding is called one-hot encoding. You can also have multiple cities. For example, if a person lives in Moskau and London then you can use is_new_york=0, is_london=1, is_moskau=1.


Intuitive explanation of Convolutional layers in Deep CNN

In each layer of a CNN you have a number of filters. Suppose for the first convolutional layer you have $C_{out} = 10$ filters. The customary dimension for each of these filters is $(C_{input}, H, W)$ where $C_{input}$ represents the number of channels of the inputs of the current convolutional layer. $H$ and $W$ represents the height and width of the window of the filter. Consider this point that you are convolving each filter with the inputs. An input is a volume with $(C_{input}, H^{'}, W^{'})$ dimension. What you do as convolution is convolving these volumes, each filter and the input. The height and the width of the input $H^{'}$ and $W^{'}$ are the same or bigger than the height and width of each filter, but the depth or the number of the channels of each filter has to be the same as the input channel. After convolving each filter you will have a two-dimensional activation map. By stacking the result of each filter you will have $C_{output}$ number of output feature maps which is equal to the number of filters.


Sklearn Pipeline for mixed features: numerical and (skewed) categorical

You can use handle_unknown='ignore' in the OneHotEncoder; levels present in the test set but not the train set will be encoded as all-zeros rather than raising an error.

But... that example you provide, I rather doubt it's worth it to keep all the levels.  The coefficient learned for such a small level's dummy variable will be overly specialized (overfit).  Consider alternatives.


How to tell if the “clusters” I see in my pair plots are statistically significant or occurring by random chance?

The easiest test is equality of means of two distributions. Student's t-test


Time series forecasting with RNN(stateful LSTM) produces constant values

Is this the correct way to forecast time series with LSTMs?

  Train data independent matrix (X)=Sequences of previous 30 day values Train (Y)=The 31st day value for each of previous 30 day values

You method could work in theory. But typically, for vanilla LSTM, the setup looks like $x = [x_0, x_1, ..., x_{n-1}]$ and $y = [x_1, x_2 ..., x_n]$. I suspect you will get much better result if you do it this way.

Highly recommend Colah's excellent explaination if you haven't read it already.

  I followed the following methodology to forecast: Y for t+1 is first forecasted, Then X matrix row is shifted by 1 day and the forecasted Y is appended to the end of this row, then use this row to predict t+2 value and continue. However in each sequence after (usually) t+3 days the forecasted values become constant for the rest of the t+n days.

This is not recommended because any error in prediction is compounded over time. please find my answer in the following post for a detailed explanation
Is time series multi-step ahead forecasting a sequence to sequence problem?

How can this behaviour be explained?

If you are getting constant prediction value, the chance is you are having so-called "gradient explosion problem", try to clip the gradient before backprop might be the solution.

Correction. Had a bit better thinking about it.  it is not a gradient explosion problem here. More likely it is due to saturated neuron. Because LSTM uses sigmoid function, if the input is not properly scaled or weight is initialised inproperly, it could cause neuron saturation very quickly.


How to detect misclasification data after multiclass classification?

Sadly, this isn't possible. Simply put, if you could know for sure that your model made a mistake, then, your model will never make mistakes.
This is the painful reality of deploying models in practice. You will need manual validations to confirm whether your model performed correctly. In some systems, you may be able to outsource this to the user itself (i.e. give them an option to say the model was wrong), but you'll likely want to run independent validation.
Now, there are indeed ways to make this part less painful. First, as you said, you should take a look at the confidence scores. It should indeed be less necessary to verify predictions where the model is highly confident. You should also use your confusion matrix to figure out which predictions are more crucial to verify. Indeed, some mistakes may be less harmful than others.
What you could also do is experiment with building auxiliary binary models that take in the input, the model output, and predicts whether the model made a mistake. This isn't guaranteed to work, but it may give you better confidence scores for your model.


If you only need to predict whether an object is a pepper or not why bother predicting whether it is a red pepper or a green pepper, you are making your problem more complicated than it needs to be. In my opinion you should try to make the problem you want to solve as simple as possible so I would advise using the shortest label possible as long as it solves your problem.


How to make a Power Spectral Density Plot in R

From further research I've discovered that the frequency is given by the index of the FFT multiplied by the sampling rate and divided by the size of the array. And the amplitude is the magnitude of the complex number.

So the full code for such a plot would be as follows

# X is some set of Wait times between spikes, below is just an example
X <- c(56, 3, 4, 119, 3, 4, 121, 3, 3, 121, 3, 4, 120, 3, 4, 4, 115)
acf <- acf(X,type="covariance")
ft <- fft(acf$acf)
freq <- (1:nrow(ft))*1000/nrow(ft) #In my case we sample by ms so 1000 hz
A <- (Re(ft)^2 + Im(ft)^2)^.5 #amplitude is magnitude
PSD <- cbind.data.frame(freq,A)
ggplot(PSD, aes(x=freq,y=A)) + geom_line()


Why is the kernel of a Convolutional layer a 4D-tensor and not a 3D one?

The first formula you quote is for an image with one input channel and one output channel, it just focuses on height and width. In this case, if we consider a 5x5 convolution, the Kernel will just have size 5x5, $m$ and $n$ and going from -2 to +2.
Now if our input has 3 channels (RGB, but could be feature maps). we need to use each channel as an input, and the weights will be different for each input map. So K becomes 3-D : 3x5x5. This new dimension corresponds to the $l$ in your formula.
If we want to have 10 outputs, we need 10 different ways of convoluting the input feature maps, so our Kernel will have one more dimension, leading to a 10x3x5x5 size for the Kernel. This last dimension corresponds to the $i$ of your formula.
To recap, the 4 dimensions of the Kernel $i, l, m, n$ stand for :

$i$ : Output dimension
$l$ : Input dimension
$m, n$ : height and width

About this sentence "giving the connection strength between a unit in channel $i$ of the output and a unit in channel $j$ of the input".
I feel it is wrong as I would rather interpret it as : "giving the connection strength between a unit in channel $i$ of the output and a unit in channel $l$ of the input"


How important is it for each row of data to have the same number of features?

A lot of blank values in row isn't a problem, you can use sparse matrices.

I suppose you have a dataset to work with. So you can use validation and check some your suspicions. For example: fit model using dataset with <40 components and test it on another part of data.

Trying to consider order of components within device is challenging task. Maybe you can add pairs of components as features...

This is just my suggestions, I'm not an experienced data scientist =)


Pandas Conditional Fill NaN Forward/Backward

Simply using the fillna method and provide a limit on how many NA values should be filled. You only want the first value to be filled, soset that it to 1:

df.ffill(limit=1)

  item  month  normal_price  final_price
0    1      1          10.0          8.0
1    1      2          12.0         12.0
2    1      3          12.0         12.0
3    2      1           NaN         25.0
4    2      2          30.0         25.0
5    3      3          30.0          NaN
6    3      4         200.0        150.0

You can chain together the above with a bfill to then fill the remaining NaN values:

df.ffill(limit=1).bfill(limit=1)

  item  month  normal_price  final_price
0    1      1          10.0          8.0
1    1      2          12.0         12.0
2    1      3          12.0         12.0
3    2      1          30.0         25.0
4    2      2          30.0         25.0
5    3      3          30.0        150.0
6    3      4         200.0        150.0

This would only not be optimal if there are column in your dataframe which you would like to leave unaffected.

In that case you can do them one column at a time - i use the in_place flag so that we do not need to do any of the ugly re-assignments:

df.final_price.ffill(inplace=True, limit=1)

df

  item  month  normal_price  final_price
0    1      1          10.0          8.0
1    1      2          12.0         12.0
2    1      3           NaN         12.0
3    2      1           NaN         25.0
4    2      2          30.0         25.0
5    3      3           NaN          NaN
6    3      4         200.0        150.0

The same idea will work the backward-filling the values, using the bfill method instead of the ffill, as I have done above.


How to add incorporate meta data into text classification?

Some models cannot really handle this, while others lend themselves for it easily. I'll explain two approaches that you could use:

Naive bayes

With Naive Bayes you can use other categorical values as well as your normal n-grams or sparse bag of words vectors. Just add them one-hot encoded to your features and it is also incorporated. With numerical features you would need to use something like Gaussian Naive Bayes, to fit a distribution to your features per target class, then you can use the likelihoods of these features per class to compute the probabilities.

Neural network

If you use a neural network approach like CNNs or RNNs, you can add any type of feature representation network and concatenate it somewhere in your original network. In your case you would have a softmax at the end of your RNN. Before this, concatenate the output of your 'normal-feature' neural network, add some dense layers and feed this to your softmax output layer. This way you can train your model end-to-end and it will learn important interactions as well.


What is the formula to calculate the precision, recall, f-measure with macro, micro, none for multi-label classification in sklearn metrics?

Generally, the scoring metrics you are looking at are defined as following (see for example Wikipedia):

$$precision = \frac{TP}{TP+FP}$$
$$recall= \frac{TP}{TP+FN}$$
$$F1 = \frac{2 \times precision \times recall}{precision + recall}$$

For the multi-class case scikit learn offers the following parameterizations (see here for example):

  'micro':
  Calculate metrics globally by counting the total true positives, false negatives and false positives.

  'macro':
  Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.

  'weighted':
  Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters ‘macro’ to account for label imbalance; it can result in an F-score that is not between precision and recall.

  'samples':
  Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from accuracy_score).

And none does the following:

  If None, the scores for each class are returned.

TLDR: "micro" calculates the overall metric, "macro" derives an average assigning each class an equal weight and "weighted" calculates an average assigning each class a weight based on the number of ocurences (its support).

Accordingly, the calculations in your example go like this:

Macro

$$precision_{macro} = \sum_{classes} \frac{precision\text{ }of \text{ }class}{number\text{ }of\text{ }classes} = \frac{(2/2) + (2/3) + (2/2)}{3} \approx 0.89$$

$$recall_{macro} = \sum_{classes} \frac{recall\text{ }of \text{ }class}{number\text{ }of\text{ }classes} = \frac{(2/3) + (2/2) + (2/3)}{3} \approx 0.78$$

$$F1_{macro}= \ \sum_{classes} \frac{F1\text{ }of \text{ }class}{number\text{ }of\text{ }classes} = \frac{1}{3} \times \frac{2 \times (2/2) \times (2/3)}{(2/2) + (2/3)} + \frac{1}{3} \times \frac{2 \times (2/3) \times (2/2)}{(2/3) + (2/3)} + \frac{1}{3} \times \frac{2 \times (2/2) \times (2/3)}{(2/2) + (2/3)} \approx 0.80$$

Note that macro means that all classes have the same weight, i.e. $\frac{1}{3}$ in your example. That is where the $\times \frac{1}{3}$ to calculate the F1 score comes from.

Micro

$$precision_{micro} = \frac{\sum_{classes} TP\text{ }of \text{ }class}{\sum_{classes} TP\text{ }of\text{ }class + FP\text{ }of\text{ }class } = \frac{2+2+2}{2+3+2} \approx 0.86$$

$$recall_{micro} = \frac{\sum_{classes} TP\text{ }of \text{ }class}{\sum_{classes} TP\text{ }of\text{ }class+FN\text{ }of\text{ }class} = \frac{2+2+2}{3+2+3} = 0.75$$

$$F1_{micro}= 2\times \frac{recall_{micro} \times precision_{micro}}{recall_{micro} + precision_{micro}} \approx 0.8$$

Weighted
$$precision_{weighted} = \sum_{classes}{weight\text{ }of \text{ }class \times precision\text{ }of\text{ }class} = \frac{3}{8}\times\frac{2}{2} + \frac{2}{8}\times\frac{2}{3} + \frac{3}{8} \times \frac{2}{2} \approx 0.92$$

$$recall_{weighted} = \sum_{classes}{weight\text{ }of \text{ }class \times recall\text{ }of\text{ }class} = \frac{3}{8} \times \frac{2}{3} + \frac{2}{8}\times\frac{2}{2} + \frac{3}{8} \times \frac{2}{3} = 0.75$$

$$F1_{weighted} = \sum_{classes}{weight\text{ }of \text{ }class \times F1\text{ }of\text{ }class} = \frac{3}{8} \times \frac{2 \times (2/2) \times (2/3)}{(2/2) + (2/3)} + \frac{2}{8} \times \frac{2 \times (2/3) \times (2/2)}{(2/3) + (2/3)} + \frac{3}{8} \times \frac{2 \times (2/2) \times (2/3)}{(2/2) + (2/3)} = 0.8$$

None

$precision_{class 1} = \frac{2}{2} = 1.0$

$precision_{class 2} = \frac{2}{2+1} \approx 0.67$

$precision_{class 3} = \frac{2}{2} = 1.0$

$recall_{class 1} = \frac{2}{2+1}  \approx 0.67$

$recall_{class 2} = \frac{2}{2} = 1.0$

$recall_{class 3} = \frac{2}{2+1} \approx 0.67$

$F1_{class 1} =  \frac{2 \times 1 \times \frac{2}{3}}{1 + \frac{2}{3}} = 0.8$

$F1_{class 2} =  \frac{2  \times \frac{2}{3}\times 1}{\frac{2}{3} + 1} = 0.8$

$F1_{class 3} =  \frac{2 \times 1 \times \frac{2}{3}}{1 + \frac{2}{3}} = 0.8$

Samples

$$Precision_{samples}= \frac{1}{Number\, of\, examples} \sum_{examples} \frac{TP\,of\,example}{TP\,of\,example + FP\,of\,example} = \frac{1}{4}[\frac{2}{2}+\frac{0}{1}+\frac{1}{1}+\frac{3}{3}] = 0.75$$

$$Recall_{samples}= \frac{1}{Number\, of\, examples} \sum_{examples} \frac{TP\,of \,example}{TP\,of\,example + FN\,of\,example} = \frac{1}{4}[\frac{2}{2}+\frac{0}{2}+\frac{1}{1}+\frac{3}{3}] = 0.75$$

$$F1_{samples}= 2\times \frac{recall_{samples} \times precision_{samples}}{recall_{samples} + precision_{samples}} = 0.75$$


Artificial Neural Networks are algorithms loosely based in how a brain functions so they shouldn't be treated as the equivalent of brain learning.

The science that studies all the details of how neurons and brains learn is Computational Neuroscience and the question you ask is pretty much an open question but it has many great hypothesis.

The method you described about how artificial neural networks update their "weights" is called backpropagation and some authors included O'Reilly, Randall; Munakata, Yuko (2000) in Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the Brain argue that backpropagation is actually the algorithm that drives learning in the brain. In later publications they propose architectures that include the Hippocampus region to re-create short term memory.

However in recent years, with the advent of mainstream deep learning new ideas have arrived along with old ideas with new approaches. The ones that look to replicate the brain and its function or at least approximate it as much as we currently can are:

Spiking neural networks
Hierarchical temporal memory / Numenta (They also have resources and solutions for data science problems like anomaly detection and classification)
Other theories in Computational Intelligence

Spiking neural networks are computationally expensive and hard to tune, their performance isn't close to what deep learning can do, however natural brains use some form of Spiking neural networks which tells you about the potential of these kind of models.

Another interesting mention is Neural Turing Machines. It's an interesting approach on how certain brain inspired processes can be used to compute diverse problems.


Not sure if I understood your question correctly, but to dodata analysis, you would need to have a proper DF.

You have something like:

time A B C
1    3 4 6
time B C D
2    9 4 6

You need one table header, so remove any other header text in the rows > 1.

Also (of course) the content of each row must match the column it belongs to.

So the DF above would change to:

time A B C D
1    3 4 6 0
2    9 4 0 6

Note the "zero" entries in time=1 -> D=0 and time=2 -> C=0. Here the "zero" is chosen to fill the gaps (zero activity in this case).

This type of data representation is what you usually use for any kind of modelling.


Opinions on an LSTM hyper-parameter tuning process I am using

First of all you might want to know there is a "new" Keras tuner, which includes BayesianOptimization, so building an LSTM with keras and optimizing its hyperparams is completely a plug-in task with keras tuner :) You can find a recent answer I posted about tuning an LSTM for time series with keras tuner here

So, 2 points I would consider:

I would not loop only once over your dataset, it does not sound like enough times to find the right weights.
I would rather control the number of possible hyperparams configurations as you said, which is something you can indicate in keras tuner via max_trials param
About using keras tuner with Bayesian tuner, you can find some code below as an example for tuning the units (nodes) in the hidden layers and the learning rate:

from tensorflow import keras
from kerastuner.tuners import BayesianOptimization

n_input = 6
def build_model(hp):
    model = Sequential()
    model.add(LSTM(units=hp.Int('units',min_value=32,
                                    max_value=512,
                                    step=32),
               activation='relu', input_shape=(n_input, 1)))
    model.add(Dense(units=hp.Int('units',min_value=32,
                                    max_value=512,
                                    step=32), activation='relu'))
    model.add(Dense(1))
    model.compile(loss='mse', metrics=['mse'], optimizer=keras.optimizers.Adam(
        hp.Choice('learning_rate',
                  values=[1e-2, 1e-3, 1e-4])))

return model

bayesian_opt_tuner = BayesianOptimization(
    build_model,
    objective='mse',
    max_trials=3,
    executions_per_trial=1,
    directory=os.path.normpath('C:/keras_tuning'),
    project_name='kerastuner_bayesian_poc',
    overwrite=True)

bayesian_opt_tuner.search(train_x, train_y,epochs=n_epochs,
     #validation_data=(X_test, y_test)
     validation_split=0.2,verbose=1)

bayes_opt_model_best_model = bayesian_opt_tuner.get_best_models(num_models=1)
model = bayes_opt_model_best_model[0]

You would get something like this, informing you about the searched configurations and evaluation metrics:


Can neural networks be adapted without recreating them completely?

The answer to your question is "Transfer Learning".

Since the datasets "cat and dog" and "mouse" are quite similar as both are images. In DeepNet for recognising "cats and dogs", any deep learning network in its early layers learn to identify low-level features like edges, etc. It learns high-level feature like eyes, ears, etc in few further layers and in very later layers of DeepNet, it starts to recognise the intended object. In DeepNet for recognising "dogs", similar pattern will follow. On comparing these two, one may find that first few layers of DeepNet in both cases produce similar low-level features like edges, etc.

Hence, first few layers of the DeepNet learned from "cats" can be used as base layers for the DeepNet for "mouse" detection. This technique is called transfer learning.

To apply transfer learning, dataset on which DeepNet has been trained on and dataset on which this technique can be applied must be similar.

This transfer learning video by Andrew Ng would also be helpful in understanding the concept


Another way of looking at 600 columns is having a vector with 600 dimensions. So, the answer to reduce the 600 dimensional vectors to fewer dimensions without losing lot of information will be dimensionality reduction.

Principal component analysis(PCA) is a commonly used one. T-distributed stochastic neighbour embedding (TSNE) has been the state of the art for reducing higher dimensional vectors. Data visualisation using TSNE

To get the quick intuition about the vectors, the best thing is to reduce it to 2 or 3 dimensional vectors and plot them in a graph. Data visualisation always helps to get an overview of the data you are dealing with.


Is machine learning suitable for detecting screen elements?

Object recognition is a subfield of Computer Vision which consists of methods/algorithms aimed at determining whether a given object is present on a 2D image. Many object recognition systems just return a 'yes/no' answer, but there are also many methods that return the location of the object on the image by means of a bounding box.

This subfield comprises a vast range of methods, from low-complexity ones (like methods based solely on applying simple filters on the image, like colour/edges/etc.) to more complex ones in which machine learning methods (like deep learning) play a very important role.

Depending on the complexity of the objects you are planning to detect, and on the overall complexity of the images, machine learning may be completely unnecessary.

This Wikipedia page summarises many of these methods. You can start exploring from there, or you can start by using any introductory computer vision text.


Please look at my comment to add more information to your post. Based on the information you provided, here are my remarks:

SpaCy is trained to find locations, not addresses per se

If you use a "common" language, SpaCy is trained using WikiNER data, where locations aren't addresses but more like geographical places like city names, country names etc. So it's quite normal to not be able to detect full addresses.

You likely need to train your own entity recognizer. They detail how to do this on their website, including code samples: https://spacy.io/usage/training#ner

Don't underestimate SpaCy's rule-based matching

Is it a fancy neural network? No. Does it matter? Also no. SpaCy allows you to create rules to find entities and in cases like addresses which are generally following a pattern across entities.


Does it make sense to concatenate datasets to improve accuracy of model?

More data is always better. It makes 100% sense to concatenate all datasets in to a larger one, assuming that they are in the same format. This will most likely improve the performance of your model.


Dataset: language audio clips and country labels

Topcoder hosted competition for identifying spoken languages. They released a database containing 176 languages. The datasets and the related resources can be found here.


I will cite explanations from "A survey of Text Classification Algorithms" by Aggarwal and Zhai. The following and more can be read here in Chapters 2.1 (Gini Index) and 2.2 (Information Gain).
Gini Index

Let $p_1(w)...p_k(w)$ be the fraction of class-label presence of the $k$ different classes for the word $w$. In other words, $p_i(w)$ is the conditional probability that a document belongs to class $i$, given the fact that it contains the word $w$. Therefore, we have:
$\sum_{i=1}^{k} p_i(w)=1$
Then, the gini-index for the word $w$, denoted by $G(w)$ is defined as follows
$G(w)= \sum_{i=1}^{k} p_i(w)^2$
The value of the gini-index $G(w)$ always lies in the range $(1/k,1)$. Highervalues of the gini-index $G(w)$ indicate a greater discriminative power of the word $w$.

Information Gain

Let $P_i$ be the global probability of class $i$,and $p_i(w)$ be the probability of class $i$, given that the document contains the word $w$.  Let $F(w)$ be the fraction of the documents containing the word $w$. The information gain measure $I(w)$ for a given word $w$ is defined as follows:
$I(w) = - \sum_{i=1}^{k} P_i * log(P_i) + F(w) * \sum_{i=1}^{k} p_i(w) * log(p_i(w)) +$
$+ (1-F(w)) * \sum_{i=1}^{k} (1 - p_i(w)) * log(1-p_i(w))$
The greater the value of the information gain $I(w)$, the greater the discriminatory power of the word $w$.


Given the number of parameters and the number of options for each parameter, a graphical tool (aka, graphical user interface or GUI) is not very practical for CNNs. Most practitioners store and manipulate parameters in text form.

There are two graphical projects for TensorFlow. Playground which is an in-browser network that can be modified and run on sample datasets. TensorBoard which visualizes the computational graph and helps with debugging models.


Cluster titles or ingredients of food into n-categories

This sounds like a job for latent dirichlet alocation (i.e. topic modeling).

https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation
https://radimrehurek.com/gensim/wiki.html


Creating radial basis for linear regression Python

I was able to solve this by myself.
First you need to create a column that contains day of year values from the timestamps.
Then apply radial_basis function for that column with month parameter being the middle day of every month. For example in January it's 15, February 45 etc.
With this method you can generate a spike for every month.
data_all['Day'] = data_all.Timestamp.dt.dayofyear

def radial_basis(x, month):
    alpha = 8
    coef = -1/(2*alpha)
    return np.exp(coef*(x-month)**2)

for i in range(12):
    col = 'RB' + str(i+1)
    data_all[col] = data_all.Day.apply(radial_basis, month=(15+30*i))


As correctly pointed out above MinHash and SimHash both belong to Locality Sensitive Hashing.
Reference: https://en.wikipedia.org/wiki/Locality-sensitive_hashing

The main difference between the two is the way collision is handled,

SimHash, uses cosine similarity
MinHash, uses Jaccard Index.

Answers to your questions:

No. They uses different collision handling techniques to validate similarity. Also there is a variant on single Hash Function for Min Hash but it works differently. For more details, check the following reference out, https://en.wikipedia.org/wiki/MinHash (Variant with a single hash function)
Yes, https://github.com/chrisjmccormick/MinHash/blob/master/runMinHashExample.py
I think the complexity can be reduced to $O(n \log n)$ with modified form of binary search while clustering.


Origin of the Boolean Model of Information Retrieval

Looking for "Boolean Retrieval" in google scholar with upper date limit finds two papers that seems related from the sixties.
https://heinonline.org/HOL/LandingPage?handle=hein.journals/juraba8&div=16&id=&page=
https://apps.dtic.mil/sti/pdfs/AD0673362.pdf
There is a reference to Non-Boolean retrieval from 1963, hinting that the term was known before.
Try checking the references in these papers.


How does a zero centered activation functions like tanh helps in gradient decent?

$\tanh$ function is scaled standard sigmoid function $y=\frac{1}{1+e^{-ax}}$. Due to that scaling it has more steep gradient that standard sigmoid function. Steep gradient is important, because it makes backprop training faster and less likely to get stuck in zero-gradient area.

Furthermore $y(0) \neq 0$ where $\tanh(0) = 0$. Sometimes it is very important, especially when dealing with normalized $[0,1]$ signals as inputs.


Prediction

If the main goal is predicting anything, say, the statisitcs of the player in the next game, game result, then I would not recommend to do any scoring. Better way to go is using the pure statistics data as an input to the model. Any scoring/rankning - is information loss.

Ranking

If the goal is ranking itself, than you still need to have some target variable to predict. As you may want to check real predictive value of those ranks. That could be, again, playser stats in the next game or game result itself.

References

Sport scores prediction and RFM scoring are probably the next directions for you to look at.


Question on Scipy - Minimize. Adding additional constraints

I assume problem is in accessing your columns, elements of the V matrix, since you say:
"it should look like:"

Well to actually write down these constraints, you just need to access your columns in the functions your defined in the cons dictionary.

Lets say you have your data V

> array([[ 0.00749589,  0.01255155,  0.02396251,  0.04750988, 0.09495377]
>        [ 0.01255155,  0.02510441,  0.04794055,  0.09502834,  0.18996269],
>        [ 0.02396251,  0.04794055,  0.09631614,  0.19092151,  0.38165151],
>        [ 0.04750988,  0.09502834,  0.19092151,  0.38341252,  0.7664427 ],
>        [ 0.09495377,  0.18996269,  0.38165151,  0.7664427,   1.53713523]])

You should define your constraints dictionary in the following way (for example):

> cons = ({'type': 'eq', 'fun': lambda x:  x[0] - 2 * x[1] + 2},
> {'type': 'ineq', 'fun': lambda x: -x[0] - 2 * x[1] + 6},
> {'type': 'ineq', 'fun': lambda x: -x[0] + 2 * x[1] + 2})

While indexing desired columns inside of the lambda function.


In the context of sentiment analysis, removing stop words can be problematic if context is affected. For example suppose your stop word corpus includes ‘not’, which is a negation that can alter the valence of the passage. So you have to be cautious of exactly what is being dropped and what consequences it can have.