What is the efficiency difference between different cost functions in case of neural networks?

Think of it in terms of navigating a landscape.  The land you move across is created by your error function (that is, the realtion between your model and your data), and the way you move across the landscape is your training function.  Especially in Neural Networks, falling in 'local minima' is a big problem.  In fact, Neural Networks have been theoretially able to approximate any function for... 20 years?  See Universal Approximation Theorem.  But the problem has always been learning how to train a given network.

More specifically, in your case comaring MAE and MSE, the difference is in the 'square' part.  The steepness of the 'fitness landscape' will be much steeper where error is greater with MSE compared to MAE.  1*1 = 1.  i.e., squaring does nothing at unit-error.  10 * 10 = 100.  Absolute error of 10 will become a squared error of 100.  That is looking at one sample, and has huge implications around falling in local minima where the error is great.

Looking across the whole dataset, another huge implication is how the NN balances the error it sees.  With MSE, samples with high error become samples with very high error.  Thus, a NN trained with MSE will 'care more' about learning samples that are very wrong at the expense of small improvements to 'easier' samples.


Theoretical work on validity of restricting movement of Centroid of K-Mean

I highly recommend finding a source explains how k-means work and understand it well. The K-means is well known, so it is hard to find a reference talk about it as an algorithm or explain how it work.

I noticed you stating "author used ~1000 "fake" data points, so that the final centroid of K-mean stays within the required range" which is always going to be true. K-means is about calculating the mean (average) of data points used, which assure (always) to end with a centroid/s within the range of data used.

The power of this algorithm (K-means) is calculating the mean iteratively to reach stability of means (centroids). In another waord, in each iterate, means shift to be centered of denses. That give, if you in case of finding 1 K (one centroid) you will find it by one iterate.

Me personally suggest start with some videos, and go forward. Here is the first result on YouTube about k-means https://youtu.be/_aWzGGNrcic.


Using scikit-learn iterative imputer with extra tree regressor eats a lot of RAM

TL;DR - use the max_depth and max_samples arguments to ExtraTreesRegressor to reduce the maximum tree size. The sizes you pick might depend on the distribution of your data. As a starting point, you could start with max_depth=5 and max_samples=0.1*data.shape[0] (10%), and compare results to what you have already. Tweak as you see fit.

Apart from the fairly large input space, the data structure built by the ExtraTreeRegressor is the main issue. It will continue to expand the tree size until each leaf reaches your criteria, namely min_samples_leaf=1. This means every single data point of your input dataset must end up in its own leaf. Apart from probably overfitting, this is going to lead to high memory consumption.
See the Note: in the relevant documentation:

The default values for the parameters controlling the size of the trees (e.g. max_depth, min_samples_leaf, etc.) lead to fully grown and unpruned trees which can potentially be very large on some data sets. To reduce memory consumption, the complexity and size of the trees should be controlled by setting those parameter values.

Each ExtraTreesRegressor that you create looks like it might make a full copy of your dataset, according to the documentation for max_samples`:
    max_samples : int or float, default=None
        If bootstrap is True, the number of samples to draw from X
        to train each base estimator.
        - If None (default), then draw `X.shape[0]` samples.

To gain a deeper understanding of how you might tune your memory usage, you could take a look at the source code of the ExtraTreesRegressor.


How to merge monthly, daily and weekly data?

when given two time series with different time steps, what is better: Using the Lowest or the biggest time step ?

For your timeseries analysis you should do both: get to the highest granularity possible with the daily dataset, and also repeat the analysis with the monthly dataset. With the monthly dataset you have 120 data points, which is sufficient to get a timeseries model even with seasonality in your data.

  For known and unknown properties, how should I proceed to go from daily to weekly/monthly data ?

To obtain say weekly or monthly data from daily data, you can use smoothing functions. For financial data, you can use moving average or exponential smoothing, but if those do not work for your data, then you can use the spline smoothing function "smooth.spline" in R: https://stat.ethz.ch/R-manual/R-patched/library/stats/html/smooth.spline.html

The model returned will have less noise than the original daily dataset, and you can get values for the desired time points. Finally, these data points can be used in your timeseries analysis.

  For known and unknown properties, how should I proceed to go from weekly/monthly to daily data ?

To obtain daily data when you have monthly or weekly data, you can use interpolation. First, you should find an equation to describe the data. In order to do this you should plot the data (e.g. price over time). When factors are known to you, this equation should be influenced by those factors. When factors are unknown, you can use a best fit equation. The simplest would be a linear function or piecewise linear function, but for financial data this won't work well. In that case, you should consider piecewise cubic spline interpolation. This link goes into more detail on possible interpolation functions: http://people.math.gatech.edu/~meyer/MA6635/chap2.pdf.

In R, there is a method for doing interpolation of timeseries data. Here you would create a vector with say weekly values and NAs in the gaps for the daily values, and then use the "interpNA" function to get the interpolated values for the NAs. However, this function uses the "approx" function to get the interpolated values, which applies either a linear or constant interpolation. To perform cubic spline interpolation in R, you should use the "splinefun" function instead.

Something to be aware of is that timeseries models typically do some sort of averaging to forecast future values whether you are looking at exponential smoothing or Auto-Regressive Integrated Moving Average (ARIMA) methods amongst others. So a timeseries model to forecast daily values may not be the best choice, but the weekly or monthly models may be better.


Varying Image sizes in Tensorflow Malaria dataset | Dealing with unclean tensorflow data

#Here image is your batch.
# Add "batch" and "channels" dimensions
image = image[tf.newaxis, ..., tf.newaxis]
image.shape.as_list()  # [batch, height, width, channels]

tf.image.resize(image, [height,width])[0,...,0].numpy()


How do I use the model generated by the R package poLCA to classify new data as belonging to one of the classes?

Using carcinoma data available in the poLCA package and a 4 latent classes solution:

 library poLCA
 data(carcinoma)
 f <- cbind(A, B, C, D, E, F, G) ~ 1
 lc4 <- poLCA(f, carcinoma, nclass = 4)

The following line give the classification in terms of predicted probabilities

 lc4$predclass

They could be usefully binded to the original data for further visualizations or analyses

 carcinoma.predclass <- cbind(carcinoma, "Predicted LC" = lc4$predclass)
 head(carcinoma.predclass)

You could have a new data frame with the same columns/variables used in the previous analysis.

new.data <- data.frame(A=c(1,2,1), B=c(2,2,1), C=c(1,2,1), D=c(1,1,1), E=c(1,2,1), F=c(2,2,1), G=c(1,2,1))

  A B C D E F G
1 1 2 1 1 1 2 1
2 2 2 2 1 2 2 2
3 1 1 1 1 1 1 1

A simple method can be link the observed data patterns in the new data frame with the estimated latent class probabilities in the previous data. In fact, the first pattern has missing prediction because it wasn't in the training data.

left_join(new.data, unique(carcinoma.predclass))

Joining, by = c("A", "B", "C", "D", "E", "F", "G")
  A B C D E F G Predicted LC
1 1 2 1 1 1 2 1           NA
2 2 2 2 1 2 2 2            1
3 1 1 1 1 1 1 1            4

The best method is to use the posterior of poLCA. From the parameters estimated by the latent class model, this function calculates the probability that a specified pattern belongs to each latent class. This function can calculate posterior class membership probabilities for new data, observed or not in the training data.

new.lc4.posterior <- poLCA.posterior(lc4, new.data)

And bind the predicted Latent Classes (the classes which the highest posterior probability) to the new data.

cbind(new.data, "Predicted LC" = apply(new.lc4.posterior,1, FUN=which.max))

  A B C D E F G Predicted LC
1 1 2 1 1 1 2 1            2
2 2 2 2 1 2 2 2            1
3 1 1 1 1 1 1 1            4


Is it normal that a classifier always wrongly predicts the same samples?

What you are experiencing is pretty normal given your approach. A random forest is an ensemble of decision trees where models are trained in parallel using bootstrapped samples (a technique called bagging). Even though the decision trees are randomized (a thorough explanation of random_state can be found in this question), they still rely on an internal criterion (such as Gini index by default in RandomForestClassifier) to split the nodes and define decision paths. The fact that some of your samples are consistently being misclassified regardless of the random state is an indication of their objective difficulty when using this specific criterion.

  Therefore, do these samples which are always wrongly predicted deserve a particular attention or is it kind of logic ?

You are absolutely correct with your first thought. Paying particular attention to wrongly predicted samples in ensembles is the goal of a technique called boosting. The main idea is to train ensemble models in sequence, with new learners focusing on data points that the rest of the ensemble has previously failed on. A great overview of ensemble approaches is presented in this answer, which I highly recommend.

As far as boosting algorithms go, there are different flavors as well: you might want to try sklearn's AdaBoost and gradient tree boosting implementations, or XGBoost. These may help you finally defeat those pesky hard-to-classify samples, but be aware that bagging (your current model) has its own perks that boosting lacks.


Probabilistic Outlier Detection (edited + clarified)

In essence anomaly detection is about finding a metric with which to measure the similarity between instances and then determining a threshold when crossed constitutes an anomaly. There are parametric anomaly detection algorithms which require some hyper-parameters to be set or information about the distribution to be known. There are also non-parametric techniques which do not have these stringent requirements.

Let's consider the simple case of identifying anomalies in a population based on height. We can assume this to be a 1D Gaussian distribution. Now we need a similarity metric, let's use Euclidean distance for simplicity (the difference in individual's heights). Next we need a threshold.

Classical statistics

Classical methods would parametrize the Gaussian distribution of your population. Let us assume that after polling our population we identify that the mean height is 170cm with a variance of 15cm. Now we can set the threshold to be any multiple of the variance $\sigma$. Typically we choose $3\sigma$ as a thresholds since the ingroup would include 99.7% of the population, all those outside of the $3\sigma$ range can thus confidently be said to be anomalous.

You can also look at the application of a generalized likelihood ratio test (GLRT) to find anomalies.

Let's visualize this

import numpy as np

data = np.random.normal(170, 15, 10000)

plt.hist(data)
plt.plot([215,215],[0,3000])
plt.plot([125,125],[0,3000])
plt.show()

Parametric techniques

The examples will require that we have prior information regarding the distribution of the data. This technique will fit an ellipse around the Gaussian distribution of our data in order to contain the elements with the highest in-group correlation.

from sklearn.covariance import EllipticEnvelope
estimator = EllipticEnvelope()
estimator.fit(data.reshape(-1,1))

plt.scatter(data, np.random.randint(0,1,10000), c=estimator.predict(data.reshape(-1,1)))
plt.show()

We see that the yellow dots are the 'normal' instances and the dark blue ones are the anomalies. You can also notice that this algorithm selects an anomaly threshold very similar to the one we obtained using classical statistics methods.

Non-parametric algorithms

This is a play on the Random Forests algorithm based on an ensemble of decision trees. The decision trees are trained to identify anomalies through information loss decisions and isolating instances of your data.

from sklearn.ensemble import IsolationForest
estimator = IsolationForest()
estimator.fit(data.reshape(-1,1))
plt.scatter(data, np.random.randint(0,1,10000), c=estimator.predict(data.reshape(-1,1)))
plt.show()

Non-parametric $p$-value estimation algorithms

These are personal favorite and have been very successful in a number of tasks.

Learning Minimum Volume Sets
http://www.stat.rice.edu/~cscott/pubs/minvol06jmlr.pdf

Anomaly Detection with Score functions based on Nearest Neighbor Graphs
https://arxiv.org/abs/0910.5461

New statistic in P-value estimation for anomaly detection
http://ieeexplore.ieee.org/document/6319713/

I don't have time tonight but tomorrow evening I will elaborate on the GLRT, and the $p$-value estimation algorithms.

Extending the above to more dimensions

Of course all the described algorithm can be extended to $n$-dimensions. This is hard to plot so we will go through a 2D example and a 3D example. However, determining the probability of these being anomalies is not so simple. For some of these algorithms you will need to devise some sort of distance metric.

2D example

Let's consider this artificial data. Where tall people above 170cm have eye colors with a blue content of mean 1 and variance 0.5. And short people have lower blue content at a mean 0 and variance 0.5 (let's just assume eyes can have negative pigmentation whatever that could possibly mean).

import numpy as np

n = 10000
X = np.zeros((n,2))
X[:,0] = np.random.normal(170, 15, n)
for i in range(n):
    if X[i,0] < 170:
        X[i,1] = np.random.normal(0, 0.5)
    else:
        X[i,1] = np.random.normal(1, 0.5)

plt.scatter(X[:,0], X[:,1])
plt.show()

Classical statistics

Any p-value test used in classical statistics can be extended to multiple dimensions. This is the easiest when the features are uncorrelated. When this is the case the probability of each feature can be multiplied together. If this is not the case then you need to compute the p-value for a 2D Gaussian distribution.

For our case this is even worse! It's bimodal (there are two means for a feature). You can see this here. Our data would look something like this image from the Wiki.

I hope you can appreciate how difficult this problem is to solve. Luckily this is no longer the classical era of statistics.

Parametric techniques

These algorithms can easily be moved onto $n$-dimensions.

from sklearn.covariance import EllipticEnvelope
estimator = EllipticEnvelope()
estimator.fit(X)
plt.scatter(X[:,0], X[:,1], c=estimator.predict(X))
plt.show()

In the 1D example we discussed how this algorithm assumes a unimodal Gaussian distribution. This is not the case for our current data and you can see the failures of this model. It seems to classify many extreme examples as being part of the nominal set when they should have likely been identified as anomalies.

Non-parametric techniques

This technique can also be applied to $n$-dimensional data. Moreover, it can be used for multimodal Gaussian distributions as you can see we get better results.

from sklearn.ensemble import IsolationForest
estimator = IsolationForest()
estimator.fit(X)
plt.scatter(X[:,0], X[:,1], c=estimator.predict(X))
plt.show()


Application of Machine learning or Neural Networks for automatic Time table scheduling

Scheduling problems might be NP-Complete problems.
It is not clear what are the specific details.
You might get lucky and have specific constraints that are leading to an easy sub problem or just an easy instance.
However, since many variations like On the Complexity of Scheduling University Courses (which might be your case), Job shop scheduling,Multiprocessor scheduling and Open-shop scheduling are NP-complete, you are probably in the same situation.

Usually, it is better to treat such problems like optimization problems and not like classification problems. If you'll try to treat the problem like a classification problem you might have severe problems in building a classified dataset that will represent well the time scheduling problem that you would like to solve.

There are general technique in order to cope with optimization problem. I also found some work related to your case. I'm not familiar with this specific case but it seems that Solving the Course Scheduling Problem Using
Simulated Annealing and the work done here might help you.


What is a basic object detection/localization ML algorithm that can be used for my relatively simple image set?

So, the following techniques are used for object detection (other can YOLO, R-CNN and other Deep Learning Techniques):

Haar Cascade Classifier

This was the first facial detection algorithm to be successfully used and is popular until today. Proposed by Viola and Jones in their paper Rapid Object Detection using a Boosted Cascade of Simple
Features (2001, CVPR) this algorithm uses Haar-like features from integral images and a cascade of boosting classifiers to detect objects in images. Lienhart proposed in 2002 a improvement to this algorithm in his paper Empirical Analysis of Detection Cascades of Boosted Classifiers for Rapid Object Detection

It can be easily trained using OpenCV Object Detection Module in Python, C and Java, also available in Matlab Computer Vision Tool Box as vision.CascadeObjectDetector and probably many other platforms.

Max-Margin Object Detection

Proposed in the paper Max-Margin Object Detection by Davis E. King, creator of dlib, where it has been implemented, this algorithm has implementations for face detection using HoG Features and CNNs features.

The abstract of the paper goes:

  Most object detection methods operate by applying a binary classifier
  to sub-windows of an image, followed by a non-maximum suppression step
  where detections on overlapping sub-windows are removed. Since the
  number of possible sub-windows in even moderately sized image datasets
  is extremely large, the classifier is typically learned from only a
  subset of the windows. This avoids the computational difficulty of
  dealing with the entire set of subwindows, however, as we will show in
  this paper, it leads to sub-optimal detector performance.

  In particular, the main contribution of this paper is the introduction
  of a new method, Max-Margin Object Detection (MMOD), for learning to
  detect objects in images. This method does not perform any
  sub-sampling, but instead optimizes over all sub-windows. MMOD can be
  used to improve any object detection method which is linear in the
  learned parameters, such as HOG or bag-of-visual-word models. Using
  this approach we show substantial performance gains on three publicly
  available datasets. Strikingly, we show that a single rigid HOG filter
  can outperform a state-of-the-art deformable part model on the Face
  Detection Data Set and Benchmark when the HOG filter is learned via
  MMOD.

I think these are the two most relevant outside of Deep Learning, reading the related work fromt he 3 cited articles might give you some light on how to proceed.


Forward pass vs backward pass vs backpropagation

In brief, backpropagation references the idea of using the difference between prediction and actual values to fit the hyperparameters of the method used. But, for applying it, previous forward proagation is always required. So, we could say that backpropagation method applies forward and backward passes, sequentially and repeteadly.
Your machine learning model starts with random hyperparameter values and makes a prediction with them (forward propagation). Then it compares with real values while adjusting those random initial values (backpropagation), trying to minimize the error (depending of your objective function and optimization method applied). And then, it starts over again, until you reach the stopping criteria.
You may find a better explanation in this question.
Furthermore, you will find more topic explanation here.


Why are mini-batches degrading my conv net MNIST classifier?

Your model is very small for a convnet. 1 conv layer, 1 maxpool and 1 fc is very shallow. Try adding more layers and batchnorm2d after each conv layer followed by relu. No pool layers.


Plotly is an excellent tool for 2D and 3D data. It is quite simple to use with python as well.
https://plotly.com/python/


Approaching a multi-class classification problem but without labels

Your specific problem can be solved by Googling.

Here is a solution that

Searches the "imdb [year] [movie name]" in Google,
Finds its IMDb address and fetches the IMDb page, and then
Searches for the genres inside the IMDb page.

I have changed "romantic" to "romance", and "emotional" to "drama" to match the IMDb vocabulary.

from requests import get
import re

titles=["2013+Conjuring", "2004+The+notebook"]
genres = ['horror', 'thriller', 'comedy', 'romance', 'drama']
matched_genres = {}
for title in titles:
  query = "https://www.google.com/search?q=imdb+" + title
  print(query)
  search_result = get(query).text.lower()
  imdb_id = re.findall("https://www.imdb.com/title/(tt\d+)/", search_result)[0]
  imdb_address = "https://www.imdb.com/title/%s/" % imdb_id
  print(imdb_address)
  imdb_result = get(imdb_address).text.lower()
  matched_genres[title] = []
  for genre in genres:
    # find ">genre<" inside tags
    if imdb_result.find(">%s<" % genre) > -1:
      matched_genres[title].append(genre)

print(matched_genres)

Output

https://www.google.com/search?q=imdb+2013+Conjuring
https://www.imdb.com/title/tt1457767/
https://www.google.com/search?q=imdb+2004+The+notebook
https://www.imdb.com/title/tt0332280/
{'2013+Conjuring': ['horror', 'thriller'], '2004+The+notebook': ['romance', 'drama']}

This solution could be improved by

Querying movie titles in parallel,
Directly querying IMDb API,
Handling edge cases (for example when the first IMDb url is irrelevant or no IMDb page is found, etc.),

etc.


Continous bag of words claimed to be unsupervised, how is it working?

The CBOW approach is unsupervised because the network learns the distribution of word co-occurrences around each word, and this doesn't require labelling or additional input, just sequences of words.

As Mikolov et al state in one of the original articles, "the training objective is to learn word vector representations that are good at predicting the nearby words; in another, Mikolov et al say their aim is "building a log-linear classifier with four future and four history words at the input, where the training criterion is to correctly classify the current (middle) word". So if you can see that for each sequence, you're taking 'x' words and training the network to predict one given the others, then no supervision is involved.

It's a little unconventional because it's not the output of the network which is important, but the weights that are learned during training - these are what is taken and used as embeddings in other tasks.

Adrian Colyer does a great general write-up of word2vec here, and the explanation from Chris McCormick here is good and quite accessible.


StandardScaler before and after splitting data

In the interest of preventing information about the distribution of the test set leaking into your model, you should go for option #2 and fit the scaler on your training data only, then standardise both training and test sets with that scaler. By fitting the scaler on the full dataset prior to splitting (option #1), information about the test set is used to transform the training set, which in turn is passed downstream.

As an example, knowing the distribution of the whole dataset might influence how you detect and process outliers, as well as how you parameterise your model. Although the data itself is not exposed, information about the distribution of the data is. As a result, your test set performance is not a true estimate of performance on unseen data. Some further discussion you might find useful is on Cross Validated.


Increase number of iterations in a logistic regression

Logistic regression in R uses the iterative re-weighted least squares algorithm. You can specify the maximum iterations and accuracy with:

m <- glm(..., family = "binomial", control = list(maxit = 2, epsilon=1))

Please read the documentation for glm.control here.

You can check the actual number of iterations used in fitting:

m$iter

BUT... The default settings should be enough.

> glm.control()

$epsilon
[1] 1e-08

$maxit
[1] 25

$trace
[1] FALSE

The default maximum number of iterations is 25, and I **doubt** you will get anything by changing it to anything larger. The accuracy is 1e-08, which is already very small.

You shouldn't blindly adjust the iteration number, most likely it won't help. Try this:

> glm(admit ~ gre + gpa + rank, data = mydata, family = "binomial", control = list(trace=TRUE))

Deviance = 458.875865177 Iterations - 1
Deviance = 458.517518881 Iterations - 2
Deviance = 458.517492476 Iterations - 3
Deviance = 458.517492476 Iterations - 4

I got the dataset from http://www.ats.ucla.edu/stat/r/dae/logit.htm (because I don't have your data file).

Do you see I have trace=TRUE? I print off the deviance for each iteration. R stops because the deviance difference between iteration 3 and 4 is too small. My model only requires 4 iterations.

My recommendation:

Print your deviance for each iteration and convince yourself you don't need more than 25 iterations
Check your model properly. Do you need more independent variables? Do you need to transform your variables to improve prediction accuracy?
Ask yourself is 60% accuracy enough? In many fields 60% is a huge success.
Does your model require interaction? Note a saturated model will give you prefect accuracy, but it has all the interaction terms. Your model assumes the effects of your independent variables are additive in predicting the log-odd. Is this a fair assumption?
Do you want to run automatic stepwise model selection algorithm in R? You might get a better model than manually done it yourself.

Some useful links you might want to consider:

https://stats.stackexchange.com/questions/5354/logistic-regression-model-does-not-converge
https://stats.stackexchange.com/questions/60958/logistic-regression-fishers-scoring-iterations-do-not-match-the-selected-itera


should we write our own algorithms by specifying the conditions?

If you know the exact conditions that result in fraud, you can implement a decision tree for that. But, I believe it is going to be really difficult to come up with that set of conditions that can give clear decision boundary. The data you have created makes the problem look easier than it actually would be. Someone who did a transaction of low amount quite a few times might not be a fraud.

  Or can we use any existing algorithms?

Yes.

This is a classification problem which can be handled through machine learning techniques like logistic regression, support vector machines, neural networks (Multilayer perceptron), Random forest etc.

You can also try Naive Bayes Classifier, which can outperform complex techniques like support vector machines, neural networks.


Python script/GUI to generate positive/negative images for CascadeClassifier?

So I wrote the script. It gave me an excuse to learn Tkinter. It's pasted below. Note this is a one-off, not a model of good programming practice! If anyone uses this and has bugs or suggestions, let me know. Here's the git link and model code is pasted below:

import Tkinter
import Image, ImageTk
from Tkinter import Tk, BOTH
from ttk import Frame, Button, Style
import cv2
import os
import time
import itertools

IMAGE_DIRECTORY =            # Directory of existing files
POSITIVE_DIRECTORY =         # Where to store 'positive' cropped images
NEGATIVE_DIRECTORY =         # Where to store 'negative' cropped images (generated automatically based on 'positive' image cropping
IMAGE_RESIZE_FACTOR =        # How much to scale images for display purposes. Images are not scaled when saved.

# Everything stuffed into one class, not exactly model programming but it works for now
class Example(Frame):

    def __init__(self, parent, list_of_files, write_file):
        Frame.__init__(self, parent)
        self.parent = parent
        self.list_of_files = list_of_files
        self.write_file = write_file
        self.image = None
        self.canvas = None
        self.corners = []
        self.index = -1
        self.loadImage()
        self.initUI()
        self.resetCanvas()

    def loadImage(self):
        self.index += 1
        img = cv2.imread(self.list_of_files[self.index])
        print(self.list_of_files[self.index])
        while not img.shape[0]:
            self.index += 1
            img = cv2.imread(self.list_of_files[self.index])
        self.cv_img = img
        img_small = cv2.resize(img, (0,0), fx = IMAGE_RESIZE_FACTOR, fy = IMAGE_RESIZE_FACTOR)
        b, g, r = cv2.split(img_small)
        img_small = cv2.merge((r,g,b))
        im = Image.fromarray(img_small)
        self.image = ImageTk.PhotoImage(image=im)

    def resetCanvas(self):
        self.canvas.create_image(0, 0, image=self.image, anchor="nw")
        self.canvas.configure(height = self.image.height(), width = self.image.width())
        self.canvas.place(x = 0, y = 0, height = self.image.height(), width = self.image.width())

    def initUI(self):
        self.style = Style()
        self.style.theme_use("default")
        self.pack(fill=BOTH, expand=1)

        print "width and height of image should be ", self.image.width(), self.image.height()
        self.canvas = Tkinter.Canvas(self, width = self.image.width(), height = self.image.height())
        self.canvas.bind("<Button-1>", self.OnMouseDown)
        self.canvas.pack()

        nextButton = Button(self, text="Next", command=self.nextButton)
        nextButton.place(x=0, y=0)

        restartButton = Button(self, text="Restart", command=self.restart)
        restartButton.place(x=0, y=22)

    def nextButton(self):
        new_img = self.cv_img[self.corners[0][1]/IMAGE_RESIZE_FACTOR:self.corners[1][1]/IMAGE_RESIZE_FACTOR, self.corners[0][0]/IMAGE_RESIZE_FACTOR:self.corners[1][0]/IMAGE_RESIZE_FACTOR]
        files = self.list_of_files[self.index].split("/")
        try:
            os.stat(POSITIVE_DIRECTORY+files[-2])
        except:
            os.mkdir(POSITIVE_DIRECTORY+files[-2])
        print("saving to ", "{}{}/{}".format(POSITIVE_DIRECTORY, files[-2], files[-1]))
        cv2.imwrite("{}{}/{}".format(POSITIVE_DIRECTORY, files[-2], files[-1]), new_img)
        self.saveNegatives(files)
        self.restart()
        self.loadImage()
        self.resetCanvas()

    def saveNegatives(self, files):
        low_x = min(self.corners[0][0], self.corners[1][0])/IMAGE_RESIZE_FACTOR
        high_x = max(self.corners[0][0], self.corners[1][0])/IMAGE_RESIZE_FACTOR
        low_y = min(self.corners[0][1], self.corners[1][1])/IMAGE_RESIZE_FACTOR
        high_y = max(self.corners[0][1], self.corners[1][1])/IMAGE_RESIZE_FACTOR

        try:
            os.stat(NEGATIVE_DIRECTORY+files[-2])
        except:
            os.mkdir(NEGATIVE_DIRECTORY+files[-2])

        new_img = self.cv_img[ :low_y, :]
        cv2.imwrite("{}{}/{}{}".format(NEGATIVE_DIRECTORY, files[-2], "LY", files[-1]), new_img)
        new_img = self.cv_img[ high_y: , :]
        cv2.imwrite("{}{}/{}{}".format(NEGATIVE_DIRECTORY, files[-2], "HY", files[-1]), new_img)

        new_img = self.cv_img[ :, :low_x ]
        cv2.imwrite("{}{}/{}{}".format(NEGATIVE_DIRECTORY, files[-2], "LX", files[-1]), new_img)
        new_img = self.cv_img[:,  high_x: ]
        cv2.imwrite("{}{}/{}{}".format(NEGATIVE_DIRECTORY, files[-2], "HX", files[-1]), new_img)

    def restart(self):
        self.corners = []
        self.index -=1
        self.canvas.delete("all")
        self.loadImage()
        self.resetCanvas()

    def OnMouseDown(self, event):
        print(event.x, event.y)
        self.corners.append([event.x, event.y])
        if len(self.corners) == 2:
            self.canvas.create_rectangle(self.corners[0][0], self.corners[0][1], self.corners[1][0], self.corners[1][1], outline ='cyan', width = 2)

def main():

    root = Tk()
    root.geometry("250x150+300+300")

    list_of_files = []
    file_names = []
    walker = iter(os.walk(IMAGE_DIRECTORY))
    next(walker)
    for dir, _, _ in walker:
        files = [dir + "/" +  file for file in os.listdir(dir)]
        list_of_files.extend(files)
        file_names.extend(os.listdir(dir))

    list_of_processed_files = []
    processed_file_names = []
    walker = iter(os.walk(POSITIVE_DIRECTORY))
    next(walker)
    for dir, _, _ in walker:
        files = [dir + "/" +  file for file in os.listdir(dir)]
        list_of_processed_files.extend(files)
        processed_file_names.extend(os.listdir(dir))

    good_names = set(file_names) - set(processed_file_names)
    list_of_files = [f for i, f in enumerate(list_of_files) if file_names[i] in good_names]

    app = Example(root, list_of_files, IMAGE_DIRECTORY+"positives")
    root.mainloop()

if __name__ == '__main__':
    main()


Effecient way to decompose multiple time series in a data frame and compare the fit of additive and multiplicative models?

One way to do this would be to fit the decompositions with the same numbers of degrees of freedom and see which fits the best. It is convenient to do this using the tsibble and feasts packages as they allow for modelling many time series at once.

I've modified your example data so that it is possible to do a multiplicative decomposition -- having negative values in the data makes multiplicative decompositions problematic.

The multiplicative decomposition uses STL on the log data, and then exponentiates the trend and seasonal terms to put them back on the original scale.

Your example has no obvious seasonality so I have arbitrarily set the seasonal period to 12 for illustration purposes. Change it to whatever it should be.

I have set the trend window to be 99 and the seasonal component to be periodic. Again, change these to suit your actual data but the two fits should have the same values.

set.seed(123)
ID = factor(letters[seq(15)])
Time = c(1000,1200,1234,980,1300,1020,1180,1908,1303,
         1045,1373,1111,1097,1167,1423)
df <- data.frame(ID = rep(ID, Time), Time = sequence(Time))
df[paste0('Var', c(1:7))] <- abs(rnorm(sum(Time)))

library(tidyverse)
library(tsibble)
library(feasts)

# Create tsibble in long form
df <- df %>%
  pivot_longer(starts_with("Var"), names_to="Series", values_to="value") %>%
  as_tsibble(index=Time, key=c(ID,Series))
# Additive decompositions
additive <- df %>%
  model(add = STL(value ~ trend(window=99) + season("periodic", period=12))) %>%
  components()
# Multiplicative decompositions
multiplicative <- df %>%
  model(mult = STL(log(value) ~ trend(window=99) + season("periodic", period=12))) %>%
  components() %>%
  mutate(remainder = df$value - exp(trend+season_12))
# Find variance of remainders
rva <- additive %>%
  as_tibble() %>%
  group_by(ID, Series) %>%
  summarise(rv = var(remainder, na.rm=TRUE)) %>%
  ungroup()
rvm <- multiplicative %>%
  as_tibble() %>%
  group_by(ID, Series) %>%
  summarise(rv = var(remainder, na.rm=TRUE)) %>%
  ungroup()
# Which remainder has lowest variance?
left_join(rva, rvm, by = c("ID","Series")) %>%
  mutate(best = if_else(rv.x < rv.y, "additive", "multiplicative"))
#> # A tibble: 105 x 5
#>    ID    Series  rv.x  rv.y best
#>    <fct> <chr>  <dbl> <dbl> <chr>
#>  1 a     Var1   0.357 0.361 additive
#>  2 a     Var2   0.357 0.361 additive
#>  3 a     Var3   0.357 0.361 additive
#>  4 a     Var4   0.357 0.361 additive
#>  5 a     Var5   0.357 0.361 additive
#>  6 a     Var6   0.357 0.361 additive
#>  7 a     Var7   0.357 0.361 additive
#>  8 b     Var1   0.338 0.341 additive
#>  9 b     Var2   0.338 0.341 additive
#> 10 b     Var3   0.338 0.341 additive
#> # … with 95 more rows

Created on 2020-04-22 by the reprex package (v0.3.0)


Comparing accuracy of models in ordinal regression / classification

Your classes express a certain order. You can classify apples to, say, "green", "red" or "yellow", and then every disagreement with a reference set is equal. After all, colours express no order. So as you already suggested, I would certainly use regression. Assume that the classes could be distributed as something like this:

Very bad = 0 - 0.25
Bad = 0.25 - 0.50
Good = 0.50 - 0.75
Very good = 0.75 - 1.00

Now, the mismatch of Very good vs. Bad is at least 0.25, where is must be at least 0.50 with Very good vs. Very bad, which gives a better and more honest impression of the performance of your model.


What if multiple final prediction values for multi-class Neural Network are equal

I'd say it's not very common to have absolutely equal probabilities with a model that has been trained for sufficient epochs and that does not overfit hard.

That said, if you happen to have equal probabilities, the choice of the class to predict should depend on the domain you're approaching, e.g:

A possible implementation is like in numpy's argmax, by choosing the first index.
You could also create a new class "unknown" specifically for this issue, in your test set.
Is it worse for your business problem if you make a mistake and predict class "2" instead of class "1"? You should then choose the class whose probability of mistake is less costly (Think about the medical domain and predicting several stages of cancer)
Is the class balance the same? You could go for the most probable class in the training set


XGBoost outputs tend towards the extremes

The first I'd ask would be "What is positive/negative ratios?". Just because I had a similar issue multiple times when classes were very imbalanced. If it's your case you can try to balance dataset or try to play with scale_pos_weight parameter of XGboost. In the case of 50/50 classes ratios, probablities might be more or less normalised.

Also, it's very likely you have overfitting at the same time. Try to tune learning rate, sampling parameters and regularization parameters (reg_alpha, reg_lambda).


When should train and test data be merged?

The answer to this is simple.

When we perform the cleaning of the dataset we'll need to do the whole cleaning process for training data first then we'll do the same data cleaning process for the test dataset too. So to avoid doing the same data cleaning process twice, we merge the training and testing data then we perform the data cleaning process and after that we separate both dataset.


Predict real world data after modelling with scaled features

I would say it depends on your training set. If you had a sufficiently large (and representative) training set, then you could say that the mean and the variance of the training set are the estimates of the population mean, thus, use these values also for the standardization of the one single testing sample. If it is not the case, then the whole idea of prediction is not really good.
Generally, it is wrong to use different transformations for training and testing sets separately. You can read about it more here


How does keras calculate accuracy for multi label classification?

Why are you using binary_crossentropy?  You should be using categorical_crossentropy.  However, if you insist on using binary_crossentropy change your metric to metrics=['binary_accuracy', 'categorical_accuracy'] (this will display both accuracies).


What is the difference between dynamic programming and Q-learning?

Both Q learning and Value Iteration (a DP technique) use similar update rules based on Bellman optimality equations:

$$v_*(s) = \text{max}_{a}\sum_{s',r} p(s',r|s,a)(r + \gamma v_*(s'))$$

$$q_*(s,a) = \sum_{s',r} p(s',r|s,a)(r + \gamma\text{max}_{a'}q_*(s',a'))$$

The main difference is that DP uses an explicit model. DP requires that you know $p(s',r|s,a)$. The update rule for DP is literally the first equation turned into an update rule:

$$v_{k+1}(s) = \text{max}_{a}\sum_{s',r} p(s',r|s,a)(r + \gamma v_{k}(s'))$$

In comparison, Q learning does not require knowing $p(s',r|s,a)$, as it is based on sampling from experience. The update rule is modified to be based on samples of observed data, which have the same values in expectation, as if you had used $p(s',r|s,a)$, but without knowing it:

$$Q_{k+1}(S_t,A_t) = Q_{k}(S_t,A_t) + \alpha(R_{t+1} + \gamma\text{max}_{a'}Q_k(S_{t+1},a') - Q_{k}(S_t,A_t))$$

This is still an important difference even when both systems are run on an internal model/simulation. DP does not need to simulate anything, it iterates over the model directly. Whilst Q learning needs to work with sampled transitions - they might be simulated, but this is not the same as iterating over all states as in DP. It can often be the case that it is easier to simulate the environment than to calculate $p(s',r|s,a)$ for the full model.

Which should you choose:

Choose Dynamic Programming when you have access to the full state transition and reward model in a simple form (i.e. you have $p(s',r|s,a)$ or equivalent), and the state space is not too large - ideally the number of states is small enough to fit in memory. However, there are ways to use DP when you have a larger state space, by modifying which states it processes. So you still can use DP on larger problems if you really want to.
Choose Q learning when you don't have a model, or when the state space is too large to iterate over in full.


It sounds like you essentially have a sparse input problem, similar to doing something like a recommendation system. Imagine trying to recommend a film to somebody based on films they have already watched and rated. There would be many many films they have not seen, so you (in a sense) have missing data. This was the case in the Netflix Prize challenge.

Facorisation methods such Single Value Decomposition are common approaches in these cases. There is a nice summary of such methods used in the Netflix Prize.

Using a neural network Auto-Enconder will be reliant on relatively large numbers of samples, which is something to bear in mind when assessing performance.

Just one other idea that might be worth trying out, depending on how many features you have and the general sparsity, could be to replace the missing values with fixed values on a feature-level. This could be as simple as replacing the missing values e.g. in "relationship status" to <RELATIONSHIP>, and then in general "feature X" to just be <X>. This is a kind of trick that is often used in NLP (natural language processing) to allow filler-words to encode some useful information.

In your case, this might at least help any model distinguish between missing values across features. I have not tested something like this myself, so unfortunately I cannot cite references nor point to results.


Can my neural network learn conditional rules when classifying?

Centering the data on zero and scaling to variance of one helps with a lot of classification problems but in this case it would remove information that's needed to solve your problem as I understand it.
Another possible problem is the loss function, which you don't mention at all in your question. I would suggest something that stays fairly high when your neural network is only learning one of the two rules.


How to approach a new data set with no dependent variable

Your assignment is basically the process we call EDA - Explorative Data Analysis.

So what should you do? Simply explore!

What is the shape of your dataset?
How do variables behave, do they have a factor structure, correlate, etc.
What are the main descriptives of your dataset, to they tell an interesting story, etc.

And once you start doing this you will find something that might be interesting to explore a bit deeper depending on your dataset. Do not just use summary functions like mean, median, etc. but also try to build simple graphs and comment everything in a neat notebook!

My tip:

Look at some EDA notebooks on Kaggle for inspiration or watch this superior video by a master at work:

https://www.youtube.com/watch?v=go5Au01Jrvs

Also here is a beginner guide as well:

https://towardsdatascience.com/exploratory-data-analysis-eda-a-practical-guide-and-template-for-structured-data-abfbf3ee3bd9


where to store embeddings for similarity search?

There's Milvus search engine that utilizes several prominent Approximate KNN libraries such as FAISS, ANNOY and HNSW. It also handles several bookkeeping, clustering, data integrity and other tasks that you probably don't want to handle yourself. All for a performance price ofc, but if you don't want to pay it, you can always pick one of the "barebones" libraries.


Loading own train data and labels in dataloader using pytorch?

Assuming both of x_data and labels are lists or numpy arrays,

train_data = []
for i in range(len(x_data)):
   train_data.append([x_data[i], labels[i]])

trainloader = torch.utils.data.DataLoader(train_data, shuffle=True, batch_size=100)
i1, l1 = next(iter(trainloader))
print(i1.shape)


Effect of Skewness and data range in machine learning

Typically, folks would transform the variable. When it is strictly greater than zero, a log transform is usually sufficient. If zero is included, as in your case, one popular alternative is the box-cox transformation.


Compare cross validation and test set results

At first sight the steps seem to be correct, nevertheless:

you did not tell how you split your dataset intro training and test sets, in case it has some influence in your final score values
you might be interested in the sklearn GridSearchCV option , with which you carry out cross validation similar to what you did but you do not need to manually refit the model on the whole train dataset, because you have the option with the refit parameter (choose the scoring metric considered so as to select the best model from the grid search cross-validation process already refit on the whole train dataset wothout splits), as follows:
Basically, the steps for your example could be:

split your dataset to leave some rows out for a final validation

apply grid search cross-validation on the rest of the data (your train set), something like:
dt_clf = DecisionTreeClassifier(random_state=0, class_weight="balanced")
hiperparams_search_space = {'criterion': ["gini", "entropy"], 'max_depth': [4,
5, 6, 8], 'min_samples_leaf': [2, 3, 5]}
dec_tree_cross_val_clf = GridSearchCV(dt_clf, hiperparams_search_space, cv=10,
scoring=['accuracy', 'recall', 'precision', 'roc_auc'],
refit='recall',return_train_score=True, n_jobs=-1)
dec_tree_cross_val_clf.fit(X_train, y_train)
where for each scoring metric, you would have train and test values across your k-folds in a dataframe via dec_tree_cross_val_clf.cv_results_:

use clf.best_estimator_ to make predictions on your validation set to calculate a final score metric:
dec_tree_cross_val_clf_best_est = dec_tree_cross_val_clf.best_estimator_


sklearn.model_selection: GridSearchCV vs. KFold

Yes, you can replace the cv=5 with cv=KFold(n_splits=5, random_state=None, shuffle=False). Leaving it set to an integer, like 5, is the equivalent of setting it to either KFold(n_splits=5) or StratifiedKFold(n_splits=5), depending on the model you pass to the estimator parameter of GridSearchCV()


Hyperparameter tuning in multiclass classification problem: which scoring metric?

You should use the same metric to evaluate and to tune the classifiers. If you wull evaluate the final classifier using accuracy, then you must use accuracy to tune the hyper parameters. If you think you should use macro-averaged F1 as the final evaluation of the classifier, use it also to tune them.
On a side, for multiclass problems I have not yet heard any convincing argument not to use accuracy, but that is just me.


This is an anomaly detection problem. In your case, we would refer to it as supervised anomaly detection problem as you have the labels of categories.
This typically involves taking a large "normal" dataset, in this case, this would be receipts which are valid. And then using a machine learning method to learn features from this dataset (e.g. the typical words used, normal amounts entered, etc.) to generate a model of the "normal" data.
You could either go down the Computer Vision direction (e.g. for physical features in the photos) or down the Natural Language direction (e.g. for textual features in the receipts), or perhaps any other features available!
Once you have this model of "normal" behaviour, you can then test it against fraudulent receipts by predicting whether the receipt is normal.
But remember, the model will only be as good as your catalogue of "normal" and "abnormal" data


short answer: assess simple models first, and then build more complex ones if necessary.

For a roadmap, I'd go like this:

get familiar with the different categories of machine learning problems: classification vs regression problems. Your problem falls in the regression category (at least at first sight, you could also reframe it into a classification problem if necessary)
get familiar with the main corresponding algorithms
get familiar with the strategies to train / test a model performance
since you are explicitly asking for online resources, you can have a look at Andrew Ng's machine learning course (especially the first weeks of the course). It's not free, but you can also have a general overview buy looking for his Machine Learning Yearning Book. Another resource coming to my mind is Machine Learning Mastery website (can't remember the guy's name) where you can find some introductory ebooks to answer those questions. It's well written and can provide you a good lift

Does this help?


Which model is better for incremental learning?

I will say, it's an either Or situation
You can pick one of "Incremental/Online" training Or "addition of new class".
 You may do a fine-tuning approach with a Neural network by adjusting the o/p layer and training the last few layers. But this approach expects the new data to be quite similar to the training set.
KNN - Can do the online stuff but it doesn't do training. It simply calculates all the distances at the time of prediction. So, no reduction in computing. But you might have to compromise on accuracy if it is not the best
Scikit-Learn SGD Classifier can help with online training but can't support new classes

For classification, a somewhat important thing to note is that although a stateless feature extraction routine may be able to cope with new/unseen attributes, the incremental learner itself may be unable to cope with new/unseen targets classes.
SGDClassifier

Other models e.g. SVM/DT doesn't support incremental learning naturally. Though there are suggested approaches on the internet. But may not be simple. See these references.
journalofbigdata$\hspace{1cm}$
A Good SE read
I am not sure how will you figure it out if it is a new Class unless you have a separate arrangement because the model will predict it an existing Class anyway. Assuming you have a setup for that. We may try below approach -
Till the time, the prediction is within "known class" - do an Online training.
When the data is for a "New Class", do a full-data training.
 With this approach, you can reduce the frequency of full-data training.
A simple Neural network can be a good candidate for both purposes. You can also get class probability with it.
Please see these useful links too
Creme
Keras + online learning


how to retrain model with periodic new features?

The way gradient boosting is constructed and trained, there is not an obvious solution for this without just training from scratch. Other models might be more suitable for this (Adding this to a neural network and retraining this will take less time than from scratch I think). Another approach would be to use these time-based lag features as a time series, that way you just input your features and it will learn relationships between the past few weeks instead.


How to impute using simple imputer (custom function)

Since you want to save the training min/max and use those to replace inf's in the test set, you need a custom transformer.  To build a robust transformer, you should use some of sklearn's validation functions.  And it's best to work in numpy, since as you point out an earlier transformer in a pipeline will have already converted an input dataframe to an nparray.  (You could stick with dataframes, and either convert in your transformer (losing some efficiency), or make sure your transformer always comes first in a pipeline (alright if you're the only one using your code).)

Then, here's a simple version.  I'm opting not to have the numeric_features as input; you're probably better off using a ColumnTransformer to do that kind of selection for you.

from sklearn.utils import check_array
from sklearn.utils.validation import check_is_fitted
from sklearn.base import TransformerMixin, BaseEstimator

class ReplaceInf(TransformerMixin, BaseEstimator):
    '''Replace +-np.inf with the max/min finite values in each column.
    Attributes
    ----------
    mins_ : np.ndarray
        Per-column minimum finite values.
    maxs_ : np.ndarray
        Per-column maximum finite values.
    '''

    def fit(self, X, y=None):
        # validate and convert if possible:
        X = check_array(X, force_all_finite=False)
        # using `where=np.isfinite(X)`, nan's won't affect the min/max calculation,
        #  and using `clip` to transform will preserve nan's as well.
        self.mins_ = np.amin(X, axis=0, where=np.isfinite(X), initial=np.inf)
        self.maxs_ = np.amax(X, axis=0, where=np.isfinite(X), initial=-np.inf)
        return self

    def transform(self, X):
        X = check_array(X, force_all_finite=False)
        return np.clip(X, self.mins_, self.maxs_)


Is it possible to predict groups vs groups in machine learning?

I think, you need to look carefully through popular ranking algorithms. I would suggest to start from Elo, at a glance, it could help you to solve the problem. I worked only with "classical" "1-vs.-1" version. But here are some thoughts on your case.

Consider every bundle as a team of two players; every customer's choice as a duel. The main issue here would be the "teams randomness", as you have no constant team compositions. But you can try to solve it, using average ratings of team members. With the large number of duels and "teams" you will get enough data to predict the outcome for brand new mixtures of fruits.


Cross-validation for model comparison: use the same folds?

There seem to be a few elements of confusion here. I'll try to answer clearly, but would definitely appreciate input from others.

1. What is cross-validation for?

Cross-validation is generally used as a mechanism for figuring out which of several models might be a better fit for data based on a training set before choosing a model to apply to the test set. This is commonly done when tuning hyperparameters (such as penalty factors in regularized regressions), and in other scenarios as well.

In such a case you would want to use the same folds for cross-validating each model, because you want to identify differences in performance attributable to the models themselves, and not due to effects of randomly partitioning observations into different folds. Importantly, cross-validation is part of the model training process, not final model evaluation.

Cross-validation is not for finding the "best" performing model overall. Model performance is based on application to a held-out test set of observations. The idea is that the model performance will often be artificially high in the training data due to idiosyncrasies among the training data set which are not present in observations overall, and application to the test data suggests how the model might generalize to totally novel observations. Even if cross-validation results are discussed in a study, they are not the appropriate value to compare a model prepared independently and trained on different data.

2. During cross-validation, how would one assess which putative model is "better"?

This question can be a bit controversial, namely in that accuracy (such as of classification) is not a very strong method of identifying the best model during cross-validation. It will work with varying degrees of quality depending on what you're modelling. But better measures are varieties of loss functions, which could be mean squared error, binomial deviance, or many others (it's very situation dependent).

How you would compare the output of those loss functions is usually fairly straightforward-- you want minimum error, minimum deviance, etc. Performing a paired t-test isn't really appropriate. Such a test would tell you something about the distributions underlying your variables for data points that are connected in time (like measures of some variable before and after a treatment is applied) in the context of a specific null hypothesis and alternative hypothesis set. That sort of paired relationship doesn't apply to observations subjected to different models one-at-a-time, and in a mutually exclusive way.

3. What should be done instead?

As outlined above, model performance is assessed on novel data rather than training data. In model development that will be the test set, but it can just as well be application of the model to totally new data. If the models to be compared are similar in complexity, a proper loss function is still ideal. But in many situations classification accuracy is necessary (if that's what your boss is asking for, that may be what you have to provide).

In that case simpler counts of correct classification (9 out of 15 were correctly classified) are appropriate. Better still are measures of model sensitivity and specificity (the model's ability to identify true positives and true negatives, respectively, as in binomial classification) and positive/negative predictive value (how reliable any given classification suggested by the model is) can be very effective.

tl;dr: Cross-validation is the wrong stage to be comparing an already-trained model to a different model that is being trained, and t-tests are not informative in comparing cross-validations. Loss functions or classification accuracy are more appropriate (to varying degrees in varying situations) and should be based on data not used in training.


Poor performance shown on Rare event modeling

If you are willing to use the caret package in R and use random forests, you can use the method in the following blog post for downsampling with unbalanced datasets: http://appliedpredictivemodeling.com/blog/2013/12/8/28rmc2lv96h8fw8700zm4nl50busep

Basically, you just add a single line to your train call. Here is the relevant part:

> rfDownsampled <- train(Class ~ ., data = training,
+                        method = "rf",
+                        ntree = 1500,
+                        tuneLength = 5,
+                        metric = "ROC",
+                        trControl = ctrl,
+                        ## Tell randomForest to sample by strata. Here,
+                        ## that means within each class
+                        strata = training$Class,
+                        ## Now specify that the number of samples selected
+                        ## within each class should be the same
+                        sampsize = rep(nmin, 2))

I have had some success with this approach in your type of situation.

For some more context, here is an in-depth post about experiments with unbalanced datasets: http://www.win-vector.com/blog/2015/02/does-balancing-classes-improve-classifier-performance/


According to your description you can only use similarities between descriptions, and since there's no labelled data it has to be unsupervised.

Option 1: heuristic (i.e. ad-hoc unsupervised method). Based on your knowledge of the specifics of the data, implement a function which returns a score representing how similar two descriptions are. For example a basic matching could simply count the number of common items between the two descriptions divided by the number of items in the longest description. For a query description return the N top similar matches after comparing it to every potential match. Naturally this can be improved in many ways.
Option 2: represent each description as a vector and use any generic similarity measure over pairs of vectors (e.g. cosine).  The representation can be some kind of direct representation, e.g. some kind of TF-IDF vector, or a more sophisticated  representation as a tree or graph embedding.


High performance R libraries for interactive graphics

My current go-to are some graphics from Rstudio's flexdashboard and a set of relatively well-maintained packages:

  Leaflet, a library for creating dynamic maps that support panning and
  zooming, with various annotations like markers, polygons, and popups.

  dygraphs, which provides rich facilities for charting time-series data
  and includes support for many interactive features including
  series/point highlighting, zooming, and panning.

  Plotly, which via its ggplotly interface allows you to easily
  translate your ggplot2 graphics to an interactive web-based version.

  rbokeh, an interface to Bokeh, a powerful declarative Bokeh framework
  for creating web-based plots.

  Highcharter, a rich R interface to the popular Highcharts JavaScript
  graphics library.

Source: Flexdashboard Components


How can I run Labeled LDA over one textual document?

A python package that you can check. It's an online max-margin topic model.


How pairplot is constructed? Based on what rule? Why people use it?

Differences:

Pairplot

If you have m attributes in your dataset, it creates a figure with (m)x(m) subplots.

The main-diagonal subplots are the univariate histograms (distributions) for each attribute.

For a non diagonal subplot, assume a position (i,j). This plots all the samples of the dataset on a coordinate system with axes the attributes i and j. In other words, it projects the dataset on these two attributes only. This is particularly interesting to visually inspect how the samples are spread with respect to these two attributes ONLY. The "shape" of the spread can give you valuable insight on the relation between the two attributes. For example, if the samples are spread on a linear pattern, it means that the two attributes are linearly related in your dataset. But you can also visually recognize nonlinear relations, such as quadratic, etc.

Correlation (Pearson)

It is a mathematical measure of strictly LINEAR correlation between two attributes. The closer it is to 1 or -1, the stronger the linear relation. The closer it is to 0, the weaker the relation. If there is a value close to 1 or -1, then you can safely assume that these two variables are linearly connected. If it is close to 0, you can assume the absence of linear relation BUT you cannot infer anything about nonlinearities (they might exist even if correlation is very low).

The main-diagonal elements of the correlation matrix show the autocorrelation of each attribute, thus a measure of its normalized spread around its mean value.

The non diagonal elements (i,j) show the linear correlation between the attributes i and j. Usually, correlation matrix is plotted on a heatmap (seaborn) that helps with the visualization.

Note

If you use Pearson correlation and you find that for the selected attributes (i,j) the value is close to 1 or -1, you should expect the samples forming a straight line in the (i,j) and (j,i) subplots of your Pairplot.


Welcome to the community!

I would use the Mutual Information between two sets as the criterion of similarity. In your special case, set the mutual information of those sets whose intersection is less than X% to $0$. Run your clustering and see what comes out. With the same similarity matrix, you can also construct a weighted graph and cluster nodes (which is called Community Detection). This link gives you a proper overview starting from slide 37 and in slide 48 you find your own idea! If you have more questions about Community Detection, e.g. implementation, please drop a comment here.


Random Forest is a very specific technique, is itself a variation of other techniques.

In this link you will find a list of techniques related to random forest:

Forest of randomized trees


Prediction on timeseries data using tensorflow

The error is caused by this line:

print('epoch',epoch,'MSE=',mse.eval())

This happens because the tensor mse also depends on the placeholders X and y. One way to fix this would be to change the training loop to be:

for X_batch,y_batch in generate_batch(input,output,batch_size):
     mse_val, _ = session.run([mse, training_op],feed_dict={X:X_batch,y:y_batch})
     if epoch % 10 == 0:
         print('epoch',epoch,'MSE=',mse_val)

Also you will need to switch X back to tf.float32 since tf.matmul is not compatible with int and float. The data will automatically be casted once you feed it in.

To add a bias variable, you can do it similarly to how you define theta.

b = tf.Variable(0.0, dtype=tf.float32, name='b')
...
y_predictions += b


Let's assume that we are talking about 2D convolutions applied on images.

In a grayscale image, the data is a matrix of dimensions $w \times h$, where $w$ is the width of the image and $h$ is its height. In a color image, we normally have 3 channels: red, green and blue; this way, a color image can be represented as a matrix of dimensions $w \times h \times c$, where $c$ is the number of channels, that is, 3.

A convolution layer receives the image ($w \times h \times c$) as input, and generates as output an activation map of dimensions $w' \times h' \times c'$. The number of input channels in the convolution is $c$, while the number of output channels is $c'$. The filter for such a convolution is a tensor of dimensions $f \times f \times c \times c'$, where $f$ is the filter size (normally 3 or 5).

This way, the number of channels is the depth of the matrices involved in the convolutions. Also, a convolution operation defines the variation in such depth by specifying input and output channels.

These explanations are directly extrapolable to 1D signals or 3D signals, but the analogy with image channels made it more appropriate to use 2D signals in the example.


How to automatically verify official documents?

Please refer the below link which is related to Machine learning in document analysis and recognition.

https://www.researchgate.net/publication/242506468_Machine_Learning_in_Document_Analysis_and_Recognition


How to treat sparse categorical features in a Neural Network for multiclass classification with Tensorflow?

Having more features than batchsize it's not a problem indeed is the rule in many fields like computer vision. Having a very sparse dataset is also the norm in NLP. Both communities rely on dropout to prevent overfitting.

You may see an example of using drouput with Tensorflow here.


You should consider hierarchical K-Means Clustering.

Link to show an idea:

https://www.datanovia.com/en/lessons/hierarchical-k-means-clustering-optimize-clusters/


How do you think about neural networks and ways to design new models?

You are right to guess that there is no solid theoretical way to design neural models. This is in part consistent with the free lunch theorem which says that there is no universal algorithm that can solve all problems efficiently.

The performance of neural models not only depends on the idea but also on the data. Batchnorm has a valid reason for normalising the input to each hidden layer in neural networks but it doesn't always improve the performance of models. One can find the limitations of current neural architecture for a given data and can come up with a fix that might improve the performance but it doesn't always mean that it will prove to be better.

As of now the field is more or less driven by an empirical approach rather than a theoretical one when it comes to designing new neural architecture.


Running out of memory while processing csv file data

Like @Sagar said, you could convert your pipeline to pyspark (So Spark with python API), and you can set your memory usage to not go above 1G of RAM for example and this will be faster because of the parallelization.

You can try to use generator with Tensorflow, that will fit back and forth your data so it never explode your RAM.

If the issue is before, like getting the data into a variable, you could use Dask. Which is simply a distributed Pandas that helps you fit your data in RAM by processing them by batch


Which machine learning algorithms are hard real time capable?

making them suitable for the use in hard real time systems

Not sure what you mean by "hard" real time systems, but what you describe is basically the context of data streams. That is a whole branch with its own algorithms. Since you don't specify much, I leave the answer here, pointing out what kind of algorithms you should research.

Hope this helps!


Best way to generate artificial handwritten text?

Found some implementation of lstm-based handwriter. Maybe I will use some parts.

https://github.com/sjvasquez/handwriting-synthesis


Why LSTM models do not require labels for each step?

Time-series analysis has two main goals:

To identifying the nature of the phenomenon represented by the sequence of observations, and
To forecast (predicting future values in time for a variable).

NOTE: In this respect, models try to get pattern using historical values. Do not confuse here between forecasting (predicting future values) and prediction.

For time-series prediction(forecasting) problems, the models ( ARIMA, LSTM) try to extract trend, seasonality, and residual from the list of historical values e.g. price from 300 historical dates or time. Therefore, there is no need to look for dependent variables here.
We must be clear, the time series forecasting algorithm extrapolate historical trend to near and far future. They do not predict target variable based on dependent variables. This is the main reason you will see time-series models only work with single variable who's historical data-points become input and future data points become target.

Reference:

http://www.statsoft.com/Textbook/Time-Series-Analysis


How to approach this data set for linear regression?

So in the first approach you want to manually choose some features that seem more important?

If so, I think you might not be able to accurately diagnose the most important features, I think it is better to first choose the most impactful features using an algorithm like backward-elimination, etc to figure out what are the most important features, then ignore the less important columns and split the dataset to training and test error.


How about using the difference of an exponential and a linear or a polynomial functions?
For example, if $x = actual - predicted$, then
$$L = w_1 (e^x-1) - w_2 x$$
The cost of error increases faster for positive values than for negative ones, and is a differentiable function of $x$. You can tweak $w_1$ and $w_2$ depending on your typical errors and your learning algorithm. This is an example, for $w_1 = w_2 = 20$:


How do I find the count of a particular column, based on another column(date) using pandas?

Try this:

df['CountSoldbyMonth'] = df.groupby(['Date','Model']).transform('count')

You don't need to select the third column as you only need the counts of groupby items.


Does running a Decision Tree classifier several times help?

The scikit-learn DecisionTreeClassifier takes a parameter called random_state. If this is set to an integer, your model should produce the same results every time.

The person suggesting you run the model many times would be correct, assuming you allow for no set random state. This means the results should be slightly different every time, because there is some random selection going on in the algorithm. Here is an example from the splitter classes:

# Draw a feature at random
f_j = rand_int(n_drawn_constants, f_i - n_found_constants,
               random_state)

If are are not setting that random state (or any other kind of random seed), I am not sure off the top of my head, why or how the results would always be identical.


How to Deploy a Scikit-learn Model into the Cloud?

If you want to deploy a scikit-learn model to the cloud and be able to access it through an API, I am guessing that you will want to access the predict method.

The API

To build an API with python, the easiest is to use Flask or Django. I prefer Flask for these types of tasks because it has a simpler interface. Django offers you more lower-level control however.

You will have to create some route that the client will call by sending you the input data, and once you receive it on the server side, you'll just have to pass it through your Sklearn model, and send back the output to the client.

Run sklearn in the cloud

The other thing you need to take care of, is run Python on whatever cloud solution you'll use. In order to deploy your Flask/Django app, I would probably use Docker as it's well supported on many platform. Here is an example on how to deploy a Docker container on Azure


Protein interaction prediction- how to input this data structure

https://en.wikipedia.org/wiki/Adjacency_matrix

For such problems, you can tabulate these connections as adjacency matrix and create a network to predict weights for the matrix given some properties of nodes (Say for a social graph; given properties of User1 and User2 [for example Zipcode, school ...] output 1, or 0),

Some examples are :

https://www.biorxiv.org/content/biorxiv/early/2018/01/14/247577.full.pdf
http://kawahara.ca/convolutional-neural-networks-for-adjacency-matrices/

Edit : Illustration of Train_X and Train_Y

Columns A through H form Train_X and Column I is Train_Y.


Turned out to be a dumb question in the end - the noise vector is simply treated as a 1x1 array with "depth" (i.e. number of channels/features) of 100, which can of course be processed by the ConvTranspose2d layer. This reshaping to 1x1x100 just has to be explicitly specified as the first layer in keras:

model.add(Reshape((1, 1, 100), input_shape=(100,)))


Spatial Data Mining and ATM banking transactions

This sounds like an unsupervised learning problem since you are trying to group observations according to some common association rather than trying to predict a target. My first impression is that you are facing an uphill battle here as your data set doesn't look comprehensive enough. As a starting point, this is what I would try:

You could assume that ATMs in the same time zone would exhibit similar seasonality patterns on both daily and weekly scales. For example, on a typical weekday, there would probably be a consistent pattern of an up-tick in usage around say 8am, probably peaking during the lunch break with another peak following the end of the workday (these are just assumptions, I haven't done any analysis on this). So you could convert your TIME column into a factor with 24 levels (one for each hour of the day), but you would need to ensure that the times reported are in some consistent time zone, as opposed to the local time of the ATM from which they come which would of course not make much sense.

That approach may allow you to cluster on the basis of time zone, but what about city? Well, if you assume that different cities have different levels of affluence, the AMOUNT column might yield some useful associations.

So in summary, I'd try a clustering algorithm based on seasonal usage patterns and the value of the transactions. I'd be interested to hear if you can get any useful results from this as my feeling is that your data set probably doesn't have enough useful features to be effective.


How to use ARIMA to predict time series?

ARIMA models try to capture autocorrelation in your time series in order to make predictions. It is composed of three main models: AR, MA and I. So, you build your model to predict future values based on a linear combination of past values, linear combination of past errors and a differencing term (I) that accounts basically for a trend. You should get more information here: Forecast: Principles and Practice
My question is: why do you believe the tank volume is predicted by its own past values? When you ask how to use ARIMA, it is straight forward, however, I am concerned that you are not think properly about your problem. Isn't there an exogenous variable you could use? Maybe try a different model? A physical one?
If you really want to use ARIMA, you need to check if your data is not a random walk, plot autocorrelation and partial autocorrelation functions to see analyze which coefficients are more appropriate for AR and MA terms. Check for stationarity. Or simply use autoarima although I don't recommend doing that for the first time.


generate graph from .eps file (preferably using R)

EPS is "Encapsulated PostScript". Its meant for embedding like an image in documents, or sending to printers. You can view it with a PostScript document viewer, and there are free PostScript document viewers for Linux, Windows, and Mac OSs. Ghostview, Evince, etc etc.

So although you can view the graphic once you've got a PostScript document viewer, you cannot load it into R as if you had just plotted it.


How to get Euclidean Distance between feature maps

The Euclidean between two images $p$ and $q$ can be calculated as follows:
$d(p, q) = \sqrt{(q_1 - p_1)^2 + (q_2 - p_2)^2 + ... + (q_{49} - p_{49})^2}$
which is the distance between the 49 (7x7) features of the two images.
This should then give you a vector of shape (1024, 1) where each value is the Euclidean distance of the feature maps of the previous image, with the first one being all NA since it's the first image. Then a convolution is applied with a window of 4 with the vector/kernel [0.1, 0.1, 0.1, 0.99], which basically multiplies 4 values of the (1024, 1) vector by the kernel values and adds them up. I.e. given the first 4 values of the (1024, 1) vector of [0.3, 0.5, 0.2, 0.4] the result of this multiplication would be:
$0.3 * 0.1 + 0.5 * 0.1 + 0.2 * 0.1 + 0.99 * 0.4 = 0.03 + 0.05 + 0.02 + 3.96 = 4.06$


Feature selection and classification accuracy relation

feature selection involves several approaches just like methods for machine learning. Idea is to keep most relevant but not redundant feature for predictive model that can yield optimal accuracy.

In your case, I can not see which method you are using for feature selection but assuming that you are not taking account of multivariate nature of feature dependency. Say you have N features, likely reason that your model accuracy drops after n top feature(s) but improves by adding n+k (where n < k < N when features are in descending order based on information gain) is due to inter-dependency (more relevance and less redundancy) of top n and k features. Univariate feature selection does not necessarily get optimal model accuracy when features are inter-dependent and not mutually exclusive. From philosophical point of view, set of optimal features is analogous to a quote by Aristotle: "The whole is greater than the sum of its parts"!

For optimal feature selection, I often is Caret package in R language where one may do feature selection using recursive feature elimination (RFE) among several other approaches. There is also a package called mRMRe to do feature selection based on maximum relevance, minimal redundancy.

Best,
Samir


Considerations for multiple hidden layers in neural networks?

You generally won't know ahead of time how many layers to use for a particular project, so you'll have to try a variety (1-5 layers, etc). However, one key constraint that will provide an upper bound is that you don't want so many layers that your model starts to overfit. 10,000 records is a very small dataset, so I probably wouldn't try more than 3 hidden layers.

Some problems (e.g., object recognition in images) can have more layers depending on the design of the network. For example, using convolutions on images reduces overfitting, so you could have more layers. However, convolutions probably won't work on financial data. Convolutions work best on spatial data. Lastly, the number of layers is independent of whether the output is classification vs regression.


Suggestions for labeling data for named entity recognition

While labeling, label with sub-categories. This enables you to work with larger set of tools / algos.

With Algos that work better with linear labels (and don't know about label hierarchies), you can pre-process to transform and use only least granular labels (E.g.: CoreNLP)
With some of Deep Learning models, you can use sub category tags as it is.

Since tagging / labeling is expensive and (potentially one-time) exercise; it is better to tag more granular categories.


Let me give you an example where Andrew's recommendation works better than yours:

Let's say that the real gradient is $(0, 0, 0)$ and the gradient you have computed is $(10^{-4}, 10^{-4}, 10^{-4})$. Then your average would return $10^{-8}$, and Andrew's recommendation would return $1$. Your metric could fool you into thinking that your gradient is computed propperly and the error is just due to a numeric issue, while Andrew's cannot fool you into that, due to the fact that it considers the fact that the gradient can be very small.

To wrap up, if your gradient doesn't have norm close to zero, it wouldn't really matter. However, when the gradient is close to zero you can be fooled into thinking that your gradient is right when it is not.


AttributeError: 'DataFrame' object has no attribute 'ix'

From the docs:

Warning: Starting in 0.20.0, the .ix indexer is deprecated, in favor
of the more strict .iloc and .loc indexers.

Just use .iloc instead (for positional indexing) or .loc (if using the values of the index).
To read more about loc/ilic/iax/iat, please visit this question on Stack Overflow. To quote the top answer there:

loc: only work on index iloc: work on position ix: You can get data
from dataframe without it being in the index at: get scalar values.
It's a very fast loc iat: Get scalar values. It's a very fast iloc
http://pyciencia.blogspot.com/2015/05/obtener-y-filtrar-datos-de-un-dataframe.html
Note: As of pandas 0.20.0, the .ix indexer is deprecated in favour of
the more strict .iloc and .loc indexers.


Determine document novelty/similarity with the aid of Latent Dirichlet allocation (LDA) or Named Entities

Short answer: Topic Models and LDA aren't suited at all for novelty detection. I did some experiments with data that show a succession of topics in temporal order, and at the boundaries of the time interval one topic gets really high percentages in the document-topic matrix, i.e., the upcoming new trend (or the outrunning old trend) is not seen by LDA.

Topic Models are also not really suited for "online" processing—you read the whole corpus of documents once and than infer the topics.


What are good sources to learn about Bootstrap?

A classic book is by B. Efron who created the technique:

Bradley Efron; Robert Tibshirani (1994). An Introduction to the Bootstrap. Chapman & Hall/CRC. ISBN 978-0-412-04231-7.


Autoencoder for anomaly detection from feature vectors

Does that mean that my model (or indeed my approach of using an AE) is ineffective

Well, it depends. Auto Encoder are a quite broad field, there are many hyperparameters to tune, width, depth, loss function, optimizer, epochs.

  How should I proceed?

From my gut feeling I would say that you don't have enough data to train the AE properly. Keep in Mind, the MNIST database contains 50,000 image. And you need enough variance in order to not overfit your training data. Tree based approaches are, at least in my experience, easier to train. If you like to stick at the anomaly detection part, which I recommend since you don't know what anomalies you will face, try the Isolation Forest Algorithm. But for a solid recommendation I would need to know how your data looks.

Btw, A good metric to use in such a case is  the ROC score, which basically tells you how likely it is that your model will classify new data points correctly. Check out the link for an visual explanationROC explained

So Baseline is, try less complex approaches until you a certain that they are not sufficient enough.


python///TypeError: unsupported operand type(s) for -: 'list' and 'list' ,i write the code and i get this massege

The problem is x[0] etc are resulting in a 1D list. You need to extract element from 1D list.. below works (I used index 0 element)

area = x[0][0] * (y[1][0] - y[b][0]) + x[b][0] * (y[0][0] - y[b-1][0])


How to check two images(one is original image and other one captured by mobile) similarity using deep learning?

There are a few approaches to this off the top of my head:

1. Trivial case

You can first check if the images are identical by simply using numpy's functions, either numpy.array_equal(A, B) or numpy.allclose(A, B). Where A and B are the two images stored as numpy arrays.

To check if images are semantically similar, you need to compute a similarity measure. First let me back up a bit...

2. Latent representations

In deep learning, we can represent data such as images, voice recordings, text etc. in a latent space. This is an unobservable ("not real") space, where we attempt to capture the essence of the data. For words, we might create a 300-dimensional vector that should encode the information that a given word represents. For images, we could use an Auto Encoder.

The idea is to distill the images into a matrix that really contains all the core information in an image. This would be your latent representation of an image,  and this is where you compare images. Here is a schematic of such a model, where the "bottleneck" in the middle is where you have achieved a dense representation of an input image:

The model is trained end to end, by trying to expand that condensed middle section into an array the same as the input array (input image).

You create the latent, encoded representation of two images to be compared by running them just through the first half of your trained model (i.e. just the encoder), and calculate a distance or similarity metric between the resulting arrays. You could perhaps use a simple cosine-similarity (of flattened arrays) or even just the $L_2$ norm (sum of squared differences).

Hopefully that all provides enough guidance and keywords for you to find relevant tutorials :-)


How to model a binary classification problem in a evolving environment

If you are trying to predict something that looks like probabilities, but don't actually need a "real" probability (for instance, if you aim at ranking observations to find those who are the most likely to switch to no), you could probably use logistic regression over the dataset. This would require the model to be refitted every once in a while.

If you are looking for actual probability, it has to be a probability to switch from NO to YES over a period of time. In that case, you are falling into time series analysis. This is a more complicated problem.

My intuition would be to consider the state change as a physical random phenomenon, such as a radioactive decay, where the probability of having decayed over a $\Delta t$ time period is $e^{-\lambda \Delta t}$. Based on cases who have switched in the past, you could build a rgression model to predict a $\lambda$ value for each observation. Of course you can use other decay functions than the exponential (find the best one through the validation step). This will eventually allow predicting a switch probability over any given time period.


What is missing from the following Curriculum Learning implementation in a Deep Neural Net?

You need a lot more information to figure out what's going on here. Some possibilities include:

Your "hard" examples are actually unlearnable. You can check this by seeing if you can at minimum overfit the hard examples, and if you as a human can label the hard examples correctly.
Your network is not large enough to learn the hard examples. That is, you are sure that the hard examples can be learned, it's just that the network isn't sufficiently complex to do it. Again you can check this by seeing if you as a human are able to figure out the correct labels for the hard instances.
Your curriculum learning may actually be hurting be the training. At the end of training, your network is only seeing examples that it has the most trouble with. These examples will cause the largest changes in the network, and you might have short-term gradient explosion. You can check if this is happening by seeing if you are getting significantly worse performance on "easy" examples at the end of training on the hard examples. One solution here might be something like slowly expanding the training set after you sort by difficulty. So if you have 10k easy examples and 10k hard examples, train on the 10k easy + 1k hard until you're satisified with the network performance, then grow the training set by adding the next 1k hardest, etc.


how to calculate recency(RFM) from a given date attribute in R

Try this

as.POSIXct(Sys.Date(), format= "%Y-%m-%d", tz="UCT") - as.POSIXct("11/25/2011", format = "%m/%d/%Y",tz="UCT")

More vectorized operation in your situation

as.POSIXct(Sys.Date(), format= "%Y-%m-%d", tz="UCT") - as.POSIXct(Last_tran, format = "%m/%d/%Y",tz="UCT")


What you need is a pandas.DataFrame.update.

Use pandas.read_csv to read the csv file to a Pandas DataFrame, then do the update. Once you have made the update you can use pandas.DataFrame.to_csv to convert it back to a csv.


Troubleshooting Neural Network Implementation

Are your inputs scaled? Not doing so can cause the weights to immediately blow up.

It's common practice to preprocess your input to be between -1 and 1 or at least in that range. Otherwise you are running the risk of your gradients exploding or vanishing (covered here).


Is There a Way to Re-Calibrate Predicted Probabilities After Using Class Weights?

A more general adjustment for resampling (not just the simple undersampling in your linked paper) exists:

Add $\ln\left(\frac{p_1(1-r_1)}{(1-p_1)r_1}\right)$ to the log-odds of each prediction, where $p_1$ is the proportion of the positive class in the original dataset, and $r_1$ is the proportion of the positive class in the resampled dataset.

Equivalently, multiply the odds by the quantity inside the logarithm.  (Unfortunately, this doesn't lead to a clean adjustment directly to the probabilities.)

Let's do a little rewriting to see the connection to your linked paper.  $1-r_1$ is the proportion of negative classes call it $r_0$, and similarly with $p_1$.  Use capitals $R_1, \dotsc$ to denote the number (or total weight) of samples rather than the proportions, and without subscripts $P,R$ to denote total numbers (or weights) of samples before and after resampling.  So the multiplier becomes
$$\frac{p_1(1-r_1)}{(1-p_1)r_1} = \frac{p_1 r_0}{p_0 r_1} = \frac{(P_1/P) (R_0/R)}{(P_0/P) (R_1/R)} = \frac{P_1 R_0}{P_0 R_1}.$$
In the context of the linked paper, positive class samples are not resampled, so $P_1=R_1$ and the adjustment simplifies to $R_0/P_0$, which is the parameter $\beta$ used in the paper.

Finally, using their equation (4), we check the change in odds:
$$\text{new odds} = \frac{p}{1-p} = \frac{1}{\frac1p - 1} =
\frac{1}{\frac{\beta p_s−p_s+ 1}{\beta p_s} - 1} =
\frac{\beta p_s}{1-p_s} = \beta\cdot\text{old odds}. $$

So, what about weightings instead of resampling?  Well, class_weights might have different effects in different algorithms, but generally the idea is that (positive) integer values of class_weights should correspond to duplicating the samples that many times, and fractional values interpolate that.  So, it should be about the same to use the multiplicative factor above.  Using the size version rather than the proportion version, we should interpret $R_0$ and $R_1$ as the total weights of the relevant classes.

I've not been able to find a reference for this version, so I put together a short experiment; it seems to verify that this shift works.
GitHub/Colab notebook

Finally, this shift in log-odds will fail to produce properly calibrated probabilities if the classifier is poorly calibrated on the weighted data.  You could look into calibration techniques, from Platt to Beta to Isotonic.  In this case, the shift above is probably superfluous.


matplotlib graph to plot values and variance

You have currently stored your numbers as strings causing matplotlib to treat your variable as categorical, hence the y-axis is not ordered as expected. Before plotting you should therefore first convert them to integers like this:

x = [float(i.replace(",", ".")) for i in dev_x]

You can then use plt.plot(x) once again to plot the values, this should give you the following plot:

Edit:

Using the csv file you've provided, I am using the following code to read in the data and create the plot:

import matplotlib.pyplot as plt
import pandas as pd

# Read in csv file
df = pd.read_csv("DATA.csv")
# Set figure size
plt.figure(figsize=(15, 5))
# Create plot
plt.plot(df["DATA"])

This should give the following plot:


If you need to clean your data, you can either drop the rows which contain invalid values, or try to correct them.

Here two examples:

# If you want to change the score, so values below 0
# are changed to 0 and values above 100 are changed
# to 100 you can do that like this:

df['score']= df['score'].clip(0, 100)

# Or alternatively (in case you have more complicated
# operations, you can also use where. For the
# correction of the scores, this would look like

df['score']= df['score'].where(df['score']<=100, 100)
df['score']= df['score'].where(df['score']>0, 0)

# If you want to drop the rows that contain undefined
# subjects, you can do that as follows:

valid_subjects= ['Math','Science','English']
# define an indexer that contains True for all rows which are invalid
invalid_subj_indexer= ~df['subject'].isin(valid_subjects)
# now drop them
df.drop(invalid_subj_indexer.index[invalid_subj_indexer], inplace=True)

The result of this looks like:

   subject  score  ...
1     Math     90  ...
2  Science     85  ...
3  English    100  ...
5     Math     80  ...
6  Science     70  ...

You can test the lines above by executing the following lines first to create the test dataframe:

import io
import pandas as pd

raw=\
"""  subject score ...
1 Math    90    ...
2 Science 85    ...
3 English 105   ...
4 Comp    95    ...
5 Math    80    ...
6 Science 70    ..."""

df= pd.read_csv(io.StringIO(raw), sep='\s+')


Which book is the best for introduction to analysis social network using python3

Russell's book is fine. You might also like Social Network Analysis for Startups. All the examples are in python. You can do all your analysis in that using packages like networkx. NodeXL is for the Excel crowd. Definitely not the ideal tool for the job; I would shy away from it.

The obvious book for NodeXL is Analyzing Social Media Networks with NodeXL, which is written by the authors of NodeXL.


How should one deal with implicit data in recommendation

Your system isn't just trained on items that are recommended right? if so you have a big feedback loop here. You want to learn from all clicks/views, I hope.

You suggest that not-looking at an item is a negative signal. I strongly suggest you do not treat it that way. Not interacting with something is almost always best treated as no information. If you have an explicit signal that indicates a dislike, like a down vote (or, maybe watched 10 seconds of a video and stopped), maybe that's valid.

I would not construe this input as rating-like data. (Although in your case, you may get away with it.) Instead think of them as weights, which is exactly the treatment in the Hu Koren Volinsky paper on ALS that @Trey mentions in a comment. This lets you record relative strength of positive/negative interactions.

Finally I would note that this paper, while is very likely to be what you're looking for, does not provide for negative weights. It is simple to extend in this way. If you get that far I can point you to the easy extension, which exists already in two implementations that I know of, in Spark and Oryx.