Parameter tuning for machine learning algorithms

In general, your approach will get stuck in local minima. This is why it is not scientifically accepted.
(Notice that this may be different in very special cases, in particular if the performance of the algorithm is a strictly convex function of all input parameters).

To see how the approach fails, suppose your machine learning algorithm has two parameters, $x$ and $y$, which can be either $0$ or $1$. The default values are $x=1$ and $y=1.$

The performance of your machine learning algorithm is $f$ and should be as high as possible.
Assume the following performance levels $f(x,y)$:

    | x=0 | x=1
----|-----|-----
y=0 | 0.9 | 0.2
y=1 | 0.1 | 0.3

Your approach would do the following:

First, choose the default value $x=1$ and compute $f(x=1, y=0) = 0.2$ and $f(x=1, y=1) = 0.3$. Second, choose the default value $y=1$ and compute $f(x=0,y=1)=0.1$ and $f(x=1, y=1) = 0.3$.
Change the default values to the best value. In this case, this requires no change since $x=1$ and $y=1$ are the best values, respectively.
The result did not change. Report $(x=1, y=1)$ as the best parameter combination.

But the global performance maximum occurs at $(x=0, y=0).$


Machine learning for predicting HTML Elements on a web page?

There are at least 2 distinct problems:

Understanding the non-programmers intention for the webpage's content
Understanding the webpage's content

The 1st problem is a problem not best solved with data science or machine learning, hard-coded rules will work better. The 2nd problem is mostly a software engineering problem. I would look into existing solutions for inspiration, for example scraper or import.io.

If you still want to sort and present the "selectors" to users, that is most frequently called a "learning to rank" problem. Learning to rank problems are common in information retrieval systems / search engines.


My estimate is that there are around 5.4b numbers in your dataset, all in the range of 0-50 according to your comment. I doubt there are much faster ways to do this on all the data than you are currently doing. However if you would just take every file and get random samples of 0.1-10% of each file and then combine these, you will still get an ecdf that is visually almost exactly the same while reducing memory issues and computing power necessary.


One hot encoding for multiple label(trainy) in .fit() method?

So now I am totally confused how people use one hot encoding for
  target variable in practice? please help me out.

This mostly comes down to the tool you are using. Sklearn, which I assume you are using, does not use one hot encoded target variables. So your y should be of dimension (1600, 1) where the classes are 0, 1, 2 and 3. Instead of applying one hot encoding you can use a LabelEncoder to get it on the correct format.

I suspect the reason for your confusion comes from having seen deep learning frameworks such as Tensorflow and Keras. With them you always one hot encode your target variable.

Short answer:

Using sklearn: Label encode
Using deep learning: one hot encode


Is there a python version of Mitchell's face pose recognition code?

I have created a Notebook (ipynb) file with an implementation of this algorithm.  It is available on Github


Try this.
item_target_enc= df.groupby(['item_id'])['item_cnt_month'].mean().to_dict()

df['item_id'] =  df['item_id'].map(item_target_enc)


Antonym search for expanding search terms

That's an interesting problem.

This according to me is the most comprehensive way(if speed is not a problem. Or you can just pull all these words and create a dictionary/database of your own).

You can this ,
https://wordsapiv1.p.mashape.com/words/love/antonyms (More info about this API at this link)

However, you can restrict results to antonyms with this api.

You can use requests to make API calls.

import requests
import simplejson
response = requests.get(url)
result = simplejson.loads(response)

Then search for the antonyms from the result. If you are getting a big list of antonyms, use only the top n results for your searching.

Although, W2V gives most commonly used words in the context of the keyword, it's hard to guess which is the antonym of the keyword.


How to leverage description data in multi-class classification (dimensionality reduction)

You have textual descriptions, i.e. unstructured data. So you should probably use one of the standard representation methods for text. There are many options including sentence embeddings and this kind of advanced methods, but I'm going to describe the simple traditional option:

Each description value can be represented as a vector of feature, one for each word in the full vocabulary (i.e. over all the values for this field).
The value of each feature can be boolean (i.e. does this word appear in this description) or better a TF-IDF weight for the word.
Obviously this would lead to too many features, so one needs to apply select the most relevant ones. This part is very experimental, you might have to try various options to find the right one:

Get rid of the stop words since they provide no semantic information.
Discard all the words which appear only once, and probably also all the words which appear less than some minimum frequency $N$ (try with $N=2,3,4...). The rationale is that rare words are more likely to cause overfitting than to really help any kind of classification.
Beyond that, you could use general feature selection (e.g. information gain) or feature clustering.

[ obsolete answer to the first version of the question ]

I would definitely try to normalize these values, because semantically they are numerical and in their original form they are almost useless, no matter the method to categorize them. Making them categorical loses a lot of information, especially the ones which actually provide a number. Since some of the strings used are very vague I would probably try using intervals, i.e. two numeric values for every original input value:

three and a half inches, three and 1/2 inches -> min 3.4 - max 3.6
27.6234 inches -> min 27.6234 - max 27.6234
tall -> large range high values
short, kinda short -> large range low values

Normally there are not that many ways to give numbers as words, only a few patterns should be enough to capture all the variants. For the other non-standard cases such as "kinda short" I would start by looking in the data their distribution: if a value is frequent enough (e.g. probably "short", "tall") then manually predefined range. Values which are not frequent can be ignored, e.g. replaced with NA (since they are not frequent that shouldn't affect the data too much).


What if my validation set is worse than my training?

This is a common high-variance problem due to overfitting.

Simply put: Good training accuracy together with low dev set (or in your terminology the validation set) accuracy it means you are expressing your training data very well, whereas your model fails to perform well on unseen data (High-variance Problem).

More general: there are few other combinations of this high-/low-bias together with high-/low-variance that are explained in length in the context of Bias-Variance Trade-off by Andrew Ng in his first Machine Learning course and even more relevant to Convolutional Neural Networks in his last DeepLearning course, see this video. You also easily find many blog posts discussing about these stuffs like this blog post, or this one.

What you see here is quite alright. If fact it is highly recommended practice that you start by a model capable of describing your training set very well first (even overfit). Then if overfit meaning there is a large difference in accuracy between train/dev sets (similar to your scenario), you start penalizing the model (aka regularization). Once again you can find tons of materials on how to regularize CNN models. On the top of your search you will find Dropout, Max-Pooling. See here for example.

I also suggest to pay attention to your data split distributions. It is a very fundamental concept that often paid less attention to. Violating this assumption (aka data split mismatch) could lead to problems that could appear in any forms like large accuracy difference of model on data splits. Maybe check Ng's short video.


Best CNN architecture for binary classification of small images with a massive dataset

It all depends on the dataset, there is no single model can be the best.

I would prefer to try a transfer learning Resnet34 or resnet50 with a custom last layer for the number of class of classification.


An interesting idea would be to train the model with data between 2010 and 2018 and then keep training it every day to keep it updated.

Interesting related works can be found here and here.

Anyway, you need to decide what you want to predict it: do you want a daily output, monthly or yearly?


The equations are not showing quite what you think. For the equations you have copied, when you see $\mathbf{x}^{(i)}$, you should not think of it as

  The ith feature of an example record $\mathbf{x}$ - [INCORRECT]

Instead it is

  The ith example record from the training dataset - [CORRECT]

So, $\mathbf{x}^{(i)}$ in this case is actually a vector including all features of a single example, and the equations show vector arithmetic. The parameter $\theta$ is also a vector.

The individual feature indexing is shown with a subscript $j$ i.e. $x^{(i)}_j$ is the jth feature of the ith record. The equations are also using bold $\mathbf{x}$ for a vector quantity and non-bold $x$ for a scalar quantity to try and make it clearer.

Take care, when you have multiple dimensions of data to work through, different document sources will use different conventions and indexing schemes. After a while, when you know what to expect is going on, you will spot the differences quickly and figure out which convention is in use. This is only a minor problem, and can crop up if you switch learning sources - e.g. watch a video lecture by one researcher then read a book by another.

  Lets suppose we have n sized input data (x1,x2,...,xn) , each input data has m attributes.

There are $m$ examples in the dataset. The data is $(\mathbf{x}^{(1)}, \mathbf{x}^{(2)},...,\mathbf{x}^{(m)})$

There are not $m$ components of each vector. It is not stated how many components there are in the equations.

  In the cost fucntion we basically have an xi from (x1,x2,...,xn), but which one? We just take one at random? And that remains standard throughout the algorithm?

Both equations sum over the whole dataset to calculate cost function or its gradient.


If you don't need it to work in real time you should not worry about your CPU that much.

There are a few models for face detection using Res Net 10, with portability to OpenCV, those might be enough if you the people you are trying to count are facing forward.

Else, you can use a Res Net 10, it runs up to 100 FPS on a Intel i5 7200u, which is not that big deal.

Update

Oh, this Face Detection library has optimization form ARM processors  using Tengine which may help your case. About face/person I've seen a detection algorithm using OpenCV that detects human heads in any position, including the back of the head. If you want to do person count I think that should suffice. I will try to find it and post here.


Are all 110 million parameter in bert are trainable

All these parameters are trainable.
Note that in normal Transformers it is typical to have fixed (non-trainable) positional embeddings, but in BERT they are learned.
Note also the "pooler" component, which is an extra projection that was not mentioned in the paper, but which the authors commented on later.


Feature selection for time series prediction

You don’t need to select variables for feeding to network, deep neural networks (DNN) will do this automatically. Actually DNN gives more importance to relevant variables by setting its weights. After setting the weights, some of the hidden nodes take 0 and some of them take 1 (because of sigmoid function). You can think of this 1 and 0’s as choosing relevant variables, too.

By the way, correlation matrix can not be used to select relevant variables directly. If you want to reduce the number of variables that are fed to DNN, you can use PCA. Actually PCA components are calculated by getting the Eigen-vectors of correlation matrix.


Do larger numbers of hidden layers have a bigger effect on a classification model's accuarcy?

Neural Networks try to approximate the function which maps the given image to its label. Changing the number of hidden layers essentially means changing the number of parameters, which would be used to approximate that input-label mapping. These parameters include weights and biases ( in the case of Dense layers ).
Depending upon the number of parameters, the model might,

Underfit, meaning it has lesser parameters than that required to solve the problem. This would result in a decrease in the model's accuracy over the test dataset.

Overfit, meaning the model has excessive parameters and now it tends to memorize the given training dataset. In this case, the model's accuracy over the training dataset would be much higher than that over the testing data.

Apart from these cases, if the model's loss, as well as accuracy ( or any metric to measure the performance of the model ), seems to improve both on the testing and training data, up to a certain number of epochs, we say that the model has generalized itself.
It depends on the problem you're trying to solve. CNNs like InceptionV3, MobileNets, ResNets have hundreds of hidden layers as they are trained on huge image datasets like CelebA or ImageNet. So, they require millions of parameters to generalize themselves and obtain a better accuracy over the testing data.
You may find a clearer definition of overfitting here.

My question is why is the drop in accuracy between 8 hidden layers and
16 hidden layers much greater than the drop between 1 hidden layer and
8 hidden layers, even though the difference in the number of hidden
layers is the same (8).

Probably the no. of parameters in those hidden layers were just up to the mark for the model to generalize itself. As the no. of hidden layers was increased to 16, the model had excessive parameters and it overfitted the training data.

How do we determine the no. of hidden layers required by a model?

One way to determine is to perform hyperparameter optimization and to search for the best possible combinations of hyperparameters ( learning rate, dropout rate ).


Mapping column values of one DataFrame to another DataFrame using a key with different header names

You can convert df2 to a dictionary and use that to replace the values in df1

cat_1 = [10, 11, 12]
cat_2 = [25, 22, 30]
cat_3 = [12, 14, 15]

df1 = pd.DataFrame({'cat1':cat_1, 'cat2':cat_2, 'cat3':cat_3})

all_cats = [10, 11, 12, 25, 22, 30, 15]
cat_codes = ['A', 'B', 'C', 'D', 'E', 'F', 'G']

df2 = pd.DataFrame({'all_cats':all_cats, 'cat_codes':cat_codes})

rename_dict = df2.set_index('all_cats').to_dict()['cat_codes']

df1 = df1.replace(rename_dict)

If you still have some values that aren't in your dictionary and want to replace them with Z, you can use a regex to replace them.

df1.astype('str').replace({'\d+': 'Z'}, regex=True)


Difference in labelling and normalizing train/test data

Ideally, there should be no information leakage between your training and test sets. You need to scale your test data using the bounds found for the training data. And, you need to calculate the median with respect to your training data. Think as if you'll have the test samples one by one, not as a batch. Then, how would you calculate your test median, and how would you scale it?


Yes and no depending on how you define "good enough samples".
You will likely end up with a chicken and egg problem: you want to use the GAN to generate training data, but the GAN doesn't have enough training data itself to generate convincing enough samples.

Other techniques exist for data synthesis of training images. For example: adding noise, flipping axis, change luminosity, change color, random cropping, random distorsion.


Is there a common strategy to measure if a difference-significance of two areas under two ROC curves

The AUC is a summary statistic of your dataset. As such, if you did the same experiment again you would get a slightly different value - and if repeated several times, you'd get a distribution of values.  You probably want to show that your distribution rarely or never includes 0.5 (indicating random classification). When you have an empirical distribution like this, or a histogram of repeated experiments, you can just count how many of your estimates are $\approx 0.5$.  If none, you're golden.

Since you don't want to do your experiment multiple times, you could 'simulate' that scenario by using samples of your data, but otherwise performing the same process. This is called 'bootstrapping'.  You can use this method to construct confidence intervals around your AUC metric, or equivalently, do a statistical test to see if the metric you get is different from 0.5.

Some packages have bootstrapped confidence intervals built in to the AUC calculation, so you could just use that.


How to train the predicting boxes in a YOLO network?

This is a good question and recaps the thinking of how YOLO works.
So first of all you say your forward path gives you a tensor of
[1, num_bounding_boxes * final_layer_grid_width * final_layer_grid_height, 15].

For simplicity let us reshape this to [grid_width, grid_height, num_boxes, x_center, y_center, bbox_width, bbox_height, object_confidence_score, class1, ..., class10], this means for each Gridcell and each Boundingboxprior you have one value for x_center, y_center, bbox_width, bbox_height.
x_center and y_center predicts a value which is sigmoid activated the offset from gridcell border to the actual midpoint of the bounding box. So b_x = sigmoid(x_center)+ c_x, where c_x is x coordinate of grid Cell, for b_y = sigmoid(y_center)+ c_y, accordingly.
bbox_width and bbox_height are values which are shifts of the BBoxprior in log space so BB_width = BB_prior*e^(bbox_width), for BB_height respectively.
You can think about it this way: If an object is in a grid cell, the object_confidence_score will tell you. So you just have to adjust the Boundingbox. First step is to calculate the bounding box with highest IOU, next step is to adjust it's center with x_center, y_center and the height/widht with bbox_width, bbox_height.
Check the Yolo-V3 Paper for the used formulas and especially figure 2 depicts the parameters you asked for.


What does the "Loss" value given by Keras mean?

In this particular case, the loss is Mean squared error
i.e. the mean of the squares of the differences of your model prediction and the real value of each sample.


What can weka do that python and sklearn can't?

Weka's decision trees are from the Quinlan family, whereas sklearn uses CART.
The most notable difference is that Quinlan trees aren't restricted to binary splits: a categorical column will be split into subtrees for each level.
Another is how missing values are dealt with, but there are some differences in individual implementations, so it's not straightforward to compare the two branches.
https://stackoverflow.com/q/9979461/10495893
TDIDT Decision Trees algorithm
Otherwise, I expect the main real difference is in whether it's easier to deal with python or java.  If you want to extract decision rules, you may be looking to post-process a decision tree; I know of skope-rules to do this in python, but not whether such a thing is easy in weka.


Weka alternative - visual data exploration with a very large dataset

I tried Azure Machine Learning (by Microsoft) and it is very nice. It can easily handle large data sets and has similar to weka pipeline for ML tasks and visualisation. They offer it free for 30 days so you can try it. You can google it or follow this link Azure ML


Terminology - regression with one output and multiple output variables

Question 1. Both. If you think in opposite to multivariate terms, than in univariate regression both input and output variables should be 1-d

Question 2. Multivariate regression where more than one independent variable (predictors) and more than one dependent variable (responses), are linearly related. So input needs to be more than 2 also.


Classifying whether a chess board square contains a piece or not?

If it's a screenshot of a computer game, where each constituent has purely that box, that is by default any given box is white or black. then, any simple matrix operation like, variance of color in that box could be a feature in determining if there is a piece or not.

If it's a hover cam image (Top view), then it's a different problem. you can simply do a kernel multiplication for edge detection, any empty box will have array of 0 as it's output. while it won't be the case if there is a piece.


Hyperparameter search for LSTM-RNN using Keras (Python)

An embedding layer turns positive integers (indexes) into dense vectors of fixed size. For instance, [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]. This representation conversion is learned automatically with the embedding layer in Keras (see the documentation).

However, it seems that your data does not need any such embedding layer to perform a conversion. Having an unnecessary embedding layer is likely why you cannot get your LSTM to work properly. If that is the case then you should simply remove the embedding layer.

The first layer in your network should then have the input_shape argument added with information on the dimensions of your data (see examples). Note that you can add this argument to any layer - it will not be present in the documentation for any specific layer.

By the way, hyperparameters are often tuned using random search or Bayesian optimization. I would use RMSProp and focus on tuning batch size (sizes like 32, 64, 128, 256 and 512), gradient clipping (on the interval 0.1-10) and dropout (on the interval of 0.1-0.6). The specifics of course depend on your data and model architecture.


In short, batch gradient descent is accurate but plays it safe, and therefore is slow. Mini-batch gradient descent is a bit less accurate, but doesn't play it safe and is much faster.

When you do gradient descent, you use an estimate of the gradient to update your weights. When you use batch gradient descent, your gradient estimate is 100% accurate since it uses all your data.

Mini-batch is considered more efficient because you might be able to get, let's say,  an ~80% accurate gradient with only 5% of the data (these numbers are made up). So, your weights may not always be updated optimally (if your estimate is not so good), but you will be able to update your weights more often since you don't need to go through all your data at once.

The idea is that you update your weights more often with an approximation of your gradient, which often is good enough. The utility of mini-batch becomes more obvious when you start dealing with very large datasets.


Why does my KNeighborsClassifier graph look like this?

To summarize from the comment thread: there are two "weird" things going on here.

1. The zig-zag.

As I addressed in the comments, and @BrianSpiering in an answer, this is probably a parity effect, arising from tied votes among the nearest neighbors when $k$ is even.

2. Training accuracy not decreasing (toward test accuracy) with increasing $k$.

This was caused by an incorrect usage of cross_val_score.  You ran that function twice, separately on the training and test sets.  But that means your results were the out-of-fold score on the training set (not the actual training score!), and scores coming from models fitted on (folds of) the test set (not scores on the test set from models trained on the training set).

Probably the cleanest way to approach this is to use GridSearchCV to search over the different values of $k$.  You can then pull the results from cv_results_ for plotting.  Otherwise, if you don't need them for anything else, you can remove either the cross-validation (just score on the train and test set) or the test set (just do cross-validation and get scores on the train and test folds).


I had the same problem. Tried to fix both "likely" causes to no avail. Strangely, everything was working absolutely fine just a few days ago. I changed the Python version from 3.6.1 to 3.6.7 and it fixed the issue. Unfortunately, I don't yet know what the problem was, but it fixed the issue for me.

This is Python's docker image on CircleCi's linux machine. Specifying this in case the problem has to do anything with it.


Where can I find resources and papers regarding Data Science in the area of Public Health

I don't think that you will learn much about data science (meaning, acquire understanding and skills) by using software tools like Tableau. Such tools are targeting mainly advanced users (not data scientists), for example analysts and other subject matter experts, who use graphical user interface (GUI) to analyze and (mostly) visualize data. Having said that, software tools like Tableau might be good enough to perform initial phase of data science workflow: exploratory data analysis (EDA).

In terms of data science self-education, there are several popular online courses (MOOCs) that you can choose from (most come in both free and paid versions). In addition to the one on Udacity that you've mentioned (https://www.udacity.com/course/ud359), there are two data science courses on Coursera: Introduction to Data Science by University of Washington (https://www.coursera.org/course/datasci) and a set of courses from Data Science specialization by Johns Hopkins University (https://www.coursera.org/specialization/jhudatascience/1). Note that you can take specialization's individual courses for free at your convenience. There are several other, albeit less popular, data science MOOCs.

In terms of data sources, I'm not sure what do you mean by "Dummy Data", but there is a wealth of open data sets, including many in the area of public health. You can review corresponding resources, listed on KDnuggets (http://www.kdnuggets.com/datasets/index.html) and choose ones that you're interested in. For a country-level analysis, the fastest way to obtain data is finding and visiting corresponding open data government websites. For example, for public health data in US, I would go to http://www.healthdata.gov and http://www.data.gov (the latter - for corresponding non-medical data that you might want to include in your analysis).

In regard to research papers in the area of public health, I have two comments: 1) most empirical research in that (or any other) area IMHO can be considered a data science study/project; 2) you need to perform a literature review in the area or on the topic of your interest, so you're on your own in that sense.

Finally, a note on software tools. If you're serious about data science, I would suggest to invest some time in learning either R, or Python (if you don't know them already), as those are two most popular open source tools among data scientists nowadays. Both have a variety of feature-rich development environments as well as large ecosystems of packages/libraries and users/developers all over the world.

You might also find useful some of my other related answers here on Data Science StackExchange site. For example, I recommend you to read this answer, this answer and this answer. Good luck!


Detect different "sections" of stationarity in time-series data?

I hear my business forecasting professor in my head right now, suggesting the Cumulative Sum (CUSUM) Control Chart for this. Basically, this is a method borrowed from control/logistics studies that tracks a value over time. If a quantity is stable over time, then the series is "in control", and it falls "out of control" when the quantity starts to deviate systematically (that is, it stops looking like noise).

In the context of time series, you might compute a windowed stationarity metric from $t_{(0)}$ to $t_{(i)}$, then $t_{(i+1)}$ and so on until $t_{(n)}$, and create a CUSUM chart to watch for when the series seems to break stationarity. The same process might be applied to tracking ARIMA model parameters over time.

Some of this thinking has already been done, of course, and it seems that R Bloggers got a relatively nice result pretty quickly by using the changepoint package. You can dig into the details in the links below.

I applied this method to see if health insurance membership changed drastically across the implementation of the Affordable Care Act in the USA. This would answer the question:

  If we had models built on past membership data in 2012, would we have given an acceptable forecast for post-ACA membership?

References:

https://www.r-bloggers.com/changepoint-analysis-of-time-series/
https://cran.r-project.org/web/packages/changepoint/changepoint.pdf
https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc323.htm
https://pdfs.semanticscholar.org/97d5/0c538be1d366f85826d440006d8de65a4f84.pdf


Does class_weight solve unbalanced input for Decision Tree?

Yes you don't need to balance your train data by hand. But your test data can still be (truly) unbalanced.


You need to provide us with more information on your variables, are they categorical or numerical? If usage time is numerical then you can build a linear regression model and the get the coefficients and associated p-values out of it.

If all variables are numerical you could calculate the correlation between them, but the linear model gives more information / is superior.


Always getting value one for a binary classifier

Did you check how many samples are there for each class. I suspect imbalance class problem here. If you have majority of images with class=1. Classifier will be biased towards predicting everything as class 1. Try to balance your classes by either duplicating minor class images or deleting major class images.


How to use Scikit-Learn Label Propagation on graph structured data?

Answering my own question here, as I hope it will be useful to some readers.

Scikit-learn is primarily designed to deal with vector structured data. Hence, if you want to perform label propagation/label spreading on graph-structured data, you're probably better off reimplementing the method yourself rather than using Scikit interface.

Here is an implementation of Label Propagation and Label Spreading in PyTorch.

The two methods overall follow the same algorithmic steps, with variations on how the adjacency matrix is normalized and how the labels are propagated at each step. Let's, therefore, create a base class for our two models.

from abc import abstractmethod
import torch

class BaseLabelPropagation:
    """Base class for label propagation models.

    Parameters
    ----------
    adj_matrix: torch.FloatTensor
        Adjacency matrix of the graph.
    """
    def __init__(self, adj_matrix):
        self.norm_adj_matrix = self._normalize(adj_matrix)
        self.n_nodes = adj_matrix.size(0)
        self.one_hot_labels = None
        self.n_classes = None
        self.labeled_mask = None
        self.predictions = None

    @staticmethod
    @abstractmethod
    def _normalize(adj_matrix):
        raise NotImplementedError("_normalize must be implemented")

    @abstractmethod
    def _propagate(self):
        raise NotImplementedError("_propagate must be implemented")

    def _one_hot_encode(self, labels):
        # Get the number of classes
        classes = torch.unique(labels)
        classes = classes[classes != -1]
        self.n_classes = classes.size(0)

        # One-hot encode labeled data instances and zero rows corresponding to unlabeled instances
        unlabeled_mask = (labels == -1)
        labels = labels.clone()  # defensive copying
        labels[unlabeled_mask] = 0
        self.one_hot_labels = torch.zeros((self.n_nodes, self.n_classes), dtype=torch.float)
        self.one_hot_labels = self.one_hot_labels.scatter(1, labels.unsqueeze(1), 1)
        self.one_hot_labels[unlabeled_mask, 0] = 0

        self.labeled_mask = ~unlabeled_mask

    def fit(self, labels, max_iter, tol):
        """Fits a semi-supervised learning label propagation model.

        labels: torch.LongTensor
            Tensor of size n_nodes indicating the class number of each node.
            Unlabeled nodes are denoted with -1.
        max_iter: int
            Maximum number of iterations allowed.
        tol: float
            Convergence tolerance: threshold to consider the system at steady state.
        """
        self._one_hot_encode(labels)

        self.predictions = self.one_hot_labels.clone()
        prev_predictions = torch.zeros((self.n_nodes, self.n_classes), dtype=torch.float)

        for i in range(max_iter):
            # Stop iterations if the system is considered at a steady state
            variation = torch.abs(self.predictions - prev_predictions).sum().item()

            if variation < tol:
                print(f"The method stopped after {i} iterations, variation={variation:.4f}.")
                break

            prev_predictions = self.predictions
            self._propagate()

    def predict(self):
        return self.predictions

    def predict_classes(self):
        return self.predictions.max(dim=1).indices

The model takes as input the adjacency matrix of the graph as well as the labels of the nodes. The labels are in the form of a vector of an integer indicating the class number of each node with a -1 at the position of unlabeled nodes.

Label Propagation algorithm is presented below.

$$
\begin{array}{l}{
\mathbf{W} \text {: adjacency matrix of the graph} \\
\text { Compute the diagonal degree matrix } \mathbf{D} \text { by } \mathbf{D}_{i i} \leftarrow \sum_{j} W_{i j}} \\ {\text { Initialize } \hat{Y}^{(0)} \leftarrow\left(y_{1}, \ldots, y_{l}, 0,0, \ldots, 0\right)} \\ {\text { Iterate }} \\ {\text { 1. } \hat{Y}^{(t+1)} \leftarrow \mathbf{D}^{-1} \mathbf{W} \hat{Y}^{(t)}} \\ {\text { 2. } \hat{Y}_{l}^{(t+1)} \leftarrow Y_{l}} \\ {\text { until convergence to } \hat{Y}^{(\infty)}} \\ {\text { Label point } x_{i} \text { by the sign of } \hat{y}_{i}^{(\infty)}}\end{array}
$$

From Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. Technical Report CMU-CALD-02-107, Carnegie Mellon University, 2002

We get the following implementation.

class LabelPropagation(BaseLabelPropagation):
    def __init__(self, adj_matrix):
        super().__init__(adj_matrix)

    @staticmethod
    def _normalize(adj_matrix):
        """Computes D^-1 * W"""
        degs = adj_matrix.sum(dim=1)
        degs[degs == 0] = 1  # avoid division by 0 error
        return adj_matrix / degs[:, None]

    def _propagate(self):
        self.predictions = torch.matmul(self.norm_adj_matrix, self.predictions)

        # Put back already known labels
        self.predictions[self.labeled_mask] = self.one_hot_labels[self.labeled_mask]

    def fit(self, labels, max_iter=1000, tol=1e-3):
        super().fit(labels, max_iter, tol)

Label Spreading algorithm is:

$$
\begin{array}{l}{
\mathbf{W} \text {: adjacency matrix of the graph} \\ \text { Compute the diagonal degree matrix } \mathbf{D} \text { by } \mathbf{D}_{i i} \leftarrow \sum_{j} W_{i j}} \\ {\text { Compute the normalized graph Laplacian } \\ \mathcal{L} \leftarrow \mathbf{D}^{-1 / 2} \mathbf{W} \mathbf{D}^{-1 / 2}} \\ {\text { Initialize } \hat{Y}^{(0)} \leftarrow\left(y_{1}, \ldots, y_{l}, 0,0, \ldots, 0\right)} \\ {\text { Choose a parameter } \alpha \in[0,1)} \\ {\text { Iterate } \hat{Y}(t+1) \leftarrow \alpha \mathcal{L} \hat{Y}^{(t)}+(1-\alpha) \hat{Y}^{(0)} \text { until convergence to } \hat{Y}^{(\infty)}} \\ {\text { Label point } x_{i} \text { by the sign of } \hat{y}_{i}^{(\infty)}}
\end{array}
$$

From Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, Bernhard Schoelkopf. Learning with local and global consistency (2004)

The implementation is, therefore, the following.

class LabelSpreading(BaseLabelPropagation):
    def __init__(self, adj_matrix):
        super().__init__(adj_matrix)
        self.alpha = None

    @staticmethod
    def _normalize(adj_matrix):
        """Computes D^-1/2 * W * D^-1/2"""
        degs = adj_matrix.sum(dim=1)
        norm = torch.pow(degs, -0.5)
        norm[torch.isinf(norm)] = 1
        return adj_matrix * norm[:, None] * norm[None, :]

    def _propagate(self):
        self.predictions = (
            self.alpha * torch.matmul(self.norm_adj_matrix, self.predictions)
            + (1 - self.alpha) * self.one_hot_labels
        )

    def fit(self, labels, max_iter=1000, tol=1e-3, alpha=0.5):
        """
        Parameters
        ----------
        alpha: float
            Clamping factor.
        """
        self.alpha = alpha
        super().fit(labels, max_iter, tol)

Let's now test our propagation models on synthetic data. To do so, we choose to use a caveman graph.

import pandas as pd
import numpy as np
import networkx as nx
import matplotlib.pyplot as plt

# Create caveman graph
n_cliques = 4
size_cliques = 10
caveman_graph = nx.connected_caveman_graph(n_cliques, size_cliques)
adj_matrix = nx.adjacency_matrix(caveman_graph).toarray()

# Create labels
labels = np.full(n_cliques * size_cliques, -1.)

# Only one node per clique is labeled. Each clique belongs to a different class.
labels[0] = 0
labels[size_cliques] = 1
labels[size_cliques * 2] = 2
labels[size_cliques * 3] = 3

# Create input tensors
adj_matrix_t = torch.FloatTensor(adj_matrix)
labels_t = torch.LongTensor(labels)

# Learn with Label Propagation
label_propagation = LabelPropagation(adj_matrix_t)
label_propagation.fit(labels_t)
label_propagation_output_labels = label_propagation.predict_classes()

# Learn with Label Spreading
label_spreading = LabelSpreading(adj_matrix_t)
label_spreading.fit(labels_t, alpha=0.8)
label_spreading_output_labels = label_spreading.predict_classes()

# Plot graphs
color_map = {-1: "orange", 0: "blue", 1: "green", 2: "red", 3: "cyan"}
input_labels_colors = [color_map[l] for l in labels]
lprop_labels_colors = [color_map[l] for l in label_propagation_output_labels.numpy()]
lspread_labels_colors = [color_map[l] for l in label_spreading_output_labels.numpy()]

plt.figure(figsize=(14, 6))
ax1 = plt.subplot(1, 4, 1)
ax2 = plt.subplot(1, 4, 2)
ax3 = plt.subplot(1, 4, 3)

ax1.title.set_text("Raw data (4 classes)")
ax2.title.set_text("Label Propagation")
ax3.title.set_text("Label Spreading")

pos = nx.spring_layout(caveman_graph)
nx.draw(caveman_graph, ax=ax1, pos=pos, node_color=input_labels_colors, node_size=50)
nx.draw(caveman_graph, ax=ax2, pos=pos, node_color=lprop_labels_colors, node_size=50)
nx.draw(caveman_graph, ax=ax3, pos=pos, node_color=lspread_labels_colors, node_size=50)

# Legend
ax4 = plt.subplot(1, 4, 4)
ax4.axis("off")
legend_colors = ["orange", "blue", "green", "red", "cyan"]
legend_labels = ["unlabeled", "class 0", "class 1", "class 2", "class 3"]
dummy_legend = [ax4.plot([], [], ls='-', c=c)[0] for c in legend_colors]
plt.legend(dummy_legend, legend_labels)

plt.show()

The implemented models work correctly and allow to detect the communities in the graph.

Note: The propagation methods presented are meant to be used on undirected graphs.

The code is available as an interactive Jupyter notebook here.


How to handle addresses of the restaurants to feed the data-set in the ML model?

You can convert each address to a pair of geographical coordinates (latitude, longitude). In this way, you'll have a rough measure of the distance between different location. This can be done with Google Maps, for example.

I suggest you to use compute Manhattan distance between locations, in order to properly estimate travel times within a city.


Can CNNs detect features of different images?

The point is that parameters sharing is not the same as reusing some learnt weights in a task for another task (a.k.a transfer learning); it is, transfer learning can be used with convolutional neural networks, but parameters sharing does not mean transferring knowledge from one task to another.
For a detailed definition of parameters sharing, look at this documentation from Stanford university , section Parameter Sharing
As explained in that page, you can think of it as a tool for reducing the number of weights while learning in a specific task (so forget about traqnsferring these weights to another task, which could be the case, but not the meaning of this concept).
An intuitive way of reasoning the concept could be:
parameter sharing assumption is relatively reasonable: If detecting a horizontal edge is important at some location in the image, it should intuitively be useful at some other location as well due to the translationally-invariant structure of images


Sampling labeled data for anomaly detection

I guess it depends on how you define an anomaly. If you already know and are sure that everything other than k_max can be defined as an anomaly then sure what you mention makes sense, assuming the fact that there is not a significant overlap between k_max and other classes in your feature space. I would train an auto encoder and learn an error function, classify anything that doesn't fit the error bounds as anomalous.

If you don't know what should be considered an anomaly, I would suggest looking at the Isolation Forrest Algorithm (https://cs.nju.edu.cn/zhouzh/zhouzh.files/publication/icdm08b.pdf) which is a tree based method used for outlier detection. This would allow you to use an unsupervised (untagged with classes) dataset and figure out what should be considered as a 'clean/healthy' example for your anomaly detection algorithm and the outliers can be used as anomalous examples.

Here is another article that you might find useful, trying to do a similar thing I am trying to suggest here: https://towardsdatascience.com/anomaly-detection-for-dummies-15f148e559c1


What is the minimum amount of data required for sales prediction with ML

As @Ethan said there is no general answer to this question. There are multiple perspectives that you need to take in consideration.

Amount of the data: The general tendency is more data will lead to better results.
Variability of the data: A very important aspect is the variability of your data. If you have millions of data points that have almost the same input and output values it is empirically equivalent to a situation in which you only have one single data point.
The timescale of data acquisition: The variability will strongly depend on the relevant timescale of the problem that you are trying to solve. If you are looking at products (e.g. tickets for public transportation) that are sold at a very high frequency the data acquisition rate should be shorter (short/fast time scale). If your data acquisition rate is too small your model will not be able to detect relevant changes e.g. tickets sales during the day. If you are looking at products (e.g. car sales) then your data acquisition rate should be longer (long/slow time scale). In the second case, I would still use the fastest possible data acquisition rate, because we can always reduce the given data. Note, that the data acquisition rate is linked to the variability of the data.
Diversity of the data conditions: This is related to the variability of the data. But the focus here is on the conditions under which your data was collected. For example, imagine a situation in which we have some time series measurements of machines in a production plant during summer. If we know that the process is heavily dependent on the temperature we must also get measurements during spring, autumn, and winter in order to increase the diversity of our data set.

As the data problem is more involved it is difficult to give concrete numbers but the following rules of thumb can be applied. These must always be considered with the previous points.

In statistics, we often have the lower bound for samples to be $30$. This implies that you should never have less than $30$ observations.
For estimating parameters it is often advised to have $50$ (worst case) or better more than $200$ observations for each parameter that you want to estimate. For example, if you need to estimate $5$ parameters that would imply $250$ observations for the worst case and more than $1000$.


Validation set performance increased, test set performance decreased

Don't optimize your model for the test set during development. You shouldn't even consider using your model on the test data until you're done tweaking your it, since the point of the test set is to have a ready example of "unseen" data for your model.

The idea with a train/validation/test split is that you could evaluate all the different iterations your model went through (that you feel like testing) against this completely unseen set of data and not have to worry about whether you were optimizing for fit to this data because it's completely new to your evaluation process.

No decisions were made during development to ensure that any of the tested models would perform well on this set because you're only checking how well you did after you're done fussing with your models. It's as clean a reference point for model performance as you can get.

If the only constraint for your final model choice is performance, then you would pick the model that performed the best on your test data. But I'd highly recommend using the train/validation/test split as intended.


Recurrent neural networks (RNNs) are designed to learn sequence data. As you guess, they can definitely take multiple features as input! Keras' RNNs take 2D inputs (T, F) of timesteps T and features F (I'm ignoring the batch dimension here).

However, you don't always need or want the intermediate timesteps, t = 1, 2 ... (T - 1). Therefore, Keras flexibly supports both modes. To have it output all T timesteps, pass return_sequences=True to your RNN (e.g., LSTM or GRU) at construction. If you only want the last timestep t = T, then use return_sequences=False (this is the default if you don't pass return_sequences to the constructor).

Below are examples of both of these modes.

Example 1: Learning the sequence

Here's a quick example of training a LSTM (type of RNN) which keeps the entire sequence around. In this example, each input data point has 2 timesteps, each with 3 features; the output data has 2 timesteps (because return_sequences=True), each with 4 data points (because that is the size I pass to LSTM).

import keras.layers as L
import keras.models as M

import numpy

# The inputs to the model.
# We will create two data points, just for the example.
data_x = numpy.array([
    # Datapoint 1
    [
        # Input features at timestep 1
        [1, 2, 3],
        # Input features at timestep 2
        [4, 5, 6]
    ],
    # Datapoint 2
    [
        # Features at timestep 1
        [7, 8, 9],
        # Features at timestep 2
        [10, 11, 12]
    ]
])

# The desired model outputs.
# We will create two data points, just for the example.
data_y = numpy.array([
    # Datapoint 1
    [
        # Target features at timestep 1
        [101, 102, 103, 104],
        # Target features at timestep 2
        [105, 106, 107, 108]
    ],
    # Datapoint 2
    [
        # Target features at timestep 1
        [201, 202, 203, 204],
        # Target features at timestep 2
        [205, 206, 207, 208]
    ]
])

# Each input data point has 2 timesteps, each with 3 features.
# So the input shape (excluding batch_size) is (2, 3), which
# matches the shape of each data point in data_x above.
model_input = L.Input(shape=(2, 3))

# This RNN will return timesteps with 4 features each.
# Because return_sequences=True, it will output 2 timesteps, each
# with 4 features. So the output shape (excluding batch size) is
# (2, 4), which matches the shape of each data point in data_y above.
model_output = L.LSTM(4, return_sequences=True)(model_input)

# Create the model.
model = M.Model(input=model_input, output=model_output)

# You need to pick appropriate loss/optimizers for your problem.
# I'm just using these to make the example compile.
model.compile('sgd', 'mean_squared_error')

# Train
model.fit(data_x, data_y)

Example 2: Learning the last timestep

If, on the other hand, you want to train an LSTM which only outputs the last timestep in the sequence, then you need to set return_sequences=False (or just remove it from the constructor entirely, since False is the default). And then your output data (data_y in the example above) needs to be rearranged, since you only need to supply the last timestep. So in this second example, each input data point still has 2 timesteps, each with 3 features. The output data, however, is just a single vector for each data point, because we have flattened everything down to a single timestep. Each of these output vectors still has 4 features, though (because that is the size I pass to LSTM).

import keras.layers as L
import keras.models as M

import numpy

# The inputs to the model.
# We will create two data points, just for the example.
data_x = numpy.array([
    # Datapoint 1
    [
        # Input features at timestep 1
        [1, 2, 3],
        # Input features at timestep 2
        [4, 5, 6]
    ],
    # Datapoint 2
    [
        # Features at timestep 1
        [7, 8, 9],
        # Features at timestep 2
        [10, 11, 12]
    ]
])

# The desired model outputs.
# We will create two data points, just for the example.
data_y = numpy.array([
    # Datapoint 1
    # Target features at timestep 2
    [105, 106, 107, 108],
    # Datapoint 2
    # Target features at timestep 2
    [205, 206, 207, 208]
])

# Each input data point has 2 timesteps, each with 3 features.
# So the input shape (excluding batch_size) is (2, 3), which
# matches the shape of each data point in data_x above.
model_input = L.Input(shape=(2, 3))

# This RNN will return timesteps with 4 features each.
# Because return_sequences=False, it will output 2 timesteps, each
# with 4 features. So the output shape (excluding batch size) is
# (2, 4), which matches the shape of each data point in data_y above.
model_output = L.LSTM(4, return_sequences=False)(model_input)

# Create the model.
model = M.Model(input=model_input, output=model_output)

# You need to pick appropriate loss/optimizers for your problem.
# I'm just using these to make the example compile.
model.compile('sgd', 'mean_squared_error')

# Train
model.fit(data_x, data_y)


How can we calculate AUC for a simple decision tree?

In order to build the ROC curve and AUC (Area under curve) you have to have a binary classifier which provides you at classification time, the distribution (or at least a score), not the classification label. To give you an example, suppose you have a binary classification model, with classes $c1$ and $c2$. For a given instance, your classifier would have to return a score for $c1$ and another for $c2$. If this score is a probability-like (preferrable), than something like $p(c1)$, $p(c2)$ would work. In plain English is translated like "I (the model) classify this instance as $c1$ with probability $p(c1)$, and as $c2$ with probability  $p(c2)=1-p(c1)$".
This applies for all type of classifiers, not only for decision trees. Having these scores you can than compute ROC or AUC by varying a threshold on $p(c1)$ values, from the smallest to the greatest value.
Now, if you have an implementation of a decision tree and you want to change that implementation to return scores instead of labels you have to compute those values. The most used way for decision trees is to use the proportion of classes from the leaf nodes. So, for example you have built a decision tree and when you predict the class for an instance you arrive at a leaf node which have (stored from the learning phase) 10 instances of class $c1$ and 15 instances of class $c2$, you can use the ratios as the scores. So, in our example, you would return $p(c1) = 10 / (10+15) = 0.4$ probability of class $c1$ and $p(c2) = 15/(10+15)=0.6$ probability of being class $c2$.
For further reading on the ROC curves, the best and inspiring source of information I found to be the Tom Fawcett paper called An Introduction to ROC Analysis, it's solid gold on this topic.


As @shadowtalker indicated, a binary features indicating the existence of co-borrower should be helpful anyway. As you certainly know, "imputing missing values" as a procedure for guessing value of missing values is absolutely meaningless as it produces values for something which do not exist. If it's about salary and similar numeric data, just simply put 0 (or the neutral value. Whatever it is.) and if it's about nominal variables, add a new value equivalent to None. at the end, all single customers must have exactly same values for these features.

Also one practical idea could be dividing your data to single borrowers and 2-borrowers at the very beginning of classification so the whole model will become like this:

Above model is somehow equivalent to the feature-selection property of some classifiers such that Decision Trees. In a Bayesian Network, such a feature could be the root node.


How to interpret upper-triangular matrix of cosine similarities

Solution is to transform the matrix:

JavaRDD<Vector> rows = jsc.parallelize(Arrays.asList(
    new DenseVector(new double[]{2, 2}),
    new DenseVector(new double[]{0, 1}),
    new DenseVector(new double[]{1, 1}),
    new DenseVector(new double[]{1, 0}),
    new DenseVector(new double[]{0, 1}),
    new DenseVector(new double[]{2, 1}),
    new DenseVector(new double[]{1, 1}),
    new DenseVector(new double[]{1, 1})
));

RowMatrix mat = new RowMatrix(rows.rdd());
List<Vector> sims = mat.columnSimilarities().toRowMatrix().rows().toJavaRDD().collect();
for(Vector v: sims) {
  System.out.println(v); //(2,[1],[0.8215838362577492])
}


When to use bayesian linear regression instead of linear regression?

The Bayesian approach should be used in the case of:

Strong priors - You have preexisting data and / or domain knowledge that you want to incorporate into the analysis.

Distributional estimates - Instead of point estimates, the result of Bayesian analysis will yield distributions. Those distributions will better quantify the uncertainty of predictions.

In the case of voltage and current for devices, Bayesian linear regression will predict a distribution of current for a given voltage. That is useful because many devices are noisy and possibly the degree of noise changes over voltages.


Higher dimension data visualization in Matlab/Octave

You can use t-SNE to perform dimensionality reduction to create a 2-D or 3-D embedding space for data visualization purposes.

For Matlab you can check the implementation and the example.


BMA is "Bayesian Model Averaged". GNET is "Generalized Elastic Net".

Have you tried reading the Ishwaran and Rao papers as mentioned in the documentation for spikeslab?  There's an article in the R Journal as well that might be worth reading too: https://journal.r-project.org/archive/2010-2/RJournal_2010-2_Ishwaran~et~al.pdf - no sense duplicating it all here.


Mutate with custom function in R does not work

Why don't you solve it in this way?

mutate(gateway, YearlyHit = case_when(Frequency == 'Year' ~ 1,
                                      Frequency == 'Month' ~ 12,
                                      Frequency == 'Week' ~ 48)
)


Mapping of categorical features into binary indicator features

Compactifying the data like this saves space in memory, but it adds false relationships that one-hot encoding doesn't.

Let's consider a categorical feature with levels A, B, C, D, that you decide to encode as 00, 01, 10, 11 respectively.

In a linear model, you only get three parameters (a constant and one for each new feature); you can fit "the right" parameters to hit A, B, and C, but then fixing those parameters determines what happens on D, and may be a very poor fit there.  (You're now assuming essentially that (B-A)+(C-A)=D-A.)

In k-NN, the distance between any two levels in the one-hot encoding is the same, 1.  In this encoding, the distance between A and B is 1, but the distance between A and D is $\sqrt{2}$.

In SVM, the set {A,D} is not separable from {B,C}.  With one-hot encoding, everything is separable.

In a tree model, using the raw categorical allows the tree to split any subset of the levels against the rest; using this binary encoding forces the tree to split (AB|CD) or (AC|BD) [missing (AD|BC)]; using one-hot encoding forces the tree to split (A|BCD) or (B|ACD) or (C|ABD) or (D|ABC).  The tree can split differently subsequently and eventually recover an arbitrary split, but a greedily built tree might never accomplish that.

Notice in particular that in these last three models, we've made A and D "more different" from each other than we might have reason to believe.  And in this small example, the catches were fairly small/few, but as the dimension increases these tend to become more exaggerated.

Now, it may still be the case that it's worth doing, but these are some things to consider.


You might want to look at Siamese CNNs depending on the size of your dataset. A good introduction can be found here.


Conditional Statement to update columns based on range

One elegant way to solve this is by using numpy.select. This function takes a list of conditions and a list of choices and then pick the choice where the first condition is true.

An advantage is that since the conditions are checked in order, only one side of the condition for the day value needs to be checked.

Assuming the input dataframe is called df:

import numpy as np
import pandas as pd

conditions = [
    df['Days'] < -280,
    df['Days'] < -195,
    df['Days'] < -103,
    df['Days'] < 0,
    df['Days'] < 120,
    df['Days'] < 210,
    df['Days'] < 300,
    True
]

outputs = [
    "Plan", "Reach out clients", "Send promotion", "Close case",
    "0-3 Months", "4-6 Months", "7-9 Months", "10+ Months"
]

df['New_Columns'] = np.select(conditions, outputs)

Result:

    ID  Ordered  Days New_Columns
0   1   101565   131  4-6 Months
1   2   202546   122  4-6 Months
2   3   459863    78  0-3 Months
3   4   328453   327  10+ Months
4   5   458975   -27  Close case


My mistake came from the fact that my target variable sample only had one class. To reduce computation time, I took a sample from the main data.


For the line $w^Tx = 0$, how do we know the direction of the vector $w$

I don't see anything in the slides that suggests the line defined by w must have a positive slope. It happens to be in this picture, but w could be any vector.


scikit-learn classification report's f1 accuracy?

I give you that this is a weird way of displaying the data, but the accuracy is the only field that don't fit the schema. For example:
              precision    recall  f1-score   support

           0       0.84      0.97      0.90    160319
           1       0.67      0.27      0.38     41010

As explained in How to interpret classification report of scikit-learn?, the precision, recall, f1-score and support are simply those metrics for both classes of your binary classification problem.
The second part of the table:
    accuracy                           0.82    201329   <--- WHAT?
   macro avg       0.75      0.62      0.64    201329
weighted avg       0.80      0.82      0.79    201329

The accuracy is the overall accuracy of the model (note that accuracy is not a measure that is relative to a certain class, but a performance across all classes). The macro average for the precision and recall score is just the harmonic mean of the two classes. ie:
recall macro avg = (recall_class_1 + recall_class_0) / 2

The weighted average is just the average metric of the two classes weighted by the support/size-of-sample. ie:
recall weighted avg = (support_class_0 * recall_class_0 + support_class_1 * recall_class_1) / (support_class_0 + support_class_1)

This is a pretty long-winded way of saying the accuracy is just the overall accuracy. It has nothing to do with the column f1-score that it happens to be under.


You may don’t need any machine learning for that, just simple math.

Calculate the average use per day, define your desired minimum amount which should be on stock and collect the time for a supplier to send you new products.

After that, you can predict when to order your product from the supplier to always have your desired amount on stock.

Search for order point system :)

With the system you could estimate when a product will be out of stock and when to order to prevent that.


Drop NA values in an imported xlxs document with R

its should be working

data_set<-na.omit(un)

If want to handle NA's to mean or median:

na.mean(un, option = "mean")

Source:
https://www.rdocumentation.org/packages/imputeTS/versions/2.7/topics/na.mean


has number of output layer of DNN any effect in speed of find the optimal answer of DNN?

If you mean by the number of outputs is the number of classes,  then the answer is yes. Increasing the number of outputs will increase the number of parameters that you need to tune. The last set of layers in the dnn are fully connected and they contribute the most in the number of parameters, usually the sizes of these fully connected layers  depend on the number of classes. The more classes the larger these fully connected layers and thus the more parameters


Basic method of optimizing hyperparameters

Hyperparameter optimization follows the same rules as model selection. Each set of hyperparameters effectively represents a different model you are considering, so the data you use to fit the model with some set of hyperparameters needs to be different from the data you use to evaluate which set of hyperparameters you want to ultimately use. A common approach for evaluating hyperparameters is nested cross-validation. This basically means that you need to treat hyperparameter selection as part of your model training process, and when you evaluate your model you evaluate the entire process front to back, i.e. treating hyperparameter tuning as a component of model training with respect to cross-validating over your training process. There's an excellent discussion of this in section 7.10.2 ("The Wrong and Right Way
to Do Cross-validation") of Elements of Statistical Learning, which you can read online and/or download for free. The general idea is that if you're not careful, you can actually overfit to your evaluation data. Play with this demo to see for yourself.

But yes, your intuition is correct. Hyperparameter tuning is often very computationally expensive. One way people sometimes try to minimize this cost is by limiting the search space of feasible parameters to a small discrete set, e.g. grid search. Another approach is to use guassian processes or KDEs to approximate the cost surface in parameter space. You can even use a multi-armed bandit approach.

Frankly, you can really use pretty much any non-linear optimization technique for hyperparameter tuning, as long as you follow the appropriate cross-validation rules. The trick is figuring out bang-for-your-buck in terms of how much time/effort/compute you are willing to expend exploring parameters vs. the potential improvement to your model. Additionally, there are concerns about the limits of repeated cross-validation/holdout evaluations, but that's a whole other rabbit hole.


Does variance and standard deviation both measure how spread out the numbers are?

Yes it is. Standard deviation is a square root of variance. Square root is a monotonic transformation, meaning that it preserves the order, e.g, if a > b then sqrt(a) > sqrt(b), assuming a and b are non-negative and they always are for variance.

Standard deviation is easier to interpret and is more commonly used, when we calculate variance we square every term, taking a square root in standard deviation, we undo that.


Apparently, you need an implementation of the a priori algorithm (or whatever algorithm you want to use) that supports sparse feature vectors.  If one doesn't already exist, you might need to implement it yourself.

Alternatively, you can use the feature hashing trick.  However, beware that it can cause loss of accuracy and loss of interpretability; nothing comes for free.


Find value and return row in excel spreadsheet with over 1 mil rows

the problem of excel being very slow in finding the value and it is
  very difficult to read information in the row

What is "slow" for you (in seconds) and how fast would you like the search to be?

In a lot of cases, the performance can be considerably improved by using INDEX and MATCH to search for values in Excel, instead of using VLOOKUP (which is known to be much slower in comparison).

Alternatively, you could install SQL Server Express on your machine, import the Excel file into a table, and use some SQL statements to find the values you want.

Power BI is a visualization tool, and doesn't seem appropriate for what you're describing which is preparing the data. You could use Power BI after, of course, to plot the final results.


Preparing custom dataset for object detection using ML

I have the following suggestion:

The size of your data-set is small, you should increase the sample size. You also have to use data augmentation methods for improving the performance, translation, orientation and blurring are highly recommended.
For your task I guess the most popular approach is what which is discussed in YOLO paper, and it is highly recommended using that if you want to have object detection not just localization and recognition for good performance.
I also suggest you providing real data for your task. The images of first row are artificially made. You have to provide real data which your classifier is going to face. The distribution of your test time should be like validation and train time if you want to have good performance.


Batch Normalization For Federated Learning

One approach would be to simply average everything, as proposed in the FedAvg preprint.
Some very recent preprints suggest only relaying the learned parameters back to the central server and keeping local batch normalization (BN) statistics separate, as proposed by the SiloBN preprint.
The authors of this paper claim that:

Keeping BN statistics local permits the federated training of a model robust to the heterogeneity of the different centers, as the local statistics ensure that the intermediate activations are centered to a similar value across centers.

To paraphrase them; they distinguish BN statistics as encodings for local domain information whereas the learned parameters are to be domain-invariant. I believe their aggregation method for the relayed information is simply averaging.
Here's their first figure for reference:


Encoder and decoder are highly overloaded terms. As a generic definition, an encoder-decoder neural architecture has a part of the network called "encoder" that receives an input and generates a code (i.e. expresses the input in a different representation space) and another part called "decoder" that takes a given code and converts it to the output representation space. Normally, the dimensionality of the code is much less than that of the input/output representation spaces.

In principle, the architecture of the encoder and the decoder is arbitrary, i.e. they can be CNN's, RNN's, multilayer perceptrons or whatever other thing. This highly depends on the task. This should answer question (1).

There are two contexts where encoder-decoders network organization is usual:

image autoencoders: normally, the encoder is a CNN and the decoder is the analogous deconvolution.
sequence-to-sequence tasks: normally, both encoder and decoder are GRUs or LSTMs.

As you mention word embeddings and a blog post about text summarization, I guess your problem relates to NLP, so you may start with the Keras blog seq2seq tutorial. This should answer question (2).

The appropriateness of using pre-trained word embeddings depends on the task and the amount of training data you have. When you don't have a lot of data, using pre-trained embeddings is a form of data augmentation to get better results. For translation, it is not frequent to see pre-trained word embeddings. This should answer question (3).


How to get pixel location in after rotating the image?

You first need to define where you are rotating your image around.
If you are rotating it around (0,0) (i.e. your image is located in the positive x and y quadrant), you can apply a rotation matrix in its vanilla form:
$\left[\begin{array}{c}x' \\y' \\1\end{array}\right] = $
$\begin{bmatrix}\cos \theta &\sin \theta &0\\-\sin \theta & \cos \theta &0 \\0 &0 &1\end{bmatrix}$
$\left[\begin{array}{c}x \\y \\1\end{array}\right] = $
$\left[\begin{array}{c}x\cos\theta + y\sin\theta \\-x\sin\theta+y\cos\theta \\1\end{array}\right]$
where x and y is your starting pixel location, x' and y' is your new pixel location, and $\theta$ is the counterclockwise angle to rotate (you can ignore the 3rd row for now). Notice that because you are using sine and cosine functions, the new pixel location is unlikely to be an integer, so you will need to do some rounding.
However, if you are rotating around the center of the image (which is the more common use-case), you will need to first translate the "origin" of your image to the center, apply your rotation, then move your image back.
This alters the calculations to this:
$\left[\begin{array}{c}x' \\y' \\1\end{array}\right]$
$=\begin{bmatrix}1 &0 &0\\0 &1 &0 \\X/2 &Y/2 &1\end{bmatrix}$
$\begin{bmatrix}\cos \theta &\sin \theta &0\\-\sin \theta & \cos \theta &0 \\0 &0 &1\end{bmatrix}$
$\begin{bmatrix}1 &0 &0\\0 &1 &0 \\-X/2 &-Y/2 &1\end{bmatrix}$
$\left[\begin{array}{c}x \\y \\1\end{array}\right]$
$=\left[\begin{array}{c}(x-X/2)\cos\theta + (y-Y/2)\sin\theta + X/2 \\-(x-X/2)\sin\theta+(y-Y/2)\cos\theta + Y/2 \\1\end{array}\right]$
where capital X and Y are the size of your image in the x and y directions.

You can read more about image transformations on the mathworks page here.


Why do we need 2 matrices for word2vec or GloVe

Might not be the answer you are seeking, but I'll still have a go:

First, quick review of word2Vec, assume we are using skip gram.

A typical Word2Vec train-able model consists of 1 input layer (for example, 10 000 long one-hot vector), a hidden layer (for example 300 neurons), an output (10 000 long one-hot vector)

Input:  10 000
Hidden:  300
Output: 10 000

There is a matrix E between Input-Hidden, describing the weights to make your one-hot into an embedding. The matrix is special because each column (or rows, depending on your preferred notation) represents pre-activations in those 300 neurons -  a response to a corresponding incoming 1-hot vector.

You don't need to perform any activation on these 300 neurons and can use their values straight away as an embedding in any future task.

However, simply squeezing a one-hot into a 300-dimensional representation isn't enough - it must have a meaning.  And we ensure this meaning is correct using an additional second matrix - which connects Hidden to Output

We don't want to activate a hidden layer because activation-function won't be needed during runtime, however, in that case we will need a second matrix, going from Hidden to Output.

This second matrix will make an entirely different one-hot from your embeding. Such a one-hot will represent a most likely word to be nearby (contextually) of your original one-hot. In other words, this output won't be your original one-hot.

That's why a second matrix is needed.
At the output, we perform a softmax, like in a classification problem.

This allows us to express a relation "word"-->embedding-->"context-neighbor-word"

Now, backpropagation can be done, to correct the Input-Hidden weights (Your first matrix E) - these are the weights we really care about. That's because Matrix E can be used during Runtime (I think), perhaps being plugged as a first fully-connected layer into some Recurrent Neural Net.

In that case you wouldn't use this:

  You don't need to perform any activation on these 300 neurons and can use their values straight away as an embedding in any future task

but instead, you would just grab the appropriate column (or row, depending on your preferred notation) from that matrix, during Runtime. The benefit is that this way you get a very cheaply pre-trained fully-connected layer, designed to work with one-hots. Usually, first layers would be longest to train, due to the vanishing-gradient issue.

Why a second matrix is needed during training:

Once again, recall, there is no activation at the hidden layer.

We can dictate the network what "one-hot" must have been created in response to your original "input one-hot", and can punish network if it fails to generate a correct answer.

We cannot put softmax directly after hidden layer, because we are interested in summoning a mechanism to convert into an embedding. That's already a responsibility of a first matrix E. Thus, we need an extra step (an extra matrix) that will give us enough room to now form a conclusion at the output layer about a different but similar (context-wise) neighbor-word

During the runtime you throw away the second matrix. But don't delete it permanently in case if you need to come back and continue training your model.


Orange reading in new features from Tab-Delimited File

This should usually not happen. It can only happen if there are some stray entries in columns, columns following the penultimate(last) column of your data. Please check. This usually happens while preparing data. Please let me know if I have solved your problem.


Combine Pandas DataFrame Rows Based on Matching Data and Boolean

I believe what you need is agg from pandas. You can pass a dictionary of the different aggregations you need for each column:

import pandas as pd

df = pd.DataFrame({'year':['2017','2018','2019','2019'],
                   'ISO Week':[1,2,3,3],
                  'Price':[5,10,15,20],
                   'quantity':[1,2,3,4],
                  'organic':[True, False, True, True]})

   ISO Week  Price  organic  quantity  year
0         1      5     True         1  2017
1         2     10    False         2  2018
2         3     15     True         3  2019   #<------ combine
3         3     20     True         4  2019   #<------ combine

df.groupby(['year','ISO Week','organic'], as_index=False).agg({'Price':'mean', 'quantity':'sum'})

   year  ISO Week  organic  Price  quantity
0  2017         1     True    5.0         1
1  2018         2    False   10.0         2
2  2019         3     True   17.5         7


How is the validation set processed in PyTorch?

With regards to the backpropagation, since the network is not trained on the validation set and parameters are not updated we do not have to use backpropagation. The calculation of gradients can be disabled by using the torch.no_grad() context manager from pytorch. By not calculation gradients the model is able to process data faster. Dropout is turned off when setting the model in evaluation mode using model.eval() and is often used in combination with torch.no_grad() to turn off the computation of gradients, see also this stackoverflow answer and the pytorch documentation.


What to do after selecting a model with cross-validation?

When 'overtraining' is not a problem (as in it will not diverge if you use more time), just use all your data and the empirically found optimal hyper parameters. In case of neural networks this is not the case (although in my experience, a lot of architectures converge to a specific test error, and take a long time to end up diverging again). I see a few options that you could try:

Most obvious one is keeping a (small) validation set around to use as indicator for early stopping (don't think of this as throwing away data, you still use it to train your network better)
Use same weight initalization as one of your folds and run for the same amount of epochs, same initialization should make convergence rate more similar than new random initialization
Keep all the cross validation models and use them in an ensemble instead of retraining the full model


Association rules in R / Subset rules based on feature

I do not recall lhs or rhs having intervals. Can you share an example?
You can use the "appearance" parameter when calling apriori. You can find an example in the documentation of APappearance-class  of arules or at
http://r-statistics.co/Association-Mining-With-R.html under section "How to Find Rules Related To Given Item/s"


How to identify similar words as input from a dictionary

So, your question concerns how to effectively translate the input words into their proposed correct words (e.g. Paruuuu --> Paru) via phonetics and spelling mistake corrections.

My first idea on this would be to use a deep sequence to sequence model. In a sequence to sequence model, we encode the input word (e.g. Paruuuu) as a sequence of characters into an encoder (effectively an RNN / LSTM, etc.) into a "hidden representation".
Then you decode your hidden representation as a sequence of phonemes (which denote how we pronounce the word) with a decoder (again, another RNN / LSTM, etc.).
Then, we can take this sequence of phonemes as input into another encoder and then decode with a neural network, where the output layer is a softmax layer, which computes the probability distribution over all words in your vocabulary (in you Dict terms) and select the word the highest probability.
 So overall, we have:

Encode input word --> Decode output phoneme sequence --> Encode output phoneme sequence --> Decode with neural network to classify word.

 The proposed method is supervised, so of course, you will need examples of input words and their correct "translations" (e.g. ("Paruuu", "Paru"))

 Here is an article which gives a good intuition behind sequence to sequence models which can classify your input words: https://towardsdatascience.com/understanding-encoder-decoder-sequence-to-sequence-model-679e04af4346


Is Clustering used in real world systems/products involving large amounts of data? How are the nuances taken care of?

Clustering (mostly) does not apply to huge data.

Since you cannot automate result evaluation and validation (forget the 'internal' measures, they are useless) nor use the automatic results in any meaningful way (partially because clustering is just way too unreliable for automatic use).

Clustering requires you to study the result, learn, and then decide what to do next. You can't automate this.

It is explorative. A tool for you to study your data; not a tool for machines to automatically predict labels. Some clusters will be good, some will be bad. This is fine, you got some good clusters! Just don't rely on everything being clustered. There will always be data that does not belong to any cluster. And often, there will be clusters that just don't make any sense to you, or that are too trivial to be of use or interest.
Results just do not improve with data size.
Methods don't scale well.

Most clustering methods (including k-means) have worst case runtimes of O(n^2) or O(n^3). That is really problematic. But since they are analyzing an NP-hard problem, that is quite decent.

So I guess, you want to work on a sample.
RAM is big these days, and vectors are compact.

You can usually squeeze a billion vectors into RAM. So anything that - like k-means - works with numerical vectors (and not, say, videos and images) just fits into main memory. No need to go larger than that.


As you say, both approaches are used.  It's called tied biases  if you use one bias per convolutional filter/kernel ((3x5x5 + 1)x32 overall parameters in your example) and untied biases if you use one bias per kernel and output location ((3x5x5 + OxO)x32 overall parameters in your example).

Untied biases increase the capacity of your model, so they can be a good idea if you are underfitting. But in this case using tied biases and more filters and/or layers might also help, see https://harmdevries89.wordpress.com/2015/03/27/tied-biases-vs-untied-biases/.


Learning parameters when loss is a piecewise function

I would consider Cubic splines to approximate your piecewise function.
https://en.wikiversity.org/wiki/Cubic_Spline_Interpolation#:~:text=Cubic%20spline%20interpolation%20is%20a,Lagrange%20polynomial%20and%20Newton%20polynomial.
You can regularize your spline by adding continuity constraint on the first and the second derivatives at the nodes so the function would be smooth and would be a good approximation to your piecewise loss.


Please help me out with this Python error - 'invalid syntax'

In your current line

print regr.predict([2000])

This will not work. The first error is the lack of brackets around the contents of your print statement which is required in Python 3. Change this first to

print(regr.predict([2000]))

However, you will see that this does not work either. I suspect you are attempting to evaluate the price for a new $size = 2000$. You will need to reshape the input to your regression for this to work.

new_size = np.array([2000]).reshape((-1, 1))
print(regr.predict(new_size))

  [ 317.78380528]


How can different classification algorithms expressed as neural networks?

I have heard that each of the different classification algorithms can be expressed as a neural network architecture.

The Universal Approximation Theorem
guarantees that neural networks can approximate any function on a closed subset of $\mathbb{R}^n$.  So we are theoretically guaranteed that any function specified by a logistic regression model, SVM, ELM, or any other model can also be expressed by a neural network.

  Is there a way to convert the vector equation of a classification algorithm into neural network architecture?

This part may be hard depending on the model.  Although the universal approximation theorem guarantees that an equivalent network exists, it offers no information about the network's weights or number of hidden units.  Nonetheless, let's give it a shot for the three models you mentioned:

ELMs

This one is easy.  ELMs are already neural networks, so there's nothing left to be done.

Logistic Regression

This one is also pretty straightforward.  For simplicity, let's consider a binary classification problem.  A logistic regression model can be written as
$$
p(Y=1) = \frac{1}{1 + e^{-(w_0 + w_1x_1 + . . . w_nx_n)}}
$$

Where the expression $p(Y=1)$ means "the probability that the class is positive".  The input features are ($x_1$, $. . .$ $x_n$) and they have corresponding weights ($w_1$, . . . $w_n$).

We can represent this function with just a single neuron!  Suppose we have a neuron with $n$ inputs, weights ($w_1$, . . . $w_n$), and bias term $w_0$.  The output of the neuron (with no activation function) is given by
$$
z = w_0 + \sum_{i=1}^{n} w_ix_i
$$

Applying a sigmoid activation function to $z$ results in the exact same expression as we had above for the logistic regression model:
$$
S(z) = \frac{1}{1+e^{-z}} = \frac{1}{1 + e^{-(w_0 + w_1x_1 + . . . w_nx_n)}} = P(Y=1)
$$

Here's a diagram showing the neuron:

The situation would be more complex for multinomial logistic regression, but you get the idea.

SVMs

This one is probably the trickiest of the three, since an SVM's kernel function can be non-linear.  Again, let's consider a binary classification problem.

A linear SVM is specified by the maximal-margin hyperplane between the two classes, which can be written as $\vec{w} \cdot \vec{x} - w_0 = 0$, where $\vec{w} = \langle w_1, . . . w_n \rangle$ and $\vec{x} = \langle x_1, . . . x_n \rangle$.

Of course, the actual classification is determined by which side of the hyperplane a point lies, so the classifier can be written $f(\vec{x}) = sign(\vec{w} \cdot \vec{x} - w_0)$, with 1 denoting the positive class and -1 denoting the negative class.

As with logistic regression, it's pretty easy to represent the decision boundary with a single neuron.  A neuron with inputs $\vec{x} = \langle x_1, . . . x_n \rangle$ with weights $\vec{w} = \langle w_1, . . . w_n \rangle$ and with bias term $\hat{w_0} = -w_0$ describes the decision boundary of a linear SVM.  This is the same neuron as the one associated with logistic regression, except there is no activation function and the bias term has its sign flipped:

However, it's exceedingly common for SVMs to use the kernel trick to find non-linear decision boundaries.  When the kernel is non-linear, the translation of an SVM to a neural network is not so straightforward.  I imagine you could find an equivalent network by using the kernel function as the activation function for each neuron, but off-hand I can't describe exactly how it would work.

Hopefully that helps!


How to fix "Expected sequence or array-like"

# train
pred = model.fit(X_train, y_train)

Here you are juts fitting the model, not making any predictions.

accuracy_score(X_test,pred)

Here you are supposed to pass y_test, and y_predict, since its the output you are comparing not the input data.

Try this-

pred = model.predict(X_test)
accuracy_score(X_test,pred)


Unsupervised Classification for documents

If you have a good amount of instances for every class, you can try using a density-based approach for clustering, with algorithms like DBSCAN.

If you can label at least some of the documents, you can use semi-supervised learning. Usually, when SSL is used for clustering, you need to specify "cannot link" and "must link" constraints for some pairs of instances, which is basically labeling some instances. One algorithm that follows this approach is HMRF-KMeans (Hidden Markov Random Fields K-Means).


Is reseating passengers a reinforcement learning problem?

Reinforcement learning is more about interacting with an environment, and while this could be posed as an RL problem, I think using Global Optimization would be a more direct approach.

Essentially you want to design a cost function that describes how good a particular seating is and then use it to search the space of possible seatings.

For example to solve the problem with Simulated Annealing:

Design a cost function $e(s)$ that measures how good a seating arrangement is. Lower cost means better seating.
Design an acceptance probability function $P(e, e', T)$ that takes the costs of seating arrangements $s$ and $s'$, and a temperature $T$, and returns a probability $P$ with the properties (a) $P>0$ even if $s'$ is worse than $s$ (b) the better $s'$ is relative to $s$, the higher $P$ is and (b) the lower the temperature, $T$, the close $P$ is to $0$ when $s'$ is worse than $s$.
Set some large initial $T$ and design an annealing schedule for decreasing $T$ with the number of iterations.
Start with some seating arrangement $S$. This can be random or chosen according to a greedy strategy that tries to satisfy as many constraints as possible.
Repeat for $i$ in $k$ steps (for some sufficiently large $k$):

Consider a neighboring state $S'$ that can be reached by randomly swapping two seated people or moving someone to an empty seat. Set $S$ to $S'$ with probability $P(S, S', T)$
Decrease $T$ according to the annealing schedule at step $i$

The simulation starting at (4) can be run once or many times saving only the best result. At the end you should arrive at close to or exactly the best possible seating arrangement.


How to visualize results/errors of multilabel classifiers?

There is something called macro-averaging and micro-averaging.

Macro-averaging is just the TPs, TNs, FPs, and FNs of each label class summed up and averaged. You should use macro-averaging for generalizing and evaluating on the overall. You should not use it to make specific decisions.

See details of micro-averaging and more evaluation techniques in the link below.
https://towardsdatascience.com/journey-to-the-center-of-multi-label-classification-384c40229bff#6aaa


Here is the simplistic way.

>>> df = pd.DataFrame(data={'Letter': list('AABBBCCCC'),
>>>                        'Number': [1,2,1,2,3,1,2,3,4]})

>>> dfx = df.groupby('Letter').agg({'Number':list})
>>> dfx

              Number
Letter
A             [1, 2]
B          [1, 2, 3]
C       [1, 2, 3, 4]

>>> dfx = dfx['Number'].apply(pd.Series)
>>> dfx

          0    1    2    3
Letter
A       1.0  2.0  NaN  NaN
B       1.0  2.0  3.0  NaN
C       1.0  2.0  3.0  4.0

>>> dfx.T.fillna(0).astype(int)

Letter  A  B  C
0       1  1  1
1       2  2  2
2       0  3  3
3       0  0  4

So basically the sequence is:

aggregate by letter and put all numbers into a single cell by using df.groupby('Letter').agg({'Number':list}).
apply(pd.Series): turn "column with lists" into 2-dimensional array
T to transpose, and cleanup with fillna and type casting.


How to predict constant failing of equipment

You should try to frame as a prediction problem. Clearly define the target and then map that target to a technique:

Mean time before failure (Survival analysis)
Failure rates per million hours (Estimate Poisson distribution)
Failure within a time window (Binary classification)

Generally, Bayesian methods are used because the sparse available data can be used as a prior.


Method for finding top-k cosine similarity based closest item on large dataset

Locality sensitive hashing is a great tool for this problem.
Pick n random 400-dimensional vectors. (Be careful or not all directions will be chosen with equal probability; pick each dimension as a standard Gaussian.) Each really defines a hyperplane through the origin cutting your space in half. The sign of the dot product of any of these vectors with some new vector tells you which side of the hyperplane it's on. So computing n dot products gives n 0/1 bits, which make an n-bit hash.
Any new vector hashing to the same value must be in the same small sliver of space from the origin. And those are exactly the vectors that have a high cosine similarity with each other since their mutual angles are very small. Likewise anything hashing to almost the same value -- differing in a few bits -- is likely to be nearby. So you can restrict your search for most-similar vectors to things within one or more buckets of hashed candidate vectors.
It doesn't help directly with memory since you might need any particular bucket to satisfy a request. You also lose some accuracy since there is not a guarantee the most similar vectors lie in the buckets you examine (though it's likely, the more you examine). It lets you trade speed for accuracy mostly. However you may find you can get away with some caching scheme, where some buckets are rarely if ever accessed and so don't stay in memory.
You can see an implementation of this in Oryx, which I think is pretty straightforward.
Most of the complexity comes because it lets you specify a target percentage of vectors to evaluate, and works out the optimal hash size based on that and your machine's number of cores.


Linear regression incorrect prediction using Matlab

You are calculating a linear regression in a nonlinear environment, which means you will need nonlinear exogenous variables.

You are calculating $Y=\beta_0+\beta_1x$ (a linear equation) when you need at least a polynominal equation. The following code calculated this equation: $Y=\beta_0+\beta_1x+\beta_2x^2+\beta_3x^3$ (maybe not the best equation but a working one)

%generate some data
x=linspace(0,2*pi,100)';
y=sin(x); %response

hold on
X=x;
y=y;
% Convert matrix values to double
X = double(X(1:79));
y = double(y(1:79));

% Plot data
plot(X, y, 'rx', 'MarkerSize', 10);

m = length(y);
% Add ones column
X = [ones(m, 1) X X.^2 X.^3];

% Gradient Descent with Normal Equation
theta = (pinv(X'*X))*X'*y;

% Predict  from 80 till last sample
test_samples = x(80:end);
test_samples_val = [ones(length(test_samples),1) test_samples test_samples.^2 ...
test_samples.^3];

% Calculate predicted value
pred_value = test_samples_val * theta;

X = vertcat(X, test_samples_val);
regressionline = X*theta;

% Plot predicted value with blue cross
plot(test_samples, pred_value, 'bx', 'MarkerSize', 10);


How to limit a Julia notebook to a single core?

If the high CPU usage is due to parallelized linear algebra routines, limiting the CPU usage of the linear algebra backend helps. For Julia 1.4 compiled to use BLAS, the following limits it to a single core:
using LinearAlgebra;
BLAS.set_num_threads(1);

(Many forum posts recommend blas_set_num_threads(1) for older versions of Julia. This seems to have been depricated)


Convert a repetitive list into Pandas Dataframe

If I understand this correctly your sequence is always 3 elements. Then you can do this:

a = ['A:1','B:2','C:3','A:5','B:5','C:6','A:2','B:5','C:7']
b = []
rep_len = 3

# Looping with step size equal to repetition length
for i in range(0,len(a),rep_len):

    # Selecting a repetition length
    c = a[i:i+rep_len]

    # Extracting everything in after letter and colon and casting to integer
    c = [int(x[2:]) for x in c]

    # Append to a list of lists
    b.append(c)

df = pd.DataFrame(b, columns=['A', 'B', 'C'])

Resulting in:

    A   B   C
0   1   2   3
1   5   5   6
2   2   5   7


AttributeError: 'numpy.ndarray' object has no attribute 'predict'

You don't want to pickle the predictions but rather the fit.

Change joblib.dump(nb_predict_train, "trained-model.pkl") to joblib.dump(nb_model, "trained-model.pkl")


incremental learning vs transfer learning

Transfer Learning: for example you want to predict price of article normally we use previous data based on that we design  model .while new data came still we use that model for prediction here we are transferring the  same model for new task or in general
When you learn how to drive a car, you learn a generic skill and you will use these similar set of skills if you want to drive a truck. This transfer of your knowledge is called transfer learning.

continuous learning:for example In face recognition or image classification system, first we train our model with some data,if new person is came our model can't recognize the person in this case we train our model only with new data and attached that weight into our model. in general
If you have learnt any new skills while driving a truck, you will continuously update your knowledge and use that for improving your driving skill. This is called incremental learning (or) continuous learning (or) online learning.


How to "reshape" into square matrix for numpy.linalg.solve()?

You should formulate your lines as follows to have $(x, y)$ as unknowns:
$$\begin{align}
\left.\begin{matrix}
a_1x-y=-b_1\\
a_2x-y=-b_2
\end{matrix}\right\}
\rightarrow
\overbrace{
\begin{bmatrix}
 a_1& -1\\
 a_2& -1
\end{bmatrix}
}^{\boldsymbol{a}}
\overbrace{
\begin{bmatrix}
 x\\
 y
\end{bmatrix}
}^{\boldsymbol{x}}
=
\overbrace{
\begin{bmatrix}
 -b_1\\
 -b_2
\end{bmatrix}
}^{\boldsymbol{b}}
\end{align}$$
Therefore, the code should be:

import numpy as np

def meeting_lines(a1, b1, a2, b2):
    a = np.array([[a1, -1], [a2, -1]])
    b = np.array([-b1, -b2])
    return np.linalg.solve(a, b)

a1=1
b1=4
a2=3
b2=2
x, y = meeting_lines(a1, b1, a2, b2)
print(x, y)

which outputs:

1.0 5.0


Maybe sequential association rules could be applied here. Please check spmf java library, section 'Sequential Rule Mining'.