How to scale or standardize data that is mostly 0 (ranges from 0-1)?

You don't specify, but assuming you are doing K-means clustering, the way the algorithm calculates similarity, or more technically distance, using Euclidean Distance, the distances on the dimension with the greatest magnitude will overwhelm distances of those where the values are orders of magnitude smaller.

To overcome this I would normalize the data.  I am not sure the method you use is such a big deal.  Use min-max if you are comfortable with it.  Be careful though because it is possible, if I remember correctly, in sk-learn to scale the whole data set in one go which doesn't fix the magnitude differences.  Scale each column individually or even possibly just the one that is vastly different if all the others are nominally between 0 and 1.


Hi, im currently working for a company that has some inventory control problems

Okay, so since the visuals are done in excel, I think you can start with simple linear regression in excel. Im not an excel expert in this regard, but I think this article will be a good starting point
https://www.ablebits.com/office-addins-blog/2018/08/01/linear-regression-analysis-excel/
So, what you can do:

Get your data and fit linear regression on it.
You need to choose the dependent variable y (this you want to predict) - this will be your inventory variable. All other features (basically other data in your disposal) will act as independent Xi variables.
You fit linear model of form y = bx + a (simplified)
After you have fit the model, you can access its coefficients. One of them will be an Intercept other ones will be coefficients for your input X features.
This coefficient will be X affect on your y (inventory). So, for example, if coefficient is 0.1 for X1, then increasing X1 for 1 will lead to 0.1 increase in y.
At this point you have dependency between input data and inventory change.

Thats a lot more to consider - how good your model is (for this you use metrics, for example MAE and test your model on unseen data), do you have a data-leak, and so on and so on.  You will also need to pre-process your data in viable format, for example extract day/week number from date to try capturing seasonality. Also you should avoid correlation between variable, thats really a lot to think of.
But most importantly, I think you should try to implement it even if you dont know all details and all the rules just to get a fealing if this task fits you, after it you can go on and lear more about all this stuff.
(I would definitely recommend going through this series one day if you are interested https://www.youtube.com/playlist?list=PLOg0ngHtcqbPTlZzRHA2ocQZqB1D_qZ5V )
Hope this helps.


Calculation of distance between samples in data mining

The way that various distances are often calculated in Data Mining is using the Euclidean distance. You can read about that further here. If I understand your question correctly, the answer is no. The Euclidean distance can only be calculated between two numerical points. Therefore it would not be possible to calculate the distance between a label and a numeric point.


predict_classes() returning only 0 or 1 for multiclass image classification

I got the answer so I'm posting it for future reference for others:

The 3 classes defined in

train_directory = 'D:\D_data\Rock_Paper_Scissors\Train'
training_datgagen = ImageDataGenerator(rescale = 1./255)
training_generator = training_datgagen.flow_from_directory(
    train_directory,
    target_size = (28,28),
    class_mode = 'categorical', classes = ["Rock", "Paper" , "Scissor"])

in class_mode should be Rock , Paper , "Scissors"
those were the names of the folders i create but i made a mistake and spelled "Scissors" wrong in the code.

Thanks a lot to everyone who commented!


How to use Automated Labelling for documents?

Semi-supervised learning. You label 1% manually, let the algorithm learn, then it labels unknown data, learns from it and labels again.


the probability distribution of dependent variables

You need to tell us what the distribution of x1 is and what f and g function are exactly? When f and g are simple, we can usually solve this analytically. But when the function is complex, this kind of problems are typically approximated by Monte Carlo simulation.


Why Continous Variable Buckets Overfitting model

When you use buckets you lose some information. Basically you assume the relationship between the variable and the target is flat within the interval. When this is probably not the case. This is point #3 on Frank Harell's list of reasons why you generally shouldn't categorise at all.
More specifically, 80% of your instances seems to be in one bucket. That mean your algo can't learn what happen inside this bucket. As a starting point you may use more buckets to check if that was the problem. More convoluted techniques would include splines to smooth the response (removing noise help not learning from it).  Honestly given that you work with a tree like algo (Xgboost) you should probably let it choose optimal splits instead of enforcing some splits trough variable categorisation. That is : don't use a priori variable categorisation at all, enforce the absence of overfitting trough other parameters (tree depht at least) and if you really need category use those found by the algo.
Edit: re-reading your question it seems that you are overfitting in both case... the solution might be more about tuning your xgboost parameters.


Checkout the playlist of Ahlad Kumar on YouTube. He has a wonderful playlist for theory and implementation of Gans. Although the implementation is using tensorflow, it will satisfy your curiosity of understanding the concept without getting your hands dirty with coding from scratch without using deep learning framework.

Conditional GANs:
https://youtu.be/7Tlk3Gql-Wg


how to represent location-code as a feature in machine learning model?

You can get as creative as you want, but here are two general approaches that work for me.

Clustering the data into known geographical divisions and create dummy variables.  For example, in the United States a person can use zip codes.
Find the center for known clusters (ie zipcodes) or by some similar unsupervised cluster and use the longitude and latitude.

How you choose to augment that information depends on what exactly you're trying to predict.


With automated hyperparameter tuning available, do we still need to learn hyperparameter tuning

So do we still need to learn how to do hyperparameter tuning

If you're saying this based on the context of acquiring a new skill, then go for it. It's always a good thing to get an idea an idea of how hyper-parameter testing is done for real. In addition to sagemaker you can use tools like weights and biases


Merging common Columns values in two DataFrame Pandas

Dropping your duplicate rows and merging on 'Patient ID' should solve your issue.

data = pd.merge(res_total_Df, key_df,
                        left_on=['patient_id'], right_on = ['patient_id'], how = 'left')


After a lucky google search, I found the function na.interpolation from the imputeTS-package. After changing the dataset to a time series (XTS), I was able to impute the right data.

library(xts)
library(imputeTS)

set.seed(1337)
df <- data.frame(x = c(seq(100, 10, -4), seq(100, 20, -3), seq(100, 60, -3),
                       seq(80, 40, -3), seq(100, 10, -3))) %>% data.table()
df[sample(seq_len(nrow(df)), size = nrow(df) * 0.3), x := NA]
df$y <- seq(1,nrow(df),1)

ts <- xts(df[, -c("y")], as.Date(df$y))
head(ts)
#>              x
#> 1970-01-02 100
#> 1970-01-03  96
#> 1970-01-04  NA
#> 1970-01-05  88
#> 1970-01-06  NA
#> 1970-01-07  80

full <- na.interpolation(ts)
plot(full, type = "p")


Cost Function for evaluating a Regression Model

A loss function and cost function are the same thing.  As you intuit, classical regression treats loss/cost as symmetric, which is not always what you want. In classification tasks, you can make an asymmetric loss matrix.  You can do a similar thing with regression if you solve it with gradient descent, but the ordinary least squares has symmetric loss baked in.

So I would consider either (1) using a numeric optimization library like sklearn or tensorflow to explicitly define the regression parameters you want to estimate, write your own custom loss function, and then do parameter estimation via gradient descent, or (2) finding a software package that allows for asymmetric loss, for example see this discussion.


Neural network is getting partially trained

I can see a problem with your error function (blue). It is treating under-predictions and over-predictions the same. The error gradient should ideally direct you towards zero error.

For your error function:

$$
E = 2 \cdot (y_{expected} - y_{predicted})
$$

The derivative w.r.t prediction is:

$$
\frac{dE}{d\;y_{predicted}} = -2
$$

Note here that the error is the same whether you under-predict or over predict (i.e. the sign of error doesn't matter in the gradient). So the updates to the weights will be in the same direction (see arrows on blue line).

But if you use a quadratic error function (orange line), over- and under- prediction errors are treated differently (i.e. the gradient depends on the sign of the $expected - predicted$ difference).

$$
E = (y_{expected} - y_{predicted})^2 \\
\frac{dE}{d\;y_{predicted}} = 2 \cdot (y_{expected} - y_{predicted}) \cdot -1
$$


This sounds like a one-class or unary classification problem where you can build a model of normality using the class that you have.

Take a look at novelty and outlier detection documentation for scikit learn. They discuss the one-class SVM there, which attempts to model the decision boundary given the observations that you have. The example that is implemented is pretty straight forward. Sadly, this implementation does not provide you with a probabilistic measure (although, there are papers on how to add this functionality).

Alternatively, you can consider an auto-associative neural network (replicator network).

There is a nice discussion on time-series anomaly detection here on DS.


The term may best be expressed as a Recurrent, Heavy Subgraph. That is, a subgraph which is both Recurrent and Heavy.

Heaviness of a subgraph refers to heavily connected vertices- that is, nodes which are connected many times ("many" being relative to the network in question).

Recurrent refers to the propensity of a subgraph to occur more than once.

Thus, a Recurrent Heavy Subgraph  is a densely connected set of vertices which occurs several times in the overall network.

These subgraphs are often used to determine properties of a network.

For example: In a network of emails interactions within a company organized into 4-person teams with one member acting as the lead, each team's email activity (if they email between themselves sufficiently to be considered "heavy") could be described as a Heavy Subgraph. The fact that these subgraphs occur many times in the network make them Recurrent Heavy Subgraphs. If one was searching for structure in the network, noticing that these recurrent, heavy subgraphs exist would go a long way toward determining the organization of the network as a whole.


What is the best machine learning model to predict a continuous variable where the predictors include categorical, numerical variables and a text?

I believe this is what you want to know, the approach to handle your text data. You can do one hot encoding for your categorical data if they are not ordinal or label encoding for ordinal. As for model, I am also a newbie but I think you can't define a rule of thumb that this model is best for regression. It depends on the data, which model will work.


You should read that :

http://www.analyticsvidhya.com/blog/2016/02/time-series-forecasting-codes-python/

and take a look at that :

http://statsmodels.sourceforge.net/0.6.0/generated/statsmodels.tsa.arima_model.ARIMA.html

The endog argument is your time serie and the exog is your other data


Data scientist vs machine learning engineer

Good question. Actually there is a lot of confusion on this subject, mainly because both are quite new jobs. But if we focus on the semantics, the real meaning of the jobs become clear.

Beforehand is better to compare apples with apples, talking about a single subject, the Data. Machine Learning and its sub-genre (Deep Learning, etc.) are just one aspect of the Data World, together with the statistic theories, the data acquisition (DAQ), the processing (which can be non-machine learning driven), the interpretation of the results, etc.

So, for my explanation, I will broad the Machine Learning Engineer role to the one of Data Engineer.

Science is about experiment, trials and fails, theory building, phenomenological understanding.
Engineering is about work on what science already knows, perfecting it and carry to the "real world".

Think about a proxy: what is the difference between a nuclear scientist and a nuclear engineer?

The nuclear scientist is the one which know the science behind the atom, the interaction between them, the one which wrote the recipe which allow to get energy from the atoms.

The nuclear engineer is the guy charged to take the recipe of the scientist, and carry it to the real world. So it's knowledge about the atomic physics is quite limited, but he also know about materials, buildings, economics, and whatever else useful to build a proper nuclear plant.

Coming back to the Data world, here another example: the guys which developed Convolutional Neural Networks (Yann LeCun) is a Data Scientist, the guy which deploy the model to recognize faces in pictures is a Machine Learning Engineer. The guy responsible of the whole process, from the data acquisition to the registration of the .JPG image, is a Data Engineer.

So, basically, 90% of the Data Scientist today are actually Data Engineers or Machine Learning Engineers, and 90% of the positions opened as Data Scientist actually need Engineers. An easy check: in the interview, you will be asked about how many ML models you deployed in production, not on how many papers on new methods you published.

Instead, when you see announces about "Machine Learning Engineer", that means that the recruiters are well aware of the difference, and they really need someone able to put some model in production.


How to create a feature vector given final set of feature maps?

After asking around about this, it seems the third option is the standard: take the mean of each feature map, and create a 2048-element feature vector. The search term for this is global pooling. That's what we are talking about that is the terminology I was missing.

Global average pooling is good b/c it reduces the dimensionality before classification. Also, by the time you have reached this level of abstraction in your feature extractor, you are probably not that interested in the finer-grained spatial aspects of the features because you are already representing a very abstract "object" (depending on how deep your feature extractor is) after a great deal of pooling of your original image.

That said, if you are worried about losing spatial information you can always try one of the other two options. I'd be curious if anyone has tried this if they have experiences to share about how they went about it.

Also, for a really nice summary from a paper that addresses this topic, see this answer from crossvalidated:
https://stats.stackexchange.com/a/308218/17624

IN particular:

  Conventional convolutional neural networks perform convolution in the
  lower layers of the network. For classification, the feature maps of
  the last convolutional layer are vectorized and fed into fully
  connected layers followed by a softmax logistic regression layer. This
  structure bridges the convolutional structure with traditional neural
  network classifiers. It treats the convolutional layers as feature
  extractors, and the resulting feature is classified in a traditional
  way.

  However, the fully connected layers are prone to overfitting, thus
  hampering the generalization ability of the overall network. Dropout
  is proposed by Hinton et al as a regularizer which randomly sets half
  of the activations to the fully connected layers to zero during
  training. It has improved the generalization ability and largely
  prevents overfitting.

  In this paper, we propose another strategy called global average
  pooling to replace the traditional fully connected layers in CNN. The
  idea is to generate one feature map for each corresponding category of
  the classification task in the last mlpconv layer. Instead of adding
  fully connected layers on top of the feature maps, we take the average
  of each feature map, and the resulting vector is fed directly into the
  softmax layer. One advantage of global average pooling over the fully
  connected layers is that it is more native to the convolution
  structure by enforcing correspondences between feature maps and
  categories. Thus the feature maps can be easily interpreted as
  categories confidence maps. Another advantage is that there is no
  parameter to optimize in the global average pooling thus overfitting
  is avoided at this layer. Futhermore, global average pooling sums out
  the spatial information, thus it is more robust to spatial
  translations of the input. We can see global average pooling as a
  structural regularizer that explicitly enforces feature maps to be
  confidence maps of concepts (categories). This is made possible by the
  mlpconv layers, as they makes better approximation to the confidence
  maps than GLMs.

Which is from this paper: https://arxiv.org/pdf/1312.4400.pdf


You're using the SGD optimizer, but you're only making one pass over the training data.  I think the default learning rate for SGD is something like 0.01 or 0.001, and, with such a low learning rate, your model won't be able to learn the pattern with a single pass over the data.

To fix this, you need to do one or both of the following:

raise the learning rate
train the model for more epochs (an epoch is one pass over the training data)

To raise the learning rate, you'll need to create a custom optimizer:

. . .
const learningRate = 0.5;
const sgdOptimizer = tf.train.sgd(learningRate);
model.compile({loss: 'meanSquaredError', optimizer: sgdOptimizer});
. . .

To train for more epochs, you just need to provide some additional arguments to the fit() function:

. . .
model.fit(xs, ys, {
    epochs: 25,
    batchSize: 8
});
. . .

Raising the learning rate will increase the magnitude of each weight update, allowing your model to learn more quickly, but if the learning rate is too high then the model may not converge.  Raising the number of epochs allows for more weight updates, but will increase training time.


Try a simple equation like this:
\begin{eqnarray}
&&\max(x-20,0)*0.1 + \max(0,x-25)*(-.047) + \\
&&\max(0, x-125)*(-.011) + \max(0,x-1000)*(-.003)
\end{eqnarray}

This should take the usage $(x)$ and walk it through steps, initially charging 0.1 for each Kwh over 20: $\max(x-20,0)*0.1$

Then, if $x$ is above 20, it takes away 0.047 (0.1-0.053) for every Kwh between 20 and 25: $\max(0, x-25)*(-0.047)$

Then, if $x$ is above 125, it takes away 0.011 (0.053-0.042) for every Kwh between 25 and 125: $\max(0, x-125)*(-0.011)$


I assume you have some training data with labels, i.e. data where the titles are already linked to a given class? This is then supervised learning (as opposed to unsupervised learning), and so you could folow the following steps:

Step 1: you have words as input, so you will need a method to create numerical representation (vectors). For that you could look into algorithms such as Word2Vec, Doc2Vec, GLoVE or something like TF-IDF. If you go for the first, you might consider trying the spaCy library in python. Here is a tutorial on Word2Vec using spaCy.

Step 2: once you have your numerical representations for each of your titles, you need to somehow classify them. You could do this a few ways. Perhaps the simplest would be something like a clustering algorithm, e.g. the DB-Scan algorithm in SciKit Learn - here is a demo.
You could try more complicated methods, such as Support Vector Machines or Neural Networks, but probably best to start with a method that will get you to some results more quickly. You are classififying titles, so be sure to form your problem as a classification as opposed to a regression problem.

Step 3: assess your results and try changing a part of the loop above.

In the above, I assumed you are talking about the semantic meaning of the conference titles, and not similarity between literal word/letter combinations. That could of course be computed analytically, without the use of a model that learns.

In response to OP's comment:
From my experience, using TF-IDF or something called minimal new sets might be a good way to get your titles into representations that allow clustering. Once clusters are formed, it would be up to you to then interpret them and assign labels. If you know that there are e.g. only 10 conference, it shouldn't be too difficult to reach results. Have a look at this master thesis that does a similar thing - instead of conferences, they want to detect topics. Disclaimer: I supervised that thesis.


Customized function for Agglomerative Clustering

Most implementations will begin with a distance matrix (and not a data matrix) as input, as the algorithms are O(n²) anyway.

For the second question, I don't understand how that is defined. If your linkage is not a Lance Williams update, you may need to reimplement hierarchical clustering completely, to make sure you have all the information you need. Lance Williams are much more elegant to compute.


Should I try to predict the probability of being 1 or zero based on recall score?

I have some bad news: your model is doing nothing useful.

From the confusion matrix you can see that the model predicts class 0 for 68% of the instances and class 1 for 32%: (1446+1355)/4093=0.68 and (676+616)/4093=0.32.
You can also see that when the true class is 0 the model predicts 68% of the instance as class 0 and 32% as class 1: 1446/2122=68% and 676/2122=32%. Almost same result when the true class is 1: 1355/1971=69% and 616/1971=31%.

This means that for any instance (no matter the true class) the model randomly predicts the label: class 0 with 68% chance, class 1 with 32% chance. You can also see this problem from the fact the sum of the top-left to bottom right diagonal (correctly classified cases) is 2062, almost the same as the bottom-left to top-right diagonal (errors) which is 2031. A good model would have a much larger difference between correct cases and incorrect cases.
This is why the accuracy is only 50.4%, barely more than a random baseline with two classes, and this is why the recall for each class corresponds exactly to the arbitrary probability that the model assigns to them.

should I use recall in my decision for choosing whether to try to predict if the outcome will be a 0 or 1?

This doesn't make sense: the recall is an evaluation score, it can only be calculated after the predictions have been made.

Also I had my model using predict_proba and using the second value from the array return which would give me the probability of the test data outcome being a 1, but from this classification report it looks like I should be trying to predict the probability of the outcome being 0 as it is more descriptive when it comes to recall.

There are at least two misunderstandings here:

By itself, the fact that the model predicts class 0 more often than class 1 is neither good or bad. It would be completely wrong to choose to evaluate the performance using only the recall on the most frequent class: it's true that it's a higher value, but it's not at all a good performance indicator. For example, if one uses a model which always predicts class 0 (100% of the time) then by definition this classifier will have recall 100% for class 0, even though this model is pointless.
In binary classification, the probability of predicting the two classes are mathematically bound together: $p(C=0)+p(C=1)=1$. So choosing to predict the outcome 0 instead of 1 doesn't change anything about what the model does, it would be exactly the same model with the same performance.

What I suggest you should try is to work on your features: normally the features should be indicators that the model uses to predict the class, but unfortunately your current features are terrible at their job. Assuming that the features "make sense" for the task, it should be possible to "present them" to the model so that it can actually use them efficiently. This is called feature engineering: it's often the hard part of the job but also the most likely to really make things work :)


How to know the state-of-the-art recommended approaches for data science?

A few comments:

There is no scientific domain of "data science", instead there are multiple fields which are related to data science: statistics, ML, NLP, computer vision, signal processing... and a lot of other fields which overlap and/or focus on specific applications, for instance bioinformatics. All of these domains are highly active and specialized, so it would just be impossible to monitor every possible advance.
There is no unique recommended way: first, people disagree all the time about the best way to do X. Second, it's very rare that a method would become completely obsolete. For example TFIDF still makes sense in many use cases, with low-resource languages or when there are efficiency constraints for instance.
In order to comprehensively follow the state of the art, one would have to follow the research publications. At best it's doable for a specific domain, for example one can more or less get an idea of what happens in NLP by browsing through the main conferences. A more realistic option is to wait for the advances to reach the mainstream professionals, for example by browsing regularly through DataScienceSE and/or CrossValidated.
Final suggestion: old books are very useful to fully understand why/how things are done a certain way. We often see errors here on DataScienceSE which are due to people trying to apply methods without understanding them.


Creating graphs of CNN models: cannot use plot_model from keras.utils.vis_utils

I had exactly the same problem today and stumbled upon your question. As it had no replies, I dug further and eventually found something that helped me. For you it may be too late but anyone else having this problem, maybe it helps if I document my finding.
For the background: my setup is Python 3.7 under Windows 10.
The keras function plot_model uses pydot which itself relies on GraphViz. Moreover, the error message Format: "ps" not recognized. Use one of: you see is caused by dot, one of the tools that belongs to GraphViz, which is called from inside pydot. After playing around with the dot tool, it eventually told me the following:
There is no layout engine support for "dot" Perhaps "dot -c" needs to be run (with installer's privileges) to register the plugins?
So, I fired up a new console with administrator priviliges and ran dot -c. This enabled dot to use its layout engines and this made the error disappear. After this,  plot_model suddenly worked for me.
Hope it helps anyone.


Getting extra characters while loading data

Try using endcoding=utf-8
Code:
import pandas as pd
mydf=pd.read_sas("\home\mypath\mydataset.sas7bdat",format = 'sas7bdat', encoding="utf-8")

If the sas dataset is small, I would recommend loading from CSV file instead of sas dataset. Performance loading from SAS using Python is poor.


How to do face recognition without using any kind of Neural Networks like CNN?

As it stands, NN-based approaches are the current state of the art. It outperforms the method described below, but in the spirit of answering the question, here it goes...

The trick for not using CNNs is to find a good representation of images with faces on them. CNNs are great because they learn good features. If you don't want to use it, you can use CNNs, you can use Eingenfaces as your features.

The main assumption is that most face images lie on a low-dimensional subspace determined by the first k (k<d) directions of maximum variance.

You can Use PCA to determine the vectors or “eigenfaces” u1, ... ,uk that span that subspace.

Then you can represent all face images in the dataset as linear combinations of eigenfaces.

For example, given these images for training:

You can learn the following eigenfaces:

These eigenfaces highlight different features in a person's face. If you average all of them you find a mean face, which would be the most common features among all people in your training set.

Then, this face:

Can be reconstructed by a linear combination of the mean face and some other eigenfaces:

Now, the weights of each component can be used as features for your classification algorithm. Which at this point can be anything, like KNN or SVM.


How to find the relationship between two categorical variables?

Traditional statistics like Chi-squared tests and Cramer's V can be used to determine relationship between two categorical features.


You can break the problem into two separate problems:

How much is a person likely to donate? That is a regression problem.
How to maximize the amount raised by the campaign after deducting the cost? That is contraine optimization problem.

Each separate problem can be solved with existing tools without creating a custom loss function.


Do we have any method which handles the imbalanced classification with sample weighting instead of class weighting?

If you assign to each instance the weight of the corresponding class the effect will be similar and sometimes exact.
I say similar because there are methods which uses sample weighting intrinsically, for example AdaBoost or almost all the types of decision trees, but they do perform also some arithmetics with those weights which can lead to different results. For example in CART trees instances have a weight and if it happens that at a test node to have a missing value, the same instance goes into both child nodes but with weights splitted (their sum is equal with the original weight). So instace weights could have a more granular effect, and perhaps a better resolution and in the end maybe a better performance.
A second way often encountered in modelling is to introduce learning bias into the problem. Take for example weighted linear regression. The learning bias is your knowledge about how much you trust the value of an observation. It can be done automatically for example judging how much variability has the observation relatively to it's neighbour observations (high variance means less trustful value). The same idea of encoding trust into observations could be pushed into any classifier if it allows you to weight instances.
A last note is that a common interpretation of class weighting is in terms of cost of error (how much it will hurt you if a class A instances is predicted as class B instance) and instance weighting is usually encoded like how much trust do you have in an observation or, alternatively, how much do you want that that observation and similar with that one to be correctly classified. And, as I said in the beggining, they can have a similar effect.
Class imbalance is just a particular problem to be solved, instance weighting allows you to solve this problem and also other issues related with your data.


Time series normalization using min max technique

To resolve the issue, I used diff() method to remove trends in diff.diff()


I am getting very minimal mse values and not sure if it is correct?

So first thing first, accuracy is a classification concept. You can't say you have 99% accuracy for a regression problem.

Your code seems ok. Cross validation is not necessary here since you are not doing any hyper-parameter tuning or model selection. The mse error is indeed low, so I would suggest you go back to normalize your data, since if your target $y$ has a very small span, i.e. low $\sigma$ in Gaussian case, you will get a meaningless low mse guaranteed.


Why does Q-learning use an actor model and critic model?

The book you are reading is being somewhat lax with terms. It uses the terms "actor" and "critic", but there is another algorithm called actor-critic which is very popular recently and is quite different from Q learning. Actor-critic does have two function estimators with the roles suggested in the quote. Q-learning has one such estimator*.

I have looked at the chapter in more detail, and where it says:

  one will be used to drive Ms. Pac-Man during training (the actor), and the other will watch the actor and learn from its trials and errors (the critic).

I would substitute:

  one will be used learn from current actions, and the other will remember results from some time steps ago in order to estimate the values for next actions.

This is not something that's inherently part of Q-learning, but it is part of DQN's adjustments when combining Q-learning with neural networks. Both experience replay and having two copies of the learning network (one a temporarily "frozen" version of the other) are important for stabilising the learning algorithm. Without them it can become numerically unstable.

  Is this a typical Q-learning implementation?

It's a typical implementation of basic DQN, which is how many people nowadays would implement Q-learning with neural networks.

You can ignore the references to "actor" and "critic". Instead it is easier to consider that there is just one "action value" network, and you keep an old copy of it around to help with stability.

* Generally in RL, the term "model" is reserved for a model of the environment - which neither Q-learning nor actor-critic provide. So you will also read that Q-learning is a "model free" algorithm. For the rest of the book, you will have seen "model" to refer to any statistical learning algorithm (or the architecture and learned parameters) . . . what you will see in RL texts is the careful use of "function estimator" or other terms for networks which learn something else other than how the environment behaves.


Which ML algorithm should I use for following use case for classification and Why?

You can check https://autogluon.mxnet.io/
from autogluon import TabularPrediction as task
predictor = task.fit(train_data=task.Dataset(file_path=TRAIN_DATA.csv), label=COLUMN_NAME)
predictions = predictor.predict(task.Dataset(file_path=TEST_DATA.csv))


Gym action space for board game with reward function

You can use each position on the board as an action (100). Then use a mask that will give you only the available positions (from your current position) as possible actions.


How to derive false positive and false negative from top-k accuracy?

I think your confusion comes from the fact that the false negative of $M$ as you thought it, is not top-$k$ accuracy but perhaps $1$ $-$ top-$k$. Moreover, how do you evaluate the condition if g' in top-$k$ most similarity graphs with g?
In any case, for your case I thought the following.
Let's say that if the model $M$ returns a value $o > o^*$, where $o, o^* \in [0, 1]$ (and $o^*$ is a threshold value decided by a human), then we assume that $g \equiv g'$; $g \neq g'$ otherwise.
Thus, a false negative of $M$ is when $o < o^*$ but in reality, $g \equiv g'$.
A false positive of $M$ instead is when $o > o^*$ but in reality, $g \neq g'$.
Of course your result will depend on what value(s) of $o^*$ you choose, but now it should be more clear what are FN and FP.


Right Way to Input Text Data in Keras Auto Encoder

Here's my take on your questions.

Yes, you can zero-pad vectors. However, I would strongly recommend you use an LSTM as part of your encoder if you're using sentences as input. An LSTM can take variable-length input - that's part of its charm!
The loss function depends entirely on your goal. It sounds like you're trying to create a sequence-to-sequence VAE. I would go about this by putting a softmax function on your output layer, to generate the next most probable word in the sentence. So, I would use categorical_crossentropy as the loss function, and I would use softmax instead of sigmoid, because you're choosing the next most likely word from a set vocabulary (this is not a binary issue but rather a categorical issue).

Hope that helps!


When thinking of rnn/lstm/gru layer keep few points in mind.

What is your inputs size? In this case we have 5 word sentence so 5 input circle going into lstm layer with that every word value will be multiple by same weight value.
How many lstm cell will be present inside lstm layer? To keep it simple as we have 5 words we are keeping 5 lstm cell for each word then carry forwarding it's memory into next lstm cell.
How many output categories we have? The last lstm cell will be connected to the all 3 dense layer neurons and at each of the neurons softmax operation will happend just like normal fully connected neural network dense layer works.


Can machine learning algorithms predict random number

No, absolutely not.
ML is not a magical process, it works by identifying patterns in the data. In the case of supervised learning (i.e. with training) it means finding in the features the most relevant indications to help predict the target variable.
Randomness is the complete opposite of "patterns", by definition something truly random is not predictible in any way.
Intuitively if a human with a lot of time cannot do a task, then ML cannot do this task either.


Identify significant features in clustering results

A similar post appears on Cross-Validated, "Estimating the most important features in a k-means cluster partition".
Quoting from that post:

One way to quantify the usefulness of each feature (= variable =
dimension), from the book Burns, Robert P., and Richard Burns.
Business research methods and statistics using SPSS. Sage, 2008.
(mirror), usefulness being defined by the features'
discriminative power to tell clusters apart.

We usually examine the means for each cluster on each dimension using
ANOVA to assess how distinct our clusters are. Ideally, we would
obtain significantly different means for most, if not all dimensions,
used in the analysis. The magnitude of the F values performed on each
dimension is an indication of how well the respective dimension
discriminates between clusters.

Another way would be to remove a specific feature and see how this
impact internal quality indices. Unlike the first solution, you
would have to redo the clustering for each feature (or set of
features) you want to analyze.
FYI:

Can a useless feature negatively impact the clustering?
Can the choice of the measurement units of the features impact the clustering?
Why vector normalization can improve the accuracy of clustering and classification?
What are the most commonly used ways to perform feature selection for k-means clustering?

Furthermore, there is a paper on Feature Selection in Clustering Problems

A novel approach to combining clustering and feature selection is
presented.   It  implements  a wrapper strategy  for  feature
selection,  in  the sense that the features are directly selected by
optimizing the discriminative power of the used partitioning
algorithm.  On the technical side, we present  an  efficient
optimization  algorithm  with  guaranteed  local  convergence
property.  The only free parameter of this method is selected by a
resampling-based stability analysis.  Experiments with real-world
datasets demonstrate that our method is able to infer both meaningful
partitions and meaningful subsets of features.


Google Earth Pro Satellite image segmentation using clustering

Whoa, are you trying to do clustering based on satellite images?!  Just use the underlying long & lat coordinates to do this.  It will be infinitely easier and so much more accurate too.  Look at the example code below and try to adapt that to your specific case.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
import seaborn as sns; sns.set()
import csv

df = pd.read_csv('C:\\your_path\\properties_2017.csv')
df.head(10)

df.dropna(axis=0,how='any',subset=['latitude','longitude'],inplace=True)

# Variable with the Longitude and Latitude
X=df.loc[:,['parcelid','latitude','longitude']]
X.head(10)

K_clusters = range(1,10)
kmeans = [KMeans(n_clusters=i)

for i in K_clusters]
Y_axis = df[['latitude']]
X_axis = df[['longitude']]
score = [kmeans[i].fit(Y_axis).score(Y_axis)

for i in range(len(kmeans))] # Visualize
plt.plot(K_clusters, score)
plt.xlabel('Number of Clusters')
plt.ylabel('Score')
plt.title('Elbow Curve')
plt.show()

kmeans = KMeans(n_clusters = 3, init ='k-means++')
kmeans.fit(X[X.columns[1:3]]) # Compute k-means clustering.X['cluster_label'] = kmeans.fit_predict(X[X.columns[1:3]])centers = kmeans.cluster_centers_ # Coordinates of cluster centers.labels = kmeans.predict(X[X.columns[1:3]]) # Labels of each pointX.head(10)

X['cluster_label'] = kmeans.fit_predict(X[X.columns[1:3]])
centers = kmeans.cluster_centers_ # Coordinates of cluster centers.
labels = kmeans.predict(X[X.columns[1:3]]) # Labels of each pointX.head(10)

X.head(5)

X = X[['parcelid','cluster_label']]
X.head(5)

clustered_data = df.merge(X, left_on='parcelid', right_on='parcelid')
clustered_data.head(5)

centers = kmeans.cluster_centers_
print(centers)

# Final Result:
   parcelid  cluster_label
0  10754147              0
1  10759547              0
2  10843547              2
3  10859147              2
4  10879947              2

The data source is here:

https://www.kaggle.com/c/zillow-prize-1/data


Large amount of Sigmoid outputs are ones and zeros

Apparently the problem was that I forgot to normalize data. As I found out, sigmoid turns any input greater than 10 into 1 and inputs less than -10 into 0. Normalization solves this problem (as normalized values are between 0 and 1)


Is there a tool/ library/ algorithm that can learn from the steps I perform in cleaning text and apply them to similar data?

Trifacta (https://www.trifacta.com/) supposedly can do that (learn from some examples provided by the user. I have no interest in the company, but it comes from academic research that I'm familiar with (http://vis.stanford.edu/wrangler/). Google had something free online (Google Refine) that can also do some intuitive things very simply, but I don't think it's as well developed.
If you try either product, please let us know about your experience!


Confusion matrix, when mistakes below diagonal are better than above the diagonal

You are incorrect when you say accuracy and precision assume the same mistakes. They are quite different in nature as to when they are applied for separate use-cases.
For more details on how they are different - https://towardsdatascience.com/accuracy-precision-recall-or-f1-331fb37c5cb9
As per your situation, you could opt for precision as you are looking only for the right answers, in case of a spam email detection.
There is no metric as such which calculates how much your model underfits or overfits but a genera idea can be made when you compare the validation curve and the training curve.


Newbie question on restricted boltzmann machine

Here is an answer which might help: https://www.quora.com/What-are-best-approaches-to-modeling-real-value-continuous-input-variables-in-RBM-pretraining


how to get prediction from trained random forest model?

Use the CountVectorizer you have fitted to preprocess your custom input then feed it to your model for prediction.

custom_input = ['insert text here']
custom_input = count_vectorizer.transform(custom_input)
custom_prediction = random_forest.predict(custom_input)


Yes. PSI uses actual and expected datasets. These can be defined however you want. Expected could be training, hold-out, last quarter, last week, artificial, etc. Think of expected as the benchmark that the current (actual) is being compared to. Defining that is up to you and the problem you are trying to solve.


Pick a model from multiple models using a decision tree

Well, you can use this approach for stacking (AFAR this is how the process you are trying to perform is called):

1) Prepare a dataset in the format:

Your original features which you used to fit the 4 estimators (remove the target variable from it)
Add new target variable in it - the categorical output which shows which model should be picked up for that specific input. To obtain it - train your 4 estimators on your input and than label each input with the number of the estimator which has the closest output to the target

2) Train your stacker - on that dataset from #1 - for example the decision tree. Or any other classifier which will be able to accurately predict the number of the estimator which should be used for that specific input.

3) When using in combat mode - first run the stacker trained in #2 on new input and obtain an id of an estimator which best suites this example. Than run that estimator against new input record and get your response.

Hope that makes sense.
I tried it and it worked nicely for one of my problems.

Also you could try to employ some kind a 'blending' of these 4 predictors. Here is an approach you could take (my usage of terminology here becomes a bit frivolous since I am an engineer and not a scientist, so if someone provides a more elaborate answer on blending - please do).

First of all, recall how ANNs for classification work - and would work for you example, if you will use ANN as a stacker. You have input layer which reads the input values, some hidden layers and an output layer with a set of neurons, each neuron is associated with a specific class. For our case it would be 4 neurons. After you feed input each of 4 output neurons will output some real value, for example:
Neuron 1 - 5.4
Neuron 2 - 0.3
Neuron 3 - 15.3
Neuron 4 - 3.3

That output means that ANN thinks that the input belongs to 'class 3' and that you should use 3rd predictor for that example to get best results.

But! Let's think about probabilities. What is the probability of input to belong to ANY of 4 classes? It is 1.0 (since you have only 4 classes). And by default ANN doesn't have the knowledge that this the case. So for us it would be good if the ANN will output probability of data belong to X class instead of just some real value. For example we would like to obtain such output:

Neuron 1 - 0.6
Neuron 2 - 0.2
Neuron 3 - 0.15
Neuron 4 - 0.05

Which will read as 'I am 60% sure that you should use estimator 1 for that input, 20% sure  - that estimator 2' etc. Such objective is called 'cross-entropy'. Modern libraries for building ANNs support it and you could tell network to train to output these probabilities.

By now I believe you already got what I suggest to do. Prepare the dataset as I described at the beginning of the answer, than train an ANN with cross-entropy objective to output the probabilities. When you run it in combat mode:

1) For new input run it through ANN and obtain probabilities, like these:
Neuron 1 - 0.6
Neuron 2 - 0.2
Neuron 3 - 0.15
Neuron 4 - 0.05

2) Run each predictor against input and obtain their predictions
3) Multiply each prediction by the associated probability and sum the results:

predictor1_result*0.6 + predictor2_result*0.2 + predictor3_result*0.15 + predictor4_result*0.05

And get you final result.

Why do that? First of all, we can't be sure that each time we realy pick up the best estimator, since our stacker (ANN in our case) could make mistakes. By using such approach we give the stacker a chance to compensate for possible mistakes - which minimizes the average prediction error.

Also, consider that you get such output from ANN:
Neuron 1 - 0.45
Neuron 2 - 0.45
Neuron 3 - 0.05
Neuron 4 - 0.05

In such case the best option for us is to almost average the predictions from predictor 1 and 2 - since ANN is not really sure which of predictors to use.


Finding similar data columns with machine learning

If I'm not mistaken, you are looking for some metrics to investigate if two columns of values represent similar information. Right?

When the order of numbers is fixed for both columns (such that the ith elements of both columns relate to the same record) and hence both columns have the same length, then you can use a wide range of techniques to measure the similarity/distance between the two vectors such as cosine similarity, Pearson correlation coefficient, mutual information, etc.

If the order of numbers in the columns are different or they basically refer to different concepts, then the above techniques are meaningless. In this case, you can look at these columns as two populations of samples. In this case, you may compare their statistical distributions using their distribution parameters (eg mean and std for a normal distribution) or statistical tests (eg ANOVA). In some cases, the number of items (population size) can be different.


How many features to sample using Random Forests

I think in the original paper they suggest using $\log_2(N +1$), but either way the idea is the following:

The number of randomly selected features can influence the generalization error in two ways: selecting many features increases the strength of the individual trees whereas reducing the number of features leads to a lower correlation among the trees increasing the strength of the forest as a whole.

What's interesting is that the authors of Random Forests (pdf) find an empirical difference between classification and regression:

  An interesting difference between regression and classification is that the
  correlation increases quite slowly as the number of features used increases.

Therefore, for regression often $N/3$ is recommended, which gives larger values than $\sqrt N$.

In general, there is no clear justification for $\sqrt N$ or $\log N$ for classification problems other than that it has shown that lower correlation among trees can decrease generalization error enough to more than offset the decrease in strength of individual trees. In particular, the authors note that the range where this trade-off can decrease the generalization error is quite large:

  The in-between
  range is usually large. In this range, as the number of features goes up, the
  correlation increases, but PE*(tree) compensates by decreasing.

(PE* being the generalization error)

As they say in Elements of Statistical Learning:

  In practice the best values for these parameters will depend on the problem,
  and they should be treated as tuning parameters.

One thing your problem can depend on is the number of categorical variables. If you have many categorical variables that are encoded as dummy-variables it usually makes sense to increase the parameter. Again, from the Random Forests paper:

  When many of the variables are categorical, using a
  low [number of features] results in low correlation, but also low strength. [The number of features] must be
  increased to about two-three times $int(log_2M+1)$ to get enough strength to
  provide good test set accuracy.


Correct theoretical regularized objective function for XGB/LGBM (regression task)

For question (a), it should be as simple as adding the usual L1 penalty:
$$\begin{align*}
\Omega(f) &= \gamma T + \alpha \|w\|_1 + \frac12\lambda\|w\|_2^2 \\
&= \gamma T + \alpha \sum |w_i| + \frac12\lambda \sum w_i^2
\end{align*}$$
For (b), see LightGBM vs XGBoost
For (c), see L1 & L2 Regularization in Light GBM


depmixS4 - How to fit HMM usgin multiple sequences of observation

No, depmixS4 supports multiple external variables to be included to forecast underlying time series.   In this case transition matrix is a function of all the other external variables. It is given in depmix S4 vignette also.-

a useful material for start -
https://machinelearningstories.blogspot.com/2017/02/hidden-markov-model-session-1.html &
http://machinelearningstories.blogspot.in/2017_03_01_archive.html

R Code snippets-

Required library

library(depmixS4)

data loading-

physician_prescrition_data <-c(12,16,45,45,56,67,78,98,120,124,156)

model execution-

HMM_model <- depmixS4::depmix(physician_prescrition_data~1, nstates = 2,ntimes=length(physician_prescrition_data))

model fitting

HMM_fm <- fit(HMM_model)

Transition probabilties-

HMM_fm@transition

posterior states-

posterior(HMM_fm)
plot(ts(posterior(HMM_fm)[,1]))

Emission probabilties-

HMM_fm@response


What is a sliding-window convolutional neural network?

Sliding window in this context is regarding what is given as input to the CNN. It is a sliding window of the input image. I have seen it being used in medical domain where the images are too large to fit into a network and reshaping them into smaller sizes doesn't help. So, a sliding window is done on the bigger image and the sliding window is fed into CNN for achieving whatever task you are trying to achieve.


Anconda R version - How to upgrade to 4.0 and later

You need to create a new environment and then you can install R 4.+ in Anaconda. Follow these steps.
conda create --name r4-base

After activating r4-base run these commands
conda activate r4-base
conda install -c conda-forge r-base
conda install -c conda-forge/label/gcc7 r-base

Finally, you will notice r-basa version 4 will be installed.

Thereafter, you can install any supported packages. But with this only, you won't have the ability to use it in the Jupyter notebook. You need to install install.packages('IRkernel') and Jupyter notebook as well if you want to use it. Otherwise you are good to go with R-Studio.
For Jupyter Installation and RKernel.
conda install jupyter

Then open the R console. Write in R console
install.packages('IRkernel')
IRkernel::installspec()

Congrats! You can use Notebook for Python and R.


In Machine Learning, what is the point of using stratified sampling in selecting test set data?

When the distribution of your data is balanced or you have enough samples of each class, a normal shuffle split will work well. But if your data distribution is unbalanced and one of the classes is in minority, you do stratified sampling so as to ensure that your train and test splits represent the true nature of your data. For more details take a look here.


In doc2vec, how to model correctly when many documents share the same label?

I've tried to explain the logic behind labels used in Document vectors in Doc2Vec - How to label the paragraphs (gensim)

To answer your questions.

1) when two documents share the same label, then doc2vec algorithm determines the semantic meaning of the label from both the documents. Note that doc2vec learns the semantic meanings of labels not individual documents.

2) Again, you are not learning documents. you are instructing doc2vec to learn the embeddings for the labels. So, if multiple labels are given for a document, all receives the same semantic meaning from the document and when a part of labels when used in other documents, keep on learning more semantic meaning from them.
For instance. doc1-> hunt, bite, eat, flesh doc2-> life, love, eat, money. It is clear that doc1 is about animals and doc2 is about human, the label eat will have semantic meanings from both of them.

3) If your goal is to find similar question such as this, then you should probably give just a single label for the whole question and then find a question with a label having closer cosine distance.

4) Never confuse labels with words. In word2vec, words learn embeddings and in doc2vec, labels learn embeddings from the words used in documents. If you would like to add some more semantic meanings to a document, then you could add it to the particular document as words. If you want to add semantic meanings to a label, the words have to be added to each document which carries that label if you want the label to have strong affinity (but adding words manually, doesnt sound like a good option to me personally).


Found the solution:

  y <- factor(levels = c(1,3,4,5,7),
              labels = c('apple','pie','nuts','cookies','some_other_class')
             )


How to approach deep learning CV/Resume parser using Convolutions?

Typically, resumes are not images. Almost all resumes are Microsoft Word or pdf. Given those document formats, a deep learning parser should be sequence-based (e.g., Recurrent Neural Network (RNN) or Long Short Term Memory (LSTM)).

To apply Deep Learning, you'll need many thousands of examples with each section labeled. There is HR-XML (Human Resources - Extensible Markup Language) which are the industry standards for labels of resume sections.


Tensorflow / Deepmind: how do I take actions from observations for math algorithms related to proofs?

I would recommend you to have a look at OpenAI gym.
It will be easier for you to implement a new environment describing your problem. Regarding the agents, there are many implementations around that are compatible with OpenAI.


Neural Network not learning when more than 1 training data is given

The input has the following shape:

X = random.uniform(-5,5)
neunetwork.forward([[X]])

print(np.array([[X]]).shape) # == (1,1)

You could use numpy to generate X and y:

X = random.uniform(-5,5,size=(10,1)
Y = X**2

It would help to get a copy of the error message to make sure this is the problem.

The forward/backprop looks right in terms of accumulating gradients. I'm not sure about the following piece of code though:

dX = np.zeros((1,self.n_inputs))
for j in range(self.n_neurons):
        dX += 2*(target[0][j]-output[0][j])*self.weights.T[j]
self.targetprevX = self.prevX + dX*Lrate/self.n_neurons

I think you can utilize a dot product here.


What is the reason behind having low results using the data augmentation technique in NLP?

Text data is at the same time:

very structured, because swapping only a few words in a sentence can make it complete gibberish,
and very flexible, because there are usually many ways to express the same idea in a sentence.

As a consequence, it's very hard to have a text sample which is representative enough of a "population" text, i.e. which covers enough cases for all the possible inputs. But augmentation methods are practically sure to fail, because either they are going to make the text gibberish or just cover minor variations which don't improve the coverage significantly.

That's why a lot of the work in NLP is about experimental design and preprocessing.


Does converting continuous variable to discrete(categorical) variable increases accuracy of a tree based model?

I actually agreed with the now-deleted answer: you're unlikely to see improvement (at least with any consistency).

As you point out, binning a continuous variable just reduces the allowable split points for a tree model.  This reduces the capacity of the overall model.  That may be beneficial or detrimental, depending on how it affects the bias-variance tradeoff: I would expect it to help a lot on a fully-grown single decision tree, but probably hurt a little in a well-tuned ensemble of trees.

There are two nice examples in the sklearn docs illustrating their binning preprocessor:
https://scikit-learn.org/stable/auto_examples/preprocessing/plot_discretization.html
https://scikit-learn.org/stable/auto_examples/preprocessing/plot_discretization_classification.html

Extending the second one to (1) also include a full decision tree and (2) add binning-then-nonlinear-models produces the following:

At least in these simplistic examples, my suggestions above hold.


How to use deep learning to add local (e.g. repairing) transformations to images?

There is a paper called Spatial Transformer Networks written by Max Jaderberg et al. What it does is trying to find the canonical shape of its input by reducing transformations, like translation and rotation, or even diminishing the distortion of the inputs. It introduces a module which helps convolutional network to be spatial invariant. One of the significant achievements of this module is that it tries to enhance distorted inputs. Take a look at here.

I edit the answer because of the request of one of our friends. First I quote something from the popular book written by Pr. Gonzalez, I hope nothing goes wrong with copyright. Then I suggest my recommendation.

Figure 2.40, the one that I've attached, shows an example of the steps in Fig. 2.39. In this case, the transform used was the Fourier transform, which we mention briefly later in this section and discuss in detail in Chapter 4. Figure 2.40(a) is an image corrupted by sinusoidal interference and Fig. 2.40(b) is the magnitude of its Fourier transform, which is the output of the first stage in Fig. 2.39. As you will learn in Chapter 4, sinusoidal interference in the spatial domain appears as bright bursts of intensity in the transform domain. In this case, the bursts are in a circular pattern that can be seen in Fig. 2.40(b). Figure 2.40(c) shows a mask
image (called a filter) with white and black representing 1 and 0, respectively.
For this example, the operation in the second box of Fig. 2.39 is to multiply the mask by the transform, thus eliminating the bursts responsible for the interference. Figure 2.40(d) shows the final result, obtained by computing the inverse of the modified transform. The interference is no longer visible, and important detail is quite clear. In fact, you can even see the fiducial marks (faint crosses) that are used for image alignment.

Okay! After those quotes, I refer to my suggestion. In the ST network, the authors have claimed that their differentiable module can learn different kinds of transformations other than affine transformation. The point about that is that we usually apply two kinds of transformations. One transformation is applied to the intensity of the image, which is popular by the means of filters. The other one is called image warping where we change the position of intensities and the intensity values do not change unless they locate between the discrete grids of image entries also called pixels, picture element. Spatial transformers are good for the second task but they also can be used for the first task. There are studies about using CNNs for evaluating the images in their frequency domain and not in the spatial domain. You can employ these differentiable modules in those nets.

Finally about your specific task, what I'm seeing in your pictures, your noise has a same behaviour. I guess if that is true for all cases, your task is not learning and can be solved using usual image processing techniques.


XGBoost multiclassification interpreting predicted probabilities

Multiclassification with xgboost is normally done One vs All. Meaning that first you try to predict if it belongs to class one or not. Then two second class or not...

You can understand as the probability of belongin to one class and not to the rest.

Let's put an example for better understanding: say that you have a ton of patients that have 3 illnes: Corona Virus, Sars and HIV.

You build your multiclassification model with xgboost or whatever and for a new patient you do a prediction and check the probabilities:

Corona [0.8]
Sars [0.1]
HIV [0.01]
Not sick [0.09]

What you do from here is select the most probable cause (corona) and that is your final class.

If you do multilabel you can get something like this too
Corona [0.9]
Sars [0.20]
HIV [0.999]

Meaning that even if it has a high probability of getting corona it also has a higher probability of having HIV. In multiclassification is different from multilabel where you can have a patient that has both of the sickness

From the sklearn documentation:

  The one-vs-the-rest meta-classifier also implements a predict_proba method,
  so long as such a method is implemented by the base classifier. This method
  returns probabilities of class membership in both the single label and
  multilabel case.  Note that in the multilabel case, probabilities are the
  marginal probability that a given sample falls in the given class. As such, in
  the multilabel case the sum of these probabilities over all possible labels
  for a given sample will not sum to unity, as they do in the single label
  case.

In summary, if sickness are independent it will predict the probability of the patient having one sickness, there has to be a class not sick just if there is the possibility for the patient not being sick

Note: In NN is different


How to sort two dimensional array in Java?

There is an overloaded sort method in java.util.Arrays class which takes two arguments: the array to sort and a java.util.Comparator object.

You can add the following lines of code with your program to get the expected result.

import java.util.Arrays;
import java.util.Comparator;

Arrays.sort(testdatset, new Comparator<double[]>() {
        @Override
        public int compare(double[] o1, double[] o2) {
            return Double.compare(o2[1], o1[1]);
        }
    });


How to find unknown number of clusters in circular data?

Instead of projecting into the circle and thus making your problem 2d, why don't you just use a cyclic distance measure?

This problem should be straightforward by doing kernel density estimation on the (cyclic) time of day. Then find the peaks, which are your clusters.


How to identify the feature that make the model misclassifed in text classification

I think we would need more information about the model architecture you are using and the features you engineered for training.
Something I like to do when starting to work on text classification problems is to gradually increase the complexity of the architecture and start from simple approaches.
So if you are using tf-idf features and some sort of probabilistic model, such as logistic regression you may have a look at libraries such as Lime or eli5 that can help you make sense and explain the prediction of your classification model.
In your case you may then look at the top-features of the misclassified text and increase your understanding of why the model derived at its decision.


How can I train my CNN to learn (numerically) smaller values better?

It seems you are working on a regression problem with a strictly positive target and use MAPE as the loss function to train the neural network. This loss function could be the culprit. As illustrated nicely in this answer, minimizing MAPE leads to predictions that are biased towards smaller values. This is a consequence of the asymmetry between errors in the positive and negative directions (negative errors are 100% at worst, while positive errors are unbounded).

A possible alternative is to minimize the mean absolute error computed with the logarithm of the target. It will converge to the logarithm of the target's median.


Assuming each coordinate axis has a fixed max and min value, and $|min_{x}| = max_{x}$, $|min_{y}| = max_{y}$, $|min_{z}| = max_{z}$, the standard normalisation procedure would be:

$$ (x,y,z) \mapsto (\frac{x}{max_{x}},\frac{y}{max_{y}},\frac{z}{max_{z}})$$

I.e. mapping the values to within the range $(-1,-1,-1)$ to $(1,1,1)$.

E.g. if your corrdinate system was fixed between $-100$ and $100$ for each axis, the coordinate $(10,-52,-8)$ would normalise to:

$$ (10,-52,-8) \mapsto (\frac{10}{100},\frac{-52}{100},\frac{-8}{100}) = (0.1, -0.52, -0.08)$$


To remove Chinese characters as features -

If you want to remove non-English characters then this regex will work, by selecting characters not in a given ASCII range (0 to 122, you can adjust this since it will allow some special characters):

([^\x00-\x7A])+

So to remove those characters:

data = re.sub("([^\x00-\x7F])+"," ",data)


import numpy as np

a = np.array(['PAIDOFF', 'COLLECTION', 'COLLECTION', 'PAIDOFF'])

f = lambda x: 1 if x == "COLLECTION" else 0

np.fromiter(map(f,a),dtype=np.int)

Alternative:
np.where(a == "COLLECTION",1,0)


Does shuffling the training data matter in a K Nearest Neighbors Classifier model?

If you are talking about K-nn algorithm per se, then, you do not have any advantage in shuffling your training set. Indeed, as you said, K-nn is just looking for the K nearest neighbors and does not care at all about the order of the samples, the algorithm will scan the entire training set for a single prediction, no matter what (unless you are using some efficient implementation like ball-tree or kd-tree).
Regarding the second point of your question, I am wondering what your pipeline is: are you augmenting the dataset before splitting in train-validation-test? If so, I think you are injecting some bias in the evaluation of you algorithm, because you are evaluating your algorithm in samples that do not exist. You should augment only the training set and then validate/test your algorithm on real data.


What is the difference between multi-layer perceptron and generalized feed forward neural network?

Well you missed the diagram they provided for the GFNN. Here is the diagram from their page:

Clearly you can see what the GFNN does, unlike MLP the inputs are applied to the hidden layers also. While in MLP the only way information can travel to hidden layers is through previous layers, in GFNN the input information is directly available to the hidden layers.

I might add this type of connections are used in ResNet CNN, which increased its performance dramatically compared to other CNN architectures.


Understanding computations of Perceptron and Multi-Layer Perceptrons on Geometric level

The two pictures you show illustrate how to interprete one perceptron  and a MLP consisting of 3 layers.
Let us discuss the geometry behind one perceptron first, before explaining the image.
We consider a perceptron with $n$ inputs. Thus let $\mathbf{x} \in \mathbb{R}^{n}$ be the input vector, $\mathbf{w} \in \mathbb{R}^{n}$ be the weights, and let $b \in \mathbb{R}$ be the bias. Let us assume that $\mathbf{w} \neq \mathbf{0}$ in all subsequent parts.
By definition, a perceptron is a function
$f(\mathbf{x}) = \begin{cases}
1 & \mathbf{w}^{T} \mathbf{x}+b >0, \\
0 & \text{otherwise.}
\end{cases}$.
Now lets simplify this for a moment and assume that $b = 0$.
The set  $H = \{\mathbf{x} \in \mathbb{R}^n \mid \mathbf{w}^T \mathbf{x} = 0\}$ is called hyperplane, which is a subspace with $\dim(H) = n-1$.
By definition, $H = \mathbf{w}^\perp$, so $H$ is the orthogonal complement of the space $\mathbb{R}\mathbf{w}$.
In simple terms, this means:
For $n = 2$, $H$ has dimension $1$, which is a line that goes through the origin. The line is orthogonal to $\mathbf{w}$. This explains how to get the line, given $\mathbf{w}$ and vice versa. For example, given $\mathbf{w}$, simply draw a line that goes through the origin and is orthogonal to $\mathbf{w}$.
For $n \in \mathbb{N}$, you proceed the same, just that the dimension of $H$ might be higher (for $n=3$ you would need to draw a plane).
In your picture: you see the line in black color.
Note however that the line does not go through the origin. This is handled in the case of $b \neq 0 $.
So let $b \neq 0 $ and let $\mathbf{x}' \in \mathbb{R}^n$ such that $\langle \mathbf{x}',\mathbf{w} \rangle  = -b$.
For any $\mathbf{x} \in H$ we have $\langle \mathbf{x}'+\mathbf{x},\mathbf{w} \rangle  = \langle \mathbf{x}',\mathbf{w} \rangle  + \langle \mathbf{x},\mathbf{w} \rangle  = -b$.
Therefore, $\{\mathbf{x}'+\mathbf{x} \in \mathbb{R}^n \mid \mathbf{x} \in H\} \subset \{\mathbf{x} \in \mathbb{R}^n \mid \mathbf{w}^T \mathbf{x} = -b\}$
Now let $\mathbf{x} \in \{\mathbf{x} \in \mathbb{R}^n \mid \mathbf{w}^T \mathbf{x} = b\}$, then $\mathbf{x} = (\mathbf{x}-\mathbf{x}')+\mathbf{x}'$. Since $\langle \mathbf{x}-\mathbf{x}',\mathbf{w} \rangle  = \langle \mathbf{x},\mathbf{w} \rangle -\langle \mathbf{x}',\mathbf{w} \rangle  = -b+b= 0$, we have
$\{\mathbf{x}'+\mathbf{x} \in \mathbb{R}^n \mid \mathbf{x} \in H\} = \{\mathbf{x} \in \mathbb{R}^n \mid \mathbf{w}^T \mathbf{x} = -b\}$
In simple terms, this means:
The set $\{\mathbf{x} \in \mathbb{R}^n \mid \mathbf{w}^T \mathbf{x} = -b\}=\{\mathbf{x} \in \mathbb{R}^n \mid \mathbf{w}^T \mathbf{x} +b= 0\}$
is nothing but the set $H$ translated by $\mathbf{x}'$.
In particular for $n=2$, the line is translated by $\mathbf{x}'$.
This explains how to describe the line that is depicted in your image.
From the Hesse normal form of the line, you get $\mathbf{w}$ and $b$.
Given $b$ and $\mathbf{w}$, you get $\mathbf{x}'$ by defining  $\mathbf{x}'$ with $\langle \mathbf{x}',\mathbf{w} \rangle  = -b$. Let $i \in \{1,\ldots,n \}$ with $w_{i} \neq 0$. Then $\mathbf{x}' := \mathbf{e}_{i}\lambda$ with  $\lambda = \frac{-b}{w_{i}}$ satisfies $\langle \mathbf{x}',\mathbf{w} \rangle  = -b$, where $\mathbf{e}_{i} \in \mathbb{R}^{n}$ is the vector that is everywhere $0$ except at position $i$, where it has the value $1$.
In simple terms this means, you know how to draw the line given $\mathbf{w}$ and $b$, and vice versa.
Finally, $H^{+} := \{ \mathbf{x} \in \mathbb{R}^{n} \mid \mathbf{w}^T \mathbf{x} +b > 0 \} = \{ \mathbf{x} \in \mathbb{R}^{n} \mid \mathbf{w}^T \mathbf{x}  > -b \}$ is the upper-half space and $H^{-} := \{ \mathbf{x} \in \mathbb{R}^{n} \mid \mathbf{w}^T \mathbf{x} +b \leq  0 \}$ is the lower half-space given by $H$. The purple area in the image is now exactly the upper half-space $H^{+}$ (the area "above" the line), and of course, $f(x) = \begin{cases}
1 & x \in H^{+} \\
0 & \text{otherwise}
\end{cases}$.
Now lets have a look at the upper picture again. It corresponds to three different "trained" perceptrons. The line $H$ seperates the 2D space into two half-spaces. Everything in the purple area gets the value $1$, everything at the opposite gets the value $0$.  Therefore, the perceptron is completely defined by drawing $H$. It defines what value each vector will be assigned to.
Thus, a perceptron is able to represent for example the OR function (left example), as you can seperate $00$, from $01,10$ and $11$.
Meanwhile, the XOR function cannot be represented by a perceptron, as you cannot seperate the points of each class by a line $H$.
Now the bottom picture is about an MLP consisting of 3 layers.
Each neuron in the hidden layer corresponds again to one hyperplane.
Such an MLP thus maintains multiple hyperplanes.
Let us assume we have $k$ neurons in the hidden layer.
Now instead of asking if a vector is within the upper half-space or lower half-space of one hyperplae, a MLP describes the location of a point $\mathbf{x} \in \mathbb{R}^{n}$ with respect to all $k$ hyperplanes.
The output of a node in the last layer (output layer) is computed as $\phi(\sum_{i = 1}^{k}{w_{i}y_{i}}+b')$, where $y_{i}$ is the output of node $i$ of the hidden layer (either 1 or 0, as described before), $\phi$ is some activation function and $w_{i}$ is the corresponding weight.
Let us assume that $w_{i} = 1$ for all $i$ (as in your example image), and let us consider $F:= \sum_{i = 1}^{k}{y_{i}}$ first.
If $F = u$, this means, there are $u$ many nodes in the hidden layer that output $1$, given the input $\mathbf{x}$. Let $l_{1},\ldots,l_{u} \in \{1,\ldots,k \}$ be the indices of these nodes. For each node $i$ of the hidden layer, let $H^{+}_{i}$ be the corresponding upper half-space and $H^{-}_{i}$ be the corresponding lower half-space.
Then, we know that $\mathbf{x} \in H^{+}_{l_{r}}$ for all $r = 1,\ldots,u$ and $\mathbf{x} \in H^{-}_{j}$, for all $j \in \{1,\ldots, k \} \setminus \{l_{1},\ldots,l_{u}\}$.
In simple terms:
If $F =u$, the input $\mathbf{x}$ must be in exactly $u$-many upper-half spaces (and $k-u$-many lower half-spaces).
Now let $\phi$ be again the heavy-side function, thus $\phi(t)=1$ if $t > 0$ and $\phi(t) = 0$ for $t \leq 0$.
Then $\phi(F+b') = 1 \Longleftrightarrow F+b' > 0 \Longleftrightarrow F > b'$.
Therefore, the network will output $1$, if $\mathbf{x}$ is contained in at least $(b'+1)$-many upper half-spaces.
In the example picture, there are 5 hyperplanes and it will output 1, if the ínput vector $\mathbf{x}$ is in the center region.
In simple terms, the MLP uses a finite  arrangement of hyperplanes, see also Stanley.
Each cell (or region) is assigned either to class $0$ or $1$. So the MLP assigns to all vectors within these regions (which are polyhedrons) the same value (either $0$ or $1$).
Now using a different activation function in the hidden layer correspond to using some sort of distance measurement.  With the  perceptron, all points within on cell are assigned the same value. With functions like sigmoid, it would take into accout, how close the vector $\mathbf{x}$ is to the border (the hyperplanes).
Using weights different than $w_{i}=1$, corresponds in grouping different cells together.
Example: Let $n=2$ with $k=3$ hidden nodes, $w_{1} = 1 = w_{2}$ and $w_{3}=-2$. Then $F \in \{-2,-1,0,1,2\}$.
If $F = 0$, then $y_{1} = y_{2} = y_{3}= 0 $ or $y_{1} = y_{2} = y_{3}$.
If $F = 1$, then $y_{3} = 0$ and (either $y_{1} = 1$ or $y_{2} = 1$).
If $F = 2$, then $y_{3} = 0$ and $y_{1} = 1 = y_{2} $.
if $F = -1$, then $y_{3} = 1$ and (either $y_{1} = 1$ or $ y_{2} = 1$).
If $F = -2$, then $y_{3} = 1$, $y_{1} = y_{2} = 0$.
If you set the weight from input to hidden layer to $1$, you will get a representation of XOR.
If you use $b' = 1.5$ you get $\phi(F+b') = 1 \Longleftrightarrow F \geq 2$. Thus $\mathbf{x} \in H^{+}_{1} \cap H^{+}_{2} \cap H^{-}_{3}$  if and only if the MLP will map $\mathbf{x}$ to $1$.
With constant $1$ weights between hidden and outputs layer however,
the MLP will map $\mathbf{x}$ to $1$, if and only if: (1),(2),(3) or (4) holds:
(1): $\mathbf{x} \in H^{+}_{1} \cap H^{+}_{2} \cap H^{-}_{3}$
(2): $\mathbf{x} \in H^{+}_{1} \cap H^{+}_{3} \cap H^{-}_{2}$
(3): $\mathbf{x} \in H^{+}_{2} \cap H^{+}_{3} \cap H^{-}_{1}$
(4): $\mathbf{x} \in H^{+}_{1} \cap H^{+}_{2} \cap H^{+}_{3}$


Clustering tendency using hopkin's statistics

I don't think the Hopkins statistic is useful for this purpose at all.

It is essentially a test for a uniform distribution.

But not having a uniform distribution does not mean the data is suitable for cluster analysis.

For example a single Gaussian distribution (unimodal!) will score high on this test, but the data doesn't have multiple clusters but all points are from the same distribution.


What do you pass for the cv parameter in the sklearn method cross_val_score

It determines the splitting strategy used by sklearn.

The default (“none”) is 3-fold CV.

Doc


We can show that the "sum above threshold" is not succinct by providing a counter example.

As you wrote the definition is

  Given A1, the set of items satisfying a succinctness constraint C, then any set S satisfying C is based on A1 , i.e., S contains a subset belonging to A1

Hence, we can provide a counter example by providing a set A1 satisfying the constraint while none of its subsets satisfying it.

Consider three items, a,b and c such that each one on them costs 1.
Let the constraint C be sum(S.Price) >= 3

For the set {a, b, c} the sum of prices is 3 and therefore the constraint C is satisfied.
For each of the subsets of {a, b, c} the sum of prices is lower than 3 and therefore C is not satisfied.
We found a counter example in which a set satisfies a "sum above threshold" while none of its subsets satisfies it. Hence, "sum above threshold" is not succinct.


Customer Segmentation: Should I use a variable, representing a product, that is unpopular in the dataset for K-Means Clustering?

I suggest you use an algorithm that accommodates categorical variable since there are missing data. You can one hot encode it so missing data will be relevant. Making it zero will be misleading.

Try algorithms like tSNE and Self Organizing Map and use the Jaccard/Tanimoto distance.


Why would a fake feature with random numbers get selected in feature importance?

Scikit-learn's random forest feature importance is based on mean decrease in impurity, which is fast to compute and faithful to the original creation of the Random Forest method. In short, the default feature_importances_ gives a numerical justification of the Random Forest's representation of the feature importance using its native metric of construction. Like you saw, this metric has the drawback that it can say noise is an important feature. So you may want to consider other feature importance methods, like permutation feature importance, which will give you a more apples-to-apples comparison with other models you will test. There are pros and cons to many of these methods so be aware of them.


Indeed, convolution and cross-correlation are closely related. The former is a bit more natural in some areas of mathematics; most notably, in the convolution theorem for the Fourier transform, which states that the Fourier transform of the convolution of two functions is equal, under certain conditions, to the product of their Fourier transforms:
$$ \mathcal{F}\{f * g\} = \mathcal{F}\{f\} \mathcal{F}\{g\}, $$
thus providing an efficient way to compute the convolution.

In the context of CNN, there difference between the convolution and the cross-correlation is irrelevant. In the discrete two-dimensional case
$$ (f {**} g)[n_1, n_2] = \sum_{m_1 = -\infty}^{+\infty}\sum_{m_2 = -\infty}^{+\infty} f[m_1, m_2] \, g[n_1 - m_1, n_2 - m_2], $$
$$ (f' {\star\!\star} g)[n_1, n_2] = \sum_{m_1 = -\infty}^{+\infty}\sum_{m_2 = -\infty}^{+\infty} \overline{f'[m_1, m_2]} \, g[n_1 + m_1, n_2 + m_2]. $$
If $f$ is the filter and $g$ is the output of the previous layer of the CNN, you can see that the convolution is equivalent to cross-correlation with a filter
$$ f'[m_1, m_2] = \overline{f[-m_1, -m_2]}, $$
which is just a reflection around the secondary diagonal since all values are real.

In short, there are no benefits to using the true convolution in CNN. The only difference would be that the learned filters would be mirrored about the secondary diagonal.


Is there a difference between on-line learning, incremental learning and sequential learning?

So, the following answer is just based on different opinions of collegues and professors from the field. I want to try to summarize it briefly:

Sequential and online learning is mostly associated with Bayesian updating. The sequential learning is used widely for an order in time of the data, meaning that $x_1$ is coming always first, then $x_2$, then $x_3$ and so on. The Dataset each has a certain order in that sense.
In contrast to that incremental may be a whole block of data at time x and another block of data at time y. While the block internally may be randomly ordered.

Concerning online learning, the people mostly referred to a data stream, hence a online learning is always incremental learning but incremental learning does not have to be online.

These are quite fuzzy definitions, and in my opinion there is not clear definition though. I still hope that helps.


From Focal Loss for Dense Object Detection paper, focal loss is defined as:

$$FL(p_t) = −(1 − p_t)^γ log(p_t)$$

It is a generalized formation of a loss function that can be directly implemented for any classification problem, including text classification.

Focal loss function is designed for problems with 1:1000 training imbalance. Does your data actually have that much imbalance? If not, you can use much simpler methods (i.e., resampling).


In U-Net, is there a non-linearity (relu) in up-convolution layer?

I'd not use activation function in these layers because that's how I saw it done first time, but it's probably worth trying since I can't come to a reason why we woudn't use activation function in this layer, it probably shoudn't change the results much.
U-Net anyway has an architecture with convolutions between each upsampling, which use activation functions, so not using activation function in upsampling layers is not a big deal.


Streamlit widgets doesn't display in Jupyter Notebook

You don't run the code from a notebook. You create a python script with your code and then use this command from a shell: streamlit run yourfile.py
the output will show up in the shell

Reason for this is that streamlit runs on the main thread so it would block the jupyter kernal, you won't be able to use the notebook while the streamlit app is running. However, if you really want to stay inside Jupyter to run the app, you can run the command in a cell, like this !streamlit run yourfile.py


Appropriate model metric for a truncated response variable?

Either RMSE or MAE will give you a sense of accuracy in prediction.  The exact value of any error metric is up to your specific use case and tolerance.

In other words, if you are predicting ages of 5-10 year olds and see a RMSE of 5 then you might think that you have a significant problem.  But if you are predicting ages expecting them to be 40-80 year olds, then a RMSE of 5 wouldn't be a big concern right?


How to run Spark python code in Jupyter Notebook via command prompt

First time when you executed your code, you might have executed it from the same location where your te.csv file was present.
Are you executing this code in the same location as before? If not, then I suggest you put in and reference the absolute path of te.csv in your code. This should then run without a hitch.


I am trying to implement Isolation forest for anomaly detection but I am not able to understand and visualise the decision function

According to this answer the range of output from scikit-learn IsolationForest decision_function is between -0.5 and 0.5, where smaller values mean more anomalous.
The predict function then applies a threshold to this function to get either -1 (anomaly) or 1 (not anomaly). The decision threshold is stored as model.threshold_ and defaults to 0.0.
So if you want to mark the decision threshold in a decision boundary plot, then you could use that value unless you have chosen another one. Note that it is quite common in Anomaly Detection to select your own threshold. Ideally it is selected by using a labeled validation set, or otherwise based on the histogram of anomaly scores.
As for interpreting the values that contribute to the anomaly scores, I would recommend using Shap for IsolationForest.


After the training phase, is it better to run neural networks on a GPU or CPU?

This depends on many factors, such as the neural network architecture (CNNs tend to be better optimized than RNN on GPU) as well as how many test samples you give as input to the neural network (GPUs can be even faster when given a batch of samples instead of a single sample).

As an example, here is a benchmark comparing CPU with GPU on different CNN-based architectures. The forward pass is much slower on a CPU in that case:

FYI: Benchmarks based on neural networks libraries to compare the performance between different GPUs


But what happens if the input has 0.99 and 0.01?

indicates the point: if the third layer is perfect, then we're already done; but we need to account for small variations in the third layer.  We need to understand how far off we can get after multiplying by $W$.

Now, it appears that the answer gets this a little wrong.  If $\vec{x}$ consists of one entry at least 0.99 and nine entries of at most 0.01, then $W\vec{x}$ is off by some number depending on the ten columns of $W$, not the four rows.  And we need lower bounds for the correct slots, and upper bounds for the incorrect slots (rather than upper bounds for both).

For example, if $\vec{x}=\langle 0.05, 0.997, 0.1, 0.01, \dotsc\rangle$, we'd get

$$W\vec{x} =  \begin{pmatrix}
 0.997 +0.01 + \dotsb \\
 0.1 +0.01 + \dotsb \\
\vdots
\end{pmatrix}.$$

We need to use bias/activation/cutoffs to "push" this to the desired $\langle 1,0,0,0\rangle$ (and similarly for other $\vec{x}$).  Well, the entries where we desire a 1 are at least 0.99 in the relevant position and at least 0 elsewhere (seemingly the activations are supposed to make everything positive?), so we get at least 0.99.  The entries where we desire a 0 are at most 0.01 in every non-zero position, so we get at most $5*0.01$ (the first row of $W$ has the most nonzero entries, 5).  So setting any cutoff (with relevant a bias and/or activation function) between 0.05 and 0.99 will suffice.

[Another example, since that might have been too wordy.  Say the output in the third row is
$$\vec{x}=\langle 0, 0.01, 0.99, 0.01, 0, 0.01, 0.01, 0.01, 0, 0.01\rangle.$$
Then
$$W\vec{x} = \langle 0.05, 1.02, 0.03, 0.01\rangle.$$
You can see how I've cooked up $\vec{x}$ to get the largest possible first entry in $W\vec{x}$ given the constraints on $\vec{x}$.]


Predict when a customer would buy from previous buying history (Mining Buying patterns)

I would try using xgboost classifier. In my case, xgboost outperformed Random Forest.

import xgboost as xgb
xgb_model = xgb.XGBClassifier(
    learning_rate=0.1, n_estimators=1000, max_depth=3, min_child_weight=1,
    gamma=0, subsample=0.8, colsample_bytree=0.8, objective='binary:logistic',
    nthread=4, scale_pos_weight=1, seed=27)

You can use the below snippet to avoid over fitting problem.

xgb_model = xgb_model.fit(X_train, y_train.astype(int), early_stopping_rounds=50,
                          eval_set=[(X_test, y_test['target'])])


This plot not use frequencies but kernel density:

freqz <- with(data.frame(x,y), MASS::kde2d(x, y, n = 50))
with(freqz, plot_ly(x = x, y = y, z = z, type = "surface"))


How to improve deep learning model having less data

First have a look at your data scale. Deep learning models are sensitive to data scaling so it would be better to preprocess your data to keep in within accetable ranges:

scale [-2,2] - apply scaling so that all features and label are within this range
mean 0: try to center your data around 0

Second, deep learning models are regularised overfitting models. One thing you can try here is to progressively increase the size of your model (i.e. more hidden layers or hidden units) and also add regularisation.
Your options are:

weight decay - keras Dense layers can be augmented with weight decay. Have a look at the documentation

layer regularisation (batch norm, dropout, layer normalisation) these are a bit more advanced and work on a case by case basis - have a look at this paper as a start.


How to process the hyphenated english words for any nlp problem?

They all sound like interesting approaches. The first one is better I think because it allows for unseen hyphenated words to be somewhat understood (as e.g. well + known ~= well-known).
For a tfidf BOW model, you might get good performance from any of the above.
For a model that is sensitive to word order I would certainly go with the first option and might tokenise the text so that I had a token to represent the hyphen too.


Using a LinearSVC() for multilabel classification with MultiOutputClassifier() in a pipeline in sci-kit learn

When you run your grid search, the clf step of the pipeline is replaced by each of RandomForestClassifier, LinearSVC, GaussianNB; you never actually use the MultiOutputClassifier.

You should be able to just wrap the two offending classifiers with a MultiOutputClassifier. You'll need to prefix your hyperparameters with estimator__ to get through the MOC into the underlying classifier: clf__estimator__C.


What is the right way to store datasets for a CNN project

You can use google drive to save these and import the drive on your code.


What is the difference between 'if the data is of good quality' and 'if the data is tidy'?

Let me try to explain by intuitively. First let me take the easy one.

Data being tidy

As per definition Tidy means Arranged in Order, Neat, Uncluttered.
All of these explain the physical aspects of the data representation. For example, data arranged in proper columns, with good headings, with relevance etc. You can think of this being syntactic in nature

Good Quality Data

This refers to actual data elements in the file. Like not having too many blank (missing) values. Every column having proper data types. Data without special (unreadable) characters. Properly encoded. Not having too much unstructured data. You can think this as the semantics of the data.

Hope this helps


Time series classification with convolutional neural network

I'm not convinced 24% accuracy is very poor, depending on what your dataset looks like. Your baseline accuracy is in the realm of 3.5% with 28 classes. But let's assume you want to do better.

CNNs for images use 2-d convolution filters, because there is an expectation that pixels that are close together should be processed together. I guess you could use a 2-d convolution on the 90x59 matrix, but it isn't clear to me that we should be looking for information across "close" sensors. Perhaps you can use some sort of "tall" 2-d convolution, so that you capture information across all sensors in the same time span.

You may want to try training a model on each channel and ensembling them instead, as you mention. If you do this, I guess you could use a 1-d convolution to process each channel, but I don't really see why you would. Is there a reason you feel the need to use CNNs? There are very likely more appropriate models specific to processing time-series data.


Weights shared by different parts of a transformer model

Updated answer
The Transformer model has 2 parts: encoder and decoder.
Both encoder and decoder are comprised of a sequence of attention layers. Each layer is comprised of a combination of multi-head attention blocks, positional feedforward layers, normalization, and residual connections.
The attention layers from the encoder and decoder are slightly different: the encoder only has self-attention blocks while the decoder alternates self-attention with encoder attention blocks. Also, the self-attention blocks are masked to ensure causal predictions (i.e. the prediction of token N only depends on the previous N - 1 tokens, and not on the future ones).
In the blocks in the attention layers no parameters are shared.
Apart from that, there are other trainable elements that we have not mentioned: the source and target embeddings and the linear projection in the decoder before the final softmax.
The source and target embeddings can be shared or not. This is a design decision. They are normally shared if the token vocabulary is shared, and this normally happens when you have languages with the same script (i.e. the Latin alphabet). If your source and target languages are e.g. English and Chinese, which have different writing systems, your token vocabularies would probably not be shared, and then the embeddings wouldn't be shared either.
Then, the linear projection before the softmax can be shared with the output embedding matrix. This is also a design decision. It is frequent to share them.
Finally, the positional embeddings (which can be either trainable or pre-computed) are shared for the source and target languages.

Old Answer (mistakenly addressing multi-encoder/decoder transformers)
In general, parameters are not shared in multi-encoder/multi-decoder transformer architectures, in the sense that each encoder/decoder has its own parameters. This is because sharing parameters defeats the very purpose of having multiple encoders/decoders: if you have two encoders that share parameters, they are effectively the same encoder.
This, of course, is not a rule, and there may be cases where it makes sense to share some or all encoder/decoder parameters. An example could be a scenario where two sentences in a certain language (or two very similar languages, like dialects) are received as input. It may be beneficial to share parameters between both encoders if the amount of data in one of them is not enough.
Background: multi-encoder and/or multi-decoder Transformers are generally not used for "simple" sequence to sequence tasks like machine translation. There are certain special cases where this kind of architectures have been used with different purposes, e.g.:

Automatic Post-Editing (APE) consists of having a primary machine translation system which output is refined with a secondary translation system that corrects the errors from the primary system. The secondary system normally receives as inputs both the original source sentence and the translation from the primary system and generates the fixed translation as output. An option for this kind of scenario is having a dual-encoder single-decoder transformer, where the output of the encoders are both fed to the decoder, either concatenating them (example) or injecting them to different attention blocks (example). In this scenario, normally there is no parameter sharing at all.

Multimodal translation: in this case, we receive different data modalities (e.g. speech and text), so we need modality-specific architectures, so normally there is no parameter sharing.

Note that here I understand that the multiplicity in encoders and/or decoders happens both at training and inference time. There are other cases where such a multiplicity happens only at training time (as in multi-task learning), but at inference time only one encoder and decoder are selected for each case, e.g.:

Multilingual translation: while many multilingual MT systems have a single model for all supported languages (normally supplying special tokens that identify the source/target language), it is possible to have language-specific encoders or decoders. You can see examples of this kind of architecture in works like this or this. In this scenario, each encoder/decoder has its own parameters and there is no sharing.


Why Epochs take longer as learning proceeds?

As the number of epochs increase the error goes down and the neural network has less to learn from the given data. The learning rate also decreases towards the end which makes the calculations of the gradients and weight updates numerically unstable. These are some of the reasons that contribute to slow learning. You can have a look here Quora for more details where Ian Goodfellow has answered a similar question.


The xyzSearchCV classes in sklearn require the estimator to be compatible with the sklearn API; in particular, the estimator('s clones)'s hyperparameters get set using set_params, which needs to be defined as a class method for the estimator.  So a Keras Sequential model will not work.  Keras does provide an sklearn wrapper: https://keras.io/scikit-learn-api/