How to decide on using xgboost with imputation or without it and keeping missing values?

The first question about missing data is always why is it missing?
Have you checked or know why the data is missing and whether it is MAR, MCAR or not missing at random?
If your data is MCAR imputation is generally fine and your lower test metric might simply indicate a suboptimal imputation strategy. In this case you could try MICE or similar more advanced imputations than simple median imputation.
That not imputing missing values actually improves your prediction might indicate that your data isn't missing completely at random however. In this case coding missing values might improve your performance and therefore be the best course of action.
As described besides simply looking at the performance metric of a properly validated test set also try to understand why data is missing and what that would mean.


First Differencing to remove seasonality and trends

Since the data is recorded every month (i.e. a data point for each month in the year) and we see a yearly seasonality trend we compare the data for the same month against previous year. This is done by taking the difference against the data from last year (.diff()), which is equal to going back 12 observations since each observation is data for a specific month.


How should labeled data from multiple annotators be prepared for ML text classification?

Most machine learning algorithms are designed with complete trust in the labels. There is no standard way to model uncertainty in data labels. Thus, create a project-specific threshold for uncertainty to omit data or labelers. For example, a trusted classification data label would require n of m ensemble voting.

One major issue is re-labeling. Systems tend to evolve over time and the definition of labels is refined. Mature data labeling system have a notion of data lineage  - "Who labeled what data when with what criteria".

The book "Human-in-the-Loop Machine Learning" by Robert Munro goes into greater detail.


What is ChunkParserI in nltk.chunk ? What exactly it has been called for?

Parsing is the process of decomposing a string into it's constituent symbols (if the string is a word or a sequence of characters) or syntactic components (if the string is a meaningful textual entity like a short story, a scientific abstract or a sentence). In an NLP context, when one talks about parsing, he/she usually refers to the latter interpretation.

Chunking (in an NLP context) is a specific form of parsing in that it extracts groups of words in so-called 'chunks'. These groups of words or chunks  are 'meaningful short phrases from the sentence (tagged with Part-of-Speech). Chunks are thus made up of words and the kinds of words are defined using the part-of-speech tags. One can even define a pattern or words that can't be a part of chunk and such words are known as 'chinks''1. The latter can be defined with chunking rules.

I assume the code you posted comes from "Natural Language Processing: Python and NLTK" by Hardeniya et al.2? From there i can find that the LocationChunker class 'starts by constructing a set of all locations in the gazetteers corpus. Then, it finds the maximum number of words in a single location string so it knows how many words it must look ahead when parsing a tagged sentence.' (cf. Chapter 5, p. 319)


How to train a machine learning algorithm with multiple labels

If you intend to use a summary statistics you would engineer it so it is well suited for your task, meaning captures most of the relevant information. For these things there is usually no best universal solution but it is problem specific. You did not specify what your problem is about so I can't help you there much, maybe use the median value.


When you have an unbalanced data set, the algorithm is going to weight its success on each data point equally, meaning the majority class comes out as much more important than the minority class. The typical solution is to sample down the majority class until it's the same size as the minority class, and an alternate (similar) solution is to adjust the cost function so that the minority class is weighted appropriately.

See these similar questions for more:

Should I go for a 'balanced' dataset or a 'representative' dataset?
Quick Guide into training highly imbalanced data sets
What are the implications for training a tree ensemble with highly biased datasets?
Skewed multi-class data
Ratio of positive to negative sample in data set for best classification


plotting histogram together with variance

The problem is that you are plotting two things of different scale on the same axis.
Your variance appears to be 6*10^9 while your counts are in the hundreds. Therefore the bins are so small you can no longer see them.

You should probably use an vline instead of an hline to get the line on the x-axis

plt.axvline(var)

Here is a sample:

import seaborn as sns
import matplotlib.pyplot as plt
diamonds = sns.load_dataset('diamonds')
plt.hist(diamonds['carat'])
plt.axvline(np.var(diamonds['carat']), color='red')
plt.show()

If you really want to do this on the y axis you should use a second y axis.

Change this

plt.axhline(var)

to this

ax2 = plt.gca().twinx()
ax2.axhline(var)

Check the documentation for more info

Otherwise consider using multiple plots since it might be really confusing to have a line with a different scale on a histogram.


Can OLS regression be used to predict from a complete sequence of data?

Ordinary least squares (OLS) is an optimization method to find the best parameter estimates for linear regression, gradient descent is another.

Regardless of the specific optimization method, linear regression is not appropriate for predicting sequence data. Generally, time series methods are used to predict sequence data.


Does it make sense to use KNN to target ticket buyers for arts events?

This is a classification problem that tries to answer the question: who is likely to buy a ticket to a particular type of show. If you have information about "the previous shows" that each patron attended, you can absolutely use KNN to find which patrons have similar aggregate viewership to a particular type of show.

This might not work if you just have demographic information because there will be a lot of people that share the same demographics. However, as you mentioned, you can use the lat-long or zip code level information to get US census demographic data.

Evaluating the accuracy of this model is a little bit tricky, since you don't know who didn't attend a show due to external factors, but maybe you just want a quick and fast heuristic (which in my opinion sometimes works best).


How is vector A converted to single value scalar in Andrew Ng's course?

$n_X$ is the number of feature, $400$ is the number of data.
Each of the entry of $A$ is the output of the sigmoid layer, it is between $0$ and $1$. We can then decide a threshold (typically $0.5$) such that if it is at least the threshold, we map it to $1$, otherwise, we map it to $0$.


Confusion matrix - determine the values of FP FN TP and TN

You can!

The trick is that you actually know two other critical variables: the number of positive and negative examples (P and N). You can then use them to algebraically solve for the confusion matrix:

$ recall=\frac{TP}{TP+FN}=1-\frac{FN}{P}\Rightarrow $

$ FN = P(1-recall) $

$ recall=\frac{TP}{TP+FN}\Rightarrow (recall)(TP+FN)=TP\Rightarrow TP(1-recall)=FN\Rightarrow$

$ TP=\frac{FN}{1-recall} $
or simply: $ TP=P-FN $

$ accuracy = \frac{TP+TN}{P+N}\Rightarrow (accuracy)(P+N)=TP+TN\Rightarrow $

$ TN=(accuracy)(P+N)-TP $

$ accuracy = \frac{TP+TN}{TP+TN+FP+FN}\Rightarrow (accuracy)(TP+TN+FP+FN)=TP+TN\Rightarrow $

$ FP=\frac{TP+FN}{accuracy}-TP-TN-FN $
or simply: $ FP=N-TN $


Can Transformer Models be used for Training Chatbots?

A transformer is just a neural network. Sure, it is way more complex than a feed forward one, but it still is a neural network. As long as you provide the correct dataset (supervised case, so the correct pair input-target) your model should be able to learn the hidden representation that make it able to answer to new questions.
I found this interesting tutorial, try to check it out.


Estimating X based on Y values in neural network

It is possible given a target value to get a range of values for the features. In classification, this is often called the decision surface or decision boundary.


Graphical Dyslexia -- building intuition for common data visualizations?

There are two books that are our "bibles" in data visualisation.

Edward Tufte, The Visual Display of Quantitative Information
David McCandless Information is Beautiful

Tufte has a wealth of examples on good and bad visualisations. McCandless has some lovely, or even beautiful graphics and ways to display data.


How can I improve the accuracy of my neural network on a very unbalanced dataset?

I'd try some kind of data augmentation, but from your question is not clear what type of data you have and it's impossible to suggest a solution.

Try to add an example of the data to your question.


Why in this case are gradient steps not perpendicular to contour lines?

That would be the case if the steps taken by gradient descent are "infinitesimal" in the mathematical sense. But in fact, it takes steps with some finite length defined by the learning rate. But the problem is, the gradient is calculated at each point, and does not really calculate how the slope changes at some finite length later. If you choose a large learning rate, you may "offshoot" from the optimal direction as shown in the figure. If you choose a small enough learning rate those oscillations will be minimal and you may move "almost" perpendicular to the contour lines and look like what you describe in the question, but it will take a very long time to complete training.

The second part of the question refers to the section titled "Conjugate Gradients" and it refers to a specific optimization method. The reason for the perpendicular lines in the second part is because of the vanishing gradient at the turning points. Quoting the text:

  The method of steepest descent involves jumping to the point of lowest
  cost along the line deﬁned by the gradient at the initial point on each
  step

Such a point is where the "directional derivative" with respect to theta, theta being the parameter in a given direction vanishes. When the derivative vanishes with respect to the parameter of the direction, The only possible way to go will be in a perpendicular direction. That is why they are called "conjugate" to each other, hence the name of the optimization method. This is detailed in the text above the figure.

To summarize the situation for that figure; An initial direction is chosen at the starting point by calculating the gradient there, and you keep moving in that direction until the gradient vanishes. That point is the minimum in that direction, so if you go any further on that straight line, the cost function will increase. The only direction you can go in which the cost function does not increase can be perpendicular to the initial direction.

To visualize this, look at image 4.5 in your first upload - page 88. Starting from that top of the saddle and following the gradient, you go down to the middle of the saddle. At that point going any further in the same direction will only take you up, while you want to go down. The only way to achieve this is to take a 90 degree turn at the middle of the saddle.


Let us assume that your corpus has n distinct keywords. For a k-means algorithm, each keyword is an axis in n dimensional space. Document is a point in that n dimensional space.

K-means algorithm will allocate each point (a document) to a single cluster. When you say a keyword is appearing in two clusters, it probably implies: that particular dimension/keyword is important for both clusters.

Let us take hypothetical example: if you have a patient's blood pressure, cholesterol levels and bunch of other medical parameters. Let us say you discretize blood pressure to 2 or 3 levels. If you run k-means on this data, each patient will be assigned a unique cluster. But it is quite possible that two (or even more) clusters all have patients with > 120 systolic blood pressure.

You need to probably read the results of the k-means more carefully.


Visualizing results of a classification problem, excluding confusion matrices?

Here are a couple of options. I prefer the stacked column chart, which could also be annotated with the actual proportions if needed.


You can replace the symbol in dataframe without iterating yourself.

df = df.replace({'R\$': ''}, regex=True)

Then change the type of columns that can be numeric. If you don't know which are those columns, use this that will automatically change the type to numeric and ignore those that cannot be changed.

df = df.apply(pd.to_numeric, errors='ignore')

When you use replace and in general many other pandas features, it doesn't update your dataframe. It creates a new, temporary dataframe. So, you either need to assign it back to your original dataframe or use inplace=True wherever it is available, like:

df.replace({'R\$': ''}, regex=True, inplace = True)

$ is a special character in regex, so you need to escape it. That's why the backslash before it.

import pandas as pd

dic = {'feature1': 'R$ aaa bb', 'feature2': 1}
df = pd.DataFrame(dic, index=[0,1])

print(df)

>>  feature1  feature2
0  R$ aaa bb         1
1  R$ aaa bb         1

df = df.replace({'R\$': ''}, regex=True)

print(df)

>>  feature1  feature2
0   aaa bb         1
1   aaa bb         1


which application domains are not well suited to deep learning?

Deep learning is generally not well suited for domains that have only very small training datasets. This includes fields like medicine where, for example, patient data can only be obtained through expensive and time consuming clinical trials. Clinical trials will often contain only several hundreds records.

By design, deep learning models have many parameters (e.g., millions), and like with any machine learning algorithm, you're likely to see overfitting when the number of parameters greatly exceeds the number of training records.


Suitable metric choice for imbalanced multi-class dataset (classes have equal importance)

Yes, a macro-average measure is the standard choice in this context: a macro-average score is simply the mean of the individual score for every class, thus it treats every class equally.
With an strongly imbalanced dataset, this means that a small class which has only a few instances instances in the data is given as much weight as the majority class. Since the former is generally harder for a classifier to correctly identify, the macro-average performance value is usually lower than a micro-average one (this is normal of course).


Retrieve user features in real time from UserId for prediction

I think most likely the return of your model wouldn’t be worth it given the amount of effort to generalize it enough.

You may have millions of users, but each users need will probably too unique for generalization. I.e., everyone’s commute is so different that what you have learned from other users are probably not applicable to other users. (Unless during rush hours where most people are heading to the core business areas. You don’t need a model for that. )
The trained model will probably be marginally better than logging the users usage history. It probably isn’t worth the effort to train and process these data for what you can gain.
Recent locations is probably good enough for most users and super easy to implement. Your model will probably have a hard time to predict the odd unusual trips anyway.

You are probably better off storing the users current location and query the most likely destination. Or simply look at all the popular trip destinations from that location. A data base with proper index should be able to handle that


How to get accuracy, F1, precision and recall, for a keras model?

Metrics have been removed from Keras core. You need to calculate them manually. They removed them on 2.0 version. Those metrics are all global metrics, but Keras works in batches. As a result, it might be more misleading than helpful.

However, if you really need them, you can do it like this

from keras import backend as K

def recall_m(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
    return recall

def precision_m(y_true, y_pred):
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision

def f1_m(y_true, y_pred):
    precision = precision_m(y_true, y_pred)
    recall = recall_m(y_true, y_pred)
    return 2*((precision*recall)/(precision+recall+K.epsilon()))

# compile the model
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc',f1_m,precision_m, recall_m])

# fit the model
history = model.fit(Xtrain, ytrain, validation_split=0.3, epochs=10, verbose=0)

# evaluate the model
loss, accuracy, f1_score, precision, recall = model.evaluate(Xtest, ytest, verbose=0)


Cross validation Vs. Train Validate Test cont'd

The answer in the linked question says to use approach 2. Specifically:

Run cross-validation on the train set separately for each choice of hyper-parameter values.
On each run, average the score across the k folds to produce a final cross-validation score.
Choose the hyper-parameters that give us the best cross-validation score.
Retrain a model with those hyper-parameters on the entire training set.
Test this model on the test set, which has not been used at all until now.


Where can I find a social network image dataset?

Google just released a beta search tool for datasets. This can help you find any kind of datasets you want: https://toolbox.google.com/datasetsearch


What is the highest accuracy for classifying cats and dogs from CIFAR-10?

Google's AutoML for large scale image classification has come up with    NASnet which now by far produces the best results(96.2% accuracy    on the top5) on ImageNet image classification (Nasnet implementation).

Imagenet has both cats and dogs as pre-trained classes but they are further divided into many sub categories of cats and dogs.


Which Regression Machine Learning algorithm to use in Azure for statistical prediction/forecasting

If I had to choose, it would be one of the last three as they are more sophisticated. If you have some time, try them all out, sweep the hyperparameters, and compare the results. This free EdX course has a segment where you do that comparison of different models in Azure ML:

https://www.edx.org/course/principles-machine-learning-microsoft-dat203-2x-4


Approach to labelling new data for training

I don't want to go through all the new cat and dog photos labelling whether each one is a cat or dog.
This is basically the reason you do ML. Your model predicts category of new cats and dogs from a set of labeled images your model has seen before.
If you mean the distribution of new input might change over time e.g. new pose or style of photos that you did not have in your previous training set start to show up more, then there are options:

First and Best: Manually label a training-set out of them and retrain the model!
Weakly Supervised Learning: Label at least some from each class beforehand (Semi-supervised Learning) or in a querying schema (Active Learning)
Last and Least: Try clustering and if your lucky and data in each class has a friendly (!!!) distribution, you can tag your data using majority of labels in that cluster (theoretically you can see this as a KNN problem as well)

I tried to throw keywords so you can go on reading yourself. Hope it helps.


multiple linear regression with 5 records and 25 features

Y comes as percent in the format. To put it into the same
dimension as  X I multiplied it with 10. Is that ok from math/data
science perspective?

As far as I can tell there's no reason to do that, and why multiply by 10?

5 records is not much but there are a lot
of features. I would like to do multiple linear regression. Do you
think this feasible with this data set? What would be objections and
risks doing that?

The fact that there are lots of features makes it harder to work with few instances, not easier. There is a very high risk of overfitting, that is of the model catching patterns which appear by chance in the features. This leads to predictions being also affected by chance, so bad performance.

Would upsampling the dataset help me with
anything here? Or could I just work with the five records?

Upsampling is unlikely to work since it's going to reproduce the patterns in the small dataset, so it's also going to reproduce patterns which appear by chance.

With
the strange shape of the dataset especially the low number of records
do you think that sufficient precision can be reached?

It depends what the data represents, if the features happen to be really good predictors for the dependent variable and are not affected by chance, it might work. But these are very optimistic assumptions, in general it's not reasonable to expect good predictions from such a small set of instances.

How could
I calculated the maximum possible precision/discriminative power
possible with this dataset? (I am looking for strong arguments why
they should give me access to the complete dataset)

In general I would suggest doing a leave-one-out experiment: use 4 instances as training set, 1 instance as test set, repeat 5 times with a different instance as test set every time. Measuring the average performance should give you an idea how far off the predictions are going to be (you could use a very simple evaluation measure such as mean absolute error).
However what you have is actually a time series apparently, so it might be worth looking at methods which take time evolution into account.


What is the dimension of the filters if the input image has only one channel?

It is better, in most cases, to convert Grayscale images to RGB images with equal corresponding values in the 3 channels.

In this case, your filter size will be NxNx3.

The reason this approach is preferred is, among other things, if some time in future, you decide to use a pre-trained CNN you would most definitely need an RGB image to use its weights.


Does it make sense to flatten two different features into a single array

I believe it does, for example, if you flatten both features into 1-D array and then concatenate them, they should still preserve most of the information. Because total number of feature vector still remains the same.


What is the concept of error bars in barplots in seaborn library?

It appears that even if you don't enter ci argument, it's set to 95 by default in barplot method, which gives you those error bars. If you don't want them, set ci = None while plotting. The bar is the confidence interval for the variable as the library describes, i.e. it's interval where the 95 % of your variable lies in, which is estimated by bootstrapping.


Create data visualization for unstructured data - Basket Market Analysis

Considering data stored in CSV format like below without headers you can use below R code to plot simple bar chart. It will plot occurrences of transactions grouped by unique transactions.

product1,product2,product3
product1,product4
product1,product2
product4,product3,product1,product2
product1,product2,product3
product1,product4
product1,product4

R Code -

transactions <- read.csv("filepath/transactions.csv", header = FALSE)
transactions$V1
plot(transactions$V1)

Bar Chart-


Practical way to convert jupyter notebook to MS Word document?

The easiest way is probably using a method similarly to what is described in this answer, that is, convert the notebook to markdown and then use any of the tools available (such as Pandoc) to convert the markdown to a Word document.


After you answer in comments:

You can add a new column with either new information (coming from another file or other source), or you can do it based on the existing columns. In fact you are already adding new columns in your code.

I will add a new column called STAGE both ways to give you an example of each.

From static data

I assume you have the stages in a list, like this:

stage_data = ['stage 1', 'final stage', 'stage 1', 'final stage',
              'stage 2','stage 1','stage 2',]

If you have the stages of each of the existing rows, you can add it simply as follows:

dataset['STAGE'] = stage_data

NOTE: The length of the list should match the length of your dataset i.e. the length of the pandas DataFrame. Check it with this:

assert len(dataset) == len(stage_data), "The lengths don't match!!"

From a condition on existing columns

# Make new column simply filled with zeros - we will fill it after
dataset['STAGE'] = np.zeros(len(dataset))

Now you can use conditions on existing columns and fill the rows that match with your corresponding stage:

# assume people with BMI above 25 must be `stage 1`
data[dataset.BMI > 25]['STAGE'] = 'stage 1'

# assume people with BMI below 15 must be `final stage`
dataset[dataset.BMI < 15]['STAGE'] = 'final stage`

# Stupid example to fill the gaps and show how you can use two conditions:
dataset[(dataset.BMI <= 25) & (dataset.BMI >= 15)]['STAGE'] = 'stage 2'


Similarity between an n-dimensional curve and selected subsets of them

So let's start!

First of all please have a look at the edit I made to your original question. It was not just an edit but implied important conceptual thing so I need to start with them:

Graphs: Graph is a reserved term in mathematics for network-like objects. You better use the term Plot for sake of clarification. So the question is absolutely not about graphs.
Time-series data: The example you made i.e. a temporal sequence of data points is called Time-series. A n-dimensional data is not necessarily a time-series and however both seem to inhibit the same structure, they are conceptually different (but not necessarily irrelevant) types of data and different sets of algorithms and problems are defined for either, so please comment here if the data is time-series or not.
Curve Plots: The last but not the least is that a plot is used to just illustrate the data and all Data Analysis algorithms and approaches use that data for inference. So you never infer anything from the plot itself but from the data that it depicts. If you have any plot-like data which you don't know the data behind you need to use Image Processing which I'm pretty sure is not the topic of this question but wanted to mention as you explicitly asked how to do the analysis with plots themselves.

After these points I get back to your question. Yes, what you are looking for is Pattern Recognition and you can use many Machine Learning approaches to solve your problem. If you already knew what are different classes of patterns, it would be called Classification but now, as I understood, you don't know the classes and you want to find similar patterns. Finding patterns of similarity in data is called Clustering.

As I don't know if your data is a time-series (e.g. amount of raining vs time) or just a structured n-dimensional data (e.g. amount of raining vs humidity) I try both.

Time-Series

When you have different segments of time-series and you need their similarity you can use Correlation Analysis or Dynamic Time Warping. If the time series is high frequency e.g. speech data or EEG, then you better convert data to frequency (or time-frequency) domain and then extract features from those segment and then use those features to determine similarity. Please note that the term similarity is pretty wide and it's defined according to the nature of data and problem. You may search for time-series clustering for more approaches.

Non-temporal Data

Similarity between the parameters of a regression model fitted to those subsets of data might help. Another approach is to apply clustering algorithms to those segments. These algorithms can be applied to the original data subsets or some features extracted from those. Dimensionality Reduction algorithms like PCA can be used for feature extraction and might be helpful for a better clustering.

I wrote my answer in rush so I would appreciate any comment or further questions.

Good Luck!


I do feature engineering on the full dataset, is this wrong?

Is it wrong to develop features using the target variable?

Not necessarily. It is called "target encoding" or "Mean encoding" and can be very useful. In your case you could, for example, use the DfS of your train data to calculate a median value per street. But you need to carefully design the target encoding to avoid overfitting (there are different strategies to do that - see below link). And for the test data you can only use the target encoding based on your train data.
The Coursera course "How to Win a Data Science Competition: Learn from Top Kagglers" has great content on target/mean encoding to be found here.

Is it wrong to do feature engineering on the full dataset?

Not necessarily. As pointed out in Nicolas' answer you need to be careful to not leak data though.
Here's an example where it would be ok: let's assume one of your features is date of enlisting which is the date when the property was published for sale. You could, for example, add a feature to the whole dataset called days since enlisting which simply calculates the days between now and when the property was published for sale. However, your median is an example which results in data leakage since it is not "per row" data engineering but "across rows" data engineering applied to train and test data.
That's why the safer approach is to first split the data, remove the target variable from the val/test data and then do feature engineering. Thereby, you avoid any unintended data leakage.


Differences between class_weight and scale_pos weight in LightGBM

You can achieve the same results by using either class_weight, scale_pos_weight and is_unbalanced for binary classification on unbalanced dataset.
Setting
class_weight = {0: (number of negative samples / number of positive samples),
                1: (number of positive samples / number of negative samples)}

is the same as setting is_unbalance = True or scale_pos_weight = (no. of negative samples / number of positive samples).


What is the difference between maximum likelihood hypothesis and maximum a posteriori hypothesis?

I'll try and provide some intuition for you here, instead of focusing on the mechanics of the math behind the methods.
Imagine you are evaluating whether a coin is fair or not, so you collect a sequence of heads and tails as your data set. In MLE, we simply look at the data we collected and find the maximum likelihood... this works well when we have no prior knowledge to leverage (i.e. we have no idea if the coin is fair or not).
By contrast, in MAP we take the same likelihood we used in MLE but now multiply by our prior knowledge. For instance, we may strongly suspect that our coin is biased and so we can influence our estimate with that knowledge through a prior distribution. This new estimate is a mixture of what we believe (our prior) and what we measured (our likelihood).
Thinking of two extreme cases here would be 1) if we very strongly believe in our prior then we would need to collect a lot of data to influence the resulting estimate away from the prior. Conversely, if we know very little up front (i.e. we have an uninformative prior) then finding the MAP estimate is equivalent to the MLE estimate because our prior did not influence the result.
Perhaps a visual representation can help too:

MLE is finding the maximum of the green curve.
MAP is 1) multiplying the blue curve by the green curve to create the red curve and 2) finding the maximum of the newly created red curve.


Has anyone succeeded in finding a good Scala/Spark kernel for Jupyter?

I am using both almond and toree. For executing only scala code, I am using almond. If you want to learn new features in scala (ex: 2.13), then use almond(Add Scala 2.13.4 support in almond v0.11.0). Almond might start supporting scala 3.0 once it is released.
For spark, I am using toree. Toree works fine for me. I am using toree 0.5.0 with spark 3.0.0. Toree uses scala 2.12.
First install spark 3.0.0 by following the steps provided here. After installation check if you are able to launch spark shell from the terminal.
#toree installation
#I used poetry for managing deps in virtual environment
pip install https://dist.apache.org/repos/dist/dev/incubator/toree/0.5.0-incubating-rc1/toree-pip/toree-0.5.0.tar.gz

# I installed/copied extracted spark binaries to /opt/spark
jupyter toree install --user --kernel_name=toree --spark_home=/opt/spark

With these steps, you should find the new kernel added to ~$HOME/.local/share/jupyter/kernels/. Launch jupyter notebook and you should be able to find the toree kernel listed when trying to create a new notebook. You could also execute jupyter kernelspec list to see if the new kernel is listed.
If you are able to create a new notebook, then you should have the kernel initialized without any issues. Try executing println("Hello world") and it should be successful.
I have captured installation summary here


When you have sensors, the values you receive change even if the signal that was recorded didn't change. This is one example of noise.
When you have a model of the world, it abstracts from the real relationships by simplifying things which are not too important. To take into account for the simplification, you model the error as noise (e.g. in a Kalman filter).
But noise sources can be anything. For example, in an image classification problem, data compression can distort an image. Images can have different resolution; low resolution signals are harder to classify than high resolution figures. Aliasing effects can also distort images.

And what is additive noise?

Suppose your system equation is
$$z = H \cdot x$$
where $z \in \mathbb{R}^{n_m}$ is the observation, $x \in \mathbb{R}^{n_x}$ is the state you're interested in and $H \in \mathbb{R}^{n_m \cdot n_x}$ is a transformation matrix. Then the noise could interact with your system in any way. But most of the time it is logical and practical that the noise is additive, meaning your model is
$$z = H \cdot x + r$$
where $r$ is sampled from a random variable of any distribution.

What is (additive) Gaussian noise?

$$r \sim \mathcal{N}(\mu, \sigma^2)$$
See normal distribution


How is error back-propagated in a multi-layer RNN

As you can see here, derivatives will be propagated by the chain rule although they are stacked. Actually, there will be two main paths. The first one will be backpropagation through time and the next one will be the backpropagation from the output of each unrolled cell which can directly be connected to the output or can be connected to the stacked unrolled cells. Also, take a look at here.


The meaning of multi-class classification rules

(Cheat, No) will be selected (cheat will be classified as No) for the rule (Refund, No).

In a binary response variable like cheat, all the information can be inferred from just the first rule:

(Refund, No) → (Cheat, No) Support = 0.4, Confidence = 0.57

The other rule is redundant.

However, in the case of a multi-class response variable, we would like to have all the rules written out so we exactly know the likelihood of the rule implying each of the different classes in the response variable. To keep things consistent, this is also done for the case when the response variable is binary.


Modelling data with Machine learning without a target variable

There cannot be a unique answer to your question. There is a discrepancy in your question though -

I am aware that this is a classification problem on which I am working on.
Could you please help me with the right step by step guide that I should follow in order to achieve an efficient clustering at the end?

However, I am assuming that you are trying to do clustering and you want methods that would give you mathematically better clusters.
clustering is an unsupervised learning problem that does not require target variables. The steps that you mentioned are pretty standard and theoretically correct but there are also other steps that you should take care of. I am listing a few :

Selection of input features - Input features that go into a clustering algorithm are of great importance. It should be noted that a variable not containing any relevant information (say, the telephone number of each person) is worse than useless because it will make the clustering less apparent. In general, the selection of “good” variables is a nontrivial task and may involve quite some trial and error
Selection of clustering algorithm - Use of a good clustering algorithm as per your data is an important step. For example, K- Means better work with numerical features, K- Modes with categorical and K- prototypes in case if you have the data which is a  mix of numerical and categorical features.


What is the best way to cluster this kind of data?

In hierarchical clustering, both agglomerative and divisive, you do not have to pre-specify the number of clusters. You can create all possible clusters and then select the number cluster to use at the end.


Difference usage of document level, sentence level and aspect level in sentiment analysis

I am building on Toros91's answer. He has said:

  Use case of all these 3 different levels are:

Sentence Level: Used to classify a sentences for instance, whether a tweet is positive or negative or neutral.
Document Level: Used to classify the whole document level to say whether that document sentiment is positive or negative or neutral.
Aspect Level: Used to select an aspect and give the outcome of that respective Aspect, for instance let us consider Iphone X as an

  aspect and mine tweets(this is an example you can use documents and so
  on) and carry out Sentiment Analysis.

Now, to answer your question further about how it can be beneficial, just take an example of some review. I am just forming my own review on something as an example, I do not know the specifics, so please do not quote me here. This has some business logic implied as well.

  I feel the latest laptop from Mac is really good overall. It has
  amazing resolution. The computer is really very sleek and can slide
  into bags easily. However, I feel the weight is a let down. The price
  is a bit expensive given the configurations. I expect an SSD storage.
  However, the processor seems really good.

At a document level, I can say the entire paragraph looks more positive than negative. So, the review, if has to be labeled as one thing, is positive.

At a sentence level, I can break the document into 6 sentences and classify each of them as positive, negative or neutral. I get a bit more specific here.

Finally, at an aspect level, I get information about price, storage, processor and so on.

Given N customer reviews, you can know how well different aspects of a particular product are received. Say, 10 customers feel the price is fine, but 4 feel it is expensive. 13 of your 14 customers really love the resolution while 1 feel it is okayish. A lot of products in day to day life have multiple things to be considered. And this is where fine grained labeling help analysts or scientists.


how to use standardization / standardscaler() for train and test?

The recommended way (see 'Elements of Statistical Learning', chapter 'The Wrong and Right Way to Do Cross-validation') is to calculate the mean and the standard deviation of the values in the training set and then apply them for standardizing both the training and testing sets.

The idea behind this is to prevent data leakage from the testing to the training set because the aim of model validation is to subject the testing data to the same conditions as the data used for the model training.


May the training set and validation set overlap?

Definitions, so we are on the same page:

Training set: the data points used to train the model.
Validation set: the data points to keep checking the performance of your model in order to know when to stop training.
Testing set: the data points used to check the performance once training is finished.

May training and validation sets overlap?

They should not.

The validation set is used to know when to stop training your model. The idea is that you check your model performance often and when there seems to be no more improvement, you stop.

Take a look at the plot below. It is plotting the loss of the model. If the loss is still decreasing means that you can keep on improving your model, but if the loss stops decreasing you stop training.

Notice how the valid loss stops decreasing before the train loss. That is because the model could keep improving to improve the performance on the training set, but if you do that, you will get overfitting. So, by having an unseen validation set, you will stop training earlier and not overfit your mode, otherwise, if you keep on training, the accuracy on the validation set will start to decrease.

That means that if the two sets overlap, the validation loss will become more similar to the training loss, so you model will keep on training and you will overfit your model.

May validation and testing sets overlap?

They should not.

You already used your validation set to stop training your model. That means that you already know the performance of your model again the validation set.

Now your model is trained and you want to test your model with unseen data points, i.e. with the testing set. If your sets overlap, you are biasing your test results towards the performance which you already know your model has.


We have $$\|x\|_2=\sqrt{\sum_{i=1}^n x_i^2}$$

Hence $$\|x\|_2^2=\sum_{i=1}^n x_i^2$$

Now let $x=y'-y$ and you obtain your formula.


Learning curve using micro F-score and macro F-score

Micro calculates F score globally by counting the total true positives, false negatives and false positives.

Macro calculates F score for each label and find their unweighted mean. Macro F score does not take label imbalance into account.

Given there is a difference in your performance between the metrics, your data is imbalanced in the base-rate for the class labels.

Adding more data to the minority class(es) will help.


How to find which features have been selected by PCA algorithm?

PCA doesn't remove any specific feature.

What PCA does it to calculate linear combinations of your variables in such way that they get "summarized" in fewer variables.

You don't eliminate variables, you reexpress them.


What is a good method to generate the KS-statistic in R?

Also consider using the builtin ks.test:

> ks.test(prd, act, alternative='greater')

        Two-sample Kolmogorov-Smirnov test

data:  prd and act
D^+ = 0.494, p-value < 2.2204e-16
alternative hypothesis: the CDF of x lies above that of y


How are Hyperplane Heatmaps created and how should they be interpreted?

I think I can answer that, since I implement such a thing in my own library, even if I really don't know how it's implemented in other libraries. Although I am confident that if there are other ways, they don't differ too much.

It took my a few weeks to understand how such a graph can be drawn.

Let's start with a general function $f:\mathbb{R} \times \mathbb{R} \to \mathbb{R}$. What you want is to draw points with a colour which signifies the value of the function. One way would be to simplify the whole problem and draw one point for each pixel. This would work, but will draw only shaded surfaces and it's impossible to draw lines with various formats (dotted lines with some colours and line widths.

The real solution which I found makes two simplifications. The first one would the that instead of colouring with a gradient you can colour bands. Suppose that your function $f$ takes values in $[-1, 1]$. You can split your co-domain into many subintervals like: $[-1, -0.9]$, $[-0.9, -0.8]$ and so on. Now what you have to do would be to paint a polygon filled with appropriate color, for each interval. So your original problem is simplified to draw multiple instances of a simpler problem. Note that when your intervals are small enough it will look like a gradient and even a trained eye would not notice.

The second simplification would be to split the space which needs to be drawn into a grid of small rectangles. So instead of drawing a polygon on the whole surface, you need to fill each small rectangle with appropriate much simpler polygon.

If it's not obvious, the problem is much much simplified. Take for example a one of the rectangle. This rectangle has four corners and you can take an additional point in the center of that rectangle (you might need that point in some certain situations).

The question is how to fill with appropriate colour the proper region?
You need to evaluate function in all four corners and in the center.

There are some specific cases:

function evaluated in all corners are smaller than the beginning of the interval => you need to do nothing
functions evaluated in all corners are greater than the end of the interval => you need to do nothing
functions evaluated in all corners are within interval => fill the whole rectangle with an appropriate color

You can stop here if you want, but your figures would looks non-smooth. You can go further:

left-up, left-down, right-down in interval, right-up bigger => there are two points, one on up side and one on right side which contains the function evaluated at max value from interval => those two points together with top-right corner forms a triangle which can be filled
many other cases which requires only common judgement to decide which polygons to be formed and should be filled.

Using this algorithm you can fill polygons or draw lines. In the specific case of SVM you need to know that the line which corresponds with $f(x,y)=0$ is the line which classifies points into positive and negative samples. Also, lines which evaluates the function at $-1$ or $1$ corresponds with the maximal margins of SVM.

After some time I found that this kind of approach is named iso lines, or iso curves. Perhaps are more similar algorithms like that.

My implementation is named mesh contour (I did not found a proper name at that time in the beginning) and you can find source here.

Some examples:


Clustering categorical variable values based on continuous target values

Since you are looking for a degree of similarity regarding $y$ and the values of $x_1,...,x_5$ do not matter you can view this as a clustering problem regarding $y$:

Let $y_1,...y_5$ be the target values with $f(x_i)=y_i$ for $i\in\{1,...5\}$ then you need to define a distance measure $d(y_i,y_j)$ which, since your variables $y$ are continuous, could be the euclidian distance: $d(y_i,y_j) = (y_i-y_j)^2$. But you could also choose the absolute difference. (note that I am assuming your $y_i$ to be one-dimensional here, i.e. $y_i\in \mathbb R$). That gives you a formula to measure "does not vary that much".

It also provides an answer to your second question: you can choose from a range of unsupervised ML algorithms to do clustering. The most popular one probably being $K$-means. The idea here is very straight forward:

1. select a number of clusters K
2. initialize the position of the K clusters randomly
3. Assign each y_i to the closest cluster
4. For each cluster k calculate the new cluster position as the mean of all y_i belonging to that cluster
5. repeat 3 and 4 until the assignments of y_i does not change anymore

Mathematically this gives you a mapping $c(y_i)=k$.

And eventually when you are done with your clustering you just pick a cluster $k$ and for each $y_i$ in that cluster you look up the corresponding $x_i$. Which will return what you asked for.


How to make edges appear in a graph when adjacency matrix value is beyond a particular threshold?

Line 7 is what you need:

set.seed(42)
iris_matrix <- as.matrix(iris[,1:4])
image(iris_matrix)
g<-cor(iris_matrix)
library(igraph)
R<-graph.adjacency(g, mode=c("undirected"), weighted=TRUE, diag=FALSE, add.colnames=NULL, add.rownames=NULL)
R[R > thresh] <- 0
plot(R)


Default value of mtry for random forests

The textbook 'The Elements of Statistical Learning' by authors Trevor Hastie, Robert Tibshirani, and Jerome Friedman discuss how to tune the number of variables to sample as you build the trees in Chapter 15, Section 3. The pdf of the text is freely available and can be found here.

Note that the values you have listed are in fact default values, as you have mentioned. In practice, it is best to tune the value of 'mtry', as the best value will depend on the specific problem and dataset you are working with.


Heat map and correlation among variables

From the info you provide, it seems you are carrying feature selection based on the correlation between your predictor variables and the target.
This is correct as a type of feature selection (see here) in the family of univariate filter selection, although not the only one. It is fast and intuitive, although you can have a look at other methods. You might also be interested in:

variance threshold selection (also per input feature, univariate filter method): it assumes that higher variance in a feature values could mean more prediction power
sequential backward selection (look here): it means more performance cost, but features are judged in subsets (not independently as above) and is ok if you don't have many features (as it seems to be)

There are many other strategies for feature selection (you might want to check for this source)


Can we use one hot encoding instead of for loops?

If you want to train the data each day and have a new model per store and per day, you should not use One Hot Encoding. If you do One Hot Encoding you will get one single trained model which will learn from the data you have in all the stores. I suppose you do not want one trained model but one model for each store.


You can optimize with non-gradient based methods. The field is called derivative-free optimization.
Local Search is one common approach for derivative-free optimization of neural networks.


DecisionTreeRegressor under the hood of GradientBoostingClassifier

After reading through more of the documentation I found the section that covers the classification case. It can be found here. Additionally this statquest was very useful.

1.11.4.5.2. Classification
Gradient boosting for classification is very similar to the regression case. However, the sum of the trees
is not homogeneous to a prediction: it cannot be a class, since the trees predict continuous values.
The mapping from the value
to a class or a probability is loss-dependent. For the deviance (or log-loss), the probability that
belongs to the positive class is modeled as
where  is the sigmoid function.
For multiclass classification, K trees (for K classes) are built at each of the  iterations. The probability that
belongs to class k is modeled as a softmax of the
values.
Note that even for a classification task, the
sub-estimator is still a regressor, not a classifier. This is because the sub-estimators are trained to predict (negative) gradients, which are always continuous quantities.


Does it make sense to "reorder" a categorical feature to make it monotonic?

I would not recommend doing this in general as it will likely lead to overfitting.

While a particular reordering might improve fit on your dataset, the goal is to create a model with good predictive power. If you received never before seen data from an independent second, third, one hundredth Titanic, would the ordering still be the same? I doubt it. You'd probably be incorrectly using the somewhat arbitrary ordering fixed on the first, real Titanic on new data where the ordering is no longer the same.

I think the story goes that the young and the elderly got preferential treatment and this is why their survival rate was higher. Therefore your AgeBand curve should be approximately U-shaped. (Which it is, approximately.) If you insist on linear regression, I don't see how you could ever (directly) include this nonlinear (U-shaped) effect.

The fewer assumptions (particularities) go into the choice of features, the better. It would then make more sense to me to reorder such that you pick bands alternately from the two ends (starting from the middle): 3, 4, 2, 5, 1, 6, 0, 7 (or in reverse order, starting from the ends). Or, even better, pool the age bands into coarser groups: {1,2,3}, {4,5,6,7}, {0}. I think these are justified by your exploratory data analysis.

When I look at the range of values in the AgeBand plot ($0.33$ to $0.55$), this variable does not actually explain as much as Pclass, Sex, FareBand. In the end you might end up dropping it from your variables anyway, so once again, the more attention you pay to it, the more you're at risk of overfitting.


How can we use mean imputation without violating feature correlation?

If you want to keep the feature correlation, use an imputer that solves for the missing values by using the feature correlation, such as KNN, regression, etc. These methods work well if there is no underlying reason that the data is missing - Missing at Random.


DBMS or Software for privacy sensitive data

Two possible options:

Have the database administer set limits on SQL queries that only allow result tables with aggregated results with minimum of N people to be returned.

Apply differential privacy which a query result cannot be used to infer much about any single individual, and therefore provides privacy.


How to include class features to linear SVM

The quick answer is yes, you can use a liner SVM in presence of an encoded categorical variable.

Short explanation: The linearity of the model has nothing to do with the features of model, and actually linear feature doesn't mean anything. The linearity refers to the model, i.e. the equation that links the target to the features.

The equation

$$y = b*x + q$$

means that y is linear with respect to x, not that the variable x is linear. x is x thats it.

What you should check when dealing with linear models is if your data is linearly separable, or in other words if you can separate the different classes with a straight line. If that is not the case you are in trouble and should probably think of changing the model.


Are there any multi-type public datasets out there?

You may want to take a look at: https://archive.ics.uci.edu/ml/index.php which is a great repository with tons of datasets.


what happens to the depth channels when convolved by multiple filters in a cnn (keras, tensorflow)

When you have a image dimension of HxWxC and you apply a convolutional layer with filter size F and kernel size KxK, you will be sliding F different KxKxC kernels over your image, resulting in the output dimension of the total operation being H'xW'xF which is what you are seeing.


Tree-based algorithms and ordinal features

Edit: I misread your post: the answer is no it shouldn't matter which interval is used. So long as the order is not changed the splits your tree finds on this data will effectively be the same.
See my original answer if you want more context:
In scikit-learn specifically (I cannot speak for other implementations of tree-based models) it does not accept categorical data as input. Meaning the user must convert this to  real numbers using hot encoding, numerical encoding etc...
When fitting on any given tree - when it's iterating through possible splits it is splitting your column using an inequality (ie is this column's value bigger or smaller than this threshold value). Due to this methodology it does matter in which order your data is encoded.
If you hot-encode your data - each column will be looked at independently (a likely split would be "is this bigger or less than 0.5 as thats effectively the only split to be made when the options are either 0 or 1).
If you numerically encode your data it will just pick the best threshold that maximizes your split criterion function (usually gini inequality). The bias you add to your system by manually determining the order or the numerical encoding can affect how well the tree is able to split on this column. If you do some logical encoding whereby the target increases in value as the value of your feature increases, your tree will be able to split on this column more effectively & you will likely see better results.
For this reason I recommend numerically encoding with a TargetEncoder (https://contrib.scikit-learn.org/category_encoders/targetencoder.html). This way the order of your numerical encoder will make sense for the decision tree.


Determine when entry/series of entries are outliers

Look into six-sigma techniques: e.g.: http://www.whatissixsigma.net/imr/

In short: create a moving range (moving average +/- 3 times moving std.deviation).

Points outside that range can be considered outliers and need inspection (not just discarding).


What's the difference of stateless LSTM and a normal feed-forward NN?

Theoretically a stateless LSTM gives the same result as a statefull LSTM, but there are few pros and cons between them. A stateless LSTM requires you to structure your data in a particular way, in turn it is vastly more performant, while a statefull LSTM you can have varying timesteps, but at a performance penalty.

The stateless LSTM does have state, it's just implemented differently. Instead of managing it yourself by constantly calling reset_states() your data is structure in such a way that it is automatically reset when the end of one series of timesteps is done.

Let's create an explicit example. Say you have some label data y[i] where i is the index, and input data x[i, t] where i corresponds to your label's index, and t is at what timestep this was observed. (The x and ys can be vectors, matrices, or any type of tensor really.)

Say you have 4 output labels (not 4 classes), and 2 to 4 timesteps for each of them. Your data might look like this:

                   x[0, 0], x[0, 1]  ->  y[0]
          x[1, 0], x[1, 1], x[1, 2]  ->  y[1]
 x[2, 0], x[2, 1], x[2, 2], x[2, 3]  ->  y[2]
                   x[3, 0], x[3, 1]  ->  y[3]

This is the kind of data a statefull LSTM can handle. But at the end of each label, you would have to call reset_states() to indicate that your no longer dependent on the previous series of xs. This is slow.

With a stateless LSTM you would pad the x so that you have a rectangular matrix:

 0        0        x[0, 0], x[0, 1]  ->  y[0]
 0        x[1, 0], x[1, 1], x[1, 2]  ->  y[1]
 x[2, 0], x[2, 1], x[2, 2], x[2, 3]  ->  y[2]
 0        0        x[3, 0], x[3, 1]  ->  y[3]

In this case, there is an implicit reset of state for each label. This way of doing it is vastly more performant. But there might occur issues with bogging down your data with empty data. Personally I tend to add a binary flag to each x vector to indicate to the network in the LSTM that it should probably ignore the data.

I'm not sure about the specifics of how stateless LSTMs are implemented, but I think it might be about unrolling the LSTM units.

Ultimately, both stateless and statefull LSTM's do the same thing, they are just implemented differently.

  why would someone use stateless LSTM instead of a feed-forward perceptron network?

Well, ironically it's because stateless LSTM have state, which means it can take into account the previous timestep input values when doing predictions.


ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)

The problem is inside the sampling functions. I had the same problem and found out the answer in the tutorial here.

my original code is:

def sampling(args):
     z_mean, z_log_sigma = args
     epsilon = K.random_normal(shape=z_mean.shape)
     return z_mean + K.exp(z_log_sigma) * epsilon

with this sampling method, I got the same error with yours.

the codes fixed the problem is:

class Sampling(layers.Layer):
  def call(self, inputs):
    z_mean, z_log_var = inputs
    batch = tf.shape(z_mean)[0]
    dim = tf.shape(z_mean)[1]
    epsilon = tf.keras.backend.random_normal(shape=(batch, dim))
    return z_mean + tf.exp(0.5 * z_log_var) * epsilon

It uses tf.shape to get the required shape parameters. It somehow fixes the problem.


Can I create a word cloud of crowdfunding donors using word cloud?

I will give you a simple solution using R which requires the wordcloud package. Of course there are many other solutions which do not require any programming skills.

The solution is a slight varient of this R-Bloggers tutorial. Feel free to have a look there for further formatting.

library(wordcloud)

words = c('Paolo Gentiloni', 'Matteo Renzi',
          'Enrico Letta', 'Mario Monti',
          'Silvio Berlusconi', 'Romano Prodi')
freq = c(100, 25, 50, 70, 95, 20)

wordcloud(words = words, freq = freq, min.freq = 1,
          max.words=200, random.order=FALSE, rot.per=0.35,
          colors=brewer.pal(8, "Dark2"))

It produces the following output:

Btw. @Lorenzo Perone: Where the ones listed Italien names? I was not sure about that.


Explanation behind the calculation of training loss in deep learning model

The line you're asking about

train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))

is basically calculating the average train_loss for the finished batches

To illustrate, suppose 4 batches have been done (with average loss named avg_loss) and current is calculated from 5th batch (with loss named new_loss)

The new average loss is from

$\frac {4 \times \text{avg_loss} + \text{new_loss}} {5}$

This is exactly the same as

$\text{avg_loss} + \frac {\text{new_loss} - \text{avg_loss}} {5}$

which is the calculation done by the code


Help improving time series prediction with LSTM on PyTorch

I think I was able to get it working now. What I did was to change the assigned nan value to 25 instead of 0 (which would be about the average) and normalize the values within the interval (-1,1) with a scaled sigmoid shifted on 25.
This is how it looks now:


Why there are so many research papers suggesting the use of Newton's
  method based optimization algorithms for deep learning instead of
  Gradient Descent?

Newton method has a faster convergence rate than gradient descent, and this is the main reason why it may be suggested as a replacement for gradient descent.

  Is Newton's method really needed if Gradient Descent can be modified
  to rectify all the problems faced during machine learning?

Existence of vanishing gradient problem depends on the choice of "activation function" and the "depth" of network. Newton method and gradient descent would both face this problem for a function like Sigmoid, since in the flat extremes of Sigmoid both first and second order derivatives are small and exponentially vanishing by depth. In other words, the problem is solved for both methods by the choice of function.

As a side note, 1st- and 2nd-order derivatives of Sigmoid go to zero at the same rate. Here is a graph of Sigmoid and its derivatives; zoom into the extremes.

Historical note. Newton method predates the vanishing gradient problem (which was faced after the introduction of Backpropagation in 60s) by centuries.


How to split data in R using dplyr if we want to have rows of the same group to belong to the same split?

Figured out an easy way to do this.

First we will just select PID from the real data.
Then we will just sample 0.75 % of these PID and save these point as training PID and the rest as testing PID.
We will thne find the intersection between this list and the real data using PID.


No it this isn't overfitting.

First of all the AUC is exactly the same between train and validation sets. The losses might have a gap but since the validation loss is still dropping (even if slowly) you are OK.


Appropriate similarity measure that highlights symmetrical values of features

Is this an issue of features?  From the explanation, it seems that the feature you are considering is distance. So the results are as expected (from a "bag of single feature distances").

You mention that order is important, so that can added as a feature.  Or since you mentioned "phase shift", time (and/or frequency) -- giving you a time series.  In both cases, you then can bring in all kinds of tools.  You could start with regression and go from there to more complex methods (simple sometimes is best).

That would allow you to yield similarity metrics between single or multiple vectors.

Vectors contained in a sliding time window vs. population, is something you maybe can also consider, we did a solution like that once.


How to tell if a problem should use regression or classification model?

A good rule of thumb is to look at the level of measurement of the target/response variable. If the response is measured on a nominal scale, the problem is a classification problem. Values on a nominal scale are for example labels of a categories where the categories have no natural order, like political parties in political science, species in biology, or parts-of-speech in grammar.

If the response is measured on a ratio or interval scale, you have a regression problem. Values on an interval scale are values where you can compare the degree of difference between values, but not the ratio between them, for instance temperature (on Farenheit or Celsius scales, but not Kelvin), or date values in a calendar. Values on ratio scales can be compared both with regards to degree of difference and ratio, like most physical quantities like mass, velocity or temperature on the Kelvin scale.

Ordinal scales are more difficult to place in either corner. I would generally say that you have a ranking problem with an ordinal response. However, the ranking problem can be approached using both classification, for instance using comparators, and and regression, like ordinal regression. Values on ordinal scales are ordered, or ranked, but you can't say anything meaningful about the degree of difference between any two values, for instance the ranking of racing drivers in a race.


How to make output dimensions match input dimensions in CNN?

You have (at least) two options. You can either:

Use a bottleneck architecture where you use pooling layers to reduce the dimensions of your data and then upcast it again using deconvolutional/upsampling layers to go back to the original dimensions. The advantage of this approach is that you use a bigger receptive field for the final output, which means it can look at a bigger part of your input sequence. You could combine this with a highway connection like structure. A very similar concept is used in 2d with segmentation, here the spatial dimensions are preserved at the output. One of the best models is called a U-Net.
Use no pooling and only padding-same convolutional layers with non-linearities. You can increase the depth but you will keep the (1000, 2) dimensions everywhere. The advantage is that it is very easy to implement, and if applicable to your problem only uses relatively local information.


How is the "base value" of SHAP values calculated?

As you say, it's the value of a feature-less model, which generally is the average of the outcome variable in the training set (often in log-odds, if classification).  With force_plot, you actually pass your desired base value as the first parameter; in that notebook's case it is explainer.expected_value[1], the average of the second class.

https://github.com/slundberg/shap/blob/06c9d18f3dd014e9ed037a084f48bfaf1bc8f75a/shap/plots/force.py#L31

https://github.com/slundberg/shap/issues/352#issuecomment-447485624


Adding hand-crafted features to a convolutional neural network (CNN) in TensorFlow

I figured it out. If we denote the additional features as x_feat, I changed the lines from

h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)

to

h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_pool2_flat = tf.concat( [h_pool2_flat, x_feat ], 1 )
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)


Try to search from the databases of official "bad words" that google publishes in this link Google's official list of bad words.
Also, here is the link for the good words Not the official list of good words
For the code, I would do it like this:
textArray = file('dir_to_your_text','r').read().split()

#Bad words should be listed like this for the split function to work
# "*** ****** **** ****" the stars are for the cenzuration :P
badArray = file('dir_to_your_bad_word_file).read().split()
goodArray = file('dir_to_your_good_word_file).read().split()

# Then you use matching algorithm from difflib on good and bad word for every word in an array of words
import difflib

goodMachingCouter = 0;
badMacihngCouter = 0;

for iGood in range(0, len(goodArray)):
    for iWord in range(0, len(textArray)):
        goodMachingCounter += difflib.SequenceMatcher(None, goodArray[iGood], textArray[iWord]).ratio()

for iBad in range(0, len(badArray)):
    for iWord in range(0, len(textArray)):
        badMachingCounter += difflib.SequenceMatcher(None, badArray[ibad], textArray[iWgoodord]).ratio()

goodMachingCouter *= 100/(len(goodArray)*len(textArray))
badMacihngCouter *= 100/(len(badArray)*len(textArray))

print('Show the good measurment of the text in %: '+goodMachingCouter)
print('Show the bad measurment of the text in %: '+badMacihngCouter)
print('Show the hootnes of the text: ' + len(textArray)*goodMachingCounter)

The code will be slow but accurate :) I didn't run and test it please do it for me and post the correct code :) because I wanna test it too :)


Which supervised learning algorithm to classify customers buying items

As per the problem statement statement it does look like 3 class classification problem.
But I don't think you can use any ML algorithms using Python without sklearn package in it. Try getting it installed and you can implement the problem using Logistic regression, Decision Tree, there are still more classification algorithms.
However if you are planning to extract a pattern from the dataset and making into cluster, you can use K-means clustering which is a un-supervised algorithm.


How to structure time series for binary classification in Keras?

you want to treat the date as a cyclical feature to capture patterns in the time stamp.

http://blog.davidkaleko.com/feature-engineering-cyclical-features.html

Keras + SK learn work nicely together. here's an example in my GitHub to extract patterns from the time stamp (just adjust to your needs and replace my Decision Tree with Keras code).

https://github.com/FrancoSwiss/Fraud_Detection/blob/master/Kaggle%20Fraud%20(1).ipynb]1


Forecasting vs non-forecasting predition for time series anomaly detection

I had the same question and ultimately tried an LSTM forecaster and LSTM autoencoder. For me, the choice of forecaster vs autoencoder had effects on the following two points:

Seasonal Pattern:
if you have a seasonal pattern (e.g. weekdays vs weekend) in your data, and the timeframe you are looking at has the pattern -- but is shifted by e.g. a day, you are likely to find that outlier with the Forecaster, but not as likely with Autoencoder.
This Disadvantage of the Autoencoder can be circumvented by also feeding in the weekday and time for each datapoint.

Needed Datapoints:
With the Autoencoder, you don't need datapoints before the timeframe you want to examine. You can just take the series you are interested in and give it to your model.


When to choose linear regression or Decision Tree or Random Forest regression?

Let me explain it using some examples for clear intuition:

  When do you use linear regression vs Decision Trees?

Linear regression is a linear model, which means it works really nicely when the data has a linear shape. But, when the data has a non-linear shape, then a linear model cannot capture the non-linear features.

So in this case, you can use the decision trees, which do a better job at capturing the non-linearity in the data by dividing the space into smaller sub-spaces depending on the questions asked.

  When do you use Random Forest vs Decision Trees?

I guess the Quora answer here would do a better job than me, at explaining the difference between them and their applications. Let me quote that for you:

  Suppose you're very indecisive, so whenever you want to watch a movie,
  you ask your friend Willow if she thinks you'll like it. In order to
  answer, Willow first needs to figure out what movies you like, so you
  give her a bunch of movies and tell her whether you liked each one or
  not (i.e., you give her a labeled training set). Then, when you ask
  her if she thinks you'll like movie X or not, she plays a 20
  questions-like game with IMDB, asking questions like "Is X a romantic
  movie?", "Does Johnny Depp star in X?", and so on. She asks more
  informative questions first (i.e., she maximizes the information gain
  of each question), and gives you a yes/no answer at the end.

  Thus, Willow is a decision tree for your movie preferences.

  But Willow is only human, so she doesn't always generalize your
  preferences very well (i.e., she overfits). In order to get more
  accurate recommendations, you'd like to ask a bunch of your friends,
  and watch movie X if most of them say they think you'll like it. That
  is, instead of asking only Willow, you want to ask Woody, Apple, and
  Cartman as well, and they vote on whether you'll like a movie (i.e.,
  you build an ensemble classifier, aka a forest in this case).

  Now you don't want each of your friends to do the same thing and give
  you the same answer, so you first give each of them slightly different
  data. After all, you're not absolutely sure of your preferences
  yourself -- you told Willow you loved Titanic, but maybe you were just
  happy that day because it was your birthday, so maybe some of your
  friends shouldn't use the fact that you liked Titanic in making their
  recommendations. Or maybe you told her you loved Cinderella, but
  actually you really really loved it, so some of your friends should
  give Cinderella more weight. So instead of giving your friends the
  same data you gave Willow, you give them slightly perturbed versions.
  You don't change your love/hate decisions, you just say you love/hate
  some movies a little more or less (you give each of your friends a
  bootstrapped version of your original training data). For example,
  whereas you told Willow that you liked Black Swan and Harry Potter and
  disliked Avatar, you tell Woody that you liked Black Swan so much you
  watched it twice, you disliked Avatar, and don't mention Harry Potter
  at all.

  By using this ensemble, you hope that while each of your friends gives
  somewhat idiosyncratic recommendations (Willow thinks you like vampire
  movies more than you do, Woody thinks you like Pixar movies, and
  Cartman thinks you just hate everything), the errors get canceled out
  in the majority. Thus, your friends now form a bagged (bootstrap
  aggregated) forest of your movie preferences.

  There's still one problem with your data, however. While you loved
  both Titanic and Inception, it wasn't because you like movies that
  star Leonardio DiCaprio. Maybe you liked both movies for other
  reasons. Thus, you don't want your friends to all base their
  recommendations on whether Leo is in a movie or not. So when each
  friend asks IMDB a question, only a random subset of the possible
  questions is allowed (i.e., when you're building a decision tree, at
  each node you use some randomness in selecting the attribute to split
  on, say by randomly selecting an attribute or by selecting an
  attribute from a random subset). This means your friends aren't
  allowed to ask whether Leonardo DiCaprio is in the movie whenever they
  want. So whereas previously you injected randomness at the data level,
  by perturbing your movie preferences slightly, now you're injecting
  randomness at the model level, by making your friends ask different
  questions at different times.

  And so your friends now form a random forest.


What are some good methods to forecast future revenue on categorical and value based data?

If you want to use neural networks this post on Kaggle might help: https://www.kaggle.com/c/m5-forecasting-accuracy/discussion/159052
It has a short list of resources for categorical embeddings and LSTM (I think).
If you think your dataset has periodic patterns, and you only need to answer your questions (not deploy a model). I would take a look a FB Prophet: https://facebook.github.io/prophet/docs/quick_start.html#python-api
It extracts the periodic components and fits them with sine and cosine waves. You can also add additional regressors, i.e., one-hot encoded categorical variables.


Should you turn off label smoothing when validating?

The way most people gain an initial understanding of label smoothing (and what most common explanations have to say on the subject) plays a great role in how one would approach this question.
At first glance, label smoothing is exactly what the name suggests: we modify the labels or some portion of them in order to get a better, more general, more robust model. It makes sense that we don't want the model to learn from (and later predict with) extreme confidence levels, especially when we know that some of the labels are wrong, since this hurts the model's abilities to perform on unseen data. The intuitive explanation of LS mechanics, then, is that we are not feeding the model with pure 1's and 0's, but with less confident values instead, resulting in a more reserved decision function that doesn't extrapolate in an extreme manner. Now, we know that smoothed labels are not the true labels, so at this point the main concern pops up - once we have trained on smoothed labels, do we also use smoothed labels for validation?
If you think about LS purely as a data manipulation technique, the answer to the question above is not obvious, as it may go both ways depending on the argument. However, one must remember that LS is almost always regarded as a regularization technique -- you mention this in the question yourself -- and there is a good reason for that. Regularization, by definition, is when the loss function is extended with an additional regularization term, which usually has to do with penalization. In LS, this penalty term is responsible for punishing high confidence predictions. Even though it may not appear as such, LS, once applied, becomes an essential part of the loss function, which should persist between training and validation if we aim to take advantage of the technique. When we apply LS during training, we are effectively trying to minimize a loss function with the regularization term added in. Throwing it out during validation would defeat the very purpose of including it in the first place: if we decide not to apply LS to the validation set as well, we are making the mistake of expecting extreme confidence (1/0 labels) from a learner that was just a moment ago specifically trained against making overly confident predictions on previously unseen data. The correct thing to do would be to validate that the predictions are of moderate confidence, just as desired. This is why the validation set should also have the regularizer present in the loss function, i.e. have smoothed labels as well.


Change priority of cell execution in jupyter notebook?

I don't think Jupyter notebooks (even via extensions) currently offer pausing/restarting cell blocks. I would suggest putting the code of both cells into a single cell and using Python logic to determine the order of execution.

In general, however,  you cannot strictly pause the execution of something and come back to it later. If you abort the execution of the function, the intermediate results are lost (because they are stored within the namespace of the function you effectively just killed). This is because Python only runs one single process at a time (key term: Global Interpreter Lock).

The only thing you might be able to do is create a cell 0, which performs some kind of check, testing how long cell 1 might take to run, then just put cell 1 and cell 2 in and if/else construction that gives the correct order to use.

Other approaches

You might want to look into something like the multiprocessing library. There you can create a group of worker processors, to which you can send the contents of cell 1 and at the same time the contents of cell 2. They will be computed at the same time, using two different processes i.e. two different instances of the Python interpreter. This can be easy to achieve if there is no direct dependency between (in your case) cell 1 and cell 2. Multiprocessing is particularly useful in the case that your long-running cell is compute bound, meaning is has to perform a large computation.

Yet another option would be to investigate the threading module and concurrent programming in general, but this get a little more complicated and is probably beyond what you want in your situation (it also requires more effort to get working that multiprocessing). This allows Python to kind of do two things at once, but with shared state - so each running thread can change variables that the other thread might also be changing, which can require a lot of work to make safe. This approach is however beneficial when you tasks are IO bound, i.e. there is no big computation, but rather you send data e.g. to a website and wait for a return - most your time is spent waiting.


Do I Need Pretrained Weights For Keras VGG16?

It would be beneficial to use pretrained weights trained on ImageNet because ImageNet is a huge dataset, and is fairly general. So you're more likely to get good image features from it. You might have to play around with from which layer you want to extract the features from.

If you want to create your own embeddings, you need to use the same code and instead of loading the weights, train it instead on the images you have. I do not recommend this for the following reasons

You most likely have too small a dataset for the model to converge
VGG16 does classification, what will be the labels for the individual frames is a design choice and being a novice, it might be frustrating for you to get it right
It takes a lot of time and resources, not an ideal situation

P.S, if you still feel that ImageNet does not sufficiently cover your use case, you can try the pretrained Inception V3 model trained on the Open Images dataset


How to find the datasets for skill test (like java, python, c c++, etc.)

a) You may try to find an anonymized resume dataset in the openbase of Kaggle datasets

b) You may do some web scraping on professional social networks like LinkedIn (take the description of a profile as resume, and the LinkedIn skills as supervised training set) to build your own sample database (do not forget to anonymize it).

Hope this helps!


Do I need to learn Hadoop to be a Data Scientist?

Different people use different tools for different things.  Terms like Data Science are generic for a reason.  A data scientist could spend an entire career without having to learn a particular tool like hadoop.  Hadoop is widely used, but it is not the only platform that is capable of managing and manipulating data, even large scale data.

I would say that a data scientist should be familiar with concepts like MapReduce, distributed systems, distributed file systems, and the like, but I wouldn't judge someone for not knowing about such things.

It's a big field.  There is a sea of knowledge and most people are capable of learning and being an expert in a single drop.  The key to being a scientist is having the desire to learn and the motivation to know that which you don't already know.

As an example:  I could hand the right person a hundred structured CSV files containing information about classroom performance in one particular class over a decade.  A data scientist would be able to spend a year gleaning insights from the data without ever needing to spread computation across multiple machines.  You could apply machine learning algorithms, analyze it using visualizations, combine it with external data about the region, ethnic makeup, changes to environment over time, political information, weather patterns, etc.  All of that would be "data science" in my opinion.  It might take something like hadoop to test and apply anything you learned to data comprising an entire country of students rather than just a classroom, but that final step doesn't necessarily make someone a data scientist.  And not taking that final step doesn't necessarily disqualify someone from being a data scientist.


Which of the 180 algorithms in R's caret package are feasible?

Benchmarking mlr (default) learners on OpenML

The entire openml database of ML results.

Test from RStudio suggests SVM.

Mlmastery suggests LDA and Trial and Error.

Do we Need Hundreds of Classifiers to Solve Real World Classification Problems? by Fern ́andez-Delgado et al.

Paper concludes  parallel random forest (parRF_t) is best followed by  random forest, LibSVM with Gaussian kernel (svm), extreme learning machine with Gaussian kernel,    C5.0  decision  tree  and  multi-layer  perceptron  (avNNet).

  The best boosting and bagging ensembles use LibSVM as base classifiers
  (in Weka),  being slightly better than the single LibSVM classifier,
  and adaboost R (ensemble of decision trees trained using Adaboost.M1).
  The probabilistic neural network in Matlab, tuning the Gaussian kernel
  spread (pnn m), and the direct kernel perceptron in C (dkp C), a very
  simple and fast neural network proposed by us (Fern ́andez-Delgado et
  al.,2014),  are  also  very  near  to  the  top-20.

Wainer, Jacques (2016) Comparison of 14 different families of classification algorithms on 115 binary datasets  Based on Fernandez-Delgado et al. (2014). "We have shown that random forests, RBF SVM, and gradient boosting machines are classification algorithm that most likely will result in the highest accuracy"

Rich Caruana & Alexandru Niculescu-Mizil () An Empirical Comparison of Supervised Learning Algorithms (classification) concludes with Platt-Calibrated Boosted Trees as best followed RF BagT Cal.SVM NN.

Many other studies include comparisons of models used. Some papers prefer SVM  others SVM with radial-basis or polynomial kernel for classification. (maybe same thing)

From my own regressions on generated data I recommend earth(MARS) Cubist SVMlinear.

Manisha Thesis first runs tests on UCI Machine Learning Repository then soil fertility which is the focus of the thesis. Best models on UCI were :"elm-kernel is the ELM neural network but with Gaussian kernel", "svr is the support vector machine for regression, with Gaussian kernel using the Lib-SVM library with the C++ interface", extraTrees and cubist. The thesis includes great descriptions of each model and links to more papers."extraTrees achieved
the best RMSE for 7 of 10 soil problems". Paper is definitely worth a read.


how to run bert's pretrained model word embeddings faster?

There are several things that could make your code faster:

Stop using Pandas. Pandas is not designed for large-scale, fast text processing. It would be better to switch to something like Apache Arrow which is designed for efficient analytic operations on modern hardware.

Refactor code to avoid casting into memory-inefficient data types. progress_notes['vectors'].to_list() converts to a Python list which will use a lot of memory.

Refactor code to avoid casting back and forth between data types. Replace the list comprehensions and ''.join with complied regex. Complied regex processing will be much faster than equivalent Python code.

Refactor code to replace functions with inline code. Python creates a new stack for each function call. Inline code will be more memory efficient.


Correlation among features (e.g. doc length, punctuation, ... ) in classifying spam emails

First about the features i  think you could add some such as :

the time when the letter is received,
number of links in the email,
the whole structure (does it follow typical structure for email),
number of words that contains numbers in it,
what is the whole mood of the email (sales,threats,info,...-for this purpose you can use sentiment analysis),
number of attachments ,
type of attachments and so on.

After that try with feature selection (you could read more about it here). For the imbalance data you need to resample the data. I would:

add copies of spam emails(oversampling)
try to generate new spam email (smote)

You can read more here.
I hope my answer will give you some clarity.


Is there any NLP framework/ algorithm which is able to derive relationships between different entities?

What you describe touches on several tasks in NLP:

semantic role labeling
relationship extraction
paraphrase detection and textual entaiment

I might not be aware of the recent developments on these tasks, but as far as I know this is a very complex problem for which so far there is no satisfactory solution.


You could use competitive fuzzy cognitive maps


You can't calculate the conditional distribution because that distribution is defined over the entire population and assuming a probabilistic model which you do not know.

In real life you have only a sample of the population. More often than not, in real life you do not have the structure of the model also. You do not know if it is a linear model, a polynomial or another type.

The probabilistic approach is to assume a model, consider some assumptions for that model and fit the model's parameters using data. That is not the Bayes classifier, it's only an approximation.

The trick is 'all models are wrong, some models are useful'. There are many approaches to learn this approximation. You can estimate the joint probability (generative classifiers), you can estimate directly the conditional probability (discriminative classifiers) or you can use a different approach (like SVM, for example) and twist it somehow to approximate the conditional probability. But all of them are approximations. The real one, THE Bayes classifier will remain unknown.

[Later edit]

Looking at the Wikipedia page for Bayes classifier it states that: "The Bayes classifier is a useful benchmark in statistical classification.". My opinion is that this statement is confusing, if not dead wrong. Probably this is related with the wrong usage of expression "Bayes classifier" to designate the Naive Bayes classifier. If one search on google for "Bayes classifier", he will see a lot of materials related with Naive Bayes. Personally I do not agree with the statement from wikipedia. Bayes classifier is that state of nature that we do not know but we aim to approximate as well as possible.


Welcome to DataScience.SE. Let's see if I've interpreted your question correctly:

  i want to understand impact of previous reviews on current review

Keep track of which reviews a reviewer has read (randomly withhold some if you have to), and use this information together the rating of the new review to create a regressor to estimate the score of a new review given the age and score of selected previous reviews.

You can do similar regressions with the topic and sentiment, if you use embeddings.


How to avoid confusion between Percentage and Percentage Points?

Use whatever conveys the message more clearly.
This usually means to use percentages only for absolute metrics and use %-points for metrics which are presented as percentages (e.g. "retention rate increased by 5%-points" in your example).
From my experience you can expect almost everybody to know the difference. The only exception would be highly operational staff, e.g. blue collar staff on the shop floor, cashiers in a supermarket etc.
In the latter cases, or if there are language barriers (e.g. audience is not fluent in English) or technical issues (e.g. a bad connection), make the changes explicit. For example:

Retention rate increased from 10% to 15%
Potential revenue drop of 7% compared to 5% in the first scenario


Weird distribution of neural network outputs

It's not that weird. You probably have an activation function on output that only reach 0 or 1 at infinity. To reach that 0 or 1 you would need to have your last hidden layer to output +/- infinity. Which is difficult given the weight structure. The best it can do will be to output extremely large positive or negative values.
It may be a problem because during the calibration process (which is an optimisation problem) you will spend significant ressources and time in trying to reach +/- infinity. Notably this is one of the causes of the famous exploding gradient problem.
How to deal with that ? In Efficient Backpropagation (LeCun and others, 1998), the author propose to use an activation function that is able to reach desired outputs. It states that a good activation function needs to have the following properties :

$f(\pm1)=\pm1$

The second derivative is a maximum at $x=1$

The effective gain is close to 1

Your problem is that you don't have something similar to the first condition. To respect those conditions LeCun proposes to use the activation function: $1.7159*tanh(\frac{2}{3}x)$. you might not need to use it in all your layers. Now your problem would probably be to find a framework that allows for custom activation functions.


Since you did not specify your exact problem, first some general remarks. Heteroscedasticity does not affect estimated coefficients, but the (estimated) standard deviation of coefficients, p-values, and confidence intervals in linear models. Heteroscedasticity means that the model's variance is contingent on features (explanatory variables).

The Breusch-Pagan (BP) test is suitable for linear models. Say you have a linear model

$$ y = \beta X + u. $$.

You can estimate the model using $\hat{\beta} = (X'X)^{-1} X'y$ and obtain the residuals $\hat{u}=y-X\hat{\beta}$. The BP test checks if residuals $\hat{u}$ are contingent on explanatory variables $X$. To do so, an auxiliary regression is used:

$$ \hat{u}^2=\gamma X + e.$$

The general idea is that $\hat{u}^2$ should be unrelated to $X$, so that coefficients $\gamma$ (apart of the intercept) should be statistically not different from zero.

This conjecture could be used to drill up problems with heteroscedasticity. In your case you are interested in how strong heteroscedasticity is. Looking at the vector $\gamma$ may be one option. A second option would be to simply look at the outcome of the BP test (the $\tilde{\chi}^2$ and related p-value).

R code (regression and BP test):

library(MASS)
data(Boston)

# Regressions
reg1 = lm(medv~indus+lstat, data=Boston)
reg2 = lm(medv~indus+crim, data=Boston)

# BP test
library(lmtest)
bptest(reg1)
bptest(reg2)
# Null hypothesis of BP test = homoscedasticity (= variance does not depend on auxiliary regressors)
# If the p-value is "small" (<0.05), the null hypothesis is rejected
# reg1 -> heteroscedasticity
# reg2 -> no heteroscedasticity

Result:

> bptest(reg1)

    studentized Breusch-Pagan test

data:  reg1
BP = 20.542, df = 2, p-value = 3.462e-05

> bptest(reg2)

    studentized Breusch-Pagan test

data:  reg2
BP = 0.053288, df = 2, p-value = 0.9737

For reg1, the Null is rejected, for reg2, the Null is not rejected. This suggests that errors are not homoscedastic for reg1. Let us try to drill the problem up a bit.

Auxiliary regression:

# 1) Get residuals
Boston$u1 = residuals(reg1)^2
Boston$u2 = residuals(reg2)^2

# 2) Aux regression
aux1 = lm(u1~indus+lstat, data=Boston)
summary(aux1)
aux2 = lm(u2~indus+crim, data=Boston)
summary(aux2)

Result:

Regression 1

Call:
lm(formula = u1 ~ indus + lstat, data = Boston)

Residuals:
   Min     1Q Median     3Q    Max
-64.78 -36.94 -22.64  -2.13 581.64

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  62.0549     7.6859   8.074 5.06e-15 ***
indus         1.0020     0.6539   1.532    0.126
lstat        -2.7603     0.6282  -4.394 1.36e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 80.36 on 503 degrees of freedom
Multiple R-squared:  0.0406,    Adjusted R-squared:  0.03678
F-statistic: 10.64 on 2 and 503 DF,  p-value: 2.974e-05

Regression 2

Call:
lm(formula = u2 ~ indus + crim, data = Boston)

Residuals:
   Min     1Q Median     3Q    Max
-61.72 -56.21 -42.16 -16.11 995.65

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) 61.49061   13.08357   4.700 3.37e-06 ***
indus        0.01188    1.08488   0.011    0.991
crim        -0.18560    0.86527  -0.214    0.830
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 152.8 on 503 degrees of freedom
Multiple R-squared:  0.0001053, Adjusted R-squared:  -0.00387
F-statistic: 0.02649 on 2 and 503 DF,  p-value: 0.9739

It seems that lstat in regression 1 causes heteroscedasticity in this model since the coefficient is statistically different from zero. In regression 2 all coefficients (except intercept) are not statistically different from zero.

Now, if you try to mitigate heteroscedasticity, you could use the results from the auxiliary regression(s) to see if some (linear) transformation of the data helped to make the variance more harmonic (note that the interpretation of $\gamma$ will change when you make linear transformations, but p-values will give you some idea).


One method to visualize high-dimensional data in lower dimensions is t-sne, which has a python implementation in scikit-learn (here)

To calculate similarity between two sentences, having their embeddings, its common to use the cosine similarity


inputs and outputs of a fully connected layer for a note classifier

Starting with your last question: The final fully-connected layer shoud output the number of target classes you have. This output can then be passed to a softmax, which normalises the values between the range [0, 1] - allowing them to be interpreted as probabilities.

Remember, you train using the following dimensions:

[batch_size, height, width, channels]        # channels = 1 (in your case) so is removed

You always need to maintain the batch_size through one loop forwards and backwards through your neuiral network. Normally it set to None which allows the model to accept any batch size you decide on.

What people mean by "make a vector" is that the output of your convolutional layers will be a 3D (including channels) and so 2D in your case. You need to flatten these into a single vector. The conv layer output, which then flows into the fully-connected layers will then be:

[batch_size, height * width]         # height and width are those from the final conv layer

Depending on your framework, there will be a layer called Flatten or view etc., which gives you the vector for each image, keeping the batch_size. This is performed as part of the model structure and so is applied to all data as it passes through the model (training, validation and test data).