Interpreting vertical and horizontal parts of ROC curve

The explanation follows from understanding what a ROC curve is made of.
First, a reminder: a ROC curve represents the performance of a soft binary classifier, i.e. a classifier which predicts a numerical value (usually a probability) which represents the likelihood of the instance to be positive.
The points of a ROC curves correspond to what happens when splitting the instances at different thresholds of their predicted value/probability. For example let's say we have 10 instances with the following predictions and status, ordered by decreasing predicted value:
predicted value   gold status
  1.0                  P
  0.95                 P
  0.95                 P
  0.85                 N
  0.70                 P
  0.55                 P
  0.52                 N
  0.47                 P
  0.26                 N
  0.14                 N

If the threshold is set at 0.60, we have 4 TP, 1 FP, 2 FN and 3 TN.
If the threshold is set at 0.75, we have 3 TP, 1 FP, 3 FN and 3 TN.
etc.

If the classifier does a decent job, the top part should contain only gold positive instances and the bottom part only negative instances, since the classifier predicts a very high (resp. very low) value for these instances. By contrast the middle part is more likely to contain errors.
This means that with a decent classifier, we encounter only positive instances until a certain threshold from the top: in my example, there are 3 P instances above threshold = 0.85, so any threshold chosen between 0.85 and 1 will have no FP instances. This implies FPR=0 with TPR ranging from 0 to 3/10, hence a vertical line on the ROC curve from the origin to (0,0.3).
Applying the same observation for the lowest values in my example, we see than when the threshold is between 0 and 0.47 (excluded) we have no FN, so we have a perfect TPR=1 and therefore an horizontal line from (1,1) to (0.8, 1).
These two extreme parts of the ranking are what the author calls "most confident predictions" and "least most confident predictions". Btw the latter is not a very good term because actually the bottom part is where the classifier is very confident about the instance being negative. If anything, the part where the classifier is the least confident is around the middle of the ranking.
[edit: forgot to answer this question]

What important information can I gain from this?

So these two straight lines show how far the classifier can go:

without making any false positive error (bottom left vertical line)
without making any false negative error (top right horizontal line)

It can be useful in applications where one wants to have perfect precision at the cost of recall, or the converse. However in general (optimizing F1-score for instance) the optimal threshold is somewhere in the middle of the curve.
Of course the longer the line, the better the classifier in general. Geometrically, it's clear that when these two lines are long then the area under the curve (AUC) is large. Note that in theory it's possible for a classifier to have a few errors close to the extremes (i.e. short lines) while still performing well in the middle range, but in practice it's quite rare.


How much of a disadvantage is a small sample size?

When I read this, I guess you are up to a causal model?

When you use methods like linear regression or logistic regression, $n=630$ is a healthy sample size. So no problem with that. What could be a problem is that you have censored data (you don't know about the remining two constituencies).

IMO the best way to deal with this problem is to cleary state that you have missing data for this constituencies, so that you only can make claims for the remaining ones. It is a common problem that there is missing data. If it is really only about two constituencies and if you look at "left/right" tendence, the missing data is not a big problem as long as the missing observations are not highly important (e.g. super large) and there is no reason to believe that the missing observations come from an entirely different data generating process (e.g. extremely different to all other observations).


Technical name for this data wrangling process? Multiple columns into multi-factor single column

It is usually called reshaping! For a great description of the process, see this walkthrough, or read up on Hadley Wickham's documentation for the reshape package!


create my custom bag of words with N-grams to apply to

My initial recommendation would be to use the NLTK library for Python. NLTK offers methods for easily extracting bigrams from text or ngrams of arbitrary length, as well as methods for analyzing the frequency distribution of those items. However, all of this requires a bit of programming.

  a functionality that can be achieved over GUI

That's tricky. Have you looked into GATE? I'm not exactly sure if/how GATE does what you want, but it does offer a GUI.


If I wanted to build a classifier that takes short time to be trained, I would rather simplify the classifier than the data. In this case, I would rather train a logistic regression model (probably with Ridge or Lasso regularization), than a very complex architecture with few of the data I have.

If I had to simplify the data, I would take the average of the three channels, thus having a black and white image as input. I would not simplify it further (I am very confident that 4 points won't give decent accuracies).

I don't really know your circumstances, but Lasso in logistic regression is a method that might allow you to select some pixels. As the coefficients obtained by Lasso are sparse (see this answer), Lasso will select the pixels that are relevant towards the prediction of the class. You can then train a bigger network with those pixels that allows you to capture nonlinearities and more complexity.


encoding datetime features .. label or onehot?

One hot encoding removed the order information compared to the label encoder. Which one to use depends on the specific question. If you think the order of categories matters, use the label encoder. If not, use the one-hot encoder. For booking prediction, I will prefer one hot.


What is "experience replay" and what are its benefits?

The key part of the quoted text is:

  To perform experience replay we store the agent's experiences $e_t = (s_t,a_t,r_t,s_{t+1})$

This means instead of running Q-learning on state/action pairs as they occur during simulation or actual experience, the system stores the data discovered for [state, action, reward, next_state] - typically in a large table. Note this does not store associated values - this is the raw data to feed into action-value calculations later.

The learning phase is then logically separate from gaining experience, and based on taking random samples from this table. You still want to interleave the two processes - acting and learning - because improving the policy will lead to different behaviour that should explore actions closer to optimal ones, and you want to learn from those. However, you can split this how you like - e.g. take one step, learn from three random prior steps etc. The Q-Learning targets when using experience replay use the same targets as the online version, so there is no new formula for that. The loss formula given is also the one you would use for DQN without experience replay. The difference is only which s, a, r, s', a' you feed into it.

In DQN, the DeepMind team also maintained two networks and switched which one was learning and which one feeding in current action-value estimates as "bootstraps". This helped with stability of the algorithm when using a non-linear function approximator. That's what the bar stands for in ${\theta}^{\overline{\space}}_i$ - it denotes the alternate frozen version of the weights.

Advantages of experience replay:

More efficient use of previous experience, by learning with it multiple times. This is key when gaining real-world experience is costly, you can get full use of it. The Q-learning updates are incremental and do not converge quickly, so multiple passes with the same data is beneficial, especially when there is low variance in immediate outcomes (reward, next state) given the same state, action pair.
Better convergence behaviour when training a function approximator. Partly this is because the data is more like i.i.d. data assumed in most supervised learning convergence proofs.

Disadvantage of experience replay:

It is harder to use multi-step learning algorithms, such as Q($\lambda$), which can be tuned to give better learning curves by balancing between bias (due to bootstrapping) and variance (due to delays and randomness in long-term outcomes). Multi-step DQN with experience-replay DQN is one of the extensions explored in the paper Rainbow: Combining Improvements in Deep Reinforcement Learning.

The approach used in DQN is briefly outlined by David Silver in parts of this video lecture (around 01:17:00, but worth seeing sections before it). I recommend watching the whole series, which is a graduate level course on reinforcement learning, if you have time.


Perform clustering from a similarity matrix

I am not sure I fully understand your question, but generally the metric you use to build a DAG needs to be understood in terms of how you interpret relevant results.
That said, a cluster map sounds like a good match for your use case. That is, a correlation matrix with sorted values according to linkage clustering on your datapoints. See below an example:

You can easily experiment with something like this using seaborn library and seaborn.clustermap.


Using Binary Image Classification on Video

Using a multi-class network would probably be the most efficient approach.

Training two networks (as suggested in the comments: one network to detect the presence of a person and one to detect the state of the knee) is unnecessary and overly complex.  You can train the network on three classes: bent knee, straight knee, absent knee.  For the absent knee, you do not need to train on all possible images, just images you expect to see.  For instance, these negative data could be the background of your intended images; take a picture with the knee and one without.

If speed is a major requirement, the model architecture becomes critical.  I suggest exploring MobileNets as they are quite fast and do not sacrifice too much accuracy.


After reading the source code, I find out that the problem is nothing related to your memory, but the limit of integer.

In the source code, it shows both of $n\min(2k,n)$ and $\min(2k,n)*(\min(2k,n)+8)$ should be less than Integer.MAX_VALUE, which is $2^{31}-1=2147483647$. In your case, $n\min(2k,n)=36176793968>2147483647$, and $\min(2k,n)*(\min(2k,n)+8)=6781851888>2147483647$.

I'm no spark expert nor java export so maybe I can't give you a solution. But personally speaking your dataset is truly big!

References

https://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html


How to plot mean_test score and mean_train score of GridSearchCV

You could visualize them as a heatmap.

For example you could use the C values as the rows, the gamma values as the columns and the color intensity of each element in the heatmap array would correspond to the mean_test_score.

To implement this you first need to create a pandas.DataFrame like this:

$$
\begin{array}{c | c c c}
  & C   & gamma & mean\_test\_score \\ \hline
1 & 0.1 & 0.001 & 0.798 \\
2 & 1   & 0.001 & 0.813 \\
3 & 1   & 0.01  & 0.801 \\
4 & 10  & 0.001 & 0.787 \\
\end{array}
$$

To do this you need to store each run you make in a different line, which will contain all necessary hyper-parameters and the result. Then you will need to make a pivot table which will use C as the rows, gamma as the columns and mean_test_score as the values.

pivot = pd.pivot_table(df, values=df['mean_test_score'])

This pivot will be the array that will form your heatmap. Now you should select your aesthetic parameters (e.g. colormap) and proceed to make the heatmap.

sns.heatmap(pivot) # plus any other aesthetic parameters you wish


NLP - Simple approach to identify commonalities in text comments between people

Note: The questioner requests for the response to be simple. Although this approach below is not simple, it is an attempt to provide a perspective that could help.

Understanding the problem stated:
At a conceptual level, there are arguably 3 concepts in this question as given below.

Semantic similarity i.e. how similar the responses (in meaning) are to one another. In a way, this similarity could be loosely inferred as similar proposition.
Syntactic similarity i.e which aspects (tokens or entities or chunks) overlap between the responses.
Text classification i.e. "is one feedback in agreement with another or neutral or against?".

Evaluating approaches:
There could be many potential solutions. Following steps are one such attempt.

Step 1 Deploy a cosine similarity algorithm to measure the similarity between responses. In order to bring it a step closer to semantic similarity, use WORDNET to build the features for computing cosine similarity. This will ensure that tokens such as "path" are treated closer to token "road".
Step 2 Group responses (for the same question) beyond a threshold cosine value (example: 0.75) as similar responses. This can be loosely considered as a set of different responses for a question.
Step 3 Train a model to identify agreement or disagreement between responses. This training can be based on a classification algorithm or at the least based on a bag of words approach(hard coded tokens such as "i dont agree", "it is incorrect" etc). This step is perhaps the least scientific one but the only pragmatic one that the author can think of.


Machine learning model bundled with a library vs. an API

The advantages of deploying the model as a package is the code will be part of the monolith application and require no remote calls thus:

No additional operational demands
No external dependencies
Faster
Fewer security issues
Same uptime as the rest of the application

In general, most features stay in a single monolith application until there is are specific reasons to create separate services.


Is a large number of epochs good or bad idea in CNN

Assuming you track the performance with a validation set, as long as validation error is decreasing, more epochs are beneficial, model is improving on seen (training) and unseen (validation) data.

As soon as validation error starts to increase, it signals that model is over-fitting on training data, thus the learning process should be stopped. This method is independent of the details of a model.

In practice, save the model corresponding to the best validation error found so far, and stop the process only when validation error is consistently increasing. This can be done manually or with an ad-hoc threshold based on the distance of current validation error from the minimum error so far. This way, learning will not be stopped prematurely due to fluctuations in validation error.


Extremely high MSE/MAE for Ridge Regression(sklearn) when the label is directly calculated from the features

(To summarize the comment thread into an answer)

Your original scores:

Mean Absolute Error: 37216342513.01034
Root Mean Squared Error: 871869805169.7842

are based on the original-scale target variable and are between $10^{10}$ and $10^{12}$, at least significantly smaller than the mean of the features (and the target)?  So these aren't automatically bad scores, although for a perfect relationship we should hope for better.  Furthermore, a 0.69 R2 values is pretty low, no scale-consciousness needed.

That both of the model's coefficients came out positive is the most worrisome point.  I'm glad you identified the culprit rows; I don't know how I would have diagnosed that from here.

Your new ridge regression still has "large" errors, but significantly smaller than before, and quite small compared to the feature/target scale.  And now the coefficients have different signs.  (I think if you'd left the TransformedTargetRegressor in, you'd get largely the same results, but with less penalization.)

Finally, when such an exact relationship is the truth, it makes sense not to penalize the regression.  Your coefficients here are a little bit larger, and the errors drop away to nearly nothing, especially considering the scale of the target.


How can I go about building a model for large number of outputs?

Neural networks can have 25 outputs.  You would probably get a slightly more accurate result with 25 independent models, but the computation and training time would be 25x of one independent model.  One model with 25 outputs would only take slightly longer than one independent model.


Is there a network analysis tool built into Orange?

This Networks add-on for Orange data mining suite should help you. If you are open to using other solutions, I would recommend networkx Python library.


Accuracy is a measure of comparing the true label to the predicted label. K-Means is an unsupervised clustering algorithm where a predicted label does not exist. So, accuracy can not be directly applied to K-Means clustering evaluation. However, there are two examples of metrics that you could use to evaluate your clusters.

Within Cluster Sum of Squares

The first is Within Cluster Sum of Squares (WCSS), which measures the squared average distance of all the points within a cluster to the center of the cluster (known as the cluster centroid).

To calculate this, you can start by finding the Euclidean distance between a given point and the cluster center which the point is assigned to. You then repeat this process for every point in the cluster, and then sum the values for the cluster and divide by the number of points. Finally calculate the average across all clusters. This will give you the average within cluster sum of squares.

This measurement can indicate the variability of the points within the cluster in terms of the average distance to the cluster center. A large sum of squares could indicate a very large a spread out cluster. A small sum of squares could indicate a small, compact cluster with little variation in the attributes of the points. This measurement is sometimes called Cohesion because it measures similarity between the data points in that cluster.

Between Clusters Sum of Squares

The second metric is Between Clusters Sum of Squares (BCSS), which measures the squared average distance between all cluster centroids.

To calculate this, you can find the Euclidean distance from a given cluster centroid to all other cluster centroids. Repeat this for all of the clusters. Then, sum all of the values together. This will give you the Between Cluster Sum of Squares. You can divide by the number of clusters to calculate the Average Between Cluster Sum of Squares.

This measurement can indicates the variation between all clusters. A large number can indicate clusters that are spread out. A small number can indicate clusters that are close to each other. This measurement is sometimes called separation because it measures the separation of the clusters.

Other Resources

You can also look into the Silhouette Coefficient, which combines both the Cohesion and Separation. And the Elbow Method can be used to help you determine the optimal K value.

Checkout the limitations of k-means clustering. And if you want an in-depth explanation of WCSS and BCSS, check out this Khan Academy video:

https://www.khanacademy.org/math/statistics-probability/analysis-of-variance-anova-library/analysis-of-variance-anova/v/anova-2-calculating-ssw-and-ssb-total-sum-of-squares-within-and-between-avi

Depending on the language you are programming in, there may be packages available to assist you in evaluating your clusters. R has kmeans, which has attributes including withinss and betweenss. For Python, Sklearn's implementation of k-means has inertia, which is the "sum of squared distances to the closest centroid for all observations in the training set".

Consider a different algorithm

If you are looking to measure the accuracy of a prediction on a given dataset, it would help to define what the ground truth of that is. If you have a ground truth, there may be a better algorithm to use to model the outcome.


Predict customer behaviour with Transformer(attention is all you need)

Definitely, it is a good idea and has been attempted before.
Have a look at Alibaba's paper which uses transformers:
https://www.kdnuggets.com/2019/08/order-matters-alibabas-transformer-based-recommender-system.html
https://arxiv.org/abs/1905.06874
There are also various other papers that use other types of seq2seq for recommender systems.


Building a Citation Network to Analyze in R

I don't understand if you want to build a network of authors who cite each other, or a network of papers who cite other papers (which would be a much sparser network because the coauthorship relationships won't show up as edges).

I would follow a strategy similar to this one (off the top of my head):

Assign ids to your papers
Build 2 csv files : papers.csv, citations.csv
read in 2-col csv file "paper.csv" as a two-column data frame:  col1: paper_id, col2: title
read in 2-col csv file "citations.csv" as a two-column data frame:  col1: paper_id, col2: cites_id,

With R's igraph package, you can construct a  network pg (for papers_graph)
with (pseudocode)

pg <- igraph::graph_from_data_frame(citations)

Then you assign "vertex attributes" to the nodes in the network:

pg <- set_vertex_attribute(pg, "title", value= papers)
# same as:  V(pg)$title <- papers$title

(and possibly many other attributes)

Then you can use igraph's many functions (~200) to analyze the network.

For visualizations, you plot the ids, and use a diagram type which gives you the title when you move the mouse over / click on the node symbol (which is labeled only with the id to save screen space). You can use other design elements such as coloring the nodes by release year for instance).


Process melspectrograms with convolutional neural network

Leave the resolution untouched, and introduce more convolutional
  layers.

This should be the next step. Two primary reasons for it :

This should reduce number of trainable parameters
Model can learn more abstract features


Time series clustering using dynamic time warping and agglomerative clustering

Visually, there are no clusters in your data.

If your data had clusters, there would be subtrees with a small MRCA, with a long line leading to it.

There are metrics to quantity this, but you are not there yet. Can you show the time series? (or better, send them to me)

You may be lucky, and have one of ten problems that are easy to fix. See this paper.

You ask " when is z-normalising not required for DTW?" The answer is never.
eamonn


One of the OLS assumptions is the zero conditional mean assumption, which states that $E[u|X]=0$, so that errors average out to 0.

Another assumption is homoskedasticity, which means that there is no (auto)correlation in the residuals $E[u u'|X]=\sigma I$. So the covariance matrix (sometimes also called variance-covariance matrix) is:

$$ \sigma I = \sigma \left[ \begin{array}{rrrr}
1 & 0 & \cdots & 0\\
0 & 1 & \cdots & 0 \\
 \vdots & \vdots & \vdots & \vdots \\
0 & 0 & \cdots & 1 \\
\end{array}\right] = \Omega .$$

The important thing is that all elements in the matrix (apart of the diagonal elements from top left to bottom right) are zero, which means "no correlation between residuals". Also there is a constant variance equal to $\sigma$.

In case $ \Omega \neq \sigma I$, you face heteroscedasticity and you would need to "model" $\Omega$, e.g. by "Feasible Generalized Least Squares" (FGLS).

Say you have a linear model  $y=X\beta+u$ with $E(u^2)=exp(Z\gamma)$. Here $exp(Z\gamma)$ is an example for a scedastic function which describes the conditional variance. In order to obtain consistent estimates for $\gamma$, one need to obtain consistent estimates for $u$ in the first place. This can be done by using OLS to obtain $\hat{\beta}$ and $\hat{u}$.

Based on this, one can run an auxiliary linear regression $ \log \hat{u}^2 = Z \gamma + v $ to get an estimate for $\hat{\gamma}$. These estimates can be used to compute  $\hat{\omega} = (\exp(Z\hat{\gamma}))^{1/2}$. Finally, FGLS estimates of $\beta$ are obtained by OLS with weights $\hat{\omega}$ which is called feasible weighted least squares.

See Davidson/MacKinnon: "Econometric Theory and Methods", Ch. 7.4.


The equation I ended up using is listed in the link below, provided by Emre:

Linear regression with non-symmetric cost function?


What are some good books on Machine Learning and AI like Krugman, Wells and Graddy's "Essentials of Economics"

The two books that come into my mind are:

Artificial Intelligence: A Modern Approach
The Deep Learning Book

They both start from the basics and escalate while moving on.

Also thanks for your recommendation, I'll take a look at it because I want to jump to finance at some point in my career:)


Understanding why my binary classification is approaching 50% accuracy using TensorFlow and Keras

You should consider analysing your data before applying NN. For instance, unlike Decision Trees Dense layers are not supposed to work well with denormalised features.


how do tree based methods deal with missing feature columns?

You can use Parameter to handle the unknown classes called handle_unknown in sklearn.preprocessing.OneHotEncoder
Best practice for any type of encoding :
You should train an estimator for  Onehot encoding on the training data only, and when encoding test data, you should use the same estimator used on training data.
Eg : sklearn.preprocessing.OneHotEncoder does this, and it has a parameter called : handle_unknown.

handle_unknown{‘error’, ‘ignore’}, default=’error’

Whether to raise an error or ignore if an unknown categorical feature is present during transform (default is to raise). When this parameter is set to ‘ignore’ and an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros. In the inverse transform, an unknown category will be denoted as None.
Optimal option is : You could use this parameter and set it to ignore, in order to ignore the unknown feature value and avoid an error, until you retrain your model eventually and add the new feature values to your model.
from sklearn.preprocessing import OneHotEncoder

ohe=OneHotEncoder(handle_unknown='ignore')

train=ohe.fit_transform(train)

test=ohe.transform(test)


Best network structure for unsupervised learning

You are describing a problem that can be solved with Probabilistic Graphical Models (PGMs). PGMs use graphs to expresses the conditional dependence structure between random variables.

pgmpy is a Python package for PGMs.


Loss Function - Decreasing a lot at beginning of the epoch

Have you already plotted the training and validation loss over multiple epochs? What you would expect is something that looks like an exponential decay.

Chances are your models learns the mean of the target variable in the first few batches, which reduces the MSE-loss already considerably and only later learns the subtle differences in your data. You can try to use different batch sizes as well.

Are you generally happy with the performance on your model? Does it perform well on the test/validation set? If this is the case there might be no reason to be worried about.


How to input different sized images into transfer learning network

Resizing is the best option, if they are bigger downscale them, else upscale them.


Why does the overfitting decreases if we choose K to be large in K-nearest neighbors?

Overfitting is "The production of an analysis which corresponds too closely or exactly to a particular set of data, and may therefore fail to fit additional data or predict future observations reliably." (Oxford dictionary)
When you fit a ML model, you use a dataset that you assume is a sample of the real statistical distribution you want to model. This means that the dataset does not fully reflect the real distribution and will contain anomalies, exceptions or some randomness. An overfitted model would stick too much to the particularities of this dataset and be too variant, instead of learning statistical patterns that would smooth these particularities and bring the model closer to the real statistical distribution.
In the case of KNN, K controls the size of the neighborhood used to model the local statistical properties. A very small value for K makes the model more sensitive to local anomalies and exceptions, giving too many weight to these particular points. On the contrary, a too large value of K would make the model ignore the local structure of the distribution you try to learn, and produce an underfitted model.
Let's take an example to illustrate this:
In a given city, you want to predict if a household has low or high income because you have data about most of them, but not all. You decide to use KNN and predict the income category of the households with no data according to the majority category among the K closest neighbors. This is likely to work well because we know that housing prices tend to create low or high income neighborhoods.
If you use a small K, let's say K=1 (you predict based on the closest neighbor), you might end up with these kind of predictions:

In a low income neighborhood, you wrongly predict one househlod to have a high income because its closest neighbor has a high income. Indeed, this neighbor refused to move out from his childhood house though he could afford living anywhere else.
In a high income neighborhood, you wrongly predict one household to have a low income because its closest neighborhood has a low income. Indeed, though he couldn't afford the rent, the house belongs to the family since ever.

The model overfits on these particular cases. Now if you increase the number of neighbors you compare to (= increase K), you will get a majority of either low or high income households in the neighborhood and correctly classify these examples.
Finally, if you increase K too much, you will have an underfitted model. In our example, if we increase K indefinitely we will end up taking all households in the city as neighbors, and basically always predict the category that is the majority category in the city, ignoring local particularities.


How to label images for CNN use as classifier

In general in this situation you should be training to just the specific problem you are trying to solve - which from the sounds of it is just whether edges of any kind on the shape are blurry or cleanly printed.

Unless you specifically need the shape labelled as a square triangle etc you should drop it as a label as will just add unnecessary complexity and overhead to the model.


Сlassification using convolutional layers

You need Dense() layers after convolution / pooling layers. Conv and max-pool are meant to process pixel data in a format that can be later received from dense layers.
It's the Dense() output layer that, using a softmax activation, can technically perform the classification (by returning a one-hot encoded vector).


How can I transpose a high dimensional dataset?

This is possible to do in WEKA, but it only works if:

the first column of your data is of string or nominal type, and
the rest of your columns are numeric.

Open the dataset in the explorer, and first remove the class attribute by choosing 'No Class' from the drop-down menu on the right. Then, apply the filter from filters.unsupervised.attribute.Transpose. After this, you should re-apply the class attribute by re-selecting the class attribute in the drop-down menu.

If your data doesn't meet the requirements listed above, I would recommend using this short python script to do it:

import pandas as pd
df = pd.read_csv('your_file_here.csv', index=False)
df.transpose().to_csv('your_new_file_here.csv', index=False)

and then loading the new file into WEKA.


What's the meaning of precomputed features?

Pre-trained - You save your trained network with all its weight on a big and reliable dataset. Then use again for a similar requirement with further training on some additional data. This is a typical transfer-learning.

Precomputed - Pre-compute is the process of storing the output of all your layers before the last one re-use for some other purpose on the same dataset.

It makes the subsequent (beyond the first) training faster. Layer other than the last one will not change with new data.

Since only the last (classifier) layers change during training but not the earlier layers, we can convert the images to feature vectors just once and then train the classifier on these feature vectors instead of on the images.

Reference - 1    Reference - 2


Recommendation system depend on user rating and favorite list

You can use collaborative filtering. From the user ratings you can define a ratings matrix with missing ratings that your algorithm will predict to recommend items to users.

The items data and user's whishlist can be included as auxiliary information (basically you cluster Sakes given their alcohol percentage, polish percentage and acidity and users given their whishlist using a K-mean algorithm). Have a look at this paper:

https://pdfs.semanticscholar.org/a0bc/3d1c05da5b57bee066368ffd0839247832c8.pdf


exporting the model parameters after training and testing

I'm not sure if that's what you are looking for - but you should check out the package 'pickle'. You can use it to easily dump your model into a file and load it again.


Deep model ensemble giving different results

Getting varying results at the level of F1 score is possible but should be high, as your are essentially taking the average over many values (assuming a reasonably sized dataset).

There are however many sources of random behaviour that could be creeping in. From most obvious to most obscure:

Are the definitions and parameters of each of your models identical each time?
Do you use the same train/validation/test splits for each session?
Are your training hyper-parameters (number of epochs, learning rate, early stopping) the same each time?
Do you set the random seed in Numpy/Keras/Tensorflow?
Did you set any Nvidia CUDA flags for determinism?

Regarding the final point, GPU determinism - this is a very new thing in the context of deep learning and many people believed it was impossible, but there is good progress being made. Check out these video/slides from Nvidia GTC conference this year.

These are really just the components of each individual building block i.e. each deep model. You could also introduce random behaviour perhaps in the way you chain these models together to create your ensemble.


Answer 1
The reason for reshaping is to ensure that the input data to the model is in the correct shape. But you can say it using reshape is a replication of effort.

Answer 2
The reason for converting to float so that later we could normalize image between the range of 0-1 without loss of information.


Validation vs. test vs. training accuracy. Which one should I compare for claiming overfit?

Which two accuracies I compare to see if the model is overfitting or not?

You should compare the training and test accuracies to identify over-fitting. A training accuracy that is subjectively far higher than test accuracy indicates over-fitting.

Here, "accuracy" is used in a broad sense, it can be replaced with F1, AUC, error (increase becomes decrease, higher becomes lower), etc.

I suggest "Bias and Variance" and "Learning curves" parts of "Machine Learning Yearning - Andrew Ng". It presents plots and interpretations for all the cases with a clear narration.

  When I get to run 10 fold cross-validation, I get 10 accuracies that I
  can take the average/mean of. should I call this mean as validation
  accuracy?

No. It is a [estimate of] test accuracy.
The difference between validation and test sets (and their corresponding accuracies) is that validation set is used to build/select a better model, meaning it affects the final model. However, since 10-fold CV always tests an already-built model on its 10% held-out, and it is not used here to select between models, its 10% held-out is a test set not a validation set.

  Afterward, I test the model on 30% test data and get Test Accuracy.

If you don't use the K-fold to select between multiple models, this part is not needed, run K-fold on 100% of data to get the test accuracy. Otherwise, you should keep this test set, since the result of K-fold would be a validation accuracy.

  In this case, what will be training accuracy?

From each of 10 folds you can get a test accuracy on 10% of data, and a training accuracy on 90% of data. In python, method cross_val_score only calculates the test accuracies. Here is how to calculate both:

from  sklearn import model_selection
from sklearn import datasets
from sklearn import svm

iris = datasets.load_iris()
clf = svm.SVC(kernel='linear', C=1)
scores = model_selection.cross_validate(clf, iris.data, iris.target, cv=5, return_train_score=True)
print('Train scores:')
print(scores['train_score'])
print('Test scores:')
print(scores['test_score'])

Set return_estimator = True to get the trained models too.

More on validation set

Validation set shows up in two general cases: (1) building a model, and (2) selecting between multiple models,

Two examples for building a model: we (a) stop training a neural network, or (b) stop pruning a decision tree when accuracy of model on validation set starts to decrease. Then, we test the final model on a held-out set, to get the test accuracy.
Two examples for selecting between multiple models:

a. We do K-fold CV on one neural network with 3 layers, and one with 5 layers (to get K models for each), then we select the NN with the highest validation accuracy averaged over K models; suppose the 5 layer NN. Finally, we train the 5 layer NN on a 80% train, 20% validation split of combined K folds, and then test it  on a held out set to get the test accuracy.

b. We apply two already-built SVM and decision tree models on a validation set, then we select the one with the highest validation accuracy. Finally, we test the selected model on a held-out set to get the test accuracy.


How to best estimate the coefficients of a confusion matrix in case of strong class imbalance?

There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, DP, etc.

Disclaimer:

If you use python, PyCM module can help you to find and calculate these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

>>> score = cm.Kappa


out_channels can be thought of as the number of features/filters the layer can learn. So, for eg., if a layer of a CNN has out_channels set to 64, it can "learn" 64 features. These features could be edges/patterns. This is a hyper-parameters and can set to an arbitrary number.
The activation of a neuron in a layer will be high if the input to that layer contains the feature learned by the layer, that's how a convolution (or correlation) function works.
In a CNN, a neuron is the result of computing the dot product of the input and the filter/kernel at every window of the convolution.


How to represent time based periodic data for use in ML

Answering the question JahKnows asked in the comment above will help with finding the suitable solution for your use cases.

If you are not exactly sure what you are after, here are a few things you can do with a dataset like this:

One of the basic things you can do is to find customers that behave similarly. With the data that you have, you can consider the number of machines used by each customer in each month as features. In this case, each customer is described with 5 features. I would run a clustering algorithm against this data (of course leave CustId field out) and see how many types (clusters) of customers you have.
Another thing to do is to run statistical analysis over the number of machines your customers use each month and find those that stand out from the rest of the population (customers).
Another analysis you could do is to analyze the trend for each customer separately and based on their past behaviour, determine if the number of machines they have used in the past month was in accordance with their historical behaviour. In this case, you can take proper actions based on the shift of behaviour. E.g. a drop in the number of machines could mean the customer is having technical difficulties or churning.


Confusion matrix in multilabel classification of an object in more than one class simultaneously

Confusion matrix is generally not considered as a useful tool to evaluvate our model for multiclass classification and we rather use what is known as AUC-ROC curve.

AUC stands for Area Under Curve whereas ROC stands for Reciever Operating Curve. AUC - ROC curve is a performance measurement for classification problem at various thresholds settings. ROC is a probability curve and AUC represents degree or measure of separability. It tells how much model is capable of distinguishing between classes.

Here is a detailed explaination about the AUC-ROC curve.


Regression with Neural Networks in Tensorflow problem

Based on my observations

I don't see your training_observations (output) normalized. Refer
this example
I see you are using sigmoid in all your hidden layers.
       Generally tanh is used as mean of data is closer to zero, this actually makes the learning for next layer a bit easier.


Difference between AlphaGo's policy network and value network

In brief each net has a different purpose as you mentioned:

The value network was used at the leaf nodes to reduce the depth of the tree search.
The policy network was used to reduce the breadth of the search from a node (guiding towards promising immediate actions).

In general, you can use value function methods to find an optimal policy or directly search in the policy space to optimize a parametrized policy function (of course there are pros and cons). You can use function approximators (e.g Deep Nets) in each case. I see that mainly you are confused about the policy net so I focus my answer into this.

The policy net was first:

trained to do the moves that most likely a human would do given a board state (so input is a board state and output is a histogram that shows the probability of each action given that state). The net can approximate the probability function underlying the mapping from states to actions. It is reasonable to think to start building your policy from available data after all. After supervised training using experts moves the policy net could play the game sufficient (although far from a Master's level). Simply, you attempted to capture the general pattern of action selection of professional players.

Then,

it was trained in games with opponent itself, in order to optimize the previous-learned policy. This time its weights were updated using the REINFORCE algorithm. By doing this, you update the net parameters towards maximization of expected reward. Eventually you have a net that not only selects the actions like a professional player but also towards winning the game (However it cannot plan!).

After this step, they approximated the value function of a bit more noisy version of the learned policy, by regression (input is the state board and target the result of the game). You can use this network to affect the leaf node evaluation.

Conceptually speaking, the policy net gives you a probability over actions, but this doesn't indicate that you will end up in a good, for winning the game, state. AlphaGo had some "blind spots" and during the tournament did some really bad moves but also one exceptional move that a human could never had thought.

Finally you can use your planning algorithm (MCTS) in combination with these nets. Why we took all these steps? Briefly, the simple MCTS without any "intuition" would have failed.


Effect of Stop-Word Removal on Transformers for Text Classification

Very interesting question.

Easy, but probably lazy answer

When using pre-trained models, it is always advised to feed it data similar to what it was trained with. Basically, if it matters, don't remove them, and if it doesn't matter, it doesn't hurt to keep them in. Obviously, if you can, try with or without stopwords, and see what works best for your problem.

Longer answer

You actually have two ways to "remove" your stopwords. Either you remove them from the input sequence altogether. Or, you could replace them with a mask token (i.e. <#UNKNOWN> or <#MASK>).
In the latter case, the transformer will implicitly guess what these masks are and you will have achieved the original goal of stopwords removal: make sure they don't affect the predictive outcome. Indeed, take the sentences:
"I like basketball with an audience" & "I like basketball without an audience"
These two sentences are both about basketball and you wouldn't want with/without to make your model think these sentences are about different topics. By masking with and without, you both "remove" the stopwords, and you don't potentially mess with the fact that the pre-trained model didn't use data without stopwords.
Now, what would happen if you fed an "incomplete" sentence to a transformer. The positional encodings would retain the notion that certain words are before or after each other, which is what you want. But if you remove stopwords, some words may appear "too close" to each other. But does that matter? I do not think so.
If the vector your transformer outputs for "word1 <#Mask> word2" is widely different than the one for "word1 word2", it must mean that the masked token is crucial to the overall meaning of the sentence, which would also suggest that it shouldn't be a stopword to begin with.

Final answer

I would suggest masking the stopwords instead of removing them. But, if performance is so critical that you need to feed smaller sequences, I think that you will be alright.


Why this TensorFlow Transformer model has Linear output instead of Softmax?

The key is precisely in the definition of the loss:
loss_object = tf.keras.losses.SparseCategoricalCrossentropy(
    from_logits=True, reduction='none')

As you can see, the loss is created with the flag from_logits=True which means that the input to the loss is not a probability distribution, but unnormalized log probabilities, namely "logits", which is precisely the result of the final projection, before any softmax.
When from_logits is true, the softmax itself is handled inside the loss, combining it with the sparse categorical cross-entropy into a more numerically stable form.
From the docs:

from_logits:  Whether y_pred is expected to be a logits tensor. By default, we assume that y_pred encodes a probability distribution.
Note - Using from_logits=True may be more numerically stable.


Obtaining consistent one-hot encoding of train / production data

If I understand your question correctly you want to make sure that the order of the encoding is always the same. Have you tried sklearn- to be more specific sklearn.preprocessing.OneHotEncoder?

The way it works is that you fit (or fit_transform) on your training sample. Then you save the state of your encoder (for example you can pickle it). In production you then load this encoder and transform. This should then provide the same results.

If for some reason you want to stay away from scikit you can also create a dict from your training sample and then apply this dict on production - this should be possible with pandas built in functionality.


This is often done to visualize if there is any structure in the data. Often you color the clustering differently to check if samples from the same cluster are close.
Often data contains a lot of redundant information. With many dimensions, you get the curse of dimensionality. This can lead to a few large and many small clusters. By reducing the dimensionality to a few informative features the clustering solution often improves. This is very dependent on the dataset. Perhaps all of your dimensions are informative then dimensionality reduction won't help much. Look at scoring metrics such as silhouette analysis/score, gap statistic, or elbow method.

In your case, most of the dimensions seem informative. Perhaps you can remove one or two dimensions, but it likely won't have much effect on the clustering solution.

You can choose whatever PC's you'd like but they are ordered according to the amount of variance they explain. You can look at how much the different original features contribute to the different PC's.


The issue with linear regression in your case is not whether it can be used - it can nearly always be used to build a predictive regression model - it is whether the resulting model is useful, usually whether it is accurate enough for the intended use.

The accuracy and utility of your model does not depend only on the nature of $\mathbf{x}$. Instead it depends on the true nature of the $\mathbf{x} \rightarrow y$ relationship. Your measurements and model are an approximation of that relationship. Linear models are fast and stable to compute, but can be limited if the true relationship being approximated is non-linear.

Here are some basic thoughts/feedback on your questions:

  Can Linear Regression work for such datasets?

Yes it can work. Will it be good enough? You will know after testing.

  What are other models I can use?

Almost any regression model class could be applied to this problem. For a quick comparison, to see if a non-linear model will make more accurate predictions for you, then you could try an easy-to-apply model from an existing library, such as XGBoost (which has a stand-alone command-line version).

  Should I drop one feature (some $x_{i,d}$) from each data record as I can easily calculate the value of dropped feature by using 1 - summing prob(remainingfeatures).

For a linear model, your intuition is correct that one feature is redundant, as it is itself a linear combination of the other features, so there is no impact to removing it. You could remove any single column. You should not need to though, and some model classes might work better if left in. As always, experiment if you are not sure, and use hold-out test data to verify your ideas (or even better a separate cross-validation set before final test stage)

  As I use Linear Regression all independent variables ($x_{i,0}$, $x_{i,d}$)'s are < 1 (because of the probability), the dependent variable is not. Should I use $log(y_i)$ instead (because weights $W$ for $\hat{y}_i = W\mathbf{x}_i$ calculated would be very large)?

For linear regression this scaling is not very important. However, there are a few caveats:

If you measure success by an error metric (such as mean square error), then to compare different ideas fairly, you should report the test error in the same way - e.g. you should scale back the $log(y)$-based predictions and calculate error on the same $(y - \hat{y})^2$ each time.
For some model classes, they could work better if the target variable is within a certain range. You should check documentation for this.
How useful the mapping is depends on the true nature of the relationship between $\mathbf{x}$ and $y$. Specifically for a linear model, you have completely changed the line you are trying to fit - it could as a result be a lot better or lot worse, but that is not dependent just on $\mathbf{x}$. So again you will have to give it a try and test the results.


Both of the examples are clustering examples. Clustering is about grouping of similar dataset when one is not given the data.

In the gene problem,

One possible setting is you are given the DNA micro-array data. Your task is to learn how many types of people are there. This is an unsupervised learning problem, we are not given the labels. We just group  people with similar type of genes together.

In the cocktail party problem:

There are two people in a room and there are microphones. We just record the audio and pass it to an algorithm and tell the algorithm, hey, learn the pattern if there is any. Each individual might have their own speech pattern/ language/accent. The algorithm pick it up on its own such pattern and detect that there are two people in the room and they can distinguish the speeches of the two people.

Let me give you an example of a non-clustering example.

For example, a question of interest is to detect anomaly. For example, what you are given could be thr normal operating state of a machine, say their sensor readings. From those readings, you have to learn what is normal for the machine and you have to figure out when you are given a new data point, should it be considered normal.


Is there an unsupervised learning algorithm that can cluster data based on more than two dimensions?

k-means works with more than two dimensions (e.g., the sklearn implementation of k-means does not make any assumptions on the dimensionality of the data). Assuming that k-means is the "appropriate" way to deal with your data, you may consider normalizing your columns to have unit variance first.


Generate pdf from jupyter notebook without code

Try this:
jupyter nbconvert --to pdf --TemplateExporter.exclude_input=True  my_notebook.ipynb

This also works for html output.
You will find the documentation for this and other options here.
FYI, for complex notebooks, this may generate errors depending on your version of nbconvert, LaTeX and other components. In that case try to convert to html then print to pdf using a browser.


What does the long curve-shape t-SNE mean?

The lines therefore are the time series. t-sne is a very locally sensitive algorithm so every data point is very likely to show up nearest (in 2-D) to its nearest (N-D) neighbour.

In MNIST there is no sequence, so it doesn't show up as a line in 2-D. Every digit is written in a set of similar ways that cluster together. In your time-series data, if each time point is most similar to its next time point, what we expect is for them to be in some sequence - because you'd want point t=n to be closest to points t=n-1 and t=n+1  (if they are in fact the most similar).


Convolutional Neural Network overfitting

This is because you have very little amount of data. If not enough data is provided to CNNs, they are very likely to overfit. You can do the following things in order to overcome this problem:

Data Augmentation : It is a technique to create new examples from the training examples by doing some preprocessing on them e.g. Rotation, Scaling, etc.

Rigorous Dropout : Dropout is a very powerful technique to control overfittting. You can try using more rigorous dropout in your architecture


different results with MEKA vs Scikit-learn!

According to this question

  LinearSVC is not linear SVM, do not use it if do not have to.

and furthermore

  LinearSVC [is] one of the mistakes of sklearn developers.

In this picture from the question you can observe how it is inaccurate, even compared to SVC with kernel=linear.

Further differences may be explained with differences how SMO and SVC with linear kernel are implemented.

This post compare the two, and the answer is SMO should be the fastest one, otherwise your dataset maybe is not suitable for it.

I would test SVC with linear kernel and see if things change.


Supervised Learning: time estimation of bike repair

But my concern is that my shops might be similar, it doesn't mean that they would spend the same amount of time to repair a bike. My second thought was to treat each shop independently, so basically one model per shop.

You need to decide which problem you're trying to solve, e.g.:

average duration of repair for a particular type of bike and repair id
average duration of repair for a particular shop (can be used to rank shops by their performance)
average duration of repair for a particular type of bike in a particular shop (can be used to predict the best shop for a particular bike)
...

After having clearly defined the problem, you need to format the data so that every instance corresponds to the item for which you want to predict something. For instance if the goal is to predict duration by type of bike, then an instance represents a type of bike so the feature should contain information about the bike and the target variable could be the average duration for this bike. In this case you shouldn't have several instances for the same type of bike.


SKLEARN Metrics report "Number of classes, 28, does not match size of target_names, 35. Try specifying the labels parameter"

I tried to fix it like this, not sure if it is correct.
from sklearn.utils.multiclass import unique_labels
target_names_filtered = [twenty_test.target_names[i] for i in unique_labels(twenty_test.target[:n_samples], predicted)]

then i used target_names_filtered as the target_names param.


Identify the parameter causing the anomaly in a multivariate dataset

Usually it will not be a single feature value that is responsible for the decision of a ML model. Neural Networks, Random Forests, SVM, etc. intend to transform the input in a more beneficial feature space, where making decisions is more easier for them.

As a draw back, this makes interpratability for humans more intricate. Explainability of ML methods is a whole research field.

You could check out some explainability approaches. For autoencoders for example you could use Layerwise Relevancy Propagation (LRP). https://arxiv.org/pdf/1708.08296.pdf


Compare cross validation values of Bernoulli NB and Multinomial NB

The difference is explained in the documentation:

Like MultinomialNB, this classifier is suitable for discrete data. The difference is that while MultinomialNB works with occurrence counts, BernoulliNB is designed for binary/boolean features.

So it's not a matter of number of features, it's about how the values of the features are interpreted: Multinomial can deal with multiple discrete values, whereas Bernouilli deals only with binary variables.
The doc also mentions this option:

binarize: float or None, default=0.0
Threshold for binarizing (mapping to booleans) of sample features. If None, input is presumed to already consist of binary vectors.

Since you didn't provide a value for this option in your code, the default 0.0 applies. This means that when using Bernouilli all the variables are converted to
binary variables: anything lower or equal to 0 is 0, anything higher is 1. This explains why Bernouilli works with your data, albeit not as well as Multinomial: probably for many features in your data the fact that the value is zero or not is a good indication for the label.


Definition of the Q* function in reinforcement learning

The reward of action $a$  is defined as a stationary probability distribution with mean $q_*(a)$. This is independent of time $t$.
However the estimate of $q_*(a)$ at time $t$, denoted by $Q_t(a)$, is dependent on time $t$

Or are we to understand q∗ as taking the expected reward across all t?

The expectation is not over time, but over a probability distribution with mean $q_*(a)$.
For eg., in the 10-armed bandit problem, the reward for each of the 10 actions comes from a Normal distribution with mean $q_*(a), a= 1,...,10$ and variance 1.


build a classification model under constraint

As Peter said, this is exactly the sort of problem logistic regression can work on directly. But, there are a few more things to say.

The binning that happens on your raw variables suggests there's some strong nonlinearity. You could do some univariate analysis to find suitable transformations, but in particular you could try binning, possibly with weight-of-evidence scoring and/or tree-based recursive partitioning.

If you want to insist on preserving the binning structure and the $x'$ values, then it's just straightforward logistic regression with these transformed predictors.

If you insist on preserving the binning structure but not their $x'$ values, you could again go with WOE or, one-hot encode for these bins and learn the $w_i x_i'$ products as the coefficients of the encoded variables in a logistic regression.

Finally, of course, you can replace almost any time I mentioned logistic regression with your favorite classifier.


Google prediction API: What training/prediction methods Google Prediction API employs?

If you take a look over the specifications of PMML which you can find here you can see on the left menu what options you have (like ModelTree, NaiveBayes, Neural Nets and so on).


What type of consideration can be made using clustering?

This is basic architecture of spam filter :

Statistically,spam bear lower entropy ( i.e., higher similarities) than legitimate emails.
We could use bisect k-means clustering after doing topic modelling. In k-means we had to specify k which lead to drastic change in results and it also leads to empty clusters.
I would recommend going through this paper as it highlight this approach.


Differences between a Statistician and a Data Analyst in industry

A major difference is the job market: you'll find a lot of job ads for data analysts/scientists, very few for theoretical statisticians. In most sectors (there are some exceptions, in banking for example), companies are interested in applying existing models to their data because this is what can increase their profits.

Devising new theoretical models is more on the research side of innovation. Most "pure research" job opportunities are in academia, although some big companies have research departments as well.


Looking for a strong Phd Topic in Predictive Analytics in the context of Big Data

As a fellow CS Ph.D. defending my dissertation in a Big Data-esque topic this year (I started in 2012), the best piece of material I can give you is in a link.
This is an article written by two Ph.D.s from MIT who have talked about Big Data and MOOCs. Probably, you will find this a good starting point. BTW, along this note, if you really want to come up with a valid topic (that a committee and your adviser will let you propose, research and defend) you need to read LOTS and LOTS of papers. The majority of Ph.D. students make the fatal error of thinking that some 'idea' they have is new, when it's not and has already been done. You'll have to do something truly original to earn your Ph.D. Rather than actually focus on forming an idea right now, you should do a good literature survey and the ideas will 'suggest themselves'. Good luck! It's an exciting time for you.


In SVM Algorithm, why vector w is orthogonal to the separating hyperplane?

Geometrically, the vector w is directed orthogonal to the line defined by $w^{T} x = b$. This can be understood as follows:

First take $b = 0$. Now it is clear that all vectors, $x$, with vanishing inner product with $w$ satisfy this equation, i.e. all vectors orthogonal to w satisfy this equation.

Now translate the hyperplane away from the origin over a vector a. The equation for the plane now becomes: $(x − a)^{T} w = 0$, i.e. we find that for the offset $b = a^{T} w$, which is the projection of the vector $a$ onto the vector $w$.

Without loss of generality we may thus choose a perpendicular to the plane, in which case the length $\vert\vert a \vert\vert = \vert b \vert /\vert\vert w\vert\vert$ which represents the shortest, orthogonal distance between the origin and the hyperplane.

Hence the vector $w$ is said to be orthogonal to the separating hyperplane.


Keras, re use convoluted layers training for various flattened layers training

You can simply train your convolutional layers, save the model and load weights from specific layers using the get_weights and set_weights methods (see also this previous answer). After loading the weights for you convolutional layers you can freeze those layers using the trainable attribute to make sure the weights are not changed during training.


K means visualisation after reducing dimensionality with PCA

K-means minimizes sum of squared errors, and PCA finds the projection with maximum sum of squares. So they are a quite natural fit.

Just run k-means and project it to 2d for visualization with PCA. They you are largely seeing the data the same way as k-means (if you only use the rotation, not the scaling!)

What I'd be more concerned about is the input data, as it is not particularly well suited for neither k-means nor PCA. So I wouldn't be surprised if the results are barely interpretable. Both methods make most sense if the input variables are continuous and of the same scale.


Word Frequency Analysis of Document Sets

There are many algorithm to do classification: Naïve Bayes, logistic regression, SVM, decision tree..etc. My suggestion is to try Naïve Bayes first by calculating below probabilities which a new document belongs to $class_{good}$ or $class_{bad}$. (https://web.stanford.edu/class/cs124/lec/naivebayes.pdf)

$$
P(Class_{good} \vert document_{new}) = \frac{P(document_{new} \vert Class_{good}) \cdot P(Class_{good}) }{P(document_{new})}
$$

$$
P(Class_{bad} \vert document_{new}) = \frac{P(document_{new} \vert Class_{bad}) \cdot P(Class_{bad}) }{P(document_{new})}
$$

And generally, when we are doing text mining questions, we will do several preprocess on one document:

Tokenization(1-gram/bigram/...etc)
remove stop words ('a', 'the' 'at', ... etc)
Stemming: transforms a word into its root form. (studied => study)

My suggestion is to do above preprocesses and try more features not just the words in one document, if there are some metadataes.


Machine Learning For Keeping Stable Humidity

This sounds like something you might want to do with a PID or Kalman Filter. It'd probably work better than a complex machine learning model. The humidity is probably cyclical in nature and if you can just tighten the loop a bit, you should be able to achieve your result with a simple PID.

Something like this might be what you're looking for:-

PID example

Application


There are many ways to perform chunking/phrase structure parsing. The most common methods are:

Rule-based
Statistical models
Deep Learning

Fixed rules are simple and fast. But they make many mistakes given the complexity of human language. The other methods are typically more useful because they generate probabilistic rules.

Context-free grammar (CFG) is a set of rules for the production of formal language and can be used for parsing.


What does all zero mean decrease in accuracy (MDA/permutation importance) signify?

I am assuming MDA means Mean Decrease Accuracy. Generally speaking, a good performance on a validation/testing dataset means that your model generalizes well.
On the other hand MDAs of all exactly zero means either the model always needs just one variable to get its best performance no matter what variable that is (very unlikely), or that you plain simply calculated the MDAs wrong. This is impossible to tell without code/examples/etc.


How to run only part of the code in PyCharm, having some data in operating memory

Pycharm allows you to highlight a section of run and execute that specific section in the console by right-clicking and choosing 'Execute selection in Python Console'. This section of code has now been executed and any variables you've defined are now stored in memory.
You can continue executing pieces of code in this way and reference objects previously created up until you refresh the console session.


What can be done so that 'teacher' and 'teaches' are treated similar?

Use an aggressive stemmer. The Lancaster Stemmer is one the most aggressive and popular stemmers around.
Here is the Python code:
from nltk.stem.lancaster import LancasterStemmer

lancaster_stemmer = LancasterStemmer()
assert 'teach' == lancaster_stemmer.stem('teacher') == lancaster_stemmer.stem('teaches')


DCGAN is more about network architecture alterations, while WGAN is an change to the loss function. There's nothing stopping you from using the DCGAN architecture with the WGAN objective function: all this means is minimizing an approximate Wasserstein loss, rather than a Jensen-Shannon divergence, using a particular network architecture. The WGAN (or its followups, e.g. WGAN-GP) objective is agnostic to the architecture. The only thing (that I can think of) that you need to watch for is the use of batch norm: DCGAN recommends putting it everywhere, but (at least for WGAN-GP) it messes with statistics of the critic regularization.


In general, it's never a bad idea to use logistic regression as a first stab at a classification problem. If it doesn't work great, it at least gives you a baseline. Random forest would probably work well here as well. Generally, random forests aren't considered very interpretable, but they're actually pretty interpretable if you're interested in understanding the decision process for a single prediction, which it sounds like would be sufficient for your needs here (i.e., the model can tell you a student is likely to perform a certain way because of their behavior on specific tests). A random forest will probably also handle missing data better, which might be useful to you since I'm guessing not all students took the same tests.

Another option you can try here which handles null data extremely well and would also be simple for inference, is naive bayes.


How to deal with new features values in my classification model?

Depending on the implementation, that variable could have been one-hot-encoded and in that way, the "widow" value could be assigned to "zero" in all the encoded variables (maybe equal to unknown) and the path designed to that 0 will be followed.


How to Minimize mean square error using Python

For this kind of problem, I would definitely start with scipy.otpimize methods.

I reproduce here an example on how to use it in your context:

import numpy as np
from scipy.optimize import minimize

ALPHA_TRUE = 0.5 # used only to generate some test data

def model(params, X):
    # here you need to implement your real model
    # for Predicted_Installation
    alpha = params[0]
    y_pred = np.exp(-alpha * X)
    return y_pred

def sum_of_squares(params, X, Y):
    y_pred = model(params, X)
    obj = np.sqrt(((y_pred - Y) ** 2).sum())
    return obj

# generate some test data
X = np.random.random(10) # this is "month" if I understood your problem correctly
Y = model([ALPHA_TRUE], X) # Original_Installation

# perform fit to find optimal parameters
# initial value for alpha (guess)
alpha_0 = 0.1

res = minimize(sum_of_squares, [alpha_0, ], args=(X, Y), tol=1e-3, method="Powell")
print(res)

The result looks like this:

   direc: array([[-1.12550246e-12]])
     fun: 0.0
 message: 'Optimization terminated successfully.'
    nfev: 225
     nit: 9
  status: 0
 success: True
       x: array(0.5)

You have to take a deep look at the documentation to find the best fitting method depending on whether alpha is bounded or not or whether you have constraints on your parameters.

I have also played around recently with the same kind of stuff using tensorflow gradient descent optimization (example: https://stellasia.github.io/blog/2020-02-29-custom-model-fitting-using-tensorflow/)


How to appropriately set weights for weighted KNN

Technically speaking the inverse of the distance should not pose any problem. Considering the case when you have to classify an observation (instance) whih is identical with one observation from the 'learned' data set, than the inverse distance is not defined by a direct formula since you have to divide by zero. But considering the math behind, inverse of the zero distance is positive infinity. Which means that, the weight for the identical point would dominate all other weights of the data instances. Thus the classification for that specific point could be taken as the class of the identical observation from the data set.

On the other hand the inverse of the euclidean distance is only one type of distance. You can always apply a kernel function on that to weight non-uniformly the contributions. For example you can choose a normal density with mean equals zero and variance equals 1. Than, your weight would be the value of density function for the euclidean distance.


Difference between Memory and Disk space

What you need to remember is :

Disk space : slow to read/write but cheap, any computer has > 100Go of hard disk
Memory : fast to read/write but expensive, usually a few Go

When you run an algorithm you usually use only RAM. In deep learning your dataset is often too big so you store it in the disk space and load pieces of it into RAM when needed for training.
There are tons of resources on the internet if you want to better understand how they differ (for example https://www.differencebetween.com/difference-between-hard-disk-and-vs-ram/)


Sentiment Analysis: Train Separate Models or Use One for All

I would suggest that you train a single model. The features correlated with positive or negative sentiment in a review are likely the same as those in a survey response. That is, “good” means good in both, “bad” means bad in both.

In general you should combine models and datasets when you think that the features in the different situations have the same meanings/interpretations. If they have contradictory or different meanings/interpretations, then you should consider training separate models.


'DecisionTreeClassifier' object has no attribute 'importances_'

Do Checkout this Link

  To Visualise The Tree Itself

from sklearn.tree import export_graphviz
import graphviz

export_graphviz(tree, out_file="mytree.dot")
with open("mytree.dot") as f:
    dot_graph = f.read()
graphviz.Source(dot_graph)

OR

from sklearn.tree import convert_to_graphviz
convert_to_graphviz(tree)

OR

from sklearn.tree import convert_to_graphviz
import graphviz

graphviz.Source(export_graphviz(tree))

The Visualisation You Can Get Will be Whole Tree Itself..

  To Display Feature Importances

from sklearn.tree import DecisionTreeClassifier
classifier = DecisionTreeClassifier()
classifier.fit(features, labels)
for name, importance in zip(features.columns, classifier.feature_importances_):
    print(name, importance)
    ## Now You Can Do Whatever You Want(plot them using a Barplot etc)


Is it ok to fill a pandas dataframe with NaN values?

It depends on what problems you are afraid of:

regarding "technical" issues, it should be ok having NaNs in your dataframe and, afterwards, applying the pd.isna(column_name) per attribute to get a boolean mask to find those unknown values per column, more info here
in case your problem is not knowing the actual values, one option is imputation, but not all types of variables are easily imputable (neither it makes sense to impute all of them)


CNN with Multi channel input or CNN with Multi instance learning?

One of the other approach you can try is multi input & single output approach in which passing these 2 images to 2 different layer from there different layer then concatenation of the layers to perform classification task.

It can be done using keras Functional API

input1 = keras.layers.Input(shape=(1,))
input2 = keras.layers.Input(shape=(1,))

cnn1   = keras.layers.Conv2D(32, 3, activation=keras.activation.relu)(input1)
cnn2   = keras.layers.Conv2D(32, 3, activation=keras.activation.relu)(input2)

merged = keras.layers.Concatenate(axis=1)([input1, input2])

dense1 = keras.layers.Dense(2, activation=keras.activations.relu)(merged)
output = keras.layers.Dense(1, activation=keras.activations.sigmoid)(dense1)
model10 = keras.models.Model(inputs=[input1, input2], output=output)

When passing data to model create separate arrays of input those 2 images set.

model0.fit([array_1, array_2],output, batch_size=16, epochs=100)


In designing an MLP architecture, we can restrict ourselves to 4-8 layers with 8-128 (power of 2) neurons per layer. In addition, we can assume recommended ReLU activations with He normal weight initialization and Adam or SGD with Nesterov momentum optimizers (see ipython notebook for comparison). Your loss function will depend on the problem: cross-entropy for classification and MSE for regression.

In order to avoid local optima in parameter search, it's better to use random search or bayesian optimization. For additional answers, see this post.


semi-structured text parsing using machine learning

Without a sample of your data, it's unclear what's the structure of your data and what tool is suitable to process it.

Here are some blind recommendations based on my experience:

If you just need some flexibilty parsing the text record, such as variable repeat number of certain field, or conditional parsing of fields, then you should check out this python library: http://construct.readthedocs.org/en/latest/
it allows you to first define a hirachcal structure of your data, and then apply this structure to parse information from a text file. It's especial useful when parsing binary files.
If you're looking for an algorithm that can actually "understand" your text data and "infer" the structure in a smart way. Then you might want to try graph based approach:
http://kavita-ganesan.com/opinosis


Why is the exploding/vanishing gradient problem not solved by line search?

Line search may help with exploding/vanishing gradient problem. However, line search does not work well with mini-batches and most training uses mini-batches. One of the main advantages of line search is that it tells you whether you have stepped too far. That advantage disappears when you subsample the data.


The ImageNet data is online: http://www.image-net.org/.

The data descriptions says:

  14,197,122 images, 21841 synsets indexed

However, you don't need to train a model on all of the data from scratch, since you can use pre-trained models, e.g. in TF/Keras. See the docs: https://keras.io/applications/.

I don't think the original Alexnet is available, but a lot of other models are ready to load from Keras. You may also find a Alexnet version stored as .hdf5 or so online (which you could load with Keras/TF).

I don't know what you actually want to do, but here is a simple example of how to use a pretrained model (VGG16 in this case) in Keras.


Genetic neural network to satisfy variable number of inputs and outputs

Is there a reason I shouldn't be doing it this way?

Depends on the nature of the data. There might be an element of "Scissor/Paper/Stone" in the competition you are scoring, where different strengths and weaknesses of competitors can combine such that Player A beats Player B, Player B beats Player C, but Player C beats Player A. In that case, you cannot produce reliable ranking between players by considering each entrant separately, and a network that rates each player individually will perform less well than one that can compare players.

If players are in more of a race-to-finish or score max points separately in a competition, then separately rating each player in each competition should be more reliable. And it is definitely easier to build and train a neural network to predict that.

An alternative, if your events are more like tournaments where entrants oppose each other (even if within some larger free-for-all), is to predict relative rank between pairs of players. This may not be consistent, so you will need to use a pairwise ranking method to resolve that for the final winner. If it really is a knockout tournament, and you know how the initial draw and team combinations will work, then you could maybe make a prediction by simulating the possible games.

There is nothing preventing you from combining these approaches in some way either.

Whichever method you use, you will want to think a little about what your metric is going to be to select the best approach. If you only care about predicting the winner, then accuracy of that prediction might be enough. If you care about where the eventual winner is placed, perhaps mean reciprocal rank would be better (score 1 for correct prediction, 1/2 for predicting winner as ranked second, 1/3 if third etc).


Checking regression coefficients' stability?

I had the same problem, and my data is all numeric and free from NAs.  The fix is really trivial:  instead of scope = ~ var1 + var2 + ..., you should use
scope = ~ . + var1 + var2 + ....  .  Seems like this ought to be covered in the add1 examples on its help page.


How to interpolate and check correlation of two time series with differing cardinality

I understand that your time series are unevenly spaced. In this case, why not simply use a library like traces and transform them to evenly spaced time series.


The new version (0.3.0) of Keras no longer has tied weights in AutoEncoder, and it still shows different convergence. This is because weights are initialized differently.

In the non-AE example, Dense(32,16) weights are initialized first, followed by Dense(16,32). In the AE example, Dense(32,16) weights are initialized first, followed by Dense(16,32), and then when you create the AutoEncoder instance, Dense(32,16) weights are initialized again (self.encoder.set_previous(node) will call build() to initialize weights).

Now the following two NNs converge exactly the same:

autoencoder = Sequential()
encoder = containers.Sequential([Dense(32,16,activation='tanh')])
decoder = containers.Sequential([Dense(16,32)])
autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder,
                        output_reconstruction=True))
rms = RMSprop()
autoencoder.compile(loss='mean_squared_error', optimizer=rms)
np.random.seed(0)
autoencoder.fit(trainData,trainData, nb_epoch=20, batch_size=64,
            validation_data=(testData, testData), show_accuracy=False)


Convolutional neural networks for non image dataset

Short answer:

Yes, absolutely

Longer answer:

As a rule-of-thumb, you can always use any technique on any problem, as long as you define it properly. CNNs are meant to work on signal data. 1D signals can be sound waves, financial records, etc. 2D signals are usually images. 3D signals can be videos (with time being the 3rd dimension). And obviously, you can have as many dimensions as you'd like.

So as long as you can turn your input data (let's say a single training instance) into a signal, then be my guest. However, I fail to see how this dataset can be viewed as a signal.


Create an Algorithm, that translates human language into code

A simple approach would be just syntactic conversion, as done in precompilers/preprocessors.

A second step might be interpretation of some sort of controlled language where you allow more complex phrases, as long as they are not ambiguous.

A not very different, but data-driven, approach is represented by form-filling chat bots (couldn't quickly find a generic link, but plenty of examples in search results) where you aim to identify/locate the necessary bits to build what you expect to be your queries.

A more advanced approach that might give you some interesting ideas is this screenshot-to-HTML builder, based on recurrent neural networks, namely LSTM and GRU.

This could go further with named entity extraction, context interpretation, error-correction, and so on. The list continues infinitely.

The last, simply because, as many do, you have left out much of your problem definition implicit, which reminds me of our little friend Alice in Wonderland:

Alice: Would you tell me, please, which way I ought to go from here?
The Cheshire Cat: That depends a good deal on where you want to get to.
Alice: I don't much care where.
The Cheshire Cat: Then it doesn't much matter which way you go.
Alice: ...So long as I get somewhere.
The Cheshire Cat: Oh, you're sure to do that, if only you walk long enough.


Is there an intuitive explanation why some neural networks have more than one fully connected layers?

There is a nonlinear activation function inbetween these fully connected layers. Thus the resulting function is not simply a linear combination of the nodes in the previous layer.


What is Difference Between np.zeros() and np.empty()

From the documentation:

  empty, unlike zeros, does not set the array values to zero, and may
  therefore be marginally faster. On the other hand, it requires the
  user to manually set all the values in the array, and should be used
  with caution.

np.zeros

Return a new array setting values to zero.

>>> np.zeros((2, 2))
array([[0., 0.],
       [0., 0.]])

np.empty

Return a new uninitialized array.

>>> np.empty((2, 2))
array([[1.35807735e-312, 1.35807731e-312],
       [1.99637364e-310, 8.69169476e-311]])


How to combine two logistic regression models trained on different set of data?

I can think of two options, there are probably more:

Train a binary classification "top model" which predicts whether the instance belongs to  model 1 or model 2. The two final models only need to be trained with their respective data and only predict their respective N or M classes. The performance of the top model is crucial in this option.
Train an open set classification model. It's a bit similar to what you're doing currently by adding a class "other model" to the first model, except that "other" is not a regular class and you don't need to train the model with instances from this class. This is related to one-class classification and requires a specific training method which can handle open-set classification.


Machine learning problem with only original data without test and validation data

Usually, with a predictive model, you generate a train and test sample from the original dataset, e.g. by using sklearn.
import numpy as np
from sklearn.model_selection import train_test_split
X, y = np.arange(10).reshape((5, 2)), range(5)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
Train some model on one set and test on the other by comparing predictions to actual (true) outcomes.
You could also use cross validation, in which case a model is trained on one part of the data (e.g. 4/5 of the data with 5-fold cv) and tested on the remaining 1/5 of the data. This is done for all of the „folds“. See Ch. 5.1. of „Introduction to Statistical Learning“ for more details on that. https://www.statlearning.com/
Simply make sure you obtain a test score based on data not used for training.


LeCun paper on deeplearning (Nature, 2015)

It would be helpful:
https://www.youtube.com/watch?v=d14TUNcbn1k

As far as I understand, in backpropagation, we need to multiply the local gradient of each node by the gradient from the upper level and this is the value that we send back


Class distribution discrepancy training/validation. Loss now uninterpretable?

The validation set is used to estimate how well the model is fitting to your solution space.  For instance, if your training loss continues to decrease but your validation loss remains constant or increases, then you should employ early stopping as to avoid overfitting.  This does not require a weighted loss.

If you wanted to compare the training loss to the validation loss, you cannot use the same weights if the distributions differ.  Instead, you can simply compare the average unweighted loss per set.  The weighted loss is for penalization.


Reading a json-file from an API, getting "AttributeError: 'Response' object has no attribute 'data' "error

This is not a data science question and you should try programming stack-exchange instead. But this is happening because of an API mismatch in Python, specifically the type of the GET method of the requests object. It's likely the code snippet you got is from a version that is not python 3.2 or a requests module that is not up to date. I would try to install a compatible version of the requests module from PyPI or make your python version compatible with the snippet.


I have gotten the answer to my question since I keep searching for solution.
I got the solution here enter link description here

By StefanM

So all I needed and should have done is to simply do
plt.show()
and I was typing fatalprop.head(10).show()
Thank you


What models do Create ML and Turi Create use

TuriCreate has several different models implemented. You can either use them specifically or you can use a create method which will analyze your data and pick one of them (not always the same one).

For instance in binary classification (not image) there is support for random forest, decision tree, boosted trees, logistic regression, SVMs and nearest neighbor.

I don't know exactly how it makes the selection, but I imagine it has a lot to do with the dimensions of your data.


Code Is Self Explanatory...

    from sklearn.cluster import KMeans
    from sklearn.datasets.samples_generator import make_blobs
    np.random.seed(0)
    centers = [[1, 1], [-1, -1]]
    n_clusters = len(centers)
    X, labels_true = make_blobs(n_samples=3000,
                                centers=centers,
                                cluster_std=0.5)

def plot_cluster_data(X, c=[1]*X.shape[0], mu=None):
    fig = plt.figure(figsize=(8, 8))
    ax = fig.add_subplot(1, 1, 1)
    if len(np.unique(c)) == 1:
        ax.plot(X[:,0], X[:,1], 'o')
    else:
        ix = np.where(c==1)
        ax.plot(X[ix,0], X[ix,1], 'o',
                markerfacecolor='red')
        ax.plot(mu[0,0], mu[0,1], 'o',
                markerfacecolor='red',
                markersize=12)
        ix = np.where(c==0)
        ax.plot(X[ix,0], X[ix,1], 'o',
                markerfacecolor='green')
        ax.plot(mu[1,0], mu[1,1], 'o',
                markerfacecolor='green',
                markersize=12)
    if not mu is None:
        ax.plot(mu[0,0], mu[0,1], 'o',
                markerfacecolor='red',
                markersize=12)
        ax.plot(mu[1,0], mu[1,1], 'o',
                markerfacecolor='green',
                markersize=12)
    plt.show()

plot_cluster_data(X)

clst = KMeans(n_clusters=2, random_state=2342)
clst.fit(X)
mu = clst.cluster_centers_
plot_cluster_data(X, mu = mu)


Train a point rather than a bounding box for object detection

Ususally the x,y coordinates of the top-left corner of the bounding box and the width and height of the bounding box is returned by objection detection algorithms (eg: Haar Cascades)
You can always convert a bounding box to a point via the equation:

$BoundingBoxCenter_{x}= \frac{BoundingBoxCorner_{x} +
 BoundingBoxWidth}{2}$
$BoundingBoxCenter_{y}=
 \frac{BoundingBoxCorner_{y} + BoundingBoxHeight}{2}$


Let’s look at the formula used for the calculation of tf-idf scores

You can see a log with the idf term.
The following answer explains this pretty well.
log with idf


How can I do simple machine learning without hard-coding behavior?

Not sure if this fits the scope of this SE, but here's a stab at an answer anyway.

With all AI approaches you have to decide what it is you're modelling and what kind of uncertainty there is. Once you pick a framework that allows modelling of your situation, you then see which elements are "fixed" and which are flexible. For example, the model may allow you to define your own network structure (or even learn it) with certain constraints. You have to decide whether this flexibility is sufficient for your purposes. Then within a particular network structure, you can learn parameters given a specific training dataset.

You rarely hard-code behavior in AI/ML solutions. It's all about modelling the underlying situation and accommodating different situations by tweaking elements of the model.

In your example, perhaps you might have the robot learn how to detect obstacles (by analyzing elements in the environment), or you might have it keep track of where the obstacles were and which way they were moving.


If you're interested in the classic MDS algorithm, it is spelled out very nicely on the Wikipedia page:

  Classical MDS uses the fact that the coordinate matrix can be derived
  by eigenvalue decomposition from $B=XX'$. And the matrix $B$ can be
  computed from proximity matrix $D$ by using double centering.

  Set up the squared proximity matrix $D^{(2)}=[d_{ij}^{2}]$
  Apply double centering: $B=-{\frac {1}{2}}JD^{(2)}J$ using the centering matrix $J=I-{\frac {1}{n}}11'$, where $n$ is the number of
  objects.
  Determine the $m$ largest eigenvalues $\lambda _{1},\lambda _{2},...,\lambda _{m}$ and corresponding eigenvectors $e_{1},e_{2},...,e_{m}$ of $B$ (where $m$ is the number of dimensions
  desired for the output).
  Now, $X=E_{m}\Lambda _{m}^{1/2}$, where $E_{m}$ is the matrix of $m$ eigenvectors and $\Lambda _{m}$ is the diagonal matrix of $m$
  eigenvalues of $B$.

I can't really tell what your $A$ matrix is, but if each entry is a measure of the distance from the $i^{th}$ to $j^{th}$ entry, then that would be your proximity matrix.

Also, $11'$ is an n-by-n matrix of all 1's. This should be fairly self-explanatory, however most of these terms can be Googled if you're unsure. It really shouldn't be any more than about 10 lines of code.


How to train a Machine Learning model for blocked data

I think your approach is more or less valid, but you’re overfitting the model because you don’t have a lot of data. So that’s my main point.

There are many ways in principle to deal with this, but if you have little data, perhaps you simply shouldn’t be expecting great results. If you use the averages, you get better results, but “the problem isn’t solved completely”. A priori I’m not sure you have reason to think you can do better.

Anyway, step 1 should be to simplify the model (fewer and more shallow trees). You should find that your test error deteriorates but your validation error improves, because rather than simply memorizing records, the model is now forced to learn something about the blocks. This information can be reused on records the model hasn’t seen yet.

You can also look at regularization, but it is essentially an automatic way of finding a simpler, but better model. Also, data augmentation. Is it possible to generate more data by varying the existing data a bit? Like the way you can generate more images by changing the images that you have: By cropping, flipping, rotating, etc.

Finally, make sure that your split between train-test-validate sets is random. I guess it depends on your data but it seems you can’t expect the model to learn about a block that doesn’t exist at all in the training set. If that block appears in the validation set only, I guess the model would be random on that data.


Usually predictive power refers to the model, rather than the data. I've occasionally seen some people use it in the way that the author of your book uses it (see this for example).
In the context of your book, yes, predictive power refers to whether input can be mapped to target output $X\rightarrow Y$. We can infer a dataset's "predictive power" by trying to model it (e.g. linear regression). If the model performs poorly, then there are two possibilities as the book says: either the dataset is not predictive (i.e. it does not offer a clean mapping from input to target output) or the methods we are using are unsuitable to model the mapping.

Some examples of both situations:

If you generated random data for $X$ and $Y$, the resulting dataset would (probably) have no predictive power as no model could reasonably generalize the mapping $X\rightarrow Y$.

If you have a nonlinear mapping, then linear regression would not fit it well. For example, if our dataset was such that $y_1$ is mapped to by $||\vec{x}||<\alpha $ and all other inputs map to $y_2$, then our dataset is extremely predictive, but our linear regression model cannot fit it (since the mapping is nonlinear). In this toy example, it's easy to see the predictive power of the dataset, particularly if the input is in 2D/3D since we could just plot it. However, manually observing such trends in highly dimensional space using actual data can be very difficult, hence we use the tools that you are learning to help interpret the data. Also, when there's nonlinearity, it's difficult to statistically evaluate the dataset itself. Variables with linear relationships are simple to correlate (e.g. Pearson's correlation coefficient) but nonlinearities can make correlation difficult. I assume that this is why your book defers to vague terminology as it's probably for pedagogical, rather than pedantic, purposes. After all, it gets the point across without needing to discuss the ongoing research into quantifying nonlinear correlations.


The meaning of random word dropout in NLP

It is not uncommon that we can make sense of a sentence without reading it completely. Or when you are having a quick look at a document, you tend to oversee some words and still understand the main point. This is the intuition behind the word dropout.

Generally this is done by randomly dropping each word in a sequence following for example a Bernoulli distribution:

  $X \leftarrow X \odot \vec{e}, \vec{e} ∼ B(n, p)$

where X is the index of the word token, n
is the lenth of the sequence, and
 $\vec{e}$ is a vector with each word dropout state.

This is usually done after calculating the word embeddings, and the words selected to be left out are normally changed to the <UNK> equivalent embedding.

By doing this, we allow out model to learn more flexible ways of writing/convey meaning.


How is the GridsearchCV Score calculated?

The score is based on the scorer defined in the scoring argument. Meaning, the scorer can be any of the default metrics, such as precision, accuracy or F1-score
(e.g., this); or a custom scorer.
For a scorer (by convention), higher value is better.
The value is not necessarily a percentage, but is often normalized between 0 and 1.


The idea is quite simple: If you find features that are correlated with your target variable but uncorrelated with each other, it will give you information with which you can predict your target variable.

The threshold of 0.5 is chosen by a person and should be changed depending on your data. For example, if you find 10 features that are uncorrelated with each other but all have a 0.3 absolute correlation coefficient with the S&P 500 than you found the stock market equivalent of the Holy Grail.

Also, as you already mentioned, the correlation coefficient only captures linear correlations. This could cause you to exclude strongly related features just because they have a more complex relationship to your target variable.

Therefore, I would strongly advise against this method. Instead, I recommend plotting the data in pairplots.


Here's a great example for Multivariate Regression using LSTM:

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/

Most Examples you will find will be for time-series prediction using LSTM, like the one you listed for Stock Prices.