Are there readily available models that can handle conditional correlation?

It is totally ok to drop null values here (in your case all the null values of LotFrontage), because this data isn't real. This is a standard part of data preprocessing called data cleaning. If the reason for the null values is known you could do additional steps like imputing or filling, but without this information just dropping them would be fine.
I would be careful changing null to zero however, as this is now creating data (i.e. apartments with zero size front lots).
Finally, you can see from your scatter plot that the relationship is not completely linear, so the linear model may have a large degree of error and/or low R squared values. So it may be more suitable to pick another algorithm for this dataset (e.g. 2D gaussian model).


Can all known ML algorithms be written as a sequence of matrix operations?

Linear algebra is very important for ML, but ML is not "just" linear algebra.

Any sequence of linear matrix-vector operations is linear.
Therefore any non-linear operation cannot be computed using linear algebra.

A few examples of non-linear operations that are used in ML:

The if statement: e.g. (x < 0)
The softmax activation
The rectified linear unit

  Can advanced features like convolution be expressed as basic matrix
  operations?

This statement is true actually. A single convolution can be expressed as matrix. However, any modern CNN has non-linearities following the convolution.


Catboost multiclassification evaluation metric: Kappa & WKappa

I posted the same question on the Catboost github (issues) page and got an answer.
The link can be found here: https://github.com/catboost/catboost/issues/1447
Answer:
Class weights and weights in WKappa are different.
Calculation of WKappa metric consists of two steps:

Confusion matrix calculation - here object and class weights are used.
WKappa calculation using confusion matrix - here values from confusion matrix multiplied by squared difference between classes. (i-j)^2.
In other words, weight of class 0 has the same effect when object of class 0 was classified as class 1 and when classified as class 5. But WKappa weight for misclassification 0->5 will be 25 times more than for misclassification 0->1.
So you can use both class weights and WKappa if your classes are ordinal.


Principal Component Analysis, Eigenvectors lying in the span of the observed data points?

The statement says, that (2) and (4) are equal. That means (2)$\Rightarrow$(4) and (4)$\Rightarrow$(2). The first implication is trivial, as you correctly pointed out.
$$\lambda v=Cv$$
implies
$$\lambda x^Tv=x^TCv$$
for all $v$, not just those from $D$. But the second implication is a bit more tricky, but that is what the proof is about. It tells you, that if you want to check, whether a vector is an eigenvector of $C$, you don't have to check if (2) is satisfied. It tells you, that when (4) is satisfied, (2) is satisfies as well.

Imagine you have 2 points in 3D space. Those 2 points are
$$x_1=(-1,-1,0)$$
$$x_2=(1,1,0)$$
(please excuse me for not making this "dataset" centered) Note, that both points lie in the $xy$ plane. Now the correlation matrix is
$$C=\begin{bmatrix}
       1/2 & -1/2 & 0           \\[0.3em]
       -1/2 & 1/2           & 0 \\[0.3em]
       0           & 0 & 0
     \end{bmatrix}$$
Now you want to know, whether $v=[1\ -1\ 0]^T$ is an eigenvector with eigenvalue 1. The statement tells you, that you can either just check, if (2) is satisfied (3 equations) or if
$$\frac{1}{2}x_1^Tv=x_1^TCv=0$$
and
$$\frac{1}{2}x_2^Tv=x_2^TCv=0$$
which are only two equations.


How to estimate the variance of regressors in scikit-learn?

I believe it is the probabilistic nature of a model that allows you to get the variance of predictions, or more generally defined as the uncertainty of predictions, like the Gaussian process you mentioned. This is not simply avaialble in standard regressors.
I think you should be looking at Probabilistic regressors like BayesianRidge if you would like to estimate the uncertainty of your model. An implementation  is also avaialble in scikit-learn, also this nice python package based on PyMC3 or directly via PyMC3 itself for instance. In the latter there are examples like for Bayesian regression in Jupyter Notebook with a good explanation.
In principle, Bayesian Models do not return a a single estimate for the model parameters, but a distribution that make it possible to make inferences about new observations as well as to examine our uncertainty in the model. You may find this post useful.
Note: Adding a normal prior on the weights as it is done in Bayesian regression, one turn the Least-Squares problem to regularized L2 regression under the hood as well (see the full math. derivation here).
Updated Answer: I totally forgot the classical yet simple and powerful Bootstrap Sampling method to calculate confidence intervals for machine learning algorithms. A textbook definition says:

Bootstrapping is a nonparametric approach to statistical inference
that substitutes computation for more traditional distributional
assumptions and asymptotic results. A number of advantages:

The bootstrap is quite general, although there are some cases in which it fails.
Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate
inferences when the data are not well behaved or when the sample size
is small.
It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.
It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).

Reference: Fox, John. Applied regression analysis and generalized linear
models. Sage Publications, 2015.

Please note you do not need a model with probabilistic nature. See this post, or this answer or this one.


Why does the parent connection matter during a NEAT crossover when both connections are identical?

The genes shown in the diagrams have been simplified so that they only show topology and innovation index number. In addition, the full genome contains weights for each connection and bias for each node.

The innovation numbers are not tracking changes with connection weights or biases, just the rough time of inserting a specific connection between two nodes. So each parent can have matching connections/innovations, but with different weights associated. The full genome is still subject to cross-over and mutation, the excerpt you quote helps explain how the genome is aligned between parents with different architectures, so that GA crossover and mutation can occur in each connection and node.

Which parent is used to create each offspring gene matters because of the additional connection weight data associated with each parent's gene.


'RandomizedSearchCV' object has no attribute 'best_estimator_'

You have to fit the RandomizedSearchCV first in order to access this attribute.

random_search.fit(X_train, y_train)
print(random_search.best_estimator_)


Why is Spark's LinearRegressionWithSGD very slow locally?

Spark is designed to be distributed across a cluster and use stochastic gradient descent (SGD) to optimize linear regression.
There is overhead for cluster infrastructure (even when the "cluster" is a single local node). Also, SGD is an iterative method that uses many batches to find a solution.
Given that your problem is 70k rows, it would be better to use a single node framework (e.g., scikit-learn) and ordinary least squares (OLS), a closed form solution, to optimize linear regression. Those two changes will greatly speed up training.


GridSearchCV does allow param_grid to be a list of grid-dicts, which sometimes is sufficient.  In this case, separate grids are generated and their union is searched.
There isn't quite a convenient implementation by which you provide your own list of hyperparameter points.  But looking at the source code for GridSearchCV, you'll notice that it's amazingly sparse.  Most everything is handled by the inherited class BaseSearchCV.  The grid-specific stuff is also factored out:
    def _run_search(self, evaluate_candidates):
        """Search all candidates in param_grid"""
        evaluate_candidates(ParameterGrid(self.param_grid))

The ParameterGrid class is what's responsible for turning the (possibly list of) dict(s) into a grid(s).
So you could probably easily write a custom class, inheriting from BaseSearchCV, where you pass the set of hyperparameter points you want and define _run_search to just evaluate each element from your set.  In fact, already evaluate_candidates is defined assuming that the hyperparameter points live in an iterable candidate_params; the ParameterGrid does all the hard work there, but in your case you can just feed your list directly.


What is syntax V and S standing for nominal subject?

This is explained in the Methodology section of the paper:

  We calculate for each head how often it assigns its maximum attention weight (excluding EOS) to a token with which it is in one of the aforementioned dependency relations. We count each relation separately and allow the relation to hold in either direction between the two tokens.

The nsubj dependency label is used in arcs between the verb ($v$) and a subject ($s$). In the paper, they define this "dependency score" taking the maximum attention weight and checking if it points to the expected word according to the dependency parse arc, in either of the directions. This is why they count it from verb to subject and from subject to verb.

If you want to know more about the conventions usually followed in dependency parsing, including these nsubj, amod, etc, you can take a look at the Stanford typed dependencies manual.


XGBClassifier error! ValueError: feature_names mismatch:

The problem occurs due to DMatrix..num_col() only returning the amount of non-zero columns in a sparse matrix. Hence, if both train & test data have the same amount of non-zero columns, everything works fine. Otherwise, you end up with different feature names lists. There're currently three solutions to work around this problem:

realign the columns names of the train dataframe and test dataframe using:

test_df = test_df[train_df.columns]

save the model first and then load the model
change the test data into array before feeding into the model, ie: use,

test_df.values instead of test_df


"Tensor" is a very general term, meaning N-dimensional array. A tensor can be 0-dimensional (a scalar), 1-dimensional (a vector), 2-dimensional (a matrix), and so on... Tensor is a very encompassing term, meaning all of the things listed above. Sometimes, people use "tensor" to refer to high dimensional objects, as explained here:

  While, technically, all of the above constructs are valid tensors, colloquially when we speak of tensors we are generally speaking of the generalization of the concept of a matrix to N ≥ 3 dimensions. We would, then, normally refer only to tensors of 3 dimensions or more as tensors, in order to avoid confusion [...]

But please keep in mind that this is just a practice, and N-dimensional array is technically a tensor.

Coming to your question, if a tensor is more like:

[1, 1]

or

[[1,1],
 [1,1]]

First, they look like Python list and list of lists, respectively. Python lists are not mathematical objects, strictly speaking. Provided you imported numpy, it would be more correct to describe them as:

numpy.array([ 1, 1 ])

and

numpy.array([[ 1, 1 ],
             [ 1, 1 ]])

Now these objects have the mathematical properties of vectors-matrices-tensors. I think they could both be thought as tensors of different kinds and ranks. The first is of rank 1, the other of rank 2.

Other useful resources on tensors are this question, and this explanation.


Why does Caret featurePlot returns returns NULL?

I had this problem due to my y variable not being a factor which is required. call:

typeof(df$ud)

make sure it is not interpreting the binary as character or numeric but indeed ad a factor.


Is Neural Style Transfer considered supervised or unsupervised learning?

Neural style transfer is not really machine learning, but an interesting side effect/output of machine learning on image tasks. When performing neural style transfer using a pre-trained model, then a significant amount of supervised machine learning has already occurred to enable it.

The style transfer algorithm is still an example of gradient-based cost function optimisation, which it shares with many supervised and unsupervised learning algorithms.

The output of style transfer is partially a probe into what a network has learned about different levels of structure in the problem domain it has been trained on. However, its main use has been to generate images with altered/mixed aesthetics for art and entertainment.


Which metric to use for evaluating a rating system

Performance Measures for Multi-Class Problems
For classification problems, classifier performance is typically defined according to the confusion matrix associated with the classifier. Based on the entries of the matrix, it is possible to compute sensitivity (recall), specificity, and precision. For a single cutoff, these quantities lead to balanced accuracy (sensitivity and specificity) or to the F1-score (recall and precision). To evaluate a scoring classifier at multiple cutoffs, these quantities can be used to determine the area under the ROC curve (AUC) or the area under the precision-recall curve (AUCPR).
All of these performance measures are easily obtainable for binary classification problems. Which measure is appropriate depends on the type of classifier.
F-Measure/F1-Score for Imbalanced Classification
Classification accuracy is widely used because it is one single measure used to summarize model performance. F-Score provides a way to combine both precision and recall into a single measure that captures both properties.
Alone, neither precision nor recall tells the whole story. We can have excellent precision with terrible recall, or alternately, terrible precision with excellent recall. F-measure provides a way to express both concerns with a single score.
Once precision and recall have been calculated for a binary or multiclass classification problem, the two scores can be combined into the calculation of the F-Measure. The traditional F measure is calculated as follows:
$\begin{equation}
\nonumber
F-Measure=\frac{2 * Precision * Recall}{Precision + Recall}
\end{equation}$
For a multi-class classification problem, we don’t calculate an overall F-1 score. Instead, we calculate the F-1 score per class in a one-vs-rest manner.
Here are simple Python functions that do exactly that:
def precision(y_true, y_pred):
    i = set(y_true).intersection(y_pred)
    len1 = len(y_pred)
    if len1 == 0:
        return 0
    else:
        return len(i) / len1

def recall(y_true, y_pred):
    i = set(y_true).intersection(y_pred)
    return len(i) / len(y_true)

def f1(y_true, y_pred):
    p = precision(y_true, y_pred)
    r = recall(y_true, y_pred)
    if p + r == 0:
        return 0
    else:
        return 2 * (p * r) / (p + r)

For imbalanced classes, I would suggest to go with Weighted F1-Score or Average AUC/Weighted AUC or Micro and macro averages of the F1-score
Weighted F1-Score was calculated by F1 score for each class independently but when it adds them together uses a weight that depends on the number of true labels of each class:
$F1_{class1}*W_1+F1_{class2}*W_2+\cdot\cdot\cdot+F1_{classN}*W_N$
therefore favoring the majority class.
Micro and macro averages of the F1-score
Micro and macro averages represent two ways of interpreting confusion matrices in multi-class settings. Here, we need to compute a confusion matrix for every class $g_i \in G = \{1, \ldots, K\}$ such that the $i-th$ confusion matrix considers class $g_i$ as the positive class and all other classes $g_j$ with $j≠i$ as the negative class. Since each confusion matrix pools all observations labeled with a class other than $g_i$ as the negative class, this approach leads to an increase in the number of true negatives, especially if there are many classes.
To exemplify why the increase in true negatives is problematic, imagine there are 10 classes with 10 observations each. Then the confusion matrix for one of the classes may have the following structure:
Prediction/Reference    Class 1     Other Class
Class 1                   8             10
Other Class               2             80

Based on this matrix, the specificity would be $\frac{80}{80 + 10} = 88.9\%$ although class 1 was only correctly predicted in 8 out of 18 instances (precision 44.4%). Thus, since the negative class is predominant, the specificity becomes inflated. Thus, micro- and macro averages are only defined for the F1-score and not for the balanced accuracy, which relies on the true negative rate.
In the following, we will use $TP_i$, $FP_i$, and $FN_i$ to respectively indicate true positives, false positives, and false negatives in the confusion matrix associated with the $i-th$ class. Moreover, let precision be indicated by $P$ and recall by $R$.
The micro average has its name from the fact that it pools the performance over the smallest possible unit (i.e. over all samples):
$\begin{align*}
P_{\rm{micro}} &= \frac{\sum_{i=1}^{|G|} TP_i}{\sum_{i=1}^{|G|} TP_i+FP_i} \\
R_{\rm{micro}} &= \frac{\sum_{i=1}^{|G|} TP_i}{\sum_{i=1}^{|G|} TP_i + FN_i}
\end{align*}$
The micro-averaged precision, $P_{micro}$, and recall, $R_{micro}$, give rise to the micro F1-score:
$F1_{\rm{micro}} = 2 \frac{P_{\rm{micro}}  \cdot R_{\rm{micro}}}{P_{\rm{micro}} + R_{\rm{micro}}}$
If a classifier obtains a large $F1_{micro}$, this indicates that it performs well overall. The micro-average is not sensitive to the predictive performance for individual classes. As a consequence, the micro-average can be particularly misleading when the class distribution is imbalanced.
The macro average has its name from the fact that it averages over larger groups, namely over the performance for individual classes rather than observations:
$\begin{align*}
P_{\rm{macro}} &= \frac{1}{|G|} \sum_{i=1}^{|G|} \frac{TP_i}{TP_i+FP_i} = \frac{\sum_{i=1}^{|G|} P_i}{|G|}\\
R_{\rm{macro}} &= \frac{1}{|G|} \sum_{i=1}^{|G|} \frac{TP_i}{TP_i + FN_i} = \frac{\sum_{i=1}^{|G|} R_i}{|G|}
\end{align*}$
The macro-averaged precision and recall give rise to the macro F1-score:
$F1_{\rm{macro}} = 2 \frac{P_{\rm{macro}}  \cdot R_{\rm{macro}}}{P_{\rm{macro}} + R_{\rm{macro}}}$
If $F1_{macro}$ has a large value, this indicates that a classifier performs well for each individual class. The macro-average is, therefore, more suitable for data with an imbalanced class distribution.
References:

http://text-analytics101.rxnlp.com/2014/10/computing-precision-and-recall-for.html
https://sebastianraschka.com/faq/docs/multiclass-metric.html
https://stats.stackexchange.com/questions/463224/which-performance-metrics-for-highly-imbalanced-multiclass-dataset
What's the difference between Sklearn F1 score 'micro' and 'weighted' for a multi class classification problem?
https://www.datascienceblog.net/post/machine-learning/performance-measures-multi-class-problems/
https://sebastianraschka.com/faq/docs/computing-the-f1-score.html


What is Extreme Learning Machine? Why tuning of weights is not required?

ELM is actually solving a generalized linear problem :

\begin{equation}
H\beta = Y
\end{equation}

where $H$ is the matrix of the hidden layer output, $\beta$ is the weight vector (or matrix) connecting the hidden layer and the output layer and $Y$ symbolizes the target of your training.

The goal is to find $\beta$ using the pseudo-inverse of H.

Solving this problem is equivalent of finding the $\beta$ achieving the minimum of the Least-square problem $||H\beta - T ||$ :
\begin{equation}
\beta = \underset{\beta}{\text{min}} \left \vert\left \vert H\beta - T \right\vert\right\vert
\end{equation}
To answer your question

  How is it possible to assign values to links without actually seeing the Training set?

Since $H$ depends on the activation functions, on the initialized weight matrix linking Input layer and hidden layer and on each hidden node bias, it encompasses all the input information and ensures a learning by tuning $\beta$ coherently.

You can find lot more details in ELM Theory and Application or in this ELM presentation for example.


Model threshold for classification model

p, r, thresholds = precision_recall_curve(y_test, y_scores)

For more clear understanding refer to this article:
https://towardsdatascience.com/fine-tuning-a-classifier-in-scikit-learn-66e048c21e65


Training Dataset required for Classifier

You have at least three options:

Use some of many available datasets (for example: BBC documents); if you need more, simply go to Google Scholar or any similar service and search for classification news politics sports. In the articles you may find many references to available datasets;
Crawl any news service, and use clustering techniques to group articles into clusters (this often separates the articles along their domain, such as politics, sports etc.), and label the articles by their affiliation to a cluster;
Crawl politics-specific, sports-specific news services and use them as labeled datasets.


difference in between CNN and Inception v3

The Inception models are types on Convolutional Neural Networks designed by google mainly for image classification. Each new version (v1, v2, v3, etc.) marks improvements they make upon the previous architecture.

The main difference between the Inception models and regular CNNs are the inception blocks. These involve convolving the same input tensor with multiple filters and concatenating their results. Such a block is depicted in the image below.

On the contrast, regular CNNs performs a single convolution operation on each tensor.

Inception-v3 is Deep Neural Network architecture that uses inception blocks like the one I described above. It's architecture is illustrated in the figure below.

The parts where the layers "branch off" and then are merged together again are the inception blocks described previously.

You can read more about Inception-v3 here.


How to integrate google cloud with dropbox and jupyter notebook using tensorflow

The python package nano is not the text editor you need. Instead, you need the Ubuntu package nano. Try:

$ sudo apt-get install nano


Confusion in applying k-fold Cross Validation to dataset

In 10 fold cross-validation, you split your dataset into 10 sections, 9 of them are for train and one for test set (there is no validation set), for example, if your dataset is 100 samples, inside a loop, in the first fold (first loop iter), the model train on 90 samples and the rest 10 are for testing the model, and loop is continued until all the dataset is used for training and testing.

for more, see here

and in python, you can implement 10 fold cross-validation using sklearn library here

Now, because your dataset is already split into 10 fold, you have two choices:

1- The easiest way is to combine your dataset into one set then using a specific library to do the 10 fold cross validation for you.

2- write code by yourself to loop over your 10 fold data, in the first iter use the first section for testing and the rest 9 for the training, in the second iter, use the second section for testing, and the first and other 8 sections for training, the loop should continue 10 times until all the data is used for training and testing.

this is the idea behind 10 fold cross validation if this not applicable for your dataset, I think 10 fold is not good in your case.


How to get used to matricial/vectorial operations?

tl;dr: refresh your linear algebra and check for built-in functions.
In addition to brevity/elegance, vectorization often dramatically improves performance. Tools like MATLAB and numpy use high-level vectorized wrappers for low-level optimized implementations, which in the end are composed of your traditional loop operations. Where MATLAB and Python are interpreted (nuances exist, but are irrelevant here), the low-level subroutines that they wrap are obviously much faster. Also, tools that they wrap, like BLAS (Basic Linear Algebra Subprograms), are so heavily optimized that it'd be difficult for someone write a low-level subroutine that matches or exceeds the performance achieved with such libraries; many people over many decades worked to optimize them!
Considering problems from a math-oriented perspective helps with vectorization. For example, say you wanted to find the $L_p$ norm of a vector $\vec{x}$. A traditional CS program would look something like:
norm = 0
for i in vec_x:
    norm = norm + pow(i,p)
norm = pow(norm,1/p)

but a mathematician would simply write $||\vec{x}||_p$. First, you should check to see if a built-in function exists to perform your task, as a mathematician might. Then you should look to vectorize using math operators such as matrix multiplication, element-wise operations, etc., which these high-level tools implement. Going back to the $L_p$ example, you could write vectorized code (in MATLAB): (sum(x.^p))^(1/p), but it'd be even better to write norm(x,p). Knowing that such functions exist is a combination of familiarity with the tool and Googling.


SVM - Making sense of distance derivation

I believe I have found the answer. A great thank-you to @Javier TG for pointing me to Andrew Ng's notes CS229 Suppor Vector Machines. I would not write down the full context as materials in the lecture notes and questions references do a much better job than I ever will. I will, however, try to fill in the tiny logic gap in many notes: why $w*x+b=1$ can represent any $w*x+b=\lambda, \lambda \ne 0$.
The key understanding is that the distance $d_i$from a point $x_i$ to the plane $w*x+b=0$ is invariant to scaling of $w$ and $b$. Below is a proof.
From the notes, we derived the distance as $$d_i=\frac{y_i(w*x_i+b)}{\|w\|}$$ This shows that $d_i$ is invariant to scaling of $w, b$. Meaning replacing $w, b$ with $2w, 2b$ gives the same $d_i$. You should see it for yourself. Then it is straightforward that the maximum of $d_i$,
$$
d=\max_{w} d_i, i=1,...,n
$$
Is also invariant to the scaling.
However, let $l_i$ represent the nominator in the distance equation, $l_i = y_i(w*x_i+b)$. You will see $l_i$ scales at the same magnitude as that of $w, b$. Meaning replacing $w, b$ with $2w, 2b$ gives $2l_i$. You should also see it for yourself. Then the maximum of $l_i$, $l$ as
$$
l=\max_{w}l_i, i=1,...,n
$$
scales with $w, b$.
Now revisit the optimization problem derived in the notes, $$
\begin{aligned}
\max_{w,b} \quad & d\\
\textrm{s.t.} \quad & y_i(w*x_i+b)\geq l,  i=1,...,n
\end{aligned}
$$
It is okay that we tweak $w, b$ to find the optimum solution, but we also got $d, l$ in the problem statement which makes the problem more complex. It will be great if we can get rid of $d, l$. We do this by leveraging the aforementioned findings. With $l$ scales with $w,b$, we can require that $l'=l*k=1$. To maintain the inequality constraint, we do the same scaling on the left hand side. $$y_i(kw*x_i+kb)\geq kl=1$$. Since $d$ is invariant to scaling, the objective function is not changed. We are still solving the same optimization problem.
Finally, we replace $w=kw, b=kb$, and $d=\frac{l}{\|w\|}=\frac{1}{\|w\|}$, and get the form of the same optimization problem stated in the notes:
$$
\begin{aligned}
\max_{w,b} \quad & \frac{1}{\|w\|}\\
\textrm{s.t.} \quad & y_i(w*x_i+b)\geq 1,  i=1,...,n
\end{aligned}
$$
Now we know, in our optimization problem, any constraint $y_i(w*x_i+b)\geq l$ is equivalent to $y_i(w*x_i+b)\geq 1$.


How to reconstruct a scikit-learn predictor for Gradient Boosting Regressor?

There are two estimators i.e. The initial predictor and the sub-estimators

init_estimator
The estimator that provides the initial predictions. Set via the init argument or loss.init_estimator.
estimators_
ndarray of DecisionTreeRegressor of shape (n_estimators, 1)
The collection of fitted sub-estimators.

Prediction after the first (i.e. init) estimator is controlled by the learning rate.
You can get the prediction as done in the below code -
trees = model.estimators_

x  = x_test.iloc[10,:].values # A sample X to be predicted
y_pred = model.init_.predict(x.reshape(1, -1)) # prediction from init estimator

for tree in trees:
    pred = tree[0].predict(x.reshape(1, -1)) # prediction from sub-estimator

    y_pred = y_pred + model.learning_rate*pred  # Summing with LR
y_pred


You can just use the previous column names for creating a new dataframe

# Use the previous column names from df and assign it to df2
df2 = pd.DataFrame(df2, columns=df.columns)


keras Sequential CNN for image data reshaping data issues

As far as labels are concerned, you can one-hot-encode by using (assuming ytrain is converted to numpy array) the below code. Image reshaping looks fine but if you are having issues with image reshaping then, you might be giving the first argument i.e., the number of images wrong. So try this

    xtrain = xtrain.reshape(xtrain.shape[0],img_rows,img_cols,16)

    ytrain = keras.utils.to_categorical(ytrain, num_classes)

Make sure you import to_categorical from keras.utils


It sounds like you're planning to store all the input data. That has to be sufficient if your problem can be solved. There is nothing else...

Whether it's "easy", that's a relative term and it depends on many things. It depends on the kind of analysis that you want to do and the amount of data, among other things. If you have less then a few MB of data and you just want to calculate basic statistics like the mean or the sum, then Excel is much easier than S3. But S3 is very scalable, and it can easily be read from systems that allow to train complex models, so if you have a lot of data and that's what you want to do, then it could be a good choice.


I guess while computing total error from errors of individual samples, if sample happens to be of a positive class, its error is multiplied by the scale_pos_weight factor. Therefore setting high scale_pos_weight urges optimizer to treat more important (mostly due to its rareness) class with more respect due its higher contribution to the total error value.


When unsupervised learning is more beneficial in comparison with supervised learning even the labelings are existed?

When unsupervised learning is more beneficial in comparison with supervised learning even the labeling are existed?

I would say that there are two main cases:

The task is semantically more meaningful as an unsupervised task. For instance let's consider a collection of books which have been annotated with topics: if the goal is to classify new books into the same pre-existing categories, then it makes sense to use a supervised setting. However if the goal is to discover new patterns of similarity that might not be intuitively easy to notice for a human annotator, then unsupervised topic modeling makes more sense.
The annotations are available for some subset of data (e.g. for evaluation purposes) but will not be available later in production (to some extent it's also a case of what the goal is, but here for technical reasons).

  If there is no labeling the unsupervised learning is better than supervised learning but in some cases even the labeling targets are available, the supervised learning approach works betters? What about conditions of these cases?

The problem here is what "better" means, i.e. how the task is evaluated. If the task is evaluated against the pre-existing labels, in theory the unsupervised version cannot work better than the supervised one since the supervised one has access to more information. It's possible that a particularly unsuitable supervised method would perform worse than a well chosen unsupervised one, but that's not a fair comparison and it's very unlikely in practice.

In general the two are not comparable because the tasks are fundamentally different: in a supervised setting one wants to find patterns related to some information which is known beforehand (the labels), whereas in an unsupervised setting one wants to discover unknown patterns.

  Can we say that if there is no clear dependency between variables unsupervised learning works better?

I don't think so because:

Some supervised algorithms are very good at optimizing whatever little amount of information is available in the data. Even with "no clear dependency" visible according to standard measures, some algorithms can combine the features optimally to minimize errors.
In the case of strictly no dependency at all between the features and the label, the unsupervised method might perform better in the sense that it might find meaningful patterns, but these won't be related to the labels at all (since there's no dependency). So we go back to the question of what "better" means with respect to the task: if the task was to discover unknown patterns then sure unsupervised is "better", but this has nothing to do with the labels and a supervised method would make no sense here. If the task was about predicting the labels, the unsupervised approach is just as bad as the supervised one (and there's a serious flaw in the design of the task!).


Can I use regression to solve a multiple equation problem

Yes, linear regression is perfect for this.

If you knew that all the quantities were measured with perfect accuracy (no errors), you could solve using linear algebra; the system amounts to solving $Ax=y$, where $A$ is a matrix where each row corresponds to the number of each task for one person, and $x$ is a vector that reflects the time of each task, and $y$ is a vector that records the number of days each person worked.

If you suspect there might be small errors in the measurement of $y$, then using linear regression is appropriate.

You'll need that the number of people exceeds the number of tasks.

If you have a different error model, the optimal procedure might be something other than linear regression.  For instance, if you have errors in both the measurements of $y$ and $x$, PCA (also called orthogonal regression) might be more suitable.  If you have a more complex error model, you might want to ask on Stats.SE for the most appropriate inference procedure, or consider using stochastic gradient descent to minimize an appropriate loss function chosen based on your error model.


Replace values in dataframe with another dataframes values based on condition

The easiest way to do it for your case is something like this:

In [1]: import pandas as pd

In [2]: df1 = pd.DataFrame([0, 1, 2, 3, 1, 2, 3, 1, 2], columns=["A"])

In [3]: df2 = pd.DataFrame([9] * len(df1), columns=["A"])

In [4]: df1
Out[4]:
   A
0  0
1  1
2  2
3  3
4  1
5  2
6  3
7  1
8  2

In [5]: df2
Out[5]:
   A
0  9
1  9
2  9
3  9
4  9
5  9
6  9
7  9
8  9

Wherever the row of column A in df1 is equal to 1, insert 1 into the same row of column A in df2
In [6]: df2.A[df1.A == 1] = 1

In [7]: df2
Out[7]:
   A
0  9
1  1
2  9
3  9
4  1
5  9
6  9
7  1
8  9

You could also use the pd.DataFrame.where method, to replace the part df1.A == 1.
There are also many other ways to get the same thing done, but my example is the most straighforward :)


If you want to use only time of login and place of login, there are several models that would do the work.
Take a look at Python's Sklearn. It offer lots of different options.
An example one is random forest: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

Normally you don't event need to understand the model under the hood to be able to use it, so no need to feel intimidated by the names


You would have to post more info about your data and maybe even some samples, but have you considered using tokenization for the data? It's from the NLP world and would allow you to assign a numeric value for each bit of you "nominal" data.


How to decide if gradients are vanishing?

Vanishing gradients are the updates to the weights (not the weight themselves) over the layers after each training batch.
You should check the update signal between the top-most layer and lowest layer after a single batch.


Should columns with close to zero variance be removed before or after one hot encoding?

I think 1st you need to remove the columns with near to zero variance(assuming that they are not nominal variables), only if you have more variables as the computation time can be reduced.

Let us consider a scenario where there are 2 columns col1, col2 and you checked for variance then col1 had 0.003 variance and col2 had 0.5 var. once you apply the function nearZeroVar(), it removes col1, you need to hot-encode only 1 variable.

By that you can save time and your computer performance won't be effected.


Understanding Terminology in Goodfellow's paper on GANS

Question 1

  1) From what I understand, the generator is a multi-layer perceptron input with a random vector sampled from a predefined latent space, (where wikipedia gives the example of a multivariate normal distribution). What does it mean to learn the generator's distribution $p_g$ over data x?

Generally, supervised learning means to learn a distribution $p_{model}(y\mid x)$ where $x$ are your examples and $y$ your labels. However, in unsupervised learning it is not about labels but instead you are interested in unconditional probabilities and train a model to learn $p_{model}(x)$ (e.g. see p.485/486 in "The Elements of Statistical Learning" by Hastie et al). When applying GANs this $model$ is the generator $g$, i.e. you are learning $p_{g}(x)$. Accordingly, in the original paper the authors state:

  The generator $G$ implicitly defines a probability distribution $p_g$ as the distribution of the samples $G(z)$ obtained when $z ∼ p_z$.

Question 2

  2) What does "we define a prior on input noise variables $p_z(z)$" mean? I understand that in Bayesian statistical inference, that a prior probability distribution, called a prior for short, gives a probability distribution on outcomes before some evidence is collected.

Prior means we are talking unconditional probabilities, i.e. $p(z)$ is not conditioned on anything related to the task and examples on hand. While for example, $p_d(y\mid x)$ (the distribution learned by the Discriminator) is conditioned on $x$. Often $z$ is sampled from the standard normal distribution $\mathcal{N}(0,1)$.

In "NIPS 2016 Tutorial: Generative Adversarial Networks" the authors provide a good summary of the overall concept of $G$:

  The generator is simply a differentiable function $G$. When
  $z$ is sampled from some simple prior distribution, $G(z)$ yields a sample of $x$
  drawn from $p_{model}$. Typically, a deep neural network is used to represent $G$.


You have to prepare your data as a numpy array with the following shape:

( Number of observations , Input length , Number of variables )

Assuming you are working with Keras, the input of the LSTM() layer is as above, but you don't need to report the number of observations: input_shape = (Input length , Number of variables). Input length is an hyperparameter of your choice.

I pushed this Notebook on GitHub that contains a function to preprocess your dataset for RNNs:

def univariate_processing(variable, window):
    ''' RNN preprocessing for single variables.
    Can be iterated for multidimensional datasets. '''
    import numpy as np

    # create empty 2D matrix from variable
    V = np.empty((len(variable)-window+1, window))

    # take each row/time window
    for i in range(V.shape[0]):
        V[i,:] = variable[i : i+window]

    V = V.astype(np.float32) # set common data type

    return V

def RNN_regprep(df, y, len_input, len_pred): #, test_size):
    '''
    RNN preprocessing for multivariate regression. Builds multidimensional
    dataset by iterating univariate preprocessing steps.
    Requires univariate_processing() function.
    Args:  df, y:                X and y data in numpy.array() format
           len_input, len_pred:  length of input and prediction sequences
    Returns:  X, Y matrices
    '''
    import numpy as np

    # create 3D matrix for multivariate input
    X = np.empty((df.shape[0]-len_input+1, len_input, df.shape[1]))

    # Iterate univariate preprocessing on all variables - store them in XM
    for i in range(df.shape[1]):
        X[ : , : , i ] = univariate_processing(df[:,i], len_input)

    # create 2D matrix of y sequences
    y = y.reshape((-1,))  # reshape to 1D if needed
    Y = univariate_processing(y, len_pred)

    ## Trim dataframes as explained
    X = X[ :-(len_pred + 1) , : , : ]
    Y = Y[len_input:-1 , :]

    # Set common datatype
    X = X.astype(np.float32)
    Y = Y.astype(np.float32)

    return X, Y

Let me know if that's what you were looking for.


Deriving the update rule for mean in k-means clustering

The k-th mean just means the mean of the k-th cluster. We want to minimize the sum of squares of each of the clusters, which we do by minimizing this so-called distortion metric. The gradient is a multivariate generalization of derivatives and the gradient of this distortion metric is the zero-vector when this is optimized, and after this you update your clusters.


How do I calculate the delta term of a Convolutional Layer, given the delta terms and weights of the previous Convolutional Layer?

I am first deriving the error for a convolutional layer below for simplicity for a one dimensional array (input) which can easily be transferred to a multidimensional then:
We assume here that the $y^{l-1}$ of length $N$ are the inputs of the $l-1$-th conv. layer, $m$ is the kernel-size of weights $w$ denoting each weight by $w_i$ and the output is $x^l$.
Hence we can write (note the summation from zero): $$x_i^l = \sum\limits_{a=0}^{m-1} w_a y_{a+i}^{l-1}$$
where $y_i^l = f(x_i^l)$ and $f$ the activation function (e.g. sigmoidal).
With this at hand we can now consider some error function $E$ and the error function at the convolutional layer (the one of your previous layer) given by $\partial E / \partial y_i^l $. We now want to find out the dependency of the error in one the weights in the previous layer(s):
\begin{equation} \frac{\partial E}{\partial w_a} = \sum\limits_{a=0}^{N-m} \frac{\partial E}{\partial x_i^l} \frac{\partial x_i^l}{\partial w_a} = \sum\limits_{a=0}^{N-m}\frac{\partial E}{\partial w_a} y_{i+a}^{l-1} \end{equation}
where we have the sum over all expression in which $w_a$ occurs, which are $N-m$. Note also that we know the last term arises from the fact that $\frac{\partial x_i^l}{\partial w_a}= y_{i+a}^{l-1}$ which you can see from the first equation.
To compute the gradient we need to know the first term, which can be calculated by:
$$ \frac{\partial E}{\partial x_i^l} = \frac{\partial E}{\partial y_i^l} \frac{\partial y_i^l}{\partial x_i^l} = \frac{\partial E}{\partial y_i^l} \frac{\partial}{\partial x_i^l} f(x_i^{l})$$
where again the first term is the error in the previous layer and $f$ the nonlinear activation function.
Having all necessary entities we are now able to calculate the error and propagate it back efficiently to the precious layer:
$$ \delta^{l-1}_a = \frac{\partial E}{\partial y_i^{l-1} } = \sum\limits_{a=0}^{m-1} \frac{\partial E}{\partial x_{i-a}^l} \frac{\partial x_{i-a}^l}{\partial y_i^{l-1}} = \sum\limits_{a=0}^{m-1} \frac{\partial E}{\partial x^l_{i-a}} w_a^{flipped}$$
Note that the last step can be understood easy when writing down the $x_i^l$-s w.r.t. the $y_i^{l-1}$-s. The $flipped$ refers to a transposed weight maxtrix ($T$).
Therefore you can just calculate the error in the next layer by (now in vector notation):
$$\delta^{l} = (w^{l})^{T} \delta^{l+1} f'(x^{l})$$
which becomes for a convolutional and subsampling layer:
$$\delta^{l} = upsample((w^{l})^{T} \delta^{l+1}) f'(x^{l})$$
where the $upsample$ operation propagates the error through the max pooling layer.
Please feel free to add or correct me!
For references see:

Convolutional Neural Network by Stanford
Convolutional Neural Network by Andrew Gibiansky

and for a C++ implementation this (without requirement to install)


When using numerical duplicates for categorical data, new columns should be added or values be converted?

It depends of the algorithm you are using. For a linear model (linear / logistic regression, SVM...), you need to create dummy variables meaning features "Sex_M" and "Sex_F" as you noticed.

However, if you are using tree-based technics, create an integer typed column with Sex in [0, 1, 2] should be sufficient for these algorithms. Reason is, contrary to linear model, tree-based technics are non-linear and will evaluate all possible splits to partition your observations.

However, the way you map your categorical variale into integers can lead to different trees structures. Below is an example. Suppose you want to predict y variable using indexed feature "Sex". On left chart, there is a slight difference between category 2 and categories 0 and 1 because categories are not oriented. It will results into two consecutive splits and should happen late in the tree building. However, on the right chart, the means difference between categories is larger that on left chart. So the split should happen earlier in the tree building. Moreover, you will only need 1 split to make the difference.

To conclude, I think simple indexing can be used with tree-based technics. I would also recommend you to sort your categorical features in an orderly way to easy tree learning.


Transform Categorical Variables into Numerical

Suppose that you want to have k-means algorithm, in the formulation of average, you have to take the average of each cluster, and then reassign the centers. If you have categorical data, how do you want to take mean? Changing categorical data to numeric data is for translating situations which don't have numerical features to be suited to be used for such algorithms.


Averaging two Word2vec vectors to obtain a unified representation for single word

Just averaging them might not be good because that would assume that they they have the same weight, and that is probably not the case if the capitalized and uncapitilized version appear with very different frequencies in your training data.

An incremental improvement would be to average them proportionally to their frequency in the corpus. So say Earth appears 159 times and earth 1239 times do something like:

v(Earth & earth ) = 159/(159+1239) * v(Earth) + 1239/(159+1239) *v(earth).

The vectors are supposed to encode semantics linearly, so this should give you a resonable approximation.


Your description is apt. There isn't anything especially "mathematical" happening here, aside from the AdaBoost algorithm itself.
In psuedocode, something like this is happening:
For n in 1 .. N_Estimators do
  Train classifier Tn on data X with weights W
  Compute weighted residuals E from Tn
  Update W based on E
  Renormalize W
end

In your case, Tn would be a Random Forest model, which is itself an ensemble based on bagging. So at each iteration of the "outer" AdaBoost model, an entire Random Forest model is being trained, i.e. several decision trees are fitted on random sub-samples of data points and features.
Of course, this is an unusual setup for a boosting model. But there's no conceptual or computational reason why you couldn't run the algorithm this way.
If you are curious about how exactly the weights are computed and updated, Scikit-learn uses the SAMME algorithm, which is based on but not exactly identical to the original AdaBoost. SAMME is described in "Multi-Class AdaBoost" by Zhu, Rhosset, Zhou, & Hastie (2006).


I think you need to start by thinking very carefully about what exactly your client wants to know, because there are two different things you could be testing:

Is each individual customer is more likely to buy a popsicle of the designated type on a hot day than on a cool day?
Is the client likely to sell more popsicles of the designated type on hot days than on cool days?

For the first one, you would analyze proportions; for the second, you would analyze absolute sales counts for the designated type of popsicle, and in fact you could probably ignore sales of all other types of popsicles.

If that latter question is the one you want to analyze, I do think running a regression analysis on sales of the particular type of popsicle vs temperature (both as continuous variables) is a reasonable place to start. In fact you could start by just plotting them and looking at the graph, to see if any trend (linear, logarithmic, logistic, etc.) jumps out at you. I wouldn't go for A/B testing right off the bat because you're not dealing with a categorical variable, and there's not a single natural way to convert it.

Of course it could make sense to do that conversion, but in doing so, I'd suggest looking at overall sales figures and other indicators of whether people perceive a day to be hot or cool to try to pin down the boundaries. Something like logistic regression (possibly multinomial) might come in handy, though I could easily see it giving worthless results if the data aren't clean enough. Perhaps you could exploit a clustering algorithm to identify groups of temperatures that have similar sales figures and use that to determine your categories. You could also try to correlate external data sets like community pool attendance, ice cream sales, electricity usage (for air conditioners), etc., if those are available, to further inform your analysis of what temperature class each day would fall in. (Maybe that's overkill, but it's an idea at least.)

Incidentally, A/B testing typically designates a process where you control the assignment of the classes A and B. Like in web design, you choose (or program a computer to choose, perhaps at random) whether each visitor gets version A or version B. You don't get to choose whether each day is hot or not, so it's not quite the same thing. That being said, you can probably use most of the same methods of analysis you would on a proper A/B test.


Can I download Twitter data via web scraping for research?

Last time I checked it was not allowed to store the contents of the tweets, instead one is supposed to store the tweet id and retrieve the content of the tweet dynamically.

Afaik this is because users are allowed to delete their tweets at any time, and keeping a tweet that they chose to delete would be against Twitter terms of use (and possibly illegal in some jurisdictions). Using the tweet id solves the problem since the content will simply not be available anymore if the tweet was deleted.

Since you plan to write a paper I assume that you're in academia? If yes in case of doubt it's always safer to ask the data protection and/or legal office in your institution. In this case you're using a secondary source (i.e. you're collecting data which already exists) so it should be straightforward (I think).


I think I have bumped into what works. I am posting the correct code since somebody may need this someday:

import numpy as np

import pandas as pd

r_cols = ['user_id', 'movie_id', 'rating']

ratings = pd.read_csv('C:/Users/dell/Downloads/DataScience/DataScience-Python3/ml-100k/u.data', sep='\t', engine='python', names=r_cols, usecols=range(3))  # please enter your file path here. The file is u.data

print(ratings.head())

movieProperties = ratings.groupby('movie_id').agg({'rating': [np.size, np.mean]})

print(movieProperties.head())

movieNumRatings = pd.DataFrame(movieProperties['rating']['size'])

movieNormalizedNumRatings = movieNumRatings.apply(lambda x: (x - np.min(x)) / (np.max(x) - np.min(x)))

print(movieNormalizedNumRatings.head())

movieDict = {}

with open('C:/Users/dell/Downloads/DataScience/DataScience-Python3/ml-100k/u.item') as f:     # The file is u.item

    temp = ''

    for line in f:

        fields = line.rstrip('\n').split('|')

        movieID = int(fields[0])

        name = fields[1]

        genres = fields[5:25]

        genres = map(int, genres)

        movieDict[movieID] = (name, genres, movieNormalizedNumRatings.loc[movieID].get('size'), movieProperties.loc[movieID].rating.get('mean'))

print(movieDict[1])

from scipy import spatial

def ComputeDistance(a, b):

    genresA = np.array(list(a[1]))

    genresB = np.array(list(b[1]))

    genreDistance = spatial.distance.cosine(genresA, genresB)

    popularityA = np.array(a[2])

    popularityB = np.array(b[2])

    popularityDistance = abs(popularityA - popularityB)

    return genreDistance + popularityDistance

print(ComputeDistance(movieDict[2], movieDict[4]))

I am using this opportunity to thank those who gave me one or two suggestions.


Include time as a variable in regression model

Theres an approach I'd take which consists of two steps. The second is optional buy highly recommended.

create variables for day, month, year separately.
process those numbers as cyclic features (day 31 is as close to day 1 as to day 30, for example. If you don't do anything about that, and store days from 1 to 31, there's no place considering the cyclic nature of the feature). This post exaplins this in detail.

Hope this helps.

EDIT1:
Also, not asked, but depending on the algorithm you're using, you may want to normalize the numerical features you have.


CNN accuracy and loss doesn't change over epochs for sentiment analysis

You have imbalanced Data, that's why your network always predicts the most frequent answer (he is in a local minima).

There are solutions to avoid this kind of problem:

Balance your data, use as much "good" as "bad" in your training set.
Change your cost function,  e.g. the cost of error on "good" should be higher than "bad".

(don't use both at the same times or your model will be imbalanced in the other class)

If you look for imbalanced data on google you will find lot of articles on this problem, since it is common.


Which batch size to use when Batch Normalization?

The fact that training size is not divisible by batch size does not matter (primarily)
main reasons for batch training is it requires less memory. Since you train the network using fewer samples, the overall training procedure requires less memory, also speed. Typically networks train faster with mini-batches. That's because we update the weights after each propagation.
What you have left in the last iteration will just be propagated even though its not divisible.


80-20 or 80-10-10 for training machine learning models?

1) In 80-10-10 scheme, 80% is for training, 10% is for validation and 10% is for testing. Validation set required to search for the optimal hyperparameters.

For models having no hyperparameters, it doesn't do much good to use a validation set (although, it is still useful in determining when to stop the training of the model using early stop). In such situation, one might just keep 80% as training set and 20% as testing set.

2) Yes, K fold CV can be used with simple split.


Getting results of folds in orange canvas

For AUC, you can not. Leave-one-out uses only one data instance in the training set. Computation of AUC requires more data.

For general cross-validation (say, 10-fold cross-validation), there are several ways to examine what happened in each fold. Test & Score widget that performs cross-validation outputs a data table with predictions for each of the data instances in the test set. Together with predicted probabilities, predicted class, original class, the data also include information about the fold. You can then select the results by the fold and analyze it further, say, in a box plot.

To compute evaluation statistics for each of the folds, use Data Sampler, chose "Cross Validation" for sampling, select the desired fold, and wire the connection between Data Sampler and Test & Score widget as shown below. Notice that in this setting the Data Sampler outputs the fold data as a sample (to be used as test data), and out-of-fold data as remaining data (to be used as training data).


How Retail Analytics Project Can Be Done?

Start by plotting some of your data in various ways in order to get more familiar with it and understand the big picture. For example plot the total amount of sales for every store, for every product, look at the different by store for the same product, difference promotion/not promotion for the same product, etc.


parsing a simple sentence using syntaxnet (on Ubuntu)

Have a look at the documentation from the Authors of SyntaxNet themselves (Google). That link shows all the main steps that you might want to do.

Looking at the general setup required, it seems like quite an involved process! I can't actually find any nice walkthroughs on how to really run and understand the models... only that first link above!

If you would like to simply parse sentences and are not tied to SyntaxNet, it might be worth looking into similar and (in my opinion) easier to use tools, such as spaCy. You can install it easily using one of the commands from their website:


Extracting text from few areas on product label

If the image will always be the same and your ROI will always be in the same location, then an RNN with CV2 is not necessary, you can mark the areas that you need to extract the texts. I even recommend you more like JaidedAI
/
EasyOCR.
Here is the google colab of the papers you sent: Information extraction


Will I be able to fetch a Data Analyst job if I learn everything in this image?

I don't think someone can learn about everything in the image and be good at all of them. Especially when you get out of school !

You need to master:

Fundamentals
Statistics
Programming
Machine Learning
Visualization

The rest depends on what you want to specialize into.


Using a part of a trained model in a custom loss function -Tensorflow

Model class allows you define multiple outputs: official tensorflow (keras) documentation.

...
hid = LeakyReLU(alpha=0.1)(hid)  # the layer you want to use
intermediate = hid
...
model = Model(input_layer, outputs=[out, intermediate])

Then, if you train using model.fit or model.fit_generator, you simply need to provide the labels as a tuple of (expected_output, expected_intermediate_layer).


Set for building ROC curve and and choosing logistic regression cut-off

Usually you need to generate the ROC curve and choose the threshold within the training data. Then, with the selected threshold, you have the possibility to report accuracy, sensitivity, recall results you reach on your validation set.


How to get different results running sklearn's MeanShift in a single program? (Python3)

Have you looked at smoothing your dataset with any other methods? What hyper-parameter tuning have you done? Also how did you convert your non_numerical data, just wondering if you can provide pastbin?


Should we apply normalization to test data as well?

Yes you need to apply normalisation to test data, if your algorithm works with or needs normalised training data*.

That is because your model works on the representation given by its input vectors. The scale of those numbers is part of the representation. This is a bit like converting between feet and metres . . . a model or formula would work with just one type of unit normally.

Not only do you need normalisation, but you should apply the exact same scaling as for your training data. That means storing the scale and offset used with your training data, and using that again. A common beginner mistake is to separately normalise your train and test data.

In Python and SKLearn, you might normalise your input/X values using the Standard Scaler like this:

scaler = StandardScaler()
train_X = scaler.fit_transform( train_X )
test_X = scaler.transform( test_X )

Note how the conversion of train_X using a function which fits (figures out the params) then normalises. Whilst the test_X conversion just transforms, using the same params that it learned from the train data.

The tf-idf normalisation you are applying should work similarly, as it learns some parameters from the data set as a whole (frequency of words in all documents), as well as using ratios found in each document.

* Some algorithms (such as those based on decision trees) do not need normalised inputs, and can cope with features that have different inherent scales.


Is GEMM used in Tensorflow, Theano, Pytorch

Those Frameworks are based on Cuda, which is the parallel computing tool for GPUs. Specificaly it is cuDNN that is used by the deep learning framework.

You can find the user guide : https://docs.nvidia.com/deeplearning/sdk/pdf/cuDNN-Developer-Guide.pdf

All off the operations are described and indeed GEMM algorithms are used for convolution operations.


I want to ask you that how to predict the probabilities of each class in test images.

At line 27 in the train.py you have the following code:

correct_prediction = tf.equal(y_pred_cls, tf.argmax(y, axis=1))

It tries to find whether the predicted values are the same as the real ones. You can run y_pred_cls to see the probability of each class for your desired input.

  I want to use the code to predict the probabilities of new data's labels, how to save and load the model we have trained which used the train data.

for saving your model and its weights you can take a look at here. As you can see from there, you have to make a saver object:

import tensorflow as tf

#Prepare to feed input, i.e. feed_dict and placeholders
w1 = tf.placeholder("float", name="w1")
w2 = tf.placeholder("float", name="w2")
b1= tf.Variable(2.0,name="bias")
feed_dict ={w1:4,w2:8}

#Define a test operation that we will restore
w3 = tf.add(w1,w2)
w4 = tf.multiply(w3,b1,name="op_to_restore")
sess = tf.Session()
sess.run(tf.global_variables_initializer())

#Create a saver object which will save all the variables
saver = tf.train.Saver()

#Run the operation by feeding input
print sess.run(w4,feed_dict)
#Prints 24 which is sum of (w1+w2)*b1

#Now, save the graph
saver.save(sess, 'my_test_model',global_step=1000)

And for loading that you have to restore it:

import tensorflow as tf

sess=tf.Session()
#First let's load meta graph and restore weights
saver = tf.train.import_meta_graph('my_test_model-1000.meta')
saver.restore(sess,tf.train.latest_checkpoint('./'))

# Access saved Variables directly
print(sess.run('bias:0'))
# This will print 2, which is the value of bias that we saved

# Now, let's access and create placeholders variables and
# create feed-dict to feed new data

graph = tf.get_default_graph()
w1 = graph.get_tensor_by_name("w1:0")
w2 = graph.get_tensor_by_name("w2:0")
feed_dict ={w1:13.0,w2:17.0}

#Now, access the op that you want to run.
op_to_restore = graph.get_tensor_by_name("op_to_restore:0")

print sess.run(op_to_restore,feed_dict)
#This will print 60 which is calculated

Edit: Actually the code is a bit strange, anyway. The following part:

tf.nn.softmax_cross_entropy_with_logits(logits=output, labels=y)

can help you. It outputs the probabilities for each class.


Selecting models for ensemble from large group of models with high uncertaintly

The general principle of ensemble learning is indeed to rely on the diversity of the individual learners rather than their performance. Therefore it's ok to include models which perform well by chance (typically because they're overfit), since if there is sufficient diversity across the models it's very unlikely that two models would be overfit in the same way, i.e. that they would wrongly predict the same instances by chance. So if most models perform reasonably well in general through different mechanisms, it is expected that for any given instance most models will predict the right answer and only a few will be wrong.
The risk of using models which have a quite high correlation together is to defeat the purpose of ensemble learning and obtain a performance similar to the best individual learner.
Note also that the risk of overfitting is quite low if the individual learners have been tested with CV. It could also be worth checking the variance of the performance across CV runs (high variance indicating potential instability), but even that is not so useful in my experience: it's really the diversity of the learners which makes ensemble learning work optimally.
But as usual a lot depends on the actual data/task, so I'd suggest properly testing a few different approaches.


Training object detection models from the scratch without using pre-trained models and weights

Maybe you could first train the model on a large sized flower dataset, that way the conv layer will be optimised to detect flowers and then do transfer learning on that model using your custom dataset freezing all but the last layer.
I am not sure though, 25 images seem way too less. Try data augmentation on those images like horizontal and vertical flips, noise, shear etc.


Algortihm for making predictions from minimal data

If your feature set is really huge and most of the variables are strong features and can determine a class on their own, i would try to go with Logistic Regression using elastic net (probably less on the lasso and more on the ridge side, so you do not throw away features which happen to always co-occur in your dataset but not in reality).

Just be sure to only consider features that appear often enough in order to avoid overfitting. This can easily happen, when the number of examples is smaller than the number of features.

Random Forests could also be worth a try, but to my experience if the number of strong features is really great but they all occur relatively rarely, RFs underperform compared to Logistic Regression.

Btw. the fact that the user only reveals partial information might not be a problem at all as long as the information given can determine the class easily - as you said. I would just give it a try.


How max_features parameter works in DecisionTreeClassifier?

Max_feature is the number of features to consider each time to make the split decision. Let us say the dimension of your data is 50 and the max_feature is 10, each time you need to find the split, you randomly select 10 features and use them to decide which one of the 10 is the best feature to use.
When you go to the next node you will select randomly another 10 and so on.
This mechanism is used to control overfitting. In fact, it is similar to the technique used in random forest, except in random forest we start with sampling also from the data and we generate multiple trees.
So even if you set the number to 10, if you go deep you will end up using all the features, but each time you limit the set to 10.
If you compare the definition of the max feature in the decision tree and random forest, you will see that they are the same.
https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html
https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html


Should we first remove the unwanted independent variable or split the data into test train data?

Whether it's removed before/after splitting doesn't matter: the variable will not appear in any part of the data so it cannot cause any data leakage anyway.

However what might matter (at least in theory) is from investigating which part of the data was the decision made. Any preprocessing and design decision should be made based on the training set only in order to avoid any risk of data leakage (see example). In practice it's not really a problem in the case you mention.


Twitter Sentiment Analysis: Detecting neutral tweets despite training on only Positive and Negative Classes

The quick (and not very satisfying) answer is "it depends" -- specifically it depends upon what your underlying conceptual model of human emotion is and how it manifests in verbal/written behavior.

What is your characterization of neutrality in relation to positive and negative valence? Can documents be put on some sort of quantitative scale with neutral sandwiched between positive and negative? This position has linguistic support at least with simple phrases which express single valence states (the solution is {great > good > acceptable > poor > horrible}).

As @dmb poster suggested, if this is your conceptual model, then you might reasonablly argue that neutral falls between pos/neg and all you have to do is determine the optimal boundaries/cutoff. Thus, you can assign neutral to a test case even if the classifier wasn't trained on neutral cases (though you do need some way of determining these cutoffs).

But what about more complex cases? What happens when you move from the level of the phrasal unit to larger sentential and discourse level units? How would you rate the sentence "I love apples but hate bananas"? Do the positive and negative elements cancel each other out to create a neutral? You can easily see how muddied this gets when we start to talk about real human texts. In my opinion, I don't think it is particularly meaningful to talk about a document-level emotion score. Rather, I believe people express emotion/sentiment that is directed at an individual objects "I hate bananas" -> hate(subj, obj) that get combined into discourse-level constructions.

So, yes, I think you could reasonable defend the use of outputting a neutral category if only pos/neg are used to train... BUT you will need to justify your use of particular cutoffs as well as consider how to deal with longer documents that express multiple (conflicting) emotions. My preferred course of action would be to have people read and label tweets as pos, neg, neutral, and mixed (then ensure inter-coder reliability) to create my training data... let the algorithm do the hard work of finding the cutoff values.


Test accuracy of neural net is going up and down

Based on the image you are sharing, the training accuracy continues to increase, the validation accuracy is changing around the 50%. I think either you do not have enough data to use neural network or the network is small to capture all the information, in both cases I feel there either under fitting or over fitting problem. Can you give us some information about the database so we can help more


Why is PCA often used before t-sne for problems when the goal is only to reduce dimensionality?

t-SNE is computationally expensive, more than PCA. Many examples might use PCA just to simplify the problem.

Moreover, it is explained here:

  If the data set is high dimensional, doing principal component
  analysis is recommended because otherwise the curse of dimensionality
  can be an issue. TSNE makes the assumption of local linearity which
  might not hold in high dimensions where the manifold may be varying
  and PCA can help alleviate this issue by reducing the dimensionality
  of the data.

In other words, it seems that t-SNE suffers from high dimensionality of your data. It is typically used to represent a manifold in 2-3 dimensions, not tens or hundreds like, for example, PCA. It's mostly a visualization tool IMHO.

On the googling part, I suggest you to search only for the most relevant elements of your search, not for articulated sentences. I googled something like: "why pca before tsne" and it was enought to find useful stuff. Browsers don't need syntactic coherence, just the right keywords.


Synthetic Gradients good number of Layers & neurons

Several months later, I have a couple of insights on it:

  Also, how should we avoid our DNI overfitting, how to monitor and ensure it's not happening?

I don't think it matters, because DNI overfitting on the gradient is actually what we want. We want it to figure-out the pattern which reduces the error in the fastest way for our data.  However, as always, we should pay attention to the Validation of our entire network while doing so.

That probably means the more DNI neurons the better, as long as the designated training data stays unmodified o_O
If new or extra data is used for training, we should throw away our DNI, and just train once more, to overfit them on this new "adjusted" training-data.

Also, more on Synthetic Grads here

wish for a better answer, so won't select this one - plz post if you have a better one


Mean error (not squared) in scikit-learn cross_val_score

from sklearn.datasets import load_diabetes
from sklearn.metrics import make_scorer
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score

def mean_error(y, y_pred):
    # assuming y and y_pred are numpy arrays
    return np.mean(y_pred - y)

X, y = load_diabetes(return_X_y=True)
mean_error_scorer = make_scorer(mean_error, greater_is_better=False)

regr = LinearRegression()
cross_val_score(regr, X, y, scoring=mean_error_scorer)


What type of optimization problem is this?

Some flavor of evolutionary algorithm may suit your problem nicely, since:

The gradient of the objective function is unavailable or cannot be computed.
The objective function itself is somewhat expensive to compute.
The objective function is may have a large number of local maxima.

In short, evolutionary algorithms do optimization by generating candidate solutions and iteratively:

Checking their "fitness" (the value of the objective function)
Selecting the best candidate solutions
Generating new candidate solutions, somehow based on the best candidates from the previous iteration

Naturally, the devil is in the details of how the best candidate solutions are selected and combined to generate new candidate solutions. This gives rise to a large number of evolutionary algorithm variants. Two popular variants are genetic algorithms and evolution strategies.

Genetic algorithms

Standard genetic algorithm implementations require you to encode your input variables in a binary fashion. This may be fine for categorical inputs, but your input variables take real values in the interval $[-1,1]$. It would be possible to implement a genetic algorithm for your problem, but it wouldn't be ideal. For your problem, I'd take a look at...

Evolution strategies

On the other hand, evolution strategies handle real-valued inputs by design and, after getting used to the jargon, are fairly easy to implement. You mentioned that you plan to use MATLAB for optimization. I found this toolbox on the FEX which implements evolution strategies, but I haven't taken it for a test drive myself. Whatever the case, I think an evolution strategy would be a good option to try, given the dynamic nature of your problem.


How can I know if my NN TensorFlow model is overfitted or not?

Short answer, no it does not look like you are overfitting. Your validation (you call this test) accuracy seems to follow your training accuracy quite closely.

Easiest way to check for yourself is to plot the training and validation accuracy over each iteration, then look at how the curves behave. What you want to see is a test curve that closely follows your validation curve.

Overfitting can come in too ways, your validation curve being quite a bit lower than your training curve and/or a validation curve that is decreasing when the training curve is increasing.

Both these behaviors can be seen in this picture:

Image taken from Stanford's CS231n

Edit: As gunes points out you seem to have a programming error that should be addressed before any checks for overfitting.


Scikit-learn Random Forest - model changes as result of input scaling

After trying to recreate the issue with random numbers (and failing initially), I figured out that the problem comes from the fact that the x_train data that I'm using contains columns that have a very small, near-zero values.

To recreate, the first section is only run once:

scale = 0.0001 # making this larger eliminates the issue
x_train = np.random.uniform(0,scale,size=(1000,10))
y_train = np.random.uniform(0,1,size=(x_train.shape[0]))

Then using the same values for x_train and y_train, run the section below but with use_weights set to True and then False.

use_weights = True
model = RandomForestRegressor(n_estimators=10, random_state=0)

if use_weights:
    weights = np.arange(1,x_train.shape[1]+1)[None,:]
    model.fit(x_train * weights, y_train)
    prediction = model.predict(x_train * weights)
else:
    model.fit(x_train, y_train)
    prediction = model.predict(x_train)

print(prediction[0]) # changes based on use_weights value assuming scale is very small

As a side note, the y_train values for the real dataset are also very small and I have to multiply them by 100 or more to get the model to even run.  That is, it will not create any leaves at all without expanding the scale of the y_train values (confirmed by running the get_depth method on each tree).

I'm wondering, is this purely a numerical imprecision issue or is it something unique to the random forest calculations that are happening under the hood?


Quasi-categorical variables - any ideas?

I would suggest you to use the idea of so-called 'fuzzy clustering', where you put each of your hours of the day value into several clusters at the same time. Details in paper: http://home.deib.polimi.it/matteucc/Clustering/tutorial_html/cmeans.html

The idea is trivial:

You decide how many clusters you want to have. For example, 4 (so you divide your day hours into 4 cathegories). Instead of computing just 1 number (which defines cluster membership) for each of your day hours you compute 4 numbers which represent the degree of membership to each of 4 clusters. So for example if you 4 clusters will contain periods 12 AM-6 AM, 6 AM- 12 PM, 12 PM - 6 PM and 6 PM - 12 AM then you would replace for example 4 AM hour in original data with vector of 4 numbers, first one is the biggest, second is smaller, third one is the smallest one etc.

Then you could use these 4 numbers in your model to fit a regression line.

Of course, if you want you could use 24 clusters and in such case each your day of hour would have a high 'relation' with nearby hours and almost 0 with the distant hours.


Challenge in visualizing large coronavirus clusters in US data

How about a small multiples style visualization? A nice 2 x 3 grid would cover the six categories here. Size on X, frequency on Y.
This approach is one of the clearest ways to show this kind of data. The stacked histogram is difficult to interpret because the bars of the same color do not have common starting points. If you make six histograms arranged in a rectangle with common scale on the Y, you can quickly visualize the distribution for each category.
Here is a plot made with some simulated data like yours. The top image has all six categories on one plot, like yours. The bottom image facets each category into its own histogram with common scale. You can much more easily compare the six categories.


Isn't the depth of a convolutional layer, the number of colors (or colorspace size)?

Yes, the depth of an image is equal to the color channels (1 for gray-scale images, 3 for RGB).

However, that is only the case for the input layer of the CNN. During the first convolution layer, the image can be passed through as many filters as we select. This number becomes the depth of the first layer. The subsequent layers can have any depth we want.

For example in the CNN depicted below:

The image input to the CNN initially has a depth of 1 (because it is gray-scale). The first convolutional layer passes it through 6 filters, so the depth of the first layer becomes 6. The second layer passes the output of the first through 16 filters, meaning that the second layer has a depth of 16, etc.


Reason for adding 1 to word index for sequence modeling

First, note that they are just adding 1 to the size of the vocabulary, not to the token IDs themselves, so the predictions are not affected.
Then, why adding 1 ?
Because Tokenizer.word_index is a python dictionary that contains token keys (string) and token ID values (integer), and where the first token ID is 1 (not zero) and where the token IDs are assigned incrementally. Therefore, the greatest token ID in word_index is len(word_index). Therefore, we need vocabulary of size len(word_index) + 1 to be able to index up to the greatest token ID.
Update: note that adding 1 to the vocabulary size has nothing to do with out of vocabulary words: the words that are not pretrained are encoded as the out-of-vocabulary token (oov_token) if it was provided when building the tokenizer, or ignored if not. The oov token, if provided, has index 1.


What kind of neural network is the best for classifying a series of vectors

First, let me point that knowing how to use PyTorch is not enough to work with neural networks.
Second, your data points have about 450 elements from which 405 are binary (solid object or not), neural networks will require you to place then as real elements for training, dealing with floating points where you could make a faster processing using models that can deal with this kind of data. After training you can speed up predictions by changing the implementation of your layers to avoid the trivial multiplications with 1 and 0.
Third, neural networks are not a silver bullet, you should try simpler models first, just to check it you actually need the power of a neural network and it's non-linearity. Check some alternatives and guides for time-series analysis. I woul go with a Random Forest or a decision tree, you should be able to make them fairly simple and then do an analysis of what's relevant for solving your problem.
That said:
First Step:
Your data is composed of vectors that can be flattened to $R^{450}$ ($15 \times (3 \times 3 \times 3 + 3)$) which is a fairly small vector, few fully connected layers (two or three should be enough) may solve your problem, that should be your first try (in neural network).
Second Step
If that doesn't suffice, you can try more complex models:
Your data has the following format:
$P = (x,y,z,t)$, which is the position of a player in time.
$D_{geo}(P) \in R^{3 \times 3 \times 3}$, which is a tensor describing their surroundings.
Since, $P$ resembles a time-series, LSMTs are usually indicated for this kind of analysis. Since you plan on using fixed time-steps, a CNN could be used, keeping data format as a tensor ($T \in R^{3 \times 3 \times 3 \times t}$) and using few CNN layers to extract information from the geographical information and only then joining the positions with it (assuming you want a fast processing model).
The actual shape of your CNN layers could will have to be tunned with a grid search.


SVM, which range to use when normalizing

Typically one decides between a range based normalization technique and a mean and standard deviation based normalization technique, which is known as standardization.
Standardization is often preferred to some types of normalization as normalization can put too much weight on anomalous data at the extrema of the cohort. This is typically not a problem, but certain discretization techniques can then hide the interesting structure in the data in a very narrow region. In your case, the structure might be being missed by your grid search, which is why you always arrive at a linear kernel. I suggest first standardizing your data and retrying your grid search. Does this arrive at a non-linear kernel?
If not, then also just try choosing some different kernel types and see if they outperform the linear kernels after you have used a grid search to optimize the other parameters.
The 3D structure in your example that is not apparent when projected into 2D should not be particularly applicable to this case since dimensionality and linearity are very different concepts.
Hope this helps!


Neural network training with known erroneous data

If you want to identify and remove problematic data, then this is much better done before training.

You might have some luck with your approach, but you have no guarantees that the neural network will help you isolate the problem entries based on error values, if it has been trained on the "good" and "bad" entries at the same time. It depends on the nature of the problem data. You take a significant risk that the model will fit to the problem data well enough to cause you to reject good data.

You should instead try to think of a way to identify the bad data more directly, and prior to training for the predictive task.

Here's one idea: Are you able to identify enough good and bad data manually - enough to train and test a classifier? Then label some of the data "good" or "bad", train a classifier and test it - the test should be on some held out labeled values, to help you assess accuracy. If your classifier has a good accuracy, you can have some confidence to use it to filter the remainder of your data, and only use the "good" classified data for your original training goal.


Sklearn and PCA. Why is max n_row == max n_components?

Given m rows of n columns, I think it's natural to think of the data as n-dimensional. However the inherent dimension d of the data may be lower; d <= n. d is the rank of the m x n matrix you could form from the data. The dimensionality of the data can be reduced to d with no loss of information, even. The same actually goes for rows, which is less intuitive but true; d <= m. So, it always makes sense to reduce dimensionality to something <= d since there's no loss; we typically reduce much further. This is why it won't let you reduce to more than the number of rows.


Extracting data, sorting and removing duplicates from an array using python 2.7

I think with the volume of data you have, you can try something like this (using the file provided above)

import json
with open('data1.txt', 'r') as f:
    f_json = f.read().replace('\'', '"')
    d = json.loads(f_json)
sorted_d = list(set([''.join(sorted(x, reverse=True)) for x in d]))


is numpy isclose function returning bad answer?

No, it uses both an absolute and relative tolerance, and the default of rtol is nonzero, adding some more tolerance. See https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.isclose.html


It doesn't require any special method. The algorithm of choice depends on your data if for instance Euclidean distance works for your data or not.

Generally, you can try Kmeans or other methods on your X or PCAs; but Hierarchical Clustering may be a good choice for visualizing the clusters for high dimensional data.

Please check here if you can read/write python code.


Identifying Waveform Segments Using Training Waveforms

Statistical Data Analysis

I would start with the following:

Statistically analyse your training events and see the mean and the variance of length and width and other properties to have an overall image of what they are.
For each event take the mean and the standard deviation.
Walk through the test data and compute the error between test signal and learned events and plot it.
At the end for each event type, you will have an error curve whit many local minimums.
Choose a threshold based on standard deviation of training events and detect minimums behind that threshold as events.

Pattern Recognition

More advanced option is to learn a model to classify events. At the beginning I should say the last question is theoretically wrong because you do not generate features from test data but only compute them. Features have been extracted (generated) from training data. I assume you mean how to feed the model with test data for classification according to its size. Well, I'd say you can segment it to chunk which may lead to losing some events because of cutting points but if test data is large enough that would not be a problem.

The point is that time-series are a special kind of data and there are methods specifically for them so the classic feature extraction and learning may lead you to an ocean of confusing methods. I suggest to search the term Time-series Mining instead to see more direct solutions.

Time-Series Mining

The very first approach I proposed was kind of this! There are many more algorithms for detecting similarity based on two time-series mostly based on Euclidean distance. For example look at LCSS or DTW. These methods are what you are looking for and both are advanced version of my first suggested algorithm. Please note that for all of them normalizing your training and test data plays an important role.

See this answer as well however that is about unsupervised segmentation of time-series.


Calculate confidence score of a neural network prediction

One way to estimate the level of confidence we have about an ANN prediction is to use dropout perturbations. The idea was proposed in this paper: Dropout as a Bayesian Approximation. Representing Model Uncertainty in Deep Learning. The core idea is to use dropout as a perturbation method, and check how predictions change with varying levels of dropout. Once you sampled a sufficient number of "distorted predictions", you can estimate something analogous to a confidence interval around the initial model prediction. This technique works either for classifiers and regressors.

You can read an explanation of this approach here.

EDIT:

To be more precise:

In order to implement this technique, use Dropout() layers, they can be used in prediction phase too (not just during training). You can train your Neural Network, then transfer its weights into another ANN with the same architecture + dropout layers. Something like:

new_model.set_weights(original_model.get_weights())

Once the new ANN with Dropout() has its weights, run it and vary its dropout hyperparams, get the predictions and calculate the CIs. I know it's time consuming so do it only if you think it's really worth it. Another possible way to implement can be done with custom layers like:

model.add(Lambda(lambda x: K.dropout(x, level=0.5)))


What is the shape of the vector after it passes through the TfidfVecorizer fit_transform() method?

The inverse document frequency transformation of TFIDF does not affect the shape of your vector. The shape is only altered during the tokenization phase which, in your case, results in a shape of (4, 9). The inverse document frequency portion only scales your existing features in accordance with how frequently these tokens arise elsewhere in your corpus. This can be thought as a one-to-one mapping of your original term-frequency vector (the tokenized corpus) to the tfidf vector (scaled using information on how frequent your vectors tokens exist elsewhere in your corpus).


"Binary Encoding" in "Decision Tree" / "Random Forest" Algorithms

In general, it "okay" to apply to binary encode high cardinality datasets. In the sense of it will create numerical features that can be learned by a machine learning model.
However there are often better options, such a label encoding, frequency encoding, target encoding, or embeddings.
It is an empirical question which encoding scheme is best for your specific data and model. The best empirical coding scheme can be found through cross validation.


Feature scaling for MLP neural network sklearn

In short:

Scaling is indeed desired.
Standardizing and normalizing should both be fine. And reasonable scaling should be good.
Of course you do need to scale your test set, but you do not "train" (i.e. fit) your scaler on the test data - you scale them using a scaler fitted on the train data (it's very natural to do in SKLearn).

For example, if you're normalizing your data (like with an SKLearn StandardScaler object), you .fit it on the train data to get the mean and standard deviance from it, and you .transform both train and test data to subtract the train mean and divide by the standard deviance.


Best practices to store Python machine learning models

You may have a look at nexus or dvc or datmo.

There was recently a presentation at at meetup in berlin, zalandos AI data engineering meetup.


Is it accurate to say that "K-means clustering the vectors based on keywords weight similarity"?

K-means clusters points (in your case represented as 745 dimensional vectors) by their similarity, that is some distance measure between points (usually the Euclidean distance).
TF-IDF produces a vector from a sentence or document, where each entry (axis) represents the frequency of a word divided by the frequency of it's occurrence in all sentences or documents, hence the name. Other weighting scheme are possible as well, see here.
If two vectors are very close to each other, it means that the content of their documents is very similar. Therefore it's likely they end up in the same cluster. In contrast, if two vectors are far away, the words in each document might be completely different or the frequency of words might be different.
So the distance between vectors can be interpreted as measure of similarity between the documents content.


As per @an6u5's comment:
If you want to weight one dimension higher than others then I suggest you standardize all of your data so that the mean is zero and the standard deviation is one. Then you can multiply the less important dimensions by a factor (2-10) so that they appear farther away to the KNN distance metric and leave the most important dimension un-scaled. Note that both standardizing and scaling are completely reversible processes, so there is very little reason not to use this simple solution


How can I annotate labels to images automatically?

Automatic image annotation has its limitations. It's basically another already trained machine learning model that's doing the annotation for you.

Keep in mind that automatic annotation also requires another step in between, which is selecting elements with low prediction confidence and review them by hand. These are the most problematic elements when you use an automatic tool to label images and almost always require manual review.

Image source and suggested reading: https://medium.com/anolytics/what-are-the-limitations-of-automatic-image-annotation-vs-manual-a8ff7edb3152


I'm not familiar with sklearn NB methods but it's likely that these are actually the predicted probabilities. NB is known to often assign extreme probabilities, i.e. close to either 0 or 1. This issue implies that the probabilities predicted by NB are not really meaningful, i.e. they cannot (usually) be interpreted as "there is an x % chance that this instance belongs to this class" for example.


Taking logarithms is not a one-size-fits-all approach - what if your response is negatively skewed?

"Simple" linear regression assumes that the response variable is normally distributed. If you'd like to perform this kind of regression, then you should probably transform your response variable to be as normal as possible.

There are other approaches / model types to deal with skewed responses - check out the Tweedie family.


TL:DR, I would suggest not to optimise over the random seed. A better investment of the time would be to improve other parts of your model, such as the pipeline, the underlying algorithms, the loss function... heck, even optimise the runtime performance!  :-)

This is an interesting question, even though (in my opinion) should not be a parameter to optimise.

I can imagine that researchers, in their struggles to beat current state-of-the-art on benchmarks such as ImageNet, may well run the same experiments many times with different random seeds, and just pick/average the best. However, the difference should not be considerable.

If your algorithms has enough data, and goes through enough iterations, the impact of the random seed should tend towards zero. Of course, as you say, it may have a huge impact. Imagine I am categorising a batch of images, into cat or dog. If I have a batch size of 1, and only 2 images that are randomly sampled, and one is correctly classified, one is not, then the random seed governing which is selected will determine whether or not I get 100% or 0% acuracy on that batch.

Some more basic information:

The use of a random seed is simply to allow for results to be as (close to) reproducible as possible. All random number generators are only pseudo-random generators, as in the values appear to be random, but are not. In essence, this can be logically deduced as (non-quantum) computers are deterministic machines, and so if given the same input, will always produce the same output. Have a look here for some more information and relative links to literature.


transition to data scientist for more money

Data scientist won't actually make you better paid. You might have seen 150k+ for some data scientists, but those people usually have a Phd and they can get similar salary even they do something else.

Data science is a very big field; you can work in the field as a software engineer. But don't expect training a gradient boosting model; you'll more likely be asked to process Hadloop big data application, optimization Python workflow etc.

In my experience, a data science certificate is useless. If your next position is a software engineering in data science, you don't need a certificate; you'll just need have a good proven record that you can deal with data. If you next position will be mathematical modelling, you'll need something like a Phd.


How exactly does a validation data-set work work in machine learning?

Machine learning models output some sort of function; for example, a decision tree is a series of comparisons that results in a leaf node, and that leaf node has some associated value. This decision tree predicts survival chance on the Titanic, for example:

(Image by Steven Milborrow, from Wikipedia)

This can be used exactly like the equation generated by linear regression. You feed the new data points into the model, you get predictions out, and then you compare those predictions to the actual values.

The comparison you do is applying a "cost function," and what cost function you use is determined by your application. For example, linear regression can be done with different cost functions; typically, people use "ordinary least squares" which tries to minimize mean squared error, but other possibilities exist, like minimizing the absolute value of the error. When the result is a percentage, then something like a log probability loss function is typically a better choice than mean squared difference between the actual and predicted probability.

Even k-nearest neighbors generates a function; it splits the space into regions where the k-nearest neighbors are the same set, and then has a flat value for that region. It ends up very similar to the result generated by a decision-tree based method.

There's also a terminology point here: typically, people talk about the training set and the test set, or the training set, the validation set, and the test set. The point of the validation set is different from that of the test set; the test set is used to determine out-of-sample error (that is, how much the model has overfit and what its real world error is likely to be like) whereas the validation set is used to determine what hyperparameters minimize the expected test error.

One could fit ten different models on 80% of the data, determine which one has the lowest error on another 10% of the data, and then finally estimate real-world error on the last 10% of the data.