scipy.signal's finds_peaks should have most of what you need.

With the height argument you can set a minimum threshold to get the first peak.

With the prominence argument you should be able to approximate the second peak. This requires peak_prominences.

  The prominence of a peak measures how much a peak stands out from the
  surrounding baseline of the signal and is defined as the vertical
  distance between the peak and its lowest contour line.


Trying to get Orange connected to Postgres Database

To install tsm_system_time you need the contrib package installed too, see instructions on that page (or this page).

These are various extensions that are not needed for basic database operations but are useful for certain applications, this is apparently such a use case. The control file is like a script that tells Postgres how to install each extension when it encounters the CREATE EXTENSION command.


Distance metric for graph layout of correlation matrix?

What I have found is that it is not worth sweating the nuances of how distances are represented because they only partially determine the layout.  Other factors are geared to generating a pleasing layout so that analysts can recognize gross patterns based on the absence presence of interconnections rather than purely based on distance.


Is there a difference between "classification" and "labeling"?

Totally disagree with @Derek Janni. Be careful about notation however you should not get lost in terminology. Those papers you mentioned used the term "labeling" literally but in Machine Learning/Data Mining community, labeling is the process of preparing data for supervised learning (classification)! It has nothing to do with the ML task!

Those papers used the term to show that after supervised learning they can recognize different labels of different objects so they used the term labeling but you probably can not find in any literature that they use these two terms as synonyms.


How to create a system to detect text structure of a file?

Based on your current examples.

You have an ML that is doing work across multiple systems and users. Generating logs.
The logs can be grouped or classed (e.g. $F1, F2,...,FN$) by:

i) the operating System used (e.g. Android, Windows etc) and
ii) the error or message generated. The error messages are differentiated by tab white space.

If you cannot work out the log class using some logic, you could look to engineer features with regex to capture the source OS and the tabbed structural information in the error lines.

|---------------------|------------------|
| Error detail        |     n_tabs       |
|---------------------|------------------|
|  string_error       |         0        |
|---------------------|------------------|


How to fill missing enteries in column A, and add respective corresponding enteries to column-B with value of previous cell

try to make a sheet derived from the source. first, put all possible time in the new sheet in an ordered way. second, do vlookup from source for value. third, put vlookup inside a iferror to find missing value and for second part of iferror use previouse value.


Binning data in one of the columns of a dataframe(Using R)

Use cut:

> df = data.frame(v=sample(1:60,1000,TRUE))
> df$cat = cut(df$v,c(-Inf,6,12,Inf))
> table(df$cat)

 (-Inf,6]    (6,12] (12, Inf]
       97        92       811

Also, simple R questions are better asked on StackOverflow.


What is the dimensionality of the bias term in neural networks?

As per the general case, the bias vector must have the same dimensions as the output vector.

Please, have a look at this excellent presentation:

In this example by M.Görner, there are 10 classes, so is bias dimension. Once inputs are multiplied by weights, the bias is added pointwise (it is 'broadcasted'). And that's pretty much it.


You can check the histogram of y/x. Using only values of x far from zero would be necessary, in your case its not a problem because prices are far from zero.

If in your case the three different distributions have intercept near 0, you may have useful results. You can also try to check the histogram of (y-intercept)/x.

You should find different modes in the distribution of this quotient. You could then divide the dataset according to different values of this quotient.


Is it right to maintain the train distribution in test set for unbalanced data?

If the training set was unbalanced the chances are the model will be biased.

Not really. Depending on the loss function you use. Also, note that for data to be unbalanced at least it has to be in a proportion of 1/100.
The rest of the questions:
ML is based on the hypothesis that train and test look alike. Oversampling methods can help in training time, still in validation and test, you should not use the oversample and validate with the real data.
Use your evaluation metrics on the real test, with test distribution and don't oversample there.

we want to use the model for commercial business that we have no idea how the distribution of data would be?

If you have no idea of how the distribution will be you have a problem. The hypothesis that people make is that the future distribution resembles something of the current real distribution (last week,last month, one year ago....)


How did the author got this final result from this Gaussian Distribution formula?

Take natural logarithm of the expression. Logarithm of product of two terms can be written as sum of logarithms of each term: log(a.b) = loga + logb. e and natural logarithm cancel each other, there is left first term in the result. Logarithm of a 1/number is equal to -log(number), it gives second term in the result.
Taking logarithms of exponentials is quite common method for preventing numerical overflows and ensuring numerical stability.


Bert Fine Tuning with additional features

To add additional features using BERT, one way is to use the existing WordPiece vocab and run pre-training for more steps on the additional data, and it should learn the compositionality. The WordPiece vocabulary can be basically used to create additional features that didn't already exist before.

Another approach to include additional features would be to add more vocab while training. Following approaches are possible:

Just replace the "[unusedX]" tokens with your vocabulary. Since these were not used they are effectively randomly initialized.
Append new vocabulary words to the end of the vocab file, and update the vocab_size parameter in bert_config.json. Later, write a script which generates a new checkpoint that is identical to the pre-trained checkpoint, but with a bigger vocab where the new embeddings are randomly initialized (for initialized we used tf.truncated_normal_initializer(stddev=0.02)). This will likely require mucking around with some tf.concat() and tf.assign() calls.

Please note that I haven't tried any of these approaches myself.


How to read specific column with specific row in x_test using python

For a 2D np array x_test_n, you could use index_row, index_col = np.where(x_test_n == 0) to return the indices with 0 values.


any efficient way to find surrounding adjective/verbs with respect to the target phrase in python [updated]?

POS-tagging consist of qualifying words by attaching a Part-Of-Speech to it. Part-Of-Speech is a tag that indicates the role of a word in a sentence (e.g. a noun, a transitive verb, a comparative adjective, etc.). You need this to know if a word is an adjective, and it is easily done with the nltk package you are using [source]:

>> nltk.pos_tag("The grand jury")
>> ('The', 'AT'), ('grand', 'JJ'), ('jury', 'NN')

Here, JJ means "Adjective" and "NN" means "Common Noun".

In your case, you are interested in neighbors adjectives. Does that mean "the closest adjective" in the sentence ? Or adjectives within a radius of the target, if any?
Depending on the definition, the way to do it differs.

For adjectives within a radius, as you have already selected words within a radius using the snippet you mentioned, you can POS-tag them and then select only those with a tag that indicates an adjective tag.

>> adjective_tags = ["JJ", "JJR", "JJS"]
>> close_adjectives_list = [a[0] for a in nltk.pos_tag(" ".join(close_words_list)) if a[1] in adjective_tags ]

You can look at this answer that list most of existing POS-tags.


Handling hierarchical category independent variables

I'd suggest the following:

3 features, one for each level main_column, sub_column1, sub_column2
2 additional features representing the hierarchical relation:

main_column/sub_column1
main_column/sub_column1/sub_column2

This way the training can select the most informative level of information between main_column, main_column/sub_column1, main_column/sub_column1/sub_column2. Depending on the data and algorithm used it might also make sense to discard rare cases for a subcategory and use a kind of "misc" category instead.


naive bayes classifier for non-binary feature values

\begin{align}
&P(x^{(1)}, \ldots, x^{(m)}, y^{(1)}, \ldots, y^{(m)})\\
&=\prod_{i=1}^m P(x^{(i)},y^{(i)})\\
&= \prod_{i=1}^m P(y^{(i)})P(x^{(i)}|y^{(i)})\\
&= \prod_{i=1}^m \left[\textbf{1}(y^{(i)}=1)(1-\theta_0)\prod_{j=1}^nP(x^{(i)}_j|y^{(i)}=1)+\textbf{1}(y^{(i)}=0)\theta_0\prod_{j=1}^nP(x^{(i)}_j|y^{(i)}=1)\right]\\
&=(1-\theta_0)^{\sum_{i=1}^m\textbf1(y^{(i)}=1)}\theta_0^{\sum_{i=1}^m\textbf1(y^{(i)}=0)}\prod_{j=1}^n\prod_{k=1}^s \theta_{j,k|y=1}^{\sum_{i=1}^m\textbf1(x_j^{(i)}=k, y^{(i)}=1)}\theta_{j,k|y=0}^{\sum_{i=1}^m\textbf1(x_j^{(i)}=k, y^{(i)}=0)}
\end{align}

Let's take the logarithm of the expression above and we want to maximize the following

\begin{align}&\sum_{i=1}^m \textbf1(y^{(i)}=1)\log(1-\theta_0)+\sum_{i=1}^m \textbf1(y^{(i)}=0)\log(\theta_0)+\\& \sum_{j=1}^n \sum_{k=1}^s \sum_{i=1}^m\left[ \textbf1(x_j^{(i)}=k, y^{(i)}=1) \log \theta_{j,k|y=1} + \textbf1(x_j^{(i)}=k, y^{(i)}=0) \log \theta_{j,k|y=0}\right]\end{align}

subject to the condition that

$$\sum_{k=1}^s \theta_{j,k|y=1}=1$$
$$\sum_{k=1}^s \theta_{j,k|y=0}=1$$

as well as the nonnegative constraints.

Let the Langrange multiplier for the sum to $1$ constraint be $\lambda_{j,1}$ and $\lambda_{j,0}$.

The Langrangian that we want to optimize is

\begin{align}&\sum_{i=1}^m \textbf1(y^{(i)}=1)\log(1-\theta_0)+\sum_{i=1}^m \textbf1(y^{(i)}=0)\log(\theta_0)+\\& \sum_{j=1}^n \sum_{k=1}^s \sum_{i=1}^m\left[ \textbf1(x_j^{(i)}=k, y^{(i)}=1) \log \theta_{j,k|y=1} + \textbf1(x_j^{(i)}=k, y^{(i)}=0) \log \theta_{j,k|y=0}\right]\\
&-\sum_{j=1}^n \lambda_{j,1}\left(\sum_{k=1}^s \theta_{j,k|y=1}-1\right)-\sum_{j=1}^n \lambda_{j,0}\left(\sum_{k=1}^s \theta_{j,k|y=0}-1\right)\end{align}

Differentiate with respect to $\theta_0$ and equate it to zero, we get

$$-\frac{\sum_{i=1}^m\textbf1(y^{(i)}=1)}{1-\theta_0}+\frac{\sum_{i=1}^m\textbf1(y^{(i)}=0)}{\theta_0}=0$$

Solving it which is just solving a linear equation, we can check that $\theta_0=\frac{\sum_{i=1}^m\textbf1(y^{(i)}=0)}{m}$.

We can also differentiate with respect to $\theta_{j,k|y=1}$,

$$ \frac{\sum_{i=1}^m\textbf{1}(x_j^{(i)}=k, y^{(i)}=1)}{\theta_{j,k|y=1}}=\lambda_{j,1}$$

That is fixing $j$,$\theta_{j,k|y=1}$ is proportional to   $ \sum_{i=1}^m\textbf{1}(x_j^{(i)}=k, y^{(i)}=1)$. With the sum to $1$ constraint, we have

$$\theta_{j,k|y=1}=\frac{\sum_{i=1}^m\textbf{1}(x_j^{(i)}=k, y^{(i)}=1)}{\sum_{k=1}^s\sum_{i=1}^m\textbf{1}(x_j^{(i)}=k, y^{(i)}=1)}$$

I will leave the last part as an exercise.


GD for logistic regression isn't stable. Why?

Your implementation seems not correct. For example:

(1) in your __gd_step() function, the line

grad[j] *= X[i, j]

which uses the index i, is outside your for loop: for i in range(self.m).

(2) in your __odds() function, the denominator exp(1 + b.dot(x)) seems incorrect. 1 should be outside of exp().


How to use Tensorflow Model Zoo frozen graphs in an Estimator pipeline?

After a lot of research, I came to the conclusion that this is doable, but would require a lot of work.

Essentially, the steps to achieve such a transformation are as follows:

Use this version of the model exporter (or any other solution) to re-export the savedmodel with its variables (by default, the Model Zoo does not give the variables, only an inference-ready model).
Load the model using the SavedModel API, i.e. loaded = tf.saved_model.load(...).
Extract the inference function: inference_func = loaded.signatures['serving_default'].
Define your loss function. This part is the most complex because the existing functions are part of the Object Detection API which is pretty much outdated. Tensorflow is already working on the upgrade of the API but at the time of writing this, they still expect a couple more months to finish it.
Set up your Dataset.
Train your model.

The best option is probably to wait for the new release of the official OD API.


Dense layer: how to transform hgih dim vector to one dim? (sigmoid function)

I suggest watching 3blue1brown's YouTube series on neural networks to get a better conceptual understanding of what's going on.

It's not that the sigmoid function itself transforms the dimension of a vector - the sigmoid function is just used as an activation function that affects the output of each individual node - it's the number of units/nodes you have in a layer that does what you're describing.

If you only have one unit in your output layer it will only output one number between 0 and 1. If you have 100 units in a dense layer it will output 100 numbers, each between 0 and 1.


What GPU size do I need to fine tune BERT base cased?

Speaking about vanilla BERT.
It is currently not possible to fine-tune BERT-Large using a GPU with 12GB - 16GB of RAM, because the maximum batch size that can fit in memory is too small (even with batch size = 1).
The fine-tuning examples which use BERT-Base should be able to run on a GPU that has at least 12GB of RAM using the hyperparameters given on this page.
However, GPU training is single-GPU only.


Linear and non-linear dependence in a single DS model

The simple answer is to experiment.
You did a quite detailed analysis of the relations between your features and response variable and that's definitely a good idea, but don't be afraid to experiment with various models, even those which don't seem perfectly adapted to the task. Why? Because the one thing that such an analysis by individual features doesn't show is the patterns which happen with combinations of features, and that's what most ML algorithms are usually good at finding. This is why one often finds surprisingly good results with methods expected to perform poorly.
So my advice is to start with a range of simple methods known to be quite robust, for instance Random Forest, SVM, logistic (or even linear) regression. A good strategy is to start with simple methods and then try to improve on that based on analyzing the results. For example a more advanced approach would be to use ensemble learning: train different types of models and then a meta-model which relies on their predictions.


What is the purpose of untrainable weights in Keras

One common application is to freeze an embedding layer. Freezing this layer will prevent the embedding from updating its weight which can be a desirable thing, especially for a text embedding layer.

There also exist designs where updating a weight during a certain batch is not wanted.  For example some GAN implementations only want to train a model during the combined model phase and therefore freezes the Generator and Discriminator layers for batches.

You can also see this used for stacked autoencoder where someone may train one layer at a time.  Here is a quick link.


XGBOOST missing_value feature degrades my performance?

If you have more than 50% of values as missing across any columns you should impute only when you are an SME of the domain or know the reason for missing(Honestly I would drop that column rather imputing). Not only XGboost most of the algorithm would penalise for Missing values and one can increase the performance by imputation. Tip may be try Catboost


Predict next month's loan balance from historical data

You have quick and easy but rough predictions:

The "meteo method": next month will be the same as last month. No as bad as you may think, and you can even replace last month by an average of the last 6 month.
Linear regression: the meteo method will fail if there is a constant growth. Make a linear regression in function of time $x(t) = a \cdot t + b$. Your forecast will be $x(today+1)$.
Exponential regression: use a regression on a growth in percentage (exponential growth). You can find the best $a$, $b$ fitting $x(t) = a \exp(b \cdot t)$, or the best $c$, $d$ to fit $log(x(t)) = c \cdot t + d$. These are the same formula in disguise and you can use a linear regression for the last one (fit $y(t) = log(x(t))$ instead of $x(t)$).

Note that this a "scientific" version of you method 1.
Exponential smoothing: It is an average giving more weight to the last values. There is a technical trick that makes computation specially easy if you have to forecast every month. In reference with the wikipedia page, take $\alpha = 1/6 = 0.1667$ for a 6 month history.

In the future, you will ba able to add a trend (Double exponential smoothing) and a seasonality (Holt-Winter model).

I would avoid method 2, consisting in adding forecast of individual account. This method works when every thing is linear so that errors cancel each others. When the the process is not linear, you'll have a systematic bias which will add together.

I also mention that your main challenge will more probably be the seasonality than the main trend.


How does tree-based algorithms handle linearly combined features?

If the sum of the two feature makes sense on the domain semantically, it might be a good idea.
But while trees can handle redundant features pretty well, increasing the number of features without adding any extra "value" or "information" can lead to lower performance in certain situations. For example, if there is no added value and you aggressively sample or restrict the number of features in each tree the two related feature will take the place of a useful features when they are both selected.
If you think the sum is a better feature than the partitions you might consider addig the sum only, or the sum and one of the components.


bias and variance trade off related question

The bias of an estimator $\hat{f}(x)$ of $f(x)$ is by definition:
$$Bias(\hat{f},f)(x)=E[\hat{f}(x)]-f(x)$$
which is some definite (not-random) function $\phi(x)$.
Why this is so? The intuitive way to understand it is that estimation itself is a random process. Given random data will produce a random estimation. However the expected estimation is $E[\hat{f}(x)]$, so it is only natural to define bias as the difference between expected estimation and true value.
In case the bias of the estimator $\hat{f}(x)$ (for $f(x)$) is zero, in other words:
$$E\left[\hat{f}(x)\right] = f(x)$$
Then the estimator is called unbiased. It has no (systematic) bias in the estimation process and the expected estimation coincides with the true value. If furthermore, this estimator has minimum possible variance is called minimum variance unbiased estimator. All these are part of the basic theory of statistics.
The linked tutorial you mention simply computes the total Mean Square Error (MSE) by the various components it can be split into (a general theorem for any type of estimation process):
MSE = (irreducible noise error) + (estimator variance) + (bias squared)
or in other words:
$$MSE(x) = Var[\epsilon] + Var[\hat{f}(x)] + Bias(\hat{f},f)(x)^2$$
The estimator variance is the same formula as for any other random variable and is certainly part of the total MSE.
Derivation:

Suppose $y = f(x) + \epsilon$, $E[\epsilon]  = 0$, $Var[\epsilon] = \sigma^2$
For any random variable $r$ we have $E[r^2] = Var[r] + E[r]^2$ (by the definition of variance).
For any non-random value $c$ we have $E[c] = c$, $Var[c] = 0$
$E[y] = E[f(x) + \epsilon] = E[f(x)] + E[\epsilon] = E[f(x)] = f(x)$
Similarly $Var[y] = Var[\epsilon]$ (expand the definition and use previous facts)
Since $\hat{f}$ and $\epsilon$ can be taken as independent, we can write the total error as:

${\displaystyle {\begin{aligned}
\operatorname {E} {\big [}(y-{\hat {f}})^{2}{\big ]}
&=\operatorname {E} {\big [}(f+\varepsilon -{\hat {f}})^{2}{\big ]}\\[5pt]
&=\operatorname {E} {\big [}(f+\varepsilon -{\hat {f}}+\operatorname {E} [{\hat {f}}]-\operatorname {E} [{\hat {f}}])^{2}{\big ]}\\[5pt]
&=\operatorname {E} {\big [}(f-\operatorname {E} [{\hat {f}}])^{2}{\big ]}+\operatorname {E} [\varepsilon ^{2}]+\operatorname {E} {\big [}(\operatorname {E} [{\hat {f}}]-{\hat {f}})^{2}{\big ]}+2\operatorname {E} {\big [}(f-\operatorname {E} [{\hat {f}}])\varepsilon {\big ]}+2\operatorname {E} {\big [}\varepsilon (\operatorname {E} [{\hat {f}}]-{\hat {f}}){\big ]}+2\operatorname {E} {\big [}(\operatorname {E} [{\hat {f}}]-{\hat {f}})(f-\operatorname {E} [{\hat {f}}]){\big ]}\\[5pt]
&=(f-\operatorname {E} [{\hat {f}}])^{2}+\operatorname {E} [\varepsilon ^{2}]+\operatorname {E} {\big [}(\operatorname {E} [{\hat {f}}]-{\hat {f}})^{2}{\big ]}+2(f-\operatorname {E} [{\hat {f}}])\operatorname {E} [\varepsilon ]+2\operatorname {E} [\varepsilon ]\operatorname {E} {\big [}\operatorname {E} [{\hat {f}}]-{\hat {f}}{\big ]}+2\operatorname {E} {\big [}\operatorname {E} [{\hat {f}}]-{\hat {f}}{\big ]}(f-\operatorname {E} [{\hat {f}}])\\[5pt]
&=(f-\operatorname {E} [{\hat {f}}])^{2}+\operatorname {E} [\varepsilon ^{2}]+\operatorname {E} {\big [}(\operatorname {E} [{\hat {f}}]-{\hat {f}})^{2}{\big ]}\\[5pt]
&=(f-\operatorname {E} [{\hat {f}}])^{2}+\operatorname {Var} [\varepsilon ]+\operatorname {Var} {\big [}{\hat {f}}{\big ]}\\[5pt]&=\operatorname {Bias} [{\hat {f}}]^{2}+\operatorname {Var} [\varepsilon ]+\operatorname {Var} {\big [}{\hat {f}}{\big ]}\\[5pt].
\end{aligned}}}$
Why MSE? It is a smooth suitable metric for the accuracy which has nice analytic properties. Other types of error are also possible.
Further reading (and derivations and references therein):

Bias of an estimator
Bias–variance tradeoff
Derivation of the Bias-Variance Tradeoff


If i no longer have access to feature in a data set should i retrain my model?

Naive Bayes, as far as I know, does not have any internal method of dealing with missing data (which is what you have). Thus, there are really two options:

1) Retrain the model, discarding the variable that contains the is author known variable. This method will probably make your model worse if the is author known variable is important, but at least you won't have any missing data to deal with.

2) Impute the missing values using some sort of algorithm, like MICE, mode imputation, knn, random forest, etc. For this problem, just from a first glance there might be correlations between x1 and your other variables that you can exploit to make a good guess for what x1 would be, given the other known values. This introduces more uncertainty in your model and might even make it worse then simply dropping x1 outright (option 1), but it is worth a shot if x1 is an important predictor.


Filters are used to extract features from images in the process of convolution.

  filters: Integer, the dimensionality of the output space (i.e. the
  number of output filters in the convolution).

  kernel_size: An integer or tuple/list of 2 integers, specifying the
  height and width of the 2D convolution window. Can be a single integer
  to specify the same value for all spatial dimensions.

For detailed undestadning of filter and kernel_size, refer:
https://www.saama.com/blog/different-kinds-convolutional-filters/
https://keras.io/layers/convolutional/


Is there a way in pandas to import NA fields as a string rather than NaN?

Use the pandas.read_csv() options:

d = pandas.read_csv('foo.csv', keep_default_na=False)

  na_values : scalar, str, list-like, or dict, default None

  Additional strings to recognize as NA/NaN. If dict passed, specific >per->column NA values. By default the following values are interpreted as >NaN: ‘’, ‘#N/A’, ‘#N/A N/A’, ‘#NA’, ‘-1.#IND’, ‘-1.#QNAN’, ‘-NaN’, ‘-nan’, >‘1.#IND’, ‘1.#QNAN’, ‘N/A’, ‘NA’, ‘NULL’, ‘NaN’, ‘n/a’, ‘nan’, ‘null’.

  keep_default_na : bool, default True

  If na_values are specified and keep_default_na is False the default NaN >values are overridden, otherwise they’re appended to.

  from here


Convolution Neural Network Loss and performance

Precision

Your step change in precision looks to be almost entirely explained by the change in positive class frequency. It is reasonable to expect the proportion of false positives to increase when increasing the proportion of negative examples. Even if you assume your cv results were perfect, then you would see some increase.

As an example, assume you have cv results representative of test results
 - which means same distribution before random under-sampling, and no over-fit to the cv set.

Say you measured precision at 0.97 with a t:f ratio of 1:2, and for the sake of simplicity that this was due to the following confusion table:

      Predicted:  T    F
Real T           97    3
Real F            3  197

What precision should you expect when going to the real distribution? That is the same as multiplying the bottom row of the confusion table by 50. Precision is $\frac{TP}{TP+FP}$, so your expected precision would be $\frac{97}{97+150} \approx 0.39$

Recall

The same effect does not impact recall, because it is about the ratio between true positive and false negative. So when you change the ratio of positive to negative classes, in theory recall should be unaffected.

In your case, recall has been affected, but a lot less than precision. that is promising. A drop from 0.95 to 0.85 between cv and test is not great perhaps, but it doesn't point to a really major problem, just room for improvement.

There are a few possible causes. The ones that I can think of are:

Your test set might be too small, so estimates of precision and recall have large error. So in fact there is no problem . . .
Test distribution might be different to train and cv set.
Train/CV set split might allow some data leakage (e.g. they share some common features such as data about the same person, and should be split by that common feature). In which case CV estimates could be too high.
Your mechanism for under-sampling the negative class may be biased.

What to do?

First of all, these results are unlikely to be anything directly do with faults in the model, and are not informed much by the training curves. They are also not that bad out of context (i.e. they are much better than simply guessing which items are in the positive class) - the question is more whether you could improve on them, and what the costs are to you for the different types of error. It might be worth you actually assigning real-world comparable costs to each type of error, to help decide whether your model is successful/useful and to pick the best model later on.

One thing from the training curves is that your cv and training loss look pretty close. It implies you are not over-fitting to the training data (or you should check to a train/cv data leak). You may have room to add more parameters and improve the model in general.

It is possible you could make the model even better with different hyper-parameter choices, feature engineering etc, and that would improve the scores. There is no general advice for that though, it depends on what you have available.

It might be worth experimenting with training on the unbalanced training set (take the raw data without undersampling) and instead weighting the loss function, so that costs are larger for inaccurate classification of positive class. This is not guaranteed to fix your problem, but will increase the amount of data you use for training.

Otherwise, you should investigate whether any of the possible causes listed above is likely and try to apply fixes.

Finally, in this situation, it is not unheard of to have a four-way data split:

A ratio-adjusted set split two ways:

Training data
CV or "Dev" set A

A same as production set split two ways:

CV or "Dev" set B
Test set

CV set A is used to perform early stopping and low-level model selection.

CV set B is used to perform high-level model selection against production metric.

Test set is used to assess the chosen "best" model without bias.


Should I choose cluster based / density based approach?

Yes, but depends on how good the algorithm is, and how effectively is the algorithm identifying important clusters. There is a lot of literature where people have taken the clustering approach.

If I were to go with the clustering approach, I would go with the hierarchical clustering technique, cause it would help me re-check my clusters and build hierarchies in them so that the seriousness of the fraud can also be explored.

  How do I find rare & suspicious events that parents should be notified
  for with the help of machine learning ?

I would point out an answer which I have written for a very similar question, which also explores the aspect of fraud detection and anomaly analysis. And as I have said in the answer, the genetic algorithm still goes strong when it comes to fraud detection, owing to it's neat method of fitness calculation of the data points.

  Any suggestions on how to present the results visually ?

Generally such charts are presented through pie charts and network graphs. you can also use infographics if you have a nice idea about design and aesthetics.

  Is a week of data enough for training ?

Generally, data science teams at Paypal, Palantir, etc churn years of data for sensible insights. As you say it's a class project, data of several months should do good. You wouldn't really get anything suspicious or sensible from a week of data.

  Are there any good open sources in Java to detect anomaly ?

Yes, there are some:

Genetic algorithms library
A Distributed Online Machine Learning Framework in java


When projecting data with UMAP, should I use only the samples I need projected or the entire dataset?

Yes, more data will improve the quality of the embedding UMAP can produce. While UMAP is somewhat robust/stable under subsampling in general you will get significantly better results with more data. It is also worth noting that most UMAP implementations are not designed for very small datasets (they make some optimization choices that assume a a reasonable dataset size). In practice it is probably best not to use UMAP with less than 100 or so data samples.


Reshaping big dataset with MinMaxScaler giving error

I assume you checked for NaN and Inf values manually.

According to the solutions posted here: https://stackoverflow.com/a/44869902/6204860 and
https://datascience.stackexchange.com/a/11933/52089

if dataset1 is a Pandas DataFrame try converting it to a matrix by running this:

dataset1 = dataset1.as_matrix().astype(np.float)
np.nan_to_num(dataset1)

and then run dataset1 = scaler.fit_transform(dataset1).

Let me know if this works :)


One way to approach it is to split the sentence into tokens and count the number of tokens that are profanities.

import re

def tokenize(text):
    return re.findall(r'\w+', text.lower())

profane_tokens = {"nerfherder"}

sentence = "Why you stuck-up, half-witted, scruffy-looking nerfherder!"

tokens = tokenize(sentence)

# Rate: number of occurrences normalized by total number
degree_of_profanity = sum(1 for t in tokens if t in profane) / len(tokens)

This code will not handle multiple tokens and many profanities are multiple tokens.


I don't know much about R2 score, but having it negative all the time seems pretty strange to me. Maybe you should try to use AUC as a metric ( <0.5 classifier is worse than a random classifier, and the closer you're to 1, the best is your algorithm).
If it appears that you still can't find a model giving decent results, the direct conclusion is not that your features are not enough, because there can be plenty of other reasons.
I'd suggest you to try SMOTE, which will create new data based on the ones you already have, and try applying your models again. Sometimes, this is a way to tackle the issue of not having enough data


Is it necessary to standardize your data before clustering?

Normalization is not always required, but it rarely hurts.

Some examples:

K-means:

  K-means clustering is "isotropic" in all directions of space and
  therefore tends to produce more or less round (rather than elongated)
  clusters. In this situation leaving variances unequal is equivalent to
  putting more weight on variables with smaller variance.

Example in Matlab:

X = [randn(100,2)+ones(100,2);...
     randn(100,2)-ones(100,2)];

% Introduce denormalization
% X(:, 2) = X(:, 2) * 1000 + 500;

opts = statset('Display','final');

[idx,ctrs] = kmeans(X,2,...
                    'Distance','city',...
                    'Replicates',5,...
                    'Options',opts);

plot(X(idx==1,1),X(idx==1,2),'r.','MarkerSize',12)
hold on
plot(X(idx==2,1),X(idx==2,2),'b.','MarkerSize',12)
plot(ctrs(:,1),ctrs(:,2),'kx',...
     'MarkerSize',12,'LineWidth',2)
plot(ctrs(:,1),ctrs(:,2),'ko',...
     'MarkerSize',12,'LineWidth',2)
legend('Cluster 1','Cluster 2','Centroids',...
       'Location','NW')
title('K-means with normalization')

(FYI: How can I detect if my dataset is clustered or unclustered (i.e. forming one single cluster)

Distributed clustering:

  The comparative analysis shows that the distributed clustering results
  depend on the type of normalization procedure.

Artificial neural network (inputs):

  If the input variables are combined linearly, as in an MLP, then it is
  rarely strictly necessary to standardize the inputs, at least in
  theory. The reason is that any rescaling of an input vector can be
  effectively undone by changing the corresponding weights and biases,
  leaving you with the exact same outputs as you had before. However,
  there are a variety of practical reasons why standardizing the inputs
  can make training faster and reduce the chances of getting stuck in
  local optima. Also, weight decay and Bayesian estimation can be done
  more conveniently with standardized inputs.

Artificial neural network (inputs/outputs)

  Should you do any of these things to your data? The answer is, it
  depends.

  Standardizing either input or target variables tends to make the training
  process better behaved by improving the numerical condition (see
  ftp://ftp.sas.com/pub/neural/illcond/illcond.html) of the optimization
  problem and ensuring that various default values involved in
  initialization and termination are appropriate. Standardizing targets
  can also affect the objective function.

  Standardization of cases should be approached with caution because it
  discards information. If that information is irrelevant, then
  standardizing cases can be quite helpful. If that information is
  important, then standardizing cases can be disastrous.

Interestingly, changing the measurement units may even lead one to see a very different clustering structure: Kaufman, Leonard, and Peter J. Rousseeuw.. "Finding groups in data: An introduction to cluster analysis." (2005).

  In some applications, changing the measurement units may even lead one
  to see a very different clustering structure. For example, the age (in
  years) and height (in centimeters) of four imaginary people are given
  in Table 3 and plotted in Figure 3. It appears that {A, B ) and { C,
  0) are two well-separated clusters. On the other hand, when height is
  expressed in feet one obtains Table 4 and Figure 4, where the obvious
  clusters are now {A, C} and { B, D}. This partition is completely
  different from the first because each subject has received another
  companion. (Figure 4 would have been flattened even more if age had
  been measured in days.)

  To avoid this dependence on the choice of measurement units, one has
  the option of  standardizing the data. This converts the original
  measurements to unitless variables.

Kaufman et al. continues with some interesting considerations (page 11):

  From a philosophical point of view, standardization does not really
  solve the problem. Indeed, the choice of measurement units gives rise
  to relative weights of the variables. Expressing a variable in smaller
  units will lead to a larger range for that variable, which will then
  have a large effect on the resulting structure. On the other hand, by
  standardizing one attempts to give all variables an equal weight, in
  the hope of achieving objectivity. As such, it may be used by a
  practitioner who possesses no prior knowledge. However, it may well be
  that some variables are intrinsically more important than others in a
  particular application, and then the assignment of weights should be
  based on subject-matter knowledge (see, e.g., Abrahamowicz, 1985). On
  the other hand, there have been attempts to devise clustering
  techniques that are independent of the scale of the variables
  (Friedman and Rubin, 1967). The proposal of Hardy and Rasson (1982) is
  to search for a partition that minimizes the total volume of the
  convex hulls of the clusters. In principle such a method is invariant
  with respect to linear transformations of the data, but unfortunately
  no algorithm exists for its implementation (except for an
  approximation that is restricted to two dimensions). Therefore, the
  dilemma of standardization appears unavoidable at present and the
  programs described in this book leave the choice up to the user.


Can metadata be used to adapt parsing for an unescaped in field use of the delimiter?

So, a few of your rows will have too many columns by one or more as a result. That's easy to detect, but harder to infer where the error was -- which two columns are actually one? which delimiter is not a delimiter?

In some cases, you can use the metadata, because it helps you know when an interpretation of the columns can't be right. For example, if just the one column can have a text value, and all the others must be numeric, it's unambiguous where the error is. Any additional columns created by this error occur right after the text column.

If they're all text, this doesn't work of course.

You might be able to leverage more than the metadata's column type. For example you may know that some fields are from an enumerated set of values, and use that to determine when a column assignment is wrong.


A Loss of 55.2164 with a sparse_categorical_crossentropy in a sequential neural network?

The reason you are getting a very high loss has to do with the normalization you are performing. You are normalizing both the X_train and X_valid, however you do not normalize X_valid. Adding normalization of X_valid to your code the final loss for X_valid is much lower.

fashion_mnist = keras.datasets.fashion_mnist

(X_train_full,y_train_full), (X_test,y_test) = fashion_mnist.load_data()

X_valid , X_train = X_train_full[:5000]/255.0 , X_train_full[5000:]/255.0
y_valid , y_train = y_train_full[:5000] , y_train_full[5000:]

# Normalizing X_test as well
X_test = X_test/255.0

model = keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[28,28]))
model.add(keras.layers.Dense(300,activation="relu"))
model.add(keras.layers.Dense(100,activation="relu"))
model.add(keras.layers.Dense(10,activation="softmax"))

model.compile(loss="sparse_categorical_crossentropy",optimizer="sgd",metrics=["accuracy"])
history = model.fit(X_train,y_train,epochs=30,validation_data=(X_valid,y_valid))

model.evaluate(X_test,y_test)
# [0.328696608543396, 0.8866999745368958]


Where does the sum of squared errors function in neural networks come from?

you can trace the squared error in statistics through multivariate calculus all the way to Pythagorus. You are basically calculating the 'effective length' of the error, the hypotenuse, among errors from multiple variables $(X_1 - X_2)^2 + (Y_1 - Y_2)^2 + ...$ like in a triangle.

But where did the square root go?

Somebody realized that calculating roots of multiple variables over multiple iterations is computationally very expensive. So they decided to drop it. Checkout the squared Euclidean distance here for more details

How would a cubic error or a logarithmic error affect the outcome?

It just takes more time to converge because they are not as accurate. But we do see logarithmic errors over squares such as logistic regression where it is more optimal

All in all it is a simple case of optimization


Creating pronunciation dictionary for ASR

I had posted a similar question in reddit as well and I got a response from Nikolay Shmyrev :

"If you want to convert latin script, you can write simple rules yourself. Something like this.  Or you can use epitran as is. "

Thanks to Nikolay Shmyrev who originally answered in reddit.


Text classification analysis based on similarity

I shall give you some hints here.

It depends on your data, my advice is that you try several algorithms.

If you have binary classification you can use both classification (try Logistic regression, SVM, RandomForest) or regression (try XGBoost or any gradient boosting algorithm, there are several in scikit learn), and then seek for a boudary in order to decide wheter a value is 0 or 1. Depending on your data, it might work very well (I've got 0.8 fscore on text classification using RandomForest and performing an important pre-processing before). Do not forget to do an appropriate preprocessing. Concerning pre-processing, in scikit-learn you have CountVectorizer and TfIdfVectorizer, very useful.

Do not use Naive Bayes, not on text and NLP.

Well, the best method for that is using Sequence models (especially LSTM), parsing text as an succession of words or characters. You can use in this case Sequences available in Keras.


Decision tree classifier: possible overfitting

Perfect training AUC is the hallmark of overfitting. When searching parameters, I often use a combination of test set AUC and the difference between training AUC and test AUC. When just using test set AUC in the search loss function, I found the model to still be pretty sensitive to which random set of test rows it sampled.

I also agree with the commenters that you should get better results with much smaller max_depth, probably on the order of 1-10, and ensemble models. Random forests are an excellent place to start.


Python sklearn model.predict() gives me different results depending on the amount of data

If XGBoostClassifier is fed the same input data over and over again it will yield the same results. There is no inherent randomness in this classifier that would different results for the same input. Additionally - there should be no difference in the result of an individual prediction if it's requested in a smaller batch versus a larger batch (again the result will be identical).
On the other hand - if you train XGBoost on different data their outputs will definitely be different. If you add new data to the underlying dataset & train with that - new & different patterns will emerge that XGBoost will try to take advantage of and the entire tree network will be fit very differently.
I suspect what you are observing is a bug in structuring your input data that you are then feeding to the .predict() method. If you share a sample of your code maybe we can drill-down on the issue.


How to reduce an unknown size data into a fixed size data? Please read details

My approach would be using the Moments.
Using moments is a cool way trowing any arbitrary set of points into space, and it is used for both probability and mass calculations.
Since you have not stated what are you going to do with this vector representation I guess that a generalized "shape" like representation might work for you.

If you are using Python, then this can help.


Choosing between TensorFlow or Theano as backend for Keras

It was announced on  2017-09-28 that Theano will be discontinued:
From here (Yoshua Bengio):

After almost ten years of development, we have the regret to announce that we will put an end to our Theano development after the 1.0 release,  which is due in the next few weeks. We will continue minimal maintenance  to keep it working for one year, but we will stop actively implementing  new features. Theano will continue to be available afterwards, as per  our engagement towards open source software, but MILA does not commit to  spend time on maintenance or support after that time frame.

So TensorFlow is a better option.


I am trying to make a Classifier using Machine learning to detect malwares,am I headed in the right direction?

It is not clear in your post whether or not you are using training/test sets so I will answer in this way:

The first thing to do if you want to validate your results is to cut your set into a training set and a validation set. This way you train the K-NN method on your training set, and you use the trained classifier on the validation set. Then you can monitor the validation error, and the training error.

In the best case scenario, it should look like this :

The aim is to obtain a model that generalizes well, which means being able to accurately predict the class a sample belongs to, when it has never seen the sample before.


How to create a multi label classification network in Keras if I have the training data with various accuracy?

One approach is to have a dummy class that represents no treatment and use the accuracy score via a threshold (lower than threshold) correspond to no treatment at all.
Threshold as used above becomes a new hyper-parameter and you have new input-output pairs that are now exact (depending on threshold).


is validation and train set should be all different files?

Validation set is basically to understand how your model is behaving in terms of over-fitting, and under-fitting, and also to find the best set of hyper-parameters for your algorithm. If you use some parts of the same training data on your validation set, then this hypothesis would not be justified.
Hence, it is suggested to split your data set into train/validation/test set without any overlapping of the data.


Image classification vs medical grading problems

The number of outputs will depend on the task and the number of targets. In the case of medical grading, you can either interpret the problem as:

a classification i.e. each grade is a class (5 outputs, one for each class)
a regression: predict a grade and round the result (1 output)

Make sure to check the loss function for each task. That will usually tell you how the code writer decided to approach the problem. To come back to your example, the first task should have a classification loss (crossentropy for instance) and the second a regression (MSE). Also check for the last layer activation function: usually softmax/sigmoid for classification and None for regression.
It's complicated to say what would work without seeing the data. A recent competition (Janestreet Market predicition) also saw people approach the problem both as a classification and a regression. The highest rated public notebook on the private set so far is a regression-based approach.
I suspect that this happened because the classification (of whether a stock is going to have a positive return in the future) requires to binarize the labels which basically lead to information loss.


Multiclass classification of textual data

300 data points is enough to give good/average results with the traditional machine learning algorithms. You should test SVMs as they are considered good for text classification.

However, deep learning is very often used when you have a lot of data, which is not the case here. So I would not recommend it for your task.


How do I control for some patients providing multiple samples in my training data?

Like all models, the final outcome will depend heavily on the data you use to train.

Given you don't have many samples for each patient and that the samples are temporally irregular, I imagine you might be lumping all recorded data (from all patients) into a single training set... ? In that case, it will be important to either perform random sampling or something like stratified sampling - both without replacement - to make sure patients don't appear in both train and test sets. Stratification simply means to create training/test splits, which do their best to preserve class balance. So if you select one feature as most important, stratification will make sure that training and test sets each have a proportional number of samples exhibiting that feature.

If you are using R, look into the Caret package - here is an example of usage. That is a very powerful package in general and worth investigating (if you use R or not!).
If you are using Python, have a look at the Splitter Classes, which include: Stratified K-folds and Stratified Shuffle Split, among others.

If temporal ordering does play a big role, taking the most recent sample is usually considered best practice. For example, measurements driven by general information flow, e.g. finance, would also consider the most recent measurement as the most valueable for prediction.

This is the case in many applications, but perhaps not with your measurements - in medical data, behavioural/environmental conditions of the patient around the time of measuring may outweigh the recentness of the data.

One more observation, in the you are trying to predict some feature that is very specific to a single patient (and you say you may only have one observation from each patient!), then you might have a hard time making accurate predictions.


TypeError: Expected binary or unicode string, got [

You should use a built-in function for train_input_fn instead of writing your own. For example:

train_input_fn = tf.estimator.inputs.numpy_input_fn(
    x={"x": np.array(training_set.data)},
    y=np.array(training_set.target),
    num_epochs=None,
    shuffle=True)

There is a complete example in the TensorFlow documentation here.


Alternative distance to Dynamic Time Warping

Like suggested in one answer on this SO question, you could use elastic matching with Levenshtein distance to your task. Levenshtein distance obeys triangle inequality and is therefore a metric distance.

Use of elastic matching was suggested for time series data comparison. Levenshtein distance works with characters data.

There is an implementation of elastic matching and Levenshtein distance calculation in Python.

To put them together you most probably need to build your own implementation.


Unsupervised encoding of categorical features

High cardinality is a common issue that we run into when doing encoding for categorical variables. One hot encoding and label encoding are the most common approaches however label encoding maynot be ideal if the variable is not nominal (not inherently hierarchical) and one hot encoding will lead to curse of dimensionality when no of categories are high. REplacing with ratio or looking at spectral encoding is a good approach to circumvent this

Details of spectral encoding are :-
https://towardsdatascience.com/spectral-encoding-of-categorical-features-b4faebdf4a


There are a couple of open source options I know of -

LibOTS - http://libots.sourceforge.net/

DocSum - http://docsum.sourceforge.net/docsum/web/about.php

A couple of commercial solutions -

Intellix Summarizer Pro - http://summarizer.intellexer.com/order_summarizer_pro.php

Copernic Summarizer - http://www.copernic.com/en/products/summarizer/

And this one is a web service -

TextTeaser - http://www.textteaser.com/

I'm sure there are plenty of others out there.  I have used Copernic a good deal and it's pretty good, but I was hoping it could be automated easily, which it can't - at least it couldn't when I used it.


Human activity recognition using smartphone data set problem

The data set definitions are on the page here:

Attribute Information at the bottom

or you can see inside the ZIP folder the file named activity_labels, that has your column headings inside of it, make sure you read the README carefully, it has some good info in it. You can easily bring in a .csv file in R using the read.csv command.

For example if you name you file samsungdata you can open R and run this command:

data <- read.csv("directory/where/file/is/located/samsungdata.csv", header = TRUE)

Or if you are already inside of the working directory in R you can just run the following

data <- read.csv("samsungdata.csv", header = TRUE)

Where the name data can be changed to whatever you want to call your data set.


Padding in Keras with output half sized input

If you goal is simply to halve the size of your filters, you could think about using some different methods other than padding, such as dilated convolutions. Have a look at this paper for some ideas and nice explanations with pictures. Just thinking about your dimensions quickly, I am not sure you could go from 14 to 7 very easily. getting to 8 or 5 is simple enough though.

One thing that isn't always obvious if you just started learning KEras, is that you can mix in Tensorflow operations directly from the Tensorflow library in with your Keras code.

In TF there is a function called pad, which allows you to specify the padding manually on each side of a tensor. There are also options to say whether the padding is done with zeros, or if the values inside your original tensor are repeated/mirrored (using the mode argument).

You could try using this to pad the layers. I can show you how to pad the tensors to get the effect you want:

from keras.models import Sequential, Model
from keras.layers import (Input, Conv2D, MaxPooling2D,
                          Flatten, Dense, Reshape, Lambda)
import tensorflow as tf

input_shape = (448, 448, 3)
batch_shape = (None,) + input_shape
raw_input = tf.placeholder(dtype='float16', shape=batch_shape)
paddings = tf.constant([[0, 0],   # the batch size dimension
                        [3, 3],   # top and bottom of image
                        [3, 3],   # left and right
                        [0, 0]])  # the channels dimension

padded_input = tf.pad(raw_input, paddings, mode='CONSTANT',
                      constant_values=0.0)  # pads with 0 by default

padded_shape = padded_input.shape     # (454, 454, 3) because we add 2*3 padding

input_layer = Input(padded_shape, batch_shape, tensor=padded_input)
layer0 = Conv2D(192, kernel_size = (3,3), padding='valid')(input_layer)
layer1 = MaxPooling2D(pool_size=(2, 2), strides = 2)(layer0)
layer2 = Conv2D(192, kernel_size = (3,3), padding='valid')(layer1)
layer3 = MaxPooling2D(pool_size = (2,2),strides = 2)(layer2)
layer4 = Conv2D(512, kernel_size = (3,3), padding='valid')(layer3)
layer5 = MaxPooling2D(pool_size = (2,2), strides = 2)(layer4)
layer6 = Conv2D(256, kernel_size = (1,1), padding='valid')(layer5)
# layer6.shape --> [Dimension(None), Dimension(55), Dimension(55), Dimension(256)]

layer6_padded = tf.pad(layer6, paddings, mode='CONSTANT')
layer6_output = Input(input_shape=layer6_padded.shape)(layer6_padded)

# This will end up giving this error at compilation:
# RuntimeError: Graph disconnected: ...,

layer7 = Conv2D(1024, kernel_size=(3, 3), strides=2)(layer6_output)
layer8 = Flatten()(layer7)
layer9 = Dense(4096)(layer7)
layer10 = Dense(7*7*30)(layer8)
output_layer = Reshape((7, 7, 30))(layer10)

# The following both fail to get the graph as we would like it
model = Model(inputs=[input_layer], outputs=[output_layer])
#model = Model(inputs=[input_layer, layer6_output], outputs=[output_layer])

model.compile('adam', 'mse')
model.summary()

I have been unable to then bring this tensor back into the Keras model (as a layer, which is required) because the standard way of using the Input object forces it to be the entry point of the computational graph, but we want the padded tensors to form an intermediary layer.

If you don't force the padded tensors into a Keras layer, attributes will be missing:

# AttributeError: 'Tensor' object has no attribute '_keras_history'

Which you can hack by just adding the attribute from the layer before we padded:

#layer6_output._keras_history = layer6._keras_history

Unfortunately, I still ran into other errors.

Perhaps you can post a new question on StackOverflow asking how to to this, if you can find anything. I did have a quick try using the idea of creating two graphs and then joining them, but didn't succeed.


WEKA Random Forest and numerical attributes

Weka does use its own implementation of C4.5, named J48.

That said, its RandomForest calls on RandomTree, which appears to be implemented independently, not ever touching the J48 code.  (That's from what I can tell, and I don't normally work in Java so take it with a grain of salt.)  I can't tell from a glance whether there are substantial differences aside from the obvious (that RandomTree subsets features available to split on at each node).

Class hierarchy of module tree
tree folder of source
(click through to see J48, RandomForest, RandomTree, etc.)


Unblanced classes: classifier only predict one class

I found in the explanations that it "considers each class as important as the other" but still don't understand how. Does it duplicate rows of the minor categories?

No, it adds a weight to each example, depending on its class. The majority class will be assigned a small weight while the minority ones will be assigned larger weights. This weight are considered during the training phase of your model, so that each example of a minority class impacts the parameter updates more than one from the majority class.

For example, suppose class 1 has a weight of $0.5$, class 2 has a weight of $1.2$ and class 3 has a weight of $1.5$. Each example from class 3 will impact the parameter updates 3 times more than an example from class 1, etc.

You can see what weights scikit-learn has selected through this function. You can also select your own weights (if you feel that some class should be more important than what scikit-learn has selected) by modifying the dictionary.

  When I used that option, it was obvious that the classifier were predicting other Mark classes more frequently, but there are more number of right predictions than the wrong ones.

I'm not sure what the question is, but if you are asking "why do I have more misses now than before?".

Because you used that option, your model has learned to classify examples while treating the classes of equal importance. But the classes don't have the same frequencies. For example if you miss-classify 1 out of 4 examples, you would miss more predictions from the majority class than from the minority ones.

Is this good or bad?

Depending on your goal.

If you want to simply maximize your algorithm's correct predictions, then you might get a better result without balancing your classes. In this case you should use a metric like accuracy.

If your goal something else, for instance to have the same accuracy in each class, you should select a more appropriate metric (e.g. a macro-averaged one).

Is there an alternative to class_weights?

Yes, in most cases it is preferable to balance the number of samples in your classes. This is done by either over-sampling the minority classes, under-sampling the majority ones, or a combination of both.

If you want to try out this approach, I'd suggest using imbalanced-learn.


Strange as it sounds, graphs and graph databases are typically implemented as linked lists. As alluded to here, even the most popular/performant graph database out there (neo4j), is secretly using something akin to a doubly-linked list.

Representing a graph this way has a number of significant benefits, but also a few drawbacks. Firstly, representing a graph this way means that you can do edge-based insertions in near-constant time. Secondly, this means that traversing the graph can happen extremely quickly, if we're only looking to either step up or down a linked list.

The biggest drawback of this though comes from something sometimes called The Justin Bieber Effect, where nodes with a large number of connections tend to be extremely slow to evaluate. Imagine having to traverse a million semi-redundant links every time someone linked to Justin Bieber.

I know that the awesome folks over at Neo4j are working on the second problem, but I'm not sure how they're going about it, or how much success they've had.


I wouldn't say your model is overfitting, for two reasons:

The validation AUC is extremely high, I don't think many models end up with ~0.98 AUC in validation. This essentially means that the model is perfectly learning the task given, with the possibility of some exceptions. These exceptions might be always a thing the same, for each training experiment you've done, and for this reason you see a consistent drop of performance.
Whenever you train a model, it is normal for the validation performance to be slightly lower than the training performance, and that doesn't necessarily mean the model is overfitting, especially when the difference is around 1/1000 of AUC. I think this question might help you understand, for instance. I would say a model overfits if there is a significant drop, which I don't see here.


It might not be directly possible to shoehorn the output of your CNN directly into an LSTM (at least without being a lot more thorough with your dimensions).

Another approach is to have a look at the Keras wrapper layer: TimeDistributed. You essentially extract features using your Conv layers as usual, but do that over time-steps, not just a random mini-batch. The features the come out then are e.g. over 5 consecutive timeframes -> this would be one single sample for the LSTM. If you perform a minibatch of say 10 samples, this means 10 * 5 = 50 input frames.

Straight from the linked documentation:

  This wrapper applies a layer to every temporal slice of an input.

  The input should be at least 3D, and the dimension of index one will
  be considered to be the temporal dimension.

  Consider a batch of 32 samples, where each sample is a sequence of 10
  vectors of 16 dimensions. The batch input shape of the layer is then
  (32, 10, 16), and the input_shape, not including the samples
  dimension, is  (10, 16).

  You can then use TimeDistributed to apply a Dense layer to each of the
  10 timesteps, independently:

# as the first layer in a model
model = Sequential()
model.add(TimeDistributed(Dense(8), input_shape=(10, 16)))
# now model.output_shape == (None, 10, 8)

As a hint for dimensions, make use of model.summary() on a compiled model in order to inspect the dimensions of each layer through the model. You will likely have to get a pen and paper out to work through the dimensions, as things start to become a little complicated when using such a wrapper layer.

Unfortunately I don't have any links/tutorials to show how this is exactly done in Keras, but hopefully the description above gives you enough info for some useful Bing! searches. Just kiddin... Google searches  ;-)


Any research on relationship between the dimensions of a (word2Vec) space and how the human mind constructs meaning (or reality) through language?

Some initial steps where taken here:
Connecting concepts in the brain by mapping cortical representations of semantic relations

To represent words as vectors, we used a pretrained word2vec model21.
Briefly, this model was a shallow neural network trained to predict
the neighboring words of every word in the Google News dataset,
including about 100 billion words
(https://code.google.com/archive/p/word2vec/). After training, the
model was able to convert any English word to a vector embedded in a
300-dimensional semantic space (extracted through software package
Gensim56 in python). Note that the basis functions learned with
word2vec should not be interpreted individually, but collectively as a
space.
Applying the encoding model to the differential vector of a word pair
could effectively generate the cortical representation of the
corresponding word relation. With this notion, we used the encoding
model to predict the cortical representations of semantic relations.
For each class of semantic relation, we calculated the relation vector
of every word pair in that class, projected the relation vector onto
the cortex using the encoding model, and averaged the projected
patterns across word-pair samples in the class.


how to use tensorflow graphs in multithread？valueerror：tensor a must be from the same graph as tensor b！

TensorFlow stores all operations on an operational graph. This graph defines what functions output to where, and it links it all together so that it can follow the steps you have set up in the graph to produce your final output. If you try to input a Tensor or operation on one graph into a Tensor or operation on another graph it will fail. Everything must be on the same execution graph.

Try to use

tf.reset_default_graph()

to reset the tf graph once it executes.

For keras use K.clear_session().

For more details about the keras and tensorflow interface view Keras documentation blog.https://blog.keras.io/keras-as-a-simplified-interface-to-tensorflow-tutorial.html this helps you a lot.


How to compare two clustering solutions when their labelling differs

This is only a partial answer since I'm not familiar with HDBSCAN, hopefully somebody else can provide a more complete answer.
As far as I understand you need to find which cluster in A corresponds to which cluster in B, i.e. an alignment between the clusters labels of A and of B. It's not recommended to match based only on the size, since it could happen by chance that a cluster in A has the same size (or similar size) as a cluster in B.
Since the instances are different, you would have to rely on how the method represents the clusters.

For example probabilistic clustering methods represent each cluster as a distribution over the features, so one can use a distance/similarity measure between these distributions.
With k-means one would compare the centroids and match the pairs of clusters for which the distance is the shortest.

I'm not familiar with HDBSCAN so I don't know how the clusters are represented inside the model: whatever this is, the idea would be to compare each internal representation of a cluster in A vs. the same in B.


Interpreting lasso logistic regression feature coefficients in multiclass problem

Welcome to the site!

I think you are right that feature weights = 0 across all three classifiers mean that the features are not predictive of the outcome.

In terms of comparing magnitudes, I'm not sure the absolute values are really comparable - each model is trying to predict a different target.

However, you could rank the coefficients in some way and then compare the rank of the coefficient across the three models. You would need to choose a robust way to rank the raw coefficients since they would range from positive to negative - e.g. a strongly negative coefficient such as -2 is probably more predictive than a barely positive coefficient of 0.1.


When to use BayesianSearchCV and how it works?

As mentioned in that Kaggle notebook, you can use it pretty much as just a drop-in replacement for other search methods (grid or random).  Bayesian searches still are random searches over a predefined search space/distribution, but now the algorithm pays attention to how well hyperparameter combinations perform, and will put more emphasis on high-performing areas.

There are several ways to do this.  In BayesianSearchCV, a surrogate function keeps track of how the algorithm thinks hyperparameters will perform, with some uncertainty.  Yet another function, an acquisition function, balances the desire to exploit high-performing areas with that to decrease uncertainty about unexplored areas (to avoid falling into a local optimum).  A couple of nice descriptions:
http://neupy.com/2016/12/17/hyperparameter_optimization_for_neural_networks.html#bayesian-optimization
http://krasserm.github.io/2018/03/21/bayesian-optimization/

SciKit-Opt is specifically tailored for optimizing very expensive functions; it spends a fair amount of time updating its surrogate and acquisition functions and choosing the next hyperparameter combination to test.  By doing so, it generally requires fewer iterations to get good results, but spends longer per iteration.  This is probably a good tradeoff for, say, neural networks, but less clearly for XGBoost.  But if you really need to eke out the last ounce of performance, it might be worth doing Bayesian over a purely random search.
https://github.com/scikit-optimize/scikit-optimize/issues/334#issuecomment-290358126
http://krasserm.github.io/2018/03/21/bayesian-optimization/
https://scikit-optimize.github.io/notebooks/bayesian-optimization.html

(The search begins with a number of randomly selected hyperparameter combinations, defaulting to 10 in skopt.  So in that version of the Kaggle notebook, actually just a random search is being performed.)


1) Manual labeling--- this is not as bad as it sounds. Especially when you apply transfer learning, and for most of the datasets you have a lot of pre-trained models. There are products for that, but also inline python libraries

2) Rule based--- not advisible, since your model will just focus on these if-else rules itself. It would be the the best if these are the rules that are not that visible in the dataset, and the model cant catch it so easily. Which just implies it will than start to learn information in other features that also could be valuable.

3) Pseudo Labeling---- will add confident predicted test data to your training data. Therefore you will reinforce the clear labeled data and use them to help and label/predict other ones. Note, its a potential overfitting method.

4) Unsupervised Approach---- try to find good representation of data. The one thats discriminative enough, than apply clustering algos. If the resulting clusters are clear and different, you can look a couple of samples from every cluster and conclude what are the labels for every sample in your dataset.


Cross Validation (sklearn train test split) - ValueError: not enough values to unpack

Sklearn expects to get X and y in train_test_split().

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

You only parse X.

See the docs: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html


Reading Date Time in R (01.01.2019 00:00)

You can use this method (assuming that 'time' is the column you want, and the date format is Date.Month.Year Hour:Minute):

d <- read.csv("my_file.csv", sep = ";", dec = ",")
strptime(as.character(d$time), format="%d.%m.%Y %H:%M")


Does a precision score increasing with a higher number of folds mean the model will improve with more data?

It may or may not indicate that more data will help.

The correct way to see if model performance increases with more data is to create a learning curve. Train your model on a small subset of your data with cross-validation, and then keep increasing the amount of data used. This is not just reducing/increasing the number of folds. It is actually limiting yourself to a small amount of data for your 10-fold cross-validation and increasing it, evaluating the performance at each increase. Here's more about a learning curve.


Is sampling a valid way to reduce complexity?

I would get a sufficiently large random/representative sample and cluster that.
To see what is such a sample, you will have to get two such samples and cluster them to get cluster solutions c1 and c2. If the matching clusters of c1 and c2 have the same model parameters, then you probably have representative samples.
You can match the clusters by looking at how c1 and c2 assign drawn data to clusters.


How can I increase my accuracy avoiding overfitting? CNN-Keras-VGG16

Keras has support for image preprocessing. You can do lateral translations and shears on your images which will alter them but since the content is still the same your have more labeled data to train on. This is known as data augmentation.

Another thing you can do is introduce dropout in your network. This drops nodes in your network based on the probability you provide. This is done to prevent dependency of neurons on each other in your network.

Both of these where introduced in this paper. You can go through the reduce overfitting section to get a better idea and understanding.


Keras: confusion matrix with predict_generator

loaded_model.predict_generator(generator=test_generator)

will give us a set of probabilities

y_true = test_generator.classes

will give us true labels

Because this is a binary classification problem, you have to find predicted labels. To do that you can use:

y_pred = probabilities > 0.5

Then we have true labels and predicted labels on the test dataset. So, the confusion matrix is given by:

font = {
'family': 'Times New Roman',
'size': 12
}
matplotlib.rc('font', **font)
mat = confusion_matrix(y_true, y_pred)
plot_confusion_matrix(conf_mat=mat, figsize=(8, 8), show_normed=False)


Supervised Recommendation System trained on labeled phrase segments

CNN-Text-classification. Try this. Keep in mind that a deep learning algorithm will require a large data-set if you want a good accuracy. Since you want to give ranked recommendations to the users, a soft-max layer at the end will give you the probability of the phrase belonging to all the classes, so you can rank the labels in the order of the probability and suggest them to the user.

Your first approach is the simplest and best given that you have finite data. I don't think it will be very scalable as your data grows. Also this approach will work best only when your phrases are so limited in their context and vocabulary. Using a convnet you can train on bigger and more ambiguous phrases and get good predictions.


Do adaptive learning optimizers follow the steepest decent?

Adam and RMSprop are both optimization algorithms which make the vanilla gradient descent more robust. You still need to calculate the gradient however, these algorithms will facilitate the descent in certain directions and punish the descent in others based on some criteria.

Before delving into the details consider this analogy; imagine you are on a mountain and you want to get back to the village. To do this you will follow the downhill slope until you get to the bottom of the mountain. However imagine a mountain that looks like the following

Evidently the best path down the mountain should just be straight down the snowy path. However, if we carefully calculate the gradient we will notice that it will always have a slight left and right component. This will cause us to oscilate from left to right while going down the mountain. This is a huge waste of energy, but it's not catastrophic. However, when we start reaching the base of the mountain the downward slope will start to subside and the oscillations will become more prominent. This may cause us to never truly reach the minimum due to the oscillations becoming dominant, or cause the learning process to be dragged on excessively thus wasting resources.

The optimization techniques such as Adam and RMSprop simply make adjustments to the gradient descent step such that we can more directly find the minimum of the objective function without distraction from oscillating gradients. This is achieved in RMSprop and Adam by maintaining statistics relevant to the past computed gradients. RMSprop keeps track of the mean and Adam keeps track of both the mean and the moment of the past gradients.

An amazing reference for the inner-workings of the gradient descent variants can be found here.

Gradient descent

Vanilla gradient descent is defined as

$\theta_{t+1} = \theta_t - \eta \nabla_\theta J(\theta)$

where $\theta$ is the set of parameters which define our objective function $J(\theta)$. The learning rate is $\eta$ and $\nabla$ defines the gradient operator. We will extend this function.

Adding Momentum

Firstly, if we throw a ball down our cliff it will get some momentum as it falls. This will cause it to tend towards the steepest part of the gradient and left to right oscillations would be minimized. We can do the same to the gradient descent algorithm by introducing momentum $\gamma$ as

$\nu_{t} = \gamma \nu_{t-1} + \eta \nabla_\theta J(\theta)$

$\theta_{t+1} = \theta_{t} - \nu_t$

This is a good result but we can still do better.

Adagrad

It is preferable to have a distinct learning rate for each parameter such that we can adjust their impact on the gradient based on their past values. This is achieved by weighing the gradients by a sum of squares term of their past gradients up until the current time $t$. The matrix $G_t$ is a diagonal matrix where its entries are precisely this sum of squares of past gradients for each parameter. The Adagrad update is thus,

$\theta_{t+1,i} = \theta_{t,i} - \frac{\eta}{\sqrt{G_{t,ii} + \epsilon}}\nabla_{\theta_t}J(\theta_{t,i})$.

However, this method suffers from diminishing gradients when the sum of squares matrix makes it such that the past values are all too small and cause no more updates to be possible. Or for parameters which may be impactfull in the future to lose all relevance due to their low gradients at the start.

RMSprop

This is a solution to the diminishing gradients problem. We will use a running average of the past gradients until a time $t$ as defined by $E[g^2]_t$. The update rule is thus

$E[g^2]_t = \gamma E[g^2]_{t-1} + (1-\gamma)g^2_t$

$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{E[g^2]_t + \epsilon}}g_t$

where $g$ is the gradient for a single parameter.

Adam

In addition to keeping track of the running average of the past gradients we will also keep track of their second order moment. $m$ is the mean and $\nu$ is the moment. Thus we keep track of them as follows

$m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$

$\nu_t = \beta_2 \nu_{t-1} + (1-\beta_2)g^2_t$

We will correct these terms to remove their bias that tends towards a zero vector as

$\hat{m}_t = \frac{m_t}{1-\beta^t_1}$

$\hat{\nu}_t = \frac{\nu_t}{1-\beta^t_2}$

Finally, the update rule is

$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{\nu}} + \epsilon}\hat{m}_t$


Performance and architecture of neural network for increased dimensions

Some things to take into account:

Try to apply appropriate input space transformations, e.g. convert to polar coordinates.
Despite the fact that a single hidden layer feedforward network can be a universal approximator, there is no guarantees about the number of neurons needed to approximate an arbitrarily complex function. Instead of having a single hidden layer and making it increasingly wide, try stacking more layers. This enables the network to model non linearities in an easier way.
Do not verify the performance of your model based on visual inspection (at least not only), but based on error measurements (e.g. MSE).


GAN - why doesn't the generator nullify the noise input?

If the generator always outputs the same image, then it's easy for the discriminator to win the game and tell apart the output of the generator from random images in the training set: if the input to the discriminator is that one image, then it outputs "came from the generator", otherwise outputs "came from the training step".  The game is set up so that the generator is rewarded for fooling the discriminator.  Always outputting the same image isn't going to fool the discriminator.


Multivariate linear regression accounting for threshold / data cleaning

You could try including "Does not have a 2nd floor" as a separate categorical variable, which you could encode as 1 or 0 in your linear model.  Something like:

price ~ no_second_floor + area_of_second_floor + [Other Variables]

Event without the other variables, it's already going to be able to fit the data better than price ~ area_of_second_floor because, instead of forcing the price of the single-floor houses to $0, it would be able to fit it to the average price of all the single-floor houses in your data set - which is the best you can do for those houses until you add other variables besides area_of_second_floor.

(Actually, it wouldn't be forced to $0 if you were to include a constant term in your model, but the constant term that best lets you fit the linear portion of the data still probably isn't what you want for the single-floor houses, some of which - as you see - are quite expensive).


First of all, Snowball is not a type of stemmer, but a programming language used to implement stemmers.
The reason why the Snowball language was created was to make it more convenient to implement linguistic rules. So to answer your question, they are rule-based, not ML-based.


How to calculate steady/incremental growth?

What stops you looking at lags?
df = data.frame(c(1,2,3,4,4,3,4,5,6,7))
colnames(df)<-c("price")
df$delta=lag(df$price, 1)
df$inc = df$price-df$delta
df

Would yield:
   price delta inc
1      1    NA  NA
2      2     1   1
3      3     2   1
4      4     3   1
5      4     4   0
6      3     4  -1
7      4     3   1
8      5     4   1
9      6     5   1
10     7     6   1

The condition here would be that each value of inc (except the first one) must be $>0$.


What is the best approach for specified optical character recognition?

Optical character recognition is a well-studied problem with many possible solutions (ressources). CNNs have proven to work extremely well even for hand-written character recognition. Take a look at this two papers:

Backpropagation Applied to Handwritten zip code
Comparaison of Classifier Methods: A Case Study in Handwritten Digit Recognition

Here is a beginner tutorial to do just that with Tensorflow.
If you need extra data to train your model, take a look at the MNIST dataset.


Strategies for automatically tuning the hyper-parameters of deep learning models

I like to start with an overfitting set of hyperparameters, train to convergence - usually zero training error - then start increasing the regularization until the test error starts declining. Time is saved by the fact that I never re-initialize, but jack up the regularization parameters in an already-fit network.

My regularization parameters are dropout rate and weight decay. I usually leave one of them fixed and move along a grid of the other. When trying for a given set of hyperparameters, I usually start with large batches (faster) and then move to small batches (deeper).

I also rely on ensembling, combining the prediction of 4 - 64 models trained analogously, maybe with different architectures

Ymmv, perhaps.


SVR is giving same prediction for all features

There are a couple of parts that I think changing will help.

First, a general one for all model building: I would suggest you scale your data before putting it into the model.

It might not directly solve the problem of receiving the same predicted value in each step, but you might notice that you predictions lie somewhere in the ranges of your input values - as you are using unscaled volume, that is making things difficult for the model. It is essentially have to work on two different scales at the same time, which is cannot do very well.

Have a look at the StandardScaler in sklean for a way how to do that.

Next a few suggestions of things to change, specifically because you are working with stock prices:

I would normally predict the value of the stock market tomorrow, and not the closing prices on the same data, where you are using open/high/low/volume. For me that only make sense if you were to have high-frequency (intraday) data.
Given this, you would need to shift your y value by one step. There is a method on Pandas DataFrames to help with that, but as you dont have a date column and you only need to shift by one timestep anyway, you can just do this:

features = df.loc[:-1, df.columns != 'Closing'].values    # leave out last step
targets = df.loc[1:, df.columns  == 'Closing'].values     # start one step later

You could then even then predict the opening price of the following day, or keep closing data in the features data, as that would not introduce temporal bias.

Something that would require more setup, would be to look at shuffling your data. Again, because you want to use historical values to predict future ones, you need to keep the relevant hsitory together. Have a look at my other answer to this question and the diagram, which explains more about this idea.

EDIT

You should also scale y_train and y_test, so that the model knows to predict within that range. Do this using the same StandardScaler instance, as not to introduce bias. Have a look at this short tutorial. Your predictions will then be within the same range (e.g. [-1, +1]). You can compute errors on that range too. If you really want, you can then scale your predictions back to the original range so they look more realistic, but that isn't really necessary to validate the model. You can simply plot the predictions against ground truth in the scaled space.

Check out this thread, which explains a few reasons as to why you should use the same instance of StandardScaler on the test data.


What is Compatible Function Approximation theorem in reinforcement learning?

There are lots of questions but I will try to answer in a way that might clear things up for you and also give you some guidance. Please note that the proofs for your questions involve lots of math operations so instead I will provide you with references.

Your main reference is the paper from Sutton PG Methods with Function Approximation. I highly recommend you to read the paper a couple of times (or even more!) and do some search in the relevant literature when you will be familiar with the main objectives, notation and math around the general approach of the methods. PG Methods are not easy to get a grasp of them mainly because of their sampling nature, notation and discrete/continuous math involved.

PG Methods satisfy (or at least should) the PG theorem (eq. 2 from the paper). An interesting approach would be to substitute the true $Q^\pi (s,a)$ by some approximate function ($f_w$ in the paper, $Q_w$ in your question). Now, we are wondering what conditions should be satisfied by that proposed approximation in order to satisfy the PG Theorem.

The first thing you notice is that a natural choice for updating the parameters $w$ is to update them towards the direction that minimizes the mean squared error of the exact $Q^\pi (s,a)$ with the function approximation. In your question this is the $\epsilon$. In such a scenario the exact $Q^\pi (s,a)$ is estimated using unbiased samples such as $r_t$.
This is explained in detail in Part 2 of the paper.

For the PG theorem to hold (proof consists of the 3 lines before Part 3) the grad of your approximate function should satisfy the compatibility condition. To sum up we started from PG theorem and we found a suitable family of function approximators for our action-value function that the PG theorem holds. In Part 3 you can see an example of a compatible function. From this of course you can use even non-linear approximators such as NNs.

A clarification on on/off-policy: The David Silver's slide that you posted here has to do with theoretical guarantees and has nothing to do with an actual RL algorithm. By the way Q-learning algorithm in which you use the $max_{a'}{Q(s',a')}$ is OFF-policy as you you don't actually use for updates the ongoing policy.

Hope this helps!


Why do we use gradients instead of residuals in Gradient Boosting?

Hmmm, I am little perplexed by your question.

In gradient boosting, we do use the residuals. The residuals are the gradients.

You can check my simple implementation of gradient boosting. This is where the magic happens:

def fit(self, X, y):
    self.fs = [self.first_estimator]
    # step 0
    f = self.first_estimator.fit(X, y)
    # step 1
    for m in range(self.M):
        f = self.predict(X)
        # step 2
        U = self.loss(y, f)
        # step 3
        g = clone(self.base_estimator).fit(X, U)
        # step 4
        self.fs.append(g)

You start by a dummy model $f$. Then you create a new model $g$ based on the errors of the existing ensemble $L(f(x),y)$. The code should be pretty straight-forward. Check the algorithm in the wikipedia page for a more formal presentation.

Why am I using residuals and gradients as interchangeable words?

You have discrete mathematics and continuous mathematics.

In neural networks, we use the definition of gradient from continuous mathematics. You have a loss function that is $C_1$ continuous (meaning that the loss must be differentiable at least once).

$$f'(x) = \lim_{h\to0} \frac{f(x+h)-f(x)}{h}$$

In gradient boosting, we use the definition of gradient from discrete mathematics. It's usually called fininite differences, instead of derivative or gradient.

$$f'(x) = f(x+1)-f(x)$$

Personally, I think gradient is a misnomer. But it's marketing. Gradient boosting sounds more mathematical and sophisticated than "differences boosting" or "residuals boosting".

By the way, the term boosting already existed when gradient boosting was invented. It was used for AdaBoost, which changes the weights of the observations based on the loss, rather than training in the residuals. AdaBoost only works for weak models (each model must commit errors), while gradient boosting can do resample and work for strong models.

Important point on gradient boosting

One important difference between gradient boosting (discrete optimization) and neural networks (continuous optimization) is that gradient boosting allows you to work with functions whose derivative is constant.

In gradient boosting, you can use "weird" functions like MAE or the Pinball function. In my code, you can see that you can choose between MSE, MAE and quantile (which is the pinball loss). These latter two do not work very well for neural networks.

The pinball loss is used for quantile predictions, like wind forecasts in energy systems. This is the reason why gradient boosting is much used in these systems.

Small point: many people think that the reason why MAE and Pinball functions work bad for neural networks is because they are not continuous. While it is true they are not continuous when $y=\hat y$, it's very easy to work-around that. This is not the reason why results are so poor with those losses. The reason why they don't work well is because the optimization steps use the gradient (in gradient descent), and the gradient is constant for the entire function $L'(y,\hat y)=1$ or $L'(y,\hat y)=-1$, so the magnitude of the gradient is always 1. The optimization will depend solely on the learning rate heuristic that you are using, which usually will not work out very well.


How to compute precision and accuracy of a sequence that is not strictly binary?

Welcome to the Site!

We know that this problem is Multi-Class Classification Problem.

To get a confusion matrix for the same you can use the following command:

from mlxtend.evaluate import confusion_matrix

#import the required packages
from mlxtend.evaluate import confusion_matrix
from mlxtend.evaluate import plot_confusion_matrix

#Actual Target Values
y_target =    [-1,1,0,1,-1,1,0,1,0,-1]
#Predicted Values
y_predicted = [-1,0,1,1,-1,0,1,1,0,-1]

#creation of confusion matrix
cm = confusion_matrix(y_target=y_target,
                      y_predicted=y_predicted,
                      binary=False)
#to print the calculated values  of Confusion Matrix
cm

Outcome:

array([[3, 0, 0],
   [0, 1, 2],
   [0, 2, 2]])

For visualizing the cm you can use the following command:

fig, ax = plot_confusion_matrix(conf_mat=cm)
plt.show()

You can go through this Link for better understanding of mlextend.

You can get the Precision and Accuracy values by using the following formulas:

$\text{Precision}_{~i} = \cfrac{M_{ii}}{\sum_j M_{kji}}$

$\text{Recall}_{~i} = \cfrac{M_{ii}}{\sum_j M_{ijk}}$

Go through these Link-1,Link-2 for better understanding on how to compute the same, in the Link-3 is GitHub link which explains on how they implemented for a 1-D array, looking at that you can try expanding it for your outcome.


How to measure the distance (in generalized sense) between geographical regions?

In terms of interaction data, you might be thinking of this paper and this project

I love the idea of using flight data. You could start with this set hosted at Kaggle.

In terms of estimating distance between states, the simplest thing you could do that might work is to measure the distance between state capitals.


Tensorflow model works for classification but not for regression (all predictions equal the output layer bias)

Surprisingly, changing the optimizer from adam to tf.keras.optimizers.SGD(lr=0.01, momentum=0.9) eliminated the identical predictions, but it looks like more tinkering is required to get good results like with the classification model.

The only question is why does this change make that much of a difference?  I'd be curious to know the answer if anyone has one.


What you are looking for is called KNN algorithm, also knows as k-nearest neighbours. It’s a supervised algorithm where you have points and their clusters given and you use these to learn a pattern for test points.


Where is my error in understanding gradient descent calculated two different ways?

Your second function is not an update rule, it is just a re-statement of the approximation between using finite differences to measure a gradient and the infinitesimal ones used in calculus. Most importantly, $s$ is not a learning rate or parameter that you are free to change. It is, by definition, the (negative of the) change observed in $C$ when you alter $w$ to $w'$.

The second equation you have derived is the same as:

$$w + \Delta w \approx w + \Delta C \frac{dw}{dC}$$

In addition, for some reason you decided to drop the approximation symbol partway through and treat your manipulations as if you had created a new weight update rule. Be careful when manipulating expressions on either side of an approximation, they do not always follow the same rules as for an equality, and treating them as such can lead you down some false paths.


When should we use the principles of tidy data by Hadley Whickham, and when should we avoid using it?

In general tidy data is great... but it can quickly become unreasonably large. This is the main reason why I usually try to refactor my data in a tidy format as late as possible in the process.

Example: imagine a dataset containing $N$ instances, with columns feature1 ... featureX and result1... resultY, where the result? columns represent some value based on several methods/parameters. The tidy version would have columns feature1 ... featureX plus method (which takes as values result1,..,resultsY) and of course result for the result value. The tidy version would contain $N \times Y$ instances, i.e. for each instance it repeats the values of the $X$ features $Y$ times. If $X$ is large the dataset is going to be very large in memory (and also stored as a file).

Naturally this gets even worse when you have for example several steps involving various parameters, multiplying the instances every time is not always feasible.


Why does feature scaling improve the convergence speed for gradient descent?

The answer to this question simply lies in how one performs or applies the chain rule of differentiation. To understand this, first look at back-propagation which is simply a method to update weights by calculating the error between the predicted value and the exact value. When this error is back-propagated to optimize your objective function, derivatives are calculated in the subsequent layers and are multiplied by learning rate and the weights are then updated.

Now just imagine, if you have some features and range of one feature is from 0 to 1 and the other feature's range is from 10 to 1000. Now when you do differentiation for back-propagation for features with high range values you may get derivatives which are greater as compared to features which have low range values, this would lead to abnormal weight updating, and would add certain biasness to our model regarding those features, even when that feature as compared to the feature which has low range values may hold equal importance when statistical testing is done.

Thus to avoid this, introduction of biasness, feature scaling is used which allows us to scale features in a standard scale without associating any kind of biasness to it. This applies to various machine learning models such as SVM, KNN etc as well as neural networks. Another reason to scale features is to avoid large derivatives being back-propagated which increases the computation.

Another curious aspect of scaling is if you don't scale your features you will have different error surface (because of introduction of biasness when it is not there) as compared to scaled features error surface this may prevent you from reaching the optimum minimum value for your objective function thus that is why scaling helps you to ensure that you at least reach local minima thus it can be said scaling helps you to increase convergence speed while doing gradient descent.