Error of linear regression as a result of the coefficients

You have some inherent error in $m$, say that you predict $m=m_0+a$, where $m_0$ is the true value and where you can control the size of $a$ in terms of your error bounds on $m$. Then you get:

$$y=(m_0+a)x+b=m_0x+ax+b.$$

The $ax$ term is what will cause your error to grow as you increase/decrease $x$ toward $\pm\infty$. In other words, your predicted line will ultimately be infinitely far from the true line (as can be visualized by drawing two lines with slightly different slopes).


Is it possible to use a pretrained scikit learn model to make predictions on a dataset with different features (than those used during training)

Technically, the only constraint would be the number of features that has to be the same than during the training phase (in most scikit learn models).
Will it perform well ?
I would say it really depends on the type of model/approach you are using and what is behind:

Note that these new features are very similar to the ones used in training

Generally speaking it will probably not perform well unless the training features and the new features are related in a way that is acceptable by your model.


How to provide classified feature to a neural network

In your problem, the label is a categorical variable (you cannot infer relation between classes just from the label value) and not ordinal (value shows relation/distance between classes).

The solution that you propose:

  1 -> [1,0,0,0], 2 -> [0,1,0,0], 3 -> [0,0,1,0], 4 -> [0,0,0,1]

is called One-Hot Encoding. This is one of the most popular ways of encoding the classes during preprocessing, in order to feed them to a classifier, thus I recommend you of doing so.

You mention that you are afraid that the

  feature gets to much weight

because of the number of samples in the dataset. This is called class imbalance. A way of circumventing it is to pre-weight your samples for your classifier, please take a look at this method implementation provided by sklearn package.


how should I measure performance if there is no test data?

If you are working with only enough data for training and validation, consider using K-Fold Cross Validation:

  One of the main reasons for using cross-validation instead of using the conventional validation (e.g. partitioning the data set into two sets of 70% for training and 30% for test) is that there is not enough data available to partition it into separate training and test sets without losing significant modelling or testing capability. In these cases, a fair way to properly estimate model prediction performance is to use cross-validation as a powerful general technique.

Assuming your "practice" data is drawn from the same distribution as your "real" data, you are on the right track by thinking about measuring the relationship between training accuracy and test accuracy in the first set to model the relationship in the second set.

However you should split your practice data into training and test only and use k-fold cross-validation on the training set. Then you should train a model on the real data with the same cross validation scheme.

For example you might get results like:

practice data:
  training accuracy: 90%
  test accuracy: 88%

real data:
  training accuracy: 89%

Since our cross-validation scheme was resistant to over-fitting, the training and test accuracy are close on the practice set, and our practice training accuracy is close to our test training accuracy. Now we can safely assume that accuracy on the unseen test set for the real data is about 2% worse than training accuracy.

However, imagine you a single validation set or no validation at all on the practice data. Now your model is more likely to overfit and you might see results like this:

practice data:
  training accuracy: 95%
  test accuracy: 75%

real data:
  training accuracy: 85%

Here we have created a model with high variance by overfitting. Training and test accuracy are far apart on the practice data, and training accuracy on the practice data does not match well with training accuracy on the real data. It is not as easy to estimate test accuracy now, because we can't really say if we have overfit by the same amount on both datasets and the second set is harder, or if we have overfit the second set by less. In the first case we might predict 65% test accuracy, and in the second case we might predict 75%.


First let's understand why the format is like this.

BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence Prediction task : determining if sen B is a random sentence with no links with A or not.
The [SEP] in the middle is here to help the model understand which token belong to which sentence.

At finetuning time, if you use a different format than pretraining format, you might confuse the model : he never saw 2 sentences formatted as [CLS] Sen A Sen B [SEP]. The model doesn't know there is 2 sentences, and will consider it as a single sentence.

If you finetune on enough data, BERT can learn the new format. This can be helpful if you need to change the input format.

But in your case, you don't need to do this. Changing the format for the sake of changing the format is just going to confuse your model, he will have to learn more thing, and there will be inconsistencies between pretraining and finetuning.

  Will the s1 in second one integrate more information from s2 than the s1 in first one does?

No. Inserting a SEP token or not will not change the amount of information exchange between the tokens of the 2 sentences. In both case the model will compute attention based on the 2 sentences. Each sentence can see the other sentence's tokens, no matter of the SEP.
The only thing you will do by removing the SEP token is confuse your model.

  Will the token embeddings change a lot between the 2 methods?

We don't know. It will definitely change, but how much ? We cannot answer. My guess is that token representations will not change a lot (because tokens are the same), but CLS representation will change a lot (instead of representing the links between 2 sentences, it will represent something else).


Why MADDPG rather than taking all cooperating agents as a single meta-agent?

MADDPG can be used to model agents that have limited observation and communication capabilities after training, which is an interesting and useful real world scenario.

why not simply treat all cooperating agents as a single meta-agent with a concatenated observation space and a concatenated action space?

Any real world implementation will then require resources to provide and manage that overview. This may not be practical or desirable in all cases.
There is no single fix for this, it is an open area of research. Whether to invest in better communication and central processing, or better autonomy for multiple agents is likely to have different answers depending on the problem and current technology limits for either approach.
MADDPG reduces the role of central processing to assessment of global reward signals during training. That means:

Each agent works with local signals only, and simpler observation and action spaces as a result. Only the reward signal processing is handled externally.

Trained agents can theoretically be used in environments where a central processor is not available.

So, for example, agents can be trained in simulation with all the oversight that allows, or with a carefully instrumented environment including high bandwidth connections between agents and central processing. They can then be deployed into matching environments where the central oversight is not available, or too costly.


Machine learning model with simultaneous function optimization

Have a look at this competition organized by the European Spatial Agency (ESA).
https://kelvins.esa.int/collision-avoidance-challenge/scoring/

Here they use a composed loss function. A binary classification whether two spatial are at risk of colliding or not. And in case that they collide a regression saying what is estimated risk between them.

This is done by constructing a loss function combining the Mean Squared Error (MSE) and the F_β score with β=2.

You could consider your problem similar. You will want to classify whether your offer gets accepted or not, and in case of acceptance the money in return that you will receive.

In the part of the discussion of the challenge, you will find some notes on how to do it. Also, I believe that they will publish a paper in the next weeks.


How to make use of POS tags as useful features for a NaiveBayesClassifier for sentiment analysis?

There are so many ways you could go about this. For starters, you could use Conditional Random Fields (CRF). There is a sweet implementation in Python. In which you can set the POS features and more. There is a website from the same source you posted on how to use CRF for your purpose (I have not read it thoroughly). Spacy is another great resource to get all the features that you need fast. Nonetheless, for SOTA you will need some NN implementations.


I have just finished my Ph.D. and have used some NLP in it. My university didn't offer any NLP courses. So I ended up teaching myself NLP. I used this book.

Which serves as a great introduction to NLP using NLTK (Natural Language Tool Kit). It gives a good introduction into programming with Python. So handy if you've never programmed in Python before too.

I would highly suggest using nltk from nltk.org (sorry can't post more than two links)

The book I used is now out of date as NLTK is now on version 3.0, the book mentioned previously is for NLTK 2.x. But the Authors are working on a new version of the book for NLTK 3.X, you can view the unfinished book here.

I would highly suggest using NLTK and if you're new to natural language processing. I would highly suggest you try and get yourself a copy of the following book:

Foundations of Statistical Natural Language Processing by Manning and Shutze

Even though it doesn't contain any code, it servers a great introduction to natural language processing.


Stability of clusters in a unsupervised machine learning

From scikit-learn document Link-

Given the knowledge of the ground truth class assignments labels_true and our clustering algorithm assignments of the same samples labels_pred, the (adjusted or unadjusted) Rand index is a function that measures the similarity of the two assignments

labels_true = [0, 0, 0, 1, 1, 1]
labels_pred = [0, 0, 1, 1, 2, 2]
metrics.rand_score(labels_true, labels_pred)

2.3.10.1.2. Drawbacks

Contrary to inertia, the (adjusted or unadjusted) Rand index requires knowledge of the ground truth classes which is almost never available in practice or requires manual assignment by human annotators (as in the supervised learning setting).


What you would typically do in your case is to apply a sentence alignment tool. Some popular options for that are:

hunalign: a classical tool that relies on a bilingual dictionary.
bleualign: it aligns based on the BLEU score similarity
vecalign: it is based on sentence embeddings, like LASER's.

I suggest you take a look at the preprocessing applied for the ParaCrawl corpus. In the article you can find an overview of the most popular methods for each processing step.
A different option altogether, as you suggest, is to translate at the document level. However, most NMT models are constrained in the length of the input text they accept, so if you go for document-level translation, you must ensure that your NMT system can handle such very long inputs. An example of NMT system that can be used for document-level NMT out of the box is Marian NMT with its gradient-checkpointing feature.


Can FTRL be applied on linear least squares? or is it just for logistic regression models?

As we can see, FTRL is nothing to do with our model, and it only serves as a way of approximating the optimal solution , just as SGD does.

Logloss is just a kind of loss pattern in classification problem,and there are other alternatives for it , such as ExpLoss. Least-squres loss is one of all loss patterns in regression problem.

The reason why FTRL was mentioned everywhere ,which mainly treated Logloss as a example, is that it was often used in classification problem , such as CTR model in online advertisement scenario.

Hopes this can helps you ,and good luck !


What's Joint Training in Neural Networks?

You are incorrect that is called Transfer learning, which focuses on storing knowledge gained while solving one problem and applying it to a different but related problem.
Joint Training is basically when the neural network trains on several different task simultaneously so optimizing more than one loss function rather than one loss function as the case with Transfer learning.
It is also called Multi task learning.
Refer this article to know more.


String similarity algorithms for string containment (rather than string equality)

My first thought was that it's difficult to formally define this concept:

"b" is likely to contain "a" with some "error"

On the one hand, there is the idea that a is a substring of b. This question should have a boolean answer: either it does contain it, or it doesn't.
On the other hand there is the idea of approximate matching: b should contain a substring c which is "similar enough" to a. In general the question of similarity between two strings is answered with a numerical value, usually a real number between 0 and 1.

As far as I know the only way to solve this issue is to consider that there is a threshold on the similarity score between a and c, where c is any substring of b. This way the answer becomes boolean.
However there might be a way around this by considering the substring operation as part of the similarity calculation. In particular the Levenshtein edit distance can account for insertions/deletions of characters, which is what a substring is w.r.t the string which contains it.
More interestingly, it is possible to assign a different cost to any particular edit operation in the Levenshtein distance. So it's probably possible to define a variant of Levenshtein where insertions at the beginning or at the end would have cost 0, thus making the final distance $x$ between a and b equivalent to "b contains a substring c which has distance $x$ against a".
The way I would try to implement this is:

Calculate the regular edit distance, keeping the matrix used for the calculation
From the matrix, count how many insertions were done but only at the beginning and at the end, then substract this value from the distance.

Note that there might be flaws in my idea, I didn't try it :D


Why do classes with fewer words have higher probability?

The Naive Bayes formula doesn't do what you say. For classes $C_k$ and words $x_1, \ldots, x_n$, the formula is

$P(C_k | x_1, \ldots, x_n) \propto P(C_k) \prod_{i=1}^N P(x_i | C_k)$

If the word newword doesn't occur, we have no prior information about $P(x_i | C_k)$, so all these conditional probabilities should be equal. So the right-hand side is just proportional to $P(C_k)$, which is independent of the number of words in $C_k$.


What is the difference between public test and private test on Kaggle

"publictest" data is used to calculate the score for the public leaderboard while competition is running.

After the competition is finished, competitors will be ranked on the data which was marked as "privatedataset".


Confusion between doing .copy() and using [[ ]]

Both are more or less the same thing.
xyz = DataFrame[["col1"]] is useful when you have to select one or more than one column from a dataframe. For example: xyz = DataFrame[["col1","col2"]]
While xyz = DataFrame["col1"].copy() or df.copy() is used for copying a complete/partial dataframe.
Note that both are exactly the same.


Is there any working alternatives to Netvizz?


What model is suitable for classification of a small data set?

The question whether to use a linear classifier depends less on the number of samples you have in your dataset and more whether your dataset is linearly separable (by the way, SVMs can be non-linear with the kernel trick).

Now with regards to confidence in the classification, In SVMs there is a method that calculates the probability that a given sample belongs to a particular class using Platt scaling ("Original Paper"). This is the approach that is used in sklearn's SVM confidence implementation. You can read more about it in the following link:

How To Compute Confidence Measure For SVM Classifiers

In both SVMs and linear regression models you can calculate the distance of a sample from the border and treat it as a confidence measurement (but it is not exactly that).

With decision trees I'm not an expert but a similar question was posted and answered in the following link:

Decision tree, how to understand or calculate the probability/confidence of prediction result

I would strongly recommend using some known embedding method like the word2vec, since as you mentioned, your dataset is too small for your model to be able to properly learn an encoding of context and vocabulary from.


Determining Important Atrributes with Feature Selection

Sounds like pretty orthodox feature importance analysis. Easy option:

FeaturePlot from caret

It basically creates a correlation matrix for you to see which seem to separate most.

Slightly fancier:

VarImpPlot from randomForest

This uses the random forest ML technique to directly tell you which features were most important. It sounds very similar to what you're looking for.

Edit:
Here is a more detailed discussion of VarImpPlot.


How to choose negative examples for recommendation system?

Yes, we should include negative samples, more so to correct selection or sampling bias.

The user would have seen only a subset that is selected by another algorithm or rules which decided that the subset is most relevant to the user. Of that, position bias kicks in and the user may not see everything. So the choices within a query or the documents within a query are likely to be very similar to one another (perhaps small differences). This makes it diff for the algorithm to learn the parameters.

Whereas while scoring in production, almost all choices/documents have to be scored for each query. UNLESS the old model is used first to select a subset of hotels and if learning-to-rank model reranks among the subset.

In my experience adding negative samples (that we can safely tell that user would not have picked/clicked the item even if he/she saw the item) helps. But it is difficult to define such base for all use-cases.

We are still experimenting with this.


How do I perform multi-step forecasting on LSTM trained with multiple observations of a sequence?

ConvLSTM, keras has an implementation, therefore you can go with Python itself.

If you want multiple outputs from the LSTM, you can have look at return_sequences and return_state feature in LSTM layers. Default values for them are None, But if you give True you can get multiple outputs for each timestep, and for everyone.

This excellent blog post helped me understand them.


Export pandas to dictionary by combining multiple row values


The objective function is the mathematical equation which combines the parameters in a useful sense. For linear regression this is a simple linear combination of weighted features.

You will need to choose which objective function is best suited for your problem. Then based on the function you selected you will know what are the model parameters that will need to be fit using an appropriate training method.

One of the skills of the data scientist is to identify the objective function which has the best likelihood of success given a problem statement.


How to keep the test data from leaking into the training process of a machine learning algorithm?

Just to clarify (and I think you've got this right, but I'm just being careful), it is best practice to:

1: Split your data into train and test

2: Split train into train and eval

3: Grid search over hyperparameters, for each combination, train on train, evaluate on eval. Select the hyperparameters which allow you to get the best score on the eval set

4: Using the best model (as ascertained in step 3), calculate the test loss, using the test data set create in step 1. Use your answer in step 4 to get the best estimate of how well your model will perform in production.

To be very clear, step 4 is not about hyperparameter selection, the hyperparameters are fixed by step 3. Step 4 is all about getting a better estimate of production performance, as what you get from step 3 is likely to be overly optimistic, as the particular hyperparameter combination used to overly optimised (overfit) to that particular train-eval split.

You can repeat steps 1-4 many times, but what you should not do, is try to use this repetition in any way, to find the best hyperparameters, you are using it to answer the question "how well is my model likely to perform on unseen data". For example, you might run it ten times and get a spread of accuracies from 65%-67%. You can then tell your boss, that if you deployed this model in production, it would likely have an accuracy of 65%-67% (or you're a bit more quantitative and you say it's the mean $\pm$ the standard error)

This leaves the question, "how do I select my best hyperparameters?". After all, you've run steps 1-4 many times and step 3 has given you a different best hyperparameter combination every time.

This is where you need to run step 3 one more time, but slightly differently. This time, you don't need a train-test split. You simply take all of your data, and do a train-eval split. You then run one further hyperparameter grid search, and that's the model you use in prod (or you could combine the train and eval sets, and train one more time, using those hyperparamters, depending on whether you're using a model that needs an eval set for early stopping like a neural net/xgboost or one that does not like a Random Forest). However, at this stage, you ignore the accuracy/peformance metrics that the training procedure is telling you. You already estimated your production accuracy/performance when you ran steps 1-4 multiple times, and that was the point of doing this.

I hope this answers your question, both on how to do the splitting, as well as illustrates why test data at no point leaks into the training data


Naive Bayes vs Full Bayes model classifiers

There's no clear definition of "Full Bayes" as a classifier. Most "real world" non-Naive Bayesian classifiers take into account some but not all dependencies between features. That is, they make independence assumptions based on the meaning of the features.

If by "full Bayesian" you mean a joint model (as your example suggests), then one of the problems is that such a model doesn't generalize: it just describes the probabilities in the training set, and that implies that it's likely to overfit badly. This is actually why NB works quite well in most cases: yes it makes unrealistic independence assumptions, but this simplification allows the model to capture basic patterns from the data. In other words, the ability of the model to generalize comes from its excessively simplified assumptions.

Note: as far as I can tell, your example is well chosen and you should see a big difference between NB and a joint model: NB should perform no better than a random baseline while the joint model should obtain near perfect accuracy. There's probably a mistake somewhere if you don't obtain these results. But while this is a good toy example, it cannot help you understand the advantage of the NB assumptions.


Classification with a lot of the classes

Try pooling in between, that will reduce size so that its compactible.
Have a look here


one-hot-encoding categorical data gives error

First of all, I noticed you have loaded the same dataframe for both train and test. Change the code like this:

import numpy as np
import pandas as pd

train = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/train.csv")
test = pd.read_csv("https://raw.githubusercontent.com/oo92/Boston-Kaggle/master/test.csv")

At this point, one-hot encode each variable you want with pandas' get_dummies() function:

# Onhe-hot encode a given variable
OHE_MSZoning = pd.get_dummies(train['MSZoning'])

It will be returned as a pandas dataframe. In my Jupyter Notebook it looks like this:

OHE_MSZoning.head()

You can repeat the same command for all the variables you want to one-hot encode.

Hope this helps, otherwise let me know.


Set value for column based on two other columns in pandas dataframe

You can use groupby together with shift and cumsum as follows:
df['header_contract'] = df['contract'] + '_' + df.sort_values(['contract', 'date']).\
  groupby('contract')["date"].\
  apply(lambda x: (x.shift() != x).cumsum()).astype(str)

In the apply, x.shift() != x is used to create a new series of booleans corresponding to if the date has changed in the next row or not. cumsum will then create a cumulative sum (treating all True as 1) which creates the suffixes for each group. This is then merged with the contract names to create the new column.
Result:
  contract prod       date  revenue header_contract
0     123A   X1 2019-04-17     5688          123A_1
1     123A   M1 2019-07-02   113932          123A_2
2     123A   V1 2019-04-17     5688          123A_1
3     123A   D1 2019-07-02    49157          123A_2
4     123B   A1 2019-04-17     5002          123B_1
5     123B   B1 2019-09-01      892          123B_2
6     123C   C1 2019-08-02     9000          123C_1


Best parameters to try while hyperparameter tuning in Decision Trees

There are no combinations that work for all cases, hyperparameter tuning is still something that is mostly done by trial and error. Things like Gridsearch and Randomsearch exist though.
A good start is always the default setting. An idea if performance is an issue is to tune on a small percentage of the training set to later switch to the full set.


Is it okay to use training data for validifying the trained model?

Scoring your model using the training data is not a best practice.  The reason is that you have already used this data to develop the "guess" at an accurate model.

This is akin to saying - 'Pick a random number between 1 and 10 (20 times)', here's a list of the last 10 numbers I picked to get you started.  You pick 20 new numbers and come up with 5 correct (25%).  What you are suggesting is that instead of reporting the 25% accuracy, that you would throw the first 10 back into the mix and report a 50% accuracy because now you've gotten 15 out of 30 right.

So there are two different things here.  The MSE of your initial model against the data you gave it - provides you and understanding of how well the model demonstrates history.  While the MSE (or other error metric) of the 'test' data shows how well you can predict the future.


How to convert input numpy data to tensorflow tf.data to train model in tensorfow?

You could try using the flow_from_directory() method on your ImageDataGenerator class, which does the augmentation - only a small change is necessary:
H = model.fit(
    aug.flow_from_directory(trainX, trainY, batch_size=BS),
    ...)

If you start using a tf.data.Dataset directly, you will get more control over how the data is read from disk (caching, number of threads etc.), but you will lose the easy image augmentation you get with the ImageDataGenerator.


xgboost in R have different results compared to boosted decision tree in Azure ML

It's hard to say, without being able to know exactly what Azure is doing.

From what they do share, they bin continuous features; you could try tree_method='hist' in xgb to be more similar there.
I can't tell how Azure deals with categoricals or missing values.
Be sure to set xgb's max_depth=0 and grow_policy='lossguide', since you want to use max_leaves instead for a direct comparison.

https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/two-class-boosted-decision-tree#usage-tips
https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/two-class-boosted-decision-tree#module-parameters


Here, you are substituting the missing values (nans) with something, it can be either the most frequent data, median, average(mean), whatever. mean modifies the nan values with the average of not-nan values.

for x in num_cols:
    imp = SimpleImputer(missing_values=np.nan, strategy='mean')
    imp.fit(np.array(ds[x]).reshape(-1,1))
    ds[x] = imp.transform(np.array(ds[x]).reshape(-1,1))

  imp = SimpleImputer(missing_values=np.nan, strategy='mean')

For transforming missing values we use SimpleImputer, here, the missing values are np.nan and we want to use the mean strategy for transforming these nans. Thereby, their values would be substituted by the average value of each feature.

  imp.fit(np.array(ds[x]).reshape(-1,1))

You have an $n * m$ (array (np.array(ds[x]))), by reshape(-1,1)) you convert it to shape $(m*n, 1)$ -1 here stands for vertical array and 1 stands for having one column.

Then you fit your transformer (imp) to your data (np.array(ds[x]).reshape(-1,1))

  ds[x] = imp.transform(np.array(ds[x]).reshape(-1,1))

Here, the transformed data would be substituted in ds[x]


Technique to determine variation in metric due to varying parameters

If the variation in values of feature then should not you remove the column? Your model will not learn from it if the variance is low.

you can Q-Q plots when you vary the features to check how close are the 2 datasets comparing their distribution


How to give breakdown of text prediction sentences in Shiny

Maybe you could put the sentences in a dataframe and used renderTable, something like this:
server <- function(input, output, session) {
  sentences <- reactive({
    data.frame(sentences=make_sentences(input$inputString))
  })
  output$prediction <- renderTable(sentences())
}

[edit]
In order to only have them line by line something like this might work:
server <- function(input, output, session) {
  sentences <- reactive({
    paste(make_sentences(input$inputString),sep='<br/>')
  })
  output$prediction <- renderTable(sentences())
}

Alternatively you could have a numbered list by using the corresponding HTML tags: insert the whole list inside <ol> ... </ol> and every sentence in a <li> .. </li>.


When is z-normalization not needed when using DTW?

That eamonn guys sounds smart.

If you do not normalize, small differecnes in the mean and/or STD rapidly swamp any shape similarity.

Here is one way to think about it.

If you do not normalize, and the means or STDs are different, then.......

EuclideanDIST(A,B) is approximately equal to DTWdist(A,B), which is approximately equal to [abs(mean(A)-mean(B)) times a constant]

If you can model the similairy with just the mean, they why do you need to even look at the shape?

BTW, you have a cool problem/dataset


Correlation is a bivariate analysis that measures the strength of association between two variables and the direction of the relationship. In terms of the strength of the relationship, the value of the correlation coefficient varies between +1 and -1. A value of ± 1 indicates a perfect degree of association between the two variables. As the correlation coefficient value goes towards 0, the relationship between the two variables will be weaker. The direction of the relationship is indicated by the sign of the coefficient; a + sign indicates a positive relationship and a – sign indicates a negative relationship.
Pearson's correlation coefficient and the others are the non-parametric method, Spearman's rank correlation coefficient and Kendall's tau coefficient.
Pearson's Correlation Coefficient
$$
r = \frac{\sum(X - \overline{X})(Y - \overline{Y})}
{\sqrt{\sum(X-\overline{X})^{2}\cdot\sum(Y-\overline{Y})^{2}}}\\
~ \\
\begin{align}
    Where, ~ \overline{X} &= mean ~ of ~ X~variable\\
    \overline{Y} &= mean ~ of ~ Y ~ variable\\
\end{align}
$$
Assumptions:

Each observation should have a pair of values.

Each variable should be continuous.

It should be the absence of outliers.

It assumes linearity and homoscedasticity.

Spearman's Rank Correlation Coefficient
$$\rho = \frac{\sum_{i=1}^{n}(R(x_i) - \overline{R(x)})(R(y_i) - \overline{R(y)})}
{\sqrt{\sum_{i=1}^{n}(R(x_i) - \overline{R(x)})^{2}\cdot\sum_{i=1}^{n}(R(y_i)-\overline{R(y)})^{2}}}
= 1 - \frac{6\sum_{i=1}^{n}(R(x_i) - R(y_i))^{2}}{n(n^{2} - 1)}\\
~ \\
\begin{align}
    Where, ~ R(x_i) &= rank ~ of ~ x_i\\
    R(y_i) &= rank ~ of ~ y_i\\
    \overline{R(x)} &=mean ~ rank ~ of ~ x\\
    \overline{R(y)} &=mean ~ rank ~ of ~ y\\
    n &= number ~ of ~ pairs
\end{align}
$$
Assumptions:

Pairs of observations are independent.

Two variables should be measured on an ordinal, interval or ratio scale.

It assumes that there is a monotonic relationship between the two variables.

Kendall's Tau Coefficient
$$
\tau = \frac{n_c - n_d}{n_c + n_d} = \frac{n_c - n_d}{n(n-1)/2}\\
~ \\
\begin{align}
    Where, ~ n_c &= number ~ of ~ concordant ~ pairs\\
    n_d &= number ~ of ~ discordant ~ pairs\\
    n &= number ~ of ~ pairs
\end{align}
$$
Assumptions:

It's the same as assumptions of Spearman's rank correlation coefficient

Comparison of Each Correlation Coefficients
Pearson correlation vs Spearman and Kendall correlation

Non-parametric correlations are less powerful because they use less information in their calculations. In the case of Pearson's correlation uses information about the mean and deviation from the mean, while non-parametric correlations use only the ordinal information and scores of pairs.

In the case of non-parametric correlation, it's possible that the X and Y values can be continuous or ordinal, and approximate normal distributions for X and Y are not required. But in the case of Pearson's correlation, it assumes the distributions of X and Y should be normal distribution and also be continuous.

Correlation coefficients only measure linear (Pearson) or monotonic (Spearman and Kendall) relationships.

Spearman correlation vs Kendall correlation

In the normal case, Kendall correlation is more robust and efficient than Spearman correlation. It means that Kendall correlation is preferred when there are small samples or some outliers.

Kendall correlation has a O(n^2) computation complexity comparing with O(n logn) of Spearman correlation, where n is the sample size.

Spearman’s rho usually is larger than Kendall’s tau.

The interpretation of Kendall’s tau in terms of the probabilities of observing the agreeable (concordant) and non-agreeable (discordant) pairs is very direct.

Example Python Implementation


What is a good explanation of Non Negative Matrix Factorization?

Non-Negative Matrix Factorization (NMF) is described well in the paper by Lee and Seung, 1999.

Simply Put

NMF takes as an input a term-document matrix and generates a set of topics that represent weighted sets of co-occurring terms. The discovered topics form a basis that provides an efficient representation of the original documents.

About NMF

NMF is used for feature extraction and is generally seen to be useful when there are many attributes, particularly when the attributes are ambiguous or are not strong predictors. By combining attributes NMF can display patterns, topics, or themes which have importance.

In practice, one encounters NMF typically where text is involved. Consider an example, where the same word (love) in a document could different meanings:

I love lettuce wraps.
I love the way I feel when I'm on vacation in Mexico.
I love my dog, Euclid.
I love being a Data Scientist.

In all 4 cases, the word 'love' is used, but it has a different meaning to the reader. By combining attributes, NMF introduces context which creates additional predictive power.

$"love" + "lettuce \ wraps" \ \Rightarrow \ "pleasure \ by \ food"$
$"love" + "vacation \ in \ Mexico" \ \Rightarrow \ "pleasure \ by \ relaxation"$
$"love" + "dog" \ \Rightarrow \ "pleasure \ by \ companionship"$
$"love" + "Data \ Scientist" \ \Rightarrow  \ "pleasure \ by  \ occupation"$

How Does It Happen

NMF breaks down the multivariate data by creating a user-defined number of features. Each one of these features is a combination of the original attribute set. It is also key to remember these coefficients of these linear combinations are non-negative.

Another way to think about it is that NMF breaks your original data features (let's call it V) into the product of two lower ranked matrices (let's call it W and H). NMF uses an iterative approach to modify the initial values of W and H so that the product approaches V. When the approximation error converges or the user-defined number of iterations is reached, NMF terminates.

NMF data preparation

Numeric attributes are normalized.
Missing numerical values are replaced with the mean.
Missing categorical values are replaced with the mode.

It is important to note that outliers can impact NMF significantly. In practice, most Data Scientist use a clipping transformation before binning or normalizing. In addition, NMF in many cases will benefit from normalization.

As in many other algorithmic cases, to improve matrix factorization, one needs to decrease the error tolerance (which will increase compute time).


How to best visualise two sets of data over several years?

Statistics of this nature are commonly normalised with respect to the population sizes by presenting them as per X. In this case per 1,000 or 100,000 people.
Absolute number of crimes makes no sense and lacks context. In terms of display, as both KPI ranges are similar (3 vs 19) a common X and Y axis is possible with two different lines.
If you want to communicate trends relative to a baseline year (e.g. 2014) you could divide all values relative to this year and express it as a % change.  I.e. crime went up 10% but police numbers went down by 5%.


Decision Tree: Efficient splitting of nodes, minimize number of gini evaluations

In general, to reduce the amount of time needed to run your dataset through the See4.5 (C4.5) algorithm, you'll want to reduce the number of nodes in the tree needing to be processed.

This can be done utilizing pruning, optimal operator selection, and incorporating a heuristic into your decision tree search.

Alpha-beta pruning, bidirectional search, and the Minmax algorithm for operator selection are a good choice when it comes to decision-tree time reduction.

I'm not going to write an entire book here, however look into artificial intelligence and see what they've accomplished so far. Alot of it needs tweaking because they keep switching (but that's another story), however, if you come across any books saying bidirectional search is in any way non-optimal, just ignore that because it's inherent that researchers can't code that well.

A good implementation of the Gini algorithm in practical use is available through Ross Quinlan's website. If you look in to and understand the C5.0 source code, you should be on decision tree research level as there isn't a clear explanation available online as far as I can tell detailing the new algorithm's additions.


Python: Fast indexing of strings in nested list without loop

This answers assumes I have correctly understood the question... I can alter my answer if OP updates the question with more details

Using your example data, you can use Pandas easily drop all duplicates.

Setup

First dump your data above into a Dataframe with three columns (one for each of the items in each row.

Import pandas:

import pandas as pd

import your data - assuming it is a list of lists - each of your rows is a list  of three items, so we have three columns:

df = pd.DataFrame.from_records(your_list_of_lists, columns=["col1", "col2", "col3"])

Have a look at the first 5 rows:

df.head()
   col1  col2     col3
0   600   900  3418309
1   600   900  3418309
2   600   900  3418314
3   600   900  3418314
4   600   900  3418319

The values will be integers by default, not strings (if they all were). But the solutions below should work the same in either case.

Solutions

If you want to get all unique values of col3, you can do one of the following:

uniques1 = set(df.col3)              # returns a Python set
uniques2 = df.col3.unique()          # returns a NumPy ndarray
uniques3 = df.col3.drop_duplicates() # returns a pandas Series object

If you want to remove only those rows where col3 repeats itself consecutively (which in some cases could produce the same result as the methods above, depending on the data), then you can look at the methods here. [An example] adapted for your columnar situation2:

def drop_consecutive_duplicates(a, col_name):      # returns the dataframe
    ...:         ar = a[col_name].values
    ...:         return a[np.concatenate(([True],ar[:-1] != ar[1:]))]

This will return the entire dataframe, which those rows of consecutive values removed.

Performance

There are many other ways to achieve the same result. Of the above, the first method is the fastest (on your small dataset example). Here are the benchmarks:

In [23]: %timeit df.col3.drop_duplicates()
263 µs ± 883 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [24]: %timeit df.col3.unique()
37.2 µs ± 3.19 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [25]: %timeit set(df.col3)
10.5 µs ± 45.7 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

The consective version:

In [26]: %timeit drop_consecutive_duplicates(df, "col3")
266 µs ± 3.61 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

I imagine the Pandas methods might be able to scale better to a DataFrame with many rows, so this example might be rather biased on the dummy dataset with only ~20 rows.

The final method clearly has a little bit of overhead as it has to perform some extra operations


Expectations of the skill set of a Data Scientist

It depends on the level of expertise that is required. Personally I know how to do most of the standard operations you would need for Excel but I don’t think a Data Scientist needs to know more than that.

If this is a job advert where they are asking for advanced knowledge of Excel as a Data Scientist and do not require much in the way of programming/machine learning skills then I would give the job a wide berth. It would indicate to me that the company putting the job out don’t understand what a Data Scientist is and are putting a job out to say they have one.


Wang, Kaijun, Baijie Wang, and Liuqing Peng. "CVAP: Validation for cluster analyses." Data Science Journal 0 (2009): 0904220071.:

To measure the quality of clustering results, there are two kinds of
validity indices: external indices and internal indices.
An external
index is a measure of agreement between two partitions where the first
partition is the a priori known clustering structure, and the second
results from the clustering procedure (Dudoit et al., 2002).
Internal
indices are used to measure the goodness of a clustering structure
without external information (Tseng et al., 2005).
For external indices, we evaluate the results of a clustering algorithm based on a known cluster structure of a data set (or cluster labels).
For internal indices, we evaluate the results using quantities and features inherent in the data set. The optimal number of clusters is usually determined based on an internal validity index.

(Dudoit et al., 2002): Dudoit, S. & Fridlyand, J. (2002) A prediction-based resampling method for estimating the number of clusters in a dataset. Genome Biology, 3(7): 0036.1-21.
(Tseng et al., 2005): Thalamuthu, A, Mukhopadhyay, I, Zheng, X, & Tseng, G. C. (2006) Evaluation and comparison of gene clustering methods in microarray analysis. Bioinformatics, 22(19):2405-12.

In your case, you need some internal indices since you have no a priori clustering structure. There exist tens of internal indices, like:

Silhouette index (implementation in MATLAB)
Davies-Bouldin
Calinski-Harabasz
Dunn index (implementation in MATLAB)
R-squared index
Hubert-Levin (C-index)
Krzanowski-Lai index
Hartigan index
Root-mean-square standard deviation (RMSSTD) index
Semi-partial R-squared (SPR) index
Distance between two clusters (CD) index
weighted inter-intra index
Homogeneity index
Separation index

Each of them have pros and cons, but at least they'll give you a more formal basis for your comparison. The MATLAB toolbox CVAP might be handy as it contains many internal validity indices.

More information:

Determining the number of clusters in a data set
How to determine which method is the most reasonable clustering results?
How can I detect if my dataset is clustered or unclustered (i.e. forming one single cluster)?
How can we choose a "good" K for K-means clustering?
Cluster analysis in R: determine the optimal number of clusters


Does it matter whether we put regularization parameter ($C$) with error or weight term in Kernel ridge regression?

Both formulations lead to the same solution if you correctly choose $C$ for both cost functions and if $C>0$.

If we have the regularized loss

$$J_1=\dfrac{1}{2}\sum_{n=1}^Ne_n^2+\dfrac{1}{2}C\sum_{k=0}^pw_k^2$$

we will have strong regularization for larger $C$ and small regularization for small positive $C$.

If we devide the loss $J_1$ by the positive $C$ we obtain the loss

$$J_2= \dfrac{1}{2C}\sum_{n=1}^Ne_n^2+\dfrac{1}{2}\sum_{k=0}^pw_k^2.$$

As $C>0$ we just scaled our loss funtion $J_1$, hence the minimum will not change. But the interpretation of $C$ will change if we replace it with its inverse as proposed in your question to obtain the loss

$$J_3= \dfrac{1}{2}\tilde{C}\sum_{n=1}^Ne_n^2+\dfrac{1}{2}\sum_{k=0}^pw_k^2.$$

With $\tilde{C}$ very small we will have strong regularization. And for large $\tilde{C}$ we will have small regularization. That is the reason why we sometimes call $\tilde{C}$ the inverse regularization parameter (e.g. see support vector machines).


ML algorithm to predict the state of state of a device (ON/ OFF) based on day, time and location

You need to predict 2 classes (on/off). The first question is: Based on what data (exactly) this happens?

Problem statement

You do have some training data, so this is a supervised learning task. The algorithm choice really depends on the question what data exactly to need and what kind of patterns you expect to detect:

is the prediction based on a single user (data will be scarce then) or on multiple users (enough to determine similar users for predictions)
does it really suffice to consider Daytime + Week/Weekend? What about public holidays or seasons, where the usage behavior likely changes
Will vacation times impact the usage behaviour beyond date/time?

Picking an algorithm

Once you now based on which data you will recognise what kind of pattern you think about algorithms that fit well. For example: If you really only need time + weekday/weekend then very simple algorithms might do. Decisions trees for example (with simple rules as if weekday + 7.a.m. then turn on).

If rules need to be a lot more complex (and I think this is the case here) you might need a more complex model. I have very little experience with time based predictions but things like recurring neural networks come to mind which are heavily used for time based predictions (e.g. web analytics in order to find anomalies in usage patterns for any given point in time).

Concerns

Your problem sounds like you need a very accurate prediction (minute-wise accuracy) while the actual user behaviour will be a bit more random (some times earlier, some times later). Many algorithms try to reduce an error which often leads to in-between values if the actual time of an event fluctuates around a mean value.

When using neural networks you can influence how the error is calculated and you can train it in order to favour one kind of error (being early) over another. This makes it more complex, and requires that being "too early" is not an issue (or that the user is always very reliable).


What's the difference of calculating features directly vs having a CNN interpret it?

what if my data is such that I can interpret, whether it's of a certain type by doing direct measurements on the data?

I love machine learning as much as the next guy, but if there is a known way to map your input to your desired output, you should use that, period.

Or perhaps it's that CNN becomes viable when there are so many measurable parameters that trying to infer all of them by manual algorithm becomes laborious?

If speed becomes an issue, then indeed, you may need to consider alternatives, including training a model. There are indeed situations where a learned function may be a bit more simplistic, but good enough and faster than an exact algorithm.

However, these scenarios are quite rare. Inference with neural networks may require non-negligible computational power, so I would guess that in most situations, the exact algorithm will be faster regardless.


Expected shape of parameter arrays
Each layer has two arrays:

one for the weights, which has a shape of (num_inputs, num_outpus)
one for the biases, which has a shape of (num_outputs)

Here the num_inputs is the number of input features to that layer and the num_outputs is the number of outputs from that layer (this is what you select when instantiating a layer).
Output of .get_weights()
Another important thing to note is the usage of .get_weights(). In fact there are to ways to use it:

From the model, i.e. model.get_weights(): This will return a flattened list containing all parameter arrays in order. For example, it could look like this: [layer1_weights, layer1_biases, layer2_weights, layer2_biases, ...]

From a layer, i.e. layer.get_weights(): This is what you used. Here it will return the parameter arrays for a given layer. For example model.layers[1].get_weights() will return the parameter arrays for layer1. If layer1 has biases then this will return two arrays, one for the weights and one for the biases.

I took the liberty of changing your code a bit to make this a bit more clear.
import numpy as np
import tensorflow as tf

f = lambda x: 2*x

Xtrain = np.random.rand(400, 5)  # 5 input features
ytrain = f(Xtrain)
Xval = np.random.rand(200, 5)  # 5 input features
yval = f(Xval)

model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(10, activation='relu'),  # this layer has 5 inputs and 10 outputs
    tf.keras.layers.Dense(1, activation='relu')  # this layer has 10 inputs and 1 output
])

model.compile(optimizer='adam',
              loss=tf.keras.losses.MeanSquaredError()
             )

model.fit(Xtrain, ytrain, epochs=1, verbose=0)

# I will be calling .get_weights() directly from the model,
# so we expect 4 arrays: 2 for each layer.

print('First layer weights:', model.get_weights()[0].shape)
print('First layer biases:', model.get_weights()[1].shape)
print('Second layer weights:', model.get_weights()[2].shape)
print('Second layer biases:', model.get_weights()[3].shape)

The output:
First layer weights: (5, 10)
First layer biases: (10,)
Second layer weights: (10, 1)
Second layer biases: (1,)


forecast monthly shipment (time series) for 300 products individually in R

It is difficult to answer this question with the limited information provided here. Nonetheless, I have following recommendations.

Do you have a (data backed) reason to believe that you do not need 300 different models? For example, are the shipments of different products correlated? Alternatively, can you instead model the problem at higher levels like product categories? If either case, you can take a look at hierarchical time series modeling. This journal article by Rob Hyndman, author of forecast package in R can get you started on hierarchical time series modeling.


Returning rows from a list of indexes in Python Pandas

I want to adress two issues. One is how you get your CSV and the other is what probably doesn't work in your code.

how to get the associated values

What you try to do is a nice task for the shift() method

# you probably don't need the semicolons at the end of the line, right?
# if you want to get rid of them, you can do:
df= pd.read_csv(
    r"CSVFILEPATH",   # your file
    engine='python',  # use the python engine instead of C to use a regex as separator
    sep=r'[,;]',      # use ; as an alternative separator
    usecols=range(3), # exclude the last column (after the ;)
    names=range(3))   # assign names, if you like you can also assign a list of more verbose column names here (this just uses numbers)

# create a dataframe that is a version of the original
# which is just one row shifted to the top
df_shifted= df.shift(-1)

# concatenate it with the original data frame and assign unique column names
df_concat=  pd.concat([df, df_shifted], axis='columns')
df_concat.columns= range(6)

# apply your filter
newdf= df.loc[df_concat[0].str.contains('LLRT|LLTX|LRRT|LTRC|LV10|LV3|LEDR|LTRV|LES2|LES1', regex = True)]

This outputs:

In [44]: newdf
Out[44]:
             0    1     2         3    4    5
0  OBSERV\LTRC  CL1   0.0  OBVAL\14  NaN  0.0
2  OBSERV\LTRC  CL1  10.0  OBVAL\14  NaN  0.0
4  OBSERV\LTRC  CL1  20.0  OBVAL\14  NaN  0.0
6  OBSERV\LTRC  CL1  30.0  OBVAL\14  NaN  0.5

From the following test data you provided:

import io
import pandas as pd

raw=\
r"""OBSERV\LTRC,CL1,0,10.00;
OBVAL\14,,0,V;
OBSERV\LTRC,CL1,10,20.00;
OBVAL\14,,0,V;
OBSERV\LTRC,CL1,20,30.00;
OBVAL\14,,0,V;
OBSERV\LTRC,CL1,30,40.00;
OBVAL\14,,0.5,V;"""

df= pd.read_csv(io.StringIO(raw), engine='python', sep=r'[,;]', usecols=range(3), names=range(3))

Code, that probably doesn't do, what you intend to do

# the following line references a method, but doesn't call it:
df.set_index
# if you execute this line, it outputs:
df.set_index
Out[18]:
<bound method DataFrame.set_index of              0    1     2
0  OBSERV\LTRC  CL1   0.0
1     OBVAL\14  NaN   0.0
...
# look at the <bound method part, this is the __repr__ string
# of the object (bound methods are objects themselfes)
# if you execute this line inbetween a script, it has no effect
# at all (just maybe slows down execution a very tiny bit)
# because you don't do anything with the returned object

You wrote, you get the message generator object <genexpr> at 0x0000028057C33648>]. This is not an error method. Like the bound method message above, this is also the __repr__ string of an obect. In this cas a generator object. If you call append on a list, it treats the argument passed in append as one object. I guess you rather wanted to add the indexes increased by one to an existing list. That can be done with the following code:

# create a copy of the list to avoid funny results
new_indices=list(observ_ind)
# add the elements of returned by the generator object
# to the list (rather than the generator object itself
new_indices.extend(i + 1 for i in observ_ind)


How to evaluate data capability to train a model?

1.2M is about 2^16.
You have 512 features plus the concept so the number of possibilities is much larger.
Therefore, you can claim that the number of samples you have is too small.

Though that, machine learning is done almost always in a situation that you have less samples the all permutations.

The VC dimension allow estimate how well a data set fits the underlying distribution. It applies to an hypothesis class (e.g., linear functions) whose size is typically much smaller than the number of permutations.

Note that both the number of permutations and the VC dimension refer to worse case scenarios. It is possible that on of the 512 is identical to the concept. In this case you will be able to learn though the other 511 features.


Divide a column by itself with mutate_at dplyr

mutate_at(my_data, vars(starts_with("INV")), sign)

Although I see your INV1 column is a date? I'm not sure how much that'll affect things..


You don't really need to implement an algorithm to achieve this. There are a few tools that will do this for you.

You can get the data assigned to buckets for further processing using Pandas, or simply count how many values fall into each bucket using NumPy.

Assign to buckets

You just need to create a Pandas DataFrame with your data and then call the handy cut function, which will put each value into a bucket/bin of your definition. From the documentation:

  Use cut when you need to segment and sort data values into bins.

In [1]: import pandas as pd
In [2]: import numpy as np    # to create dummy data

Create some dummy data, put it in a dataframe and define the bins:

In [3]: data = np.random.randint(low=1, high=10001, size=1000)
In [4]: df = pd.DataFrame(data=data, columns=["data"])
In [5]: bins = np.array([1,5,25,50,150,250,1000,5000,10000])

Pass the data, along with the bin definitions to the cut function and assign it back as a new column in the dataframe:

In [6]: df["bucket"] = pd.cut(df.data, bins)

You can then inspect the first few rows to see that the values have now been labelled with the relevant bucket:

In [7]: df.head()
Out[7]:
   data         bucket
0  8754  (5000, 10000]
1  2970   (1000, 5000]
2  6778  (5000, 10000]
3  2550   (1000, 5000]
4  5226  (5000, 10000]

Counting how many in each bucket

Here is an example using NumPy, to get an idea of the distribution, as a histogram.

Using the data and bins as defined above, we pass them to the numpy histogram function, which will count how many data points fall into each bin:

In [8]: np.histogram(data, bins)
Out[8]:
(array([  0,   2,   1,   8,   6,  61, 417, 505]),
 array([    1,     5,    25,    50,   150,   250,  1000,  5000, 10000]))

Where the first row tells you how many values fell into each bin, and the second row confirms the bins used.

You can get your dictionary of data into the same form as my dummy data above (into a numpy array) by doing this:

data = np.array([v for v in your_dict.values()])


Cross-entropy via softmax: Mathematical derivation

Hints:

$P(Z|\theta)$ is independent of $\hat{y}$ so it can be left out as merely a multiplicative factor as we are interested in $\hat{y}$.
$\theta$ is a parameter and for each model is indeed fixed, it is not a variable


What is the easiest way to scale a data science project based on scikit stack?

You generally don't.  Scikit-learn is primarily aimed to help new data scientists quickly get comfortable with data science

That being said, some strategies for scaling are discussed here: http://scikit-learn.org/stable/modules/scaling_strategies.html

This includes using out-of-core models, reducing data size with PCA, and various incremental learners

Besides that, your best bet is to use a beefier computer

Also, remember that once a model is trained, it can be pickled and shared.  Training/testing is usually the time/cpu consuming process.  So, once you have a model, you should be able to implement it on machines less beefy than the train/test machine


Needed: Java library to calculate text readability/complexity

I looked for this as well and found only the unmaintained RAT library.
But basically all of these algorithms (most are described as formulas in this paper) can be directly translated from the single Python file with no extra dependencies.
One exception was the Linsear Write formula because of its needing a syllable counting method. But if you use the CMU pronunciation dictionary or equivalent, you can easily build the countSyllables method and the rest of the algorithm is trivial.


Performing machine learning on small datasets

With 5-fold cross validation, in each fold you're reducing your training dataset to 64 observations and evaluating against 16 observations. Assuming your data is balanced and you're stratifying folds, you're only giving your models 32 observations from each class to learn from, and misclassification of a single test set observation results in a 6.25% point change in accuracy for that fold. Even if it's classified correctly in the other folds, that single misclassification will still have a 1.25% point change on the across-folds average. Yikes. It should be no surprise your models aren't performing well: not only do they not have much data to train on, but your evaluation methodology is extremely unforgiving.

So yeah, your data is pretty darn small. Off the top of my head, I can think of a handful of general strategies you can potentially use to address this:

Use more of your data for training. Instead of 5-fold CV, try leave-one-out or bootstrap.
Generate fake data. This probably won't get you very far, but it's at least an option. Check out the SMOTE algorithm.
Transfer learning. Depending on what you're doing, it might be possible to leverage a pre-trained model and then tweak it slightly to suit your needs. If this is an option it can be extremely powerful, but chances are this won't be something you can reasonably pursue. Anyway, here's an article demonstrating this on another small data medical example: a pre-trained general purpose image classifier was trained to detect cancer from just 600 images! https://arxiv.org/abs/1711.10752
Go Bayesian. Bayesian methods allow you to incorporate outside information as a "prior belief". If you have subject matter expertise (or better yet, citations) that you can use to set your expectations for values of the model parameters or hyperparameters, bayesian methods will allow you to incorporate that information explicitly, and this can in turn help your model find good parameters faster since it doesn't need to learn everything about your problem directly from the little data available to it. If you're not careful, this can result in you giving yourself the model you want to have, in which case your evaluation metrics may not be as informative as you think they are. Here there be dragons: this approach is powerful, but it can be hard to do correctly, and playing with the prior is basically an invitation for people to be skeptical of your methods even if what you're doing is sound.
Get more data. I'm guessing this isn't an option or you would have done so already. But if there's a chance it's out there: go find it. Try poking around the literature associated with this condition, maybe you'll get lucky and find a public dataset. If you're feeling bold, you could try emailing other researchers and just ask them politely if you can use their data.


About the type of problem: apparently here the goal is not to do any kind of supervised learning, it seems to be more a kind of descriptive task.
The first thing I would try to do is to align the two datasets, i.e. merge them based on events in one occuring at the same time and place as in the other one. In order to do that you need to be able to compute for any two events whether they appear in the same area around the same time. I guess you'll need a window, for example defining two events as related if they occur within 1h of each other and are distant by less than 10km. Be careful about noise in the data: the time of a telecom incident might be the time it's reported, not the time it actually happens.
Once you have linked potential related events, you can merge the two datasets into one. You should probably preserve the events which have no match in the other dataset for statistical purposes, so use an outer join.
At this stage you have an exploitable dataset. I would start by doing plots such as overall probability distribution of related events, probability of related event by location. Probability of related events across time.


Useful metrics in such scenario are:

F1 Score (and precision / recall)
ROC Curves  (Metric is : Area Under the ROC Curve (AUC))

Few articles on how to choose metrics for a specific project are:

Evaluation Metrics, ROC-Curves and imbalanced datasets by David S. Batista,
What metrics should be used for evaluating a model on an imbalanced data set? by Shir Meir Lador,
Choosing the Right Metric for Evaluating Machine Learning Models — Part 2 by Alvira Swalin.


I need sources of interrogative, exclamatory, and imperative sentences

I ultimately settled on Gutenberg Project as a source of massive amounts of data. I realized that books (fiction and non-fiction) cover pretty much every permutation of human communication that can be represented in plain text. I've written a couple of scripts to parse Gutenberg text files and next I'll be working on a support vector machine to classify each passage as regular content or other (such as headers, titles, etc).

https://www.gutenberg.org/wiki/Gutenberg:Information_About_Robot_Access_to_our_Pages


How exactly does adding a new unit work in Cascade Correlation?

I've been reading up on cascade correlation quite a bit recently and made a python implementation https://github.com/DanielSlater/CascadeCorrelation (though it still needs a bit of cleaning up/extra work and has a bunch of me mucking around with using Particle Swarm Optimization for selecting candidates, definitely not production ready).

To try and explain step 3.

Start by creating a number of candidate hidden nodes with random weights. These have incoming connection from all existing hidden nodes and input nodes.
We then use that equation $$S=\sum_o \left | \sum_p(V_p - \overline{V})(E_{p,o} - \overline{E_o}) \right |$$ to train the candidate nodes.
The residual output error is the difference between the output of the network and the target value(think sum of squared error without the square).
$S$ is the correlation between the activation of our candidate node and the the residual error.
$V_p$ is the activation of the candidate node given input $p$.
After a bit of backprop training against $S$ we choose our best candidate. This becomes a new hidden node.
This is when it's weights are frozen. That is to say after random initialization then back-prop training, then selecting the best one.


Forecasting Multiple (few hundreds) uni-variate time series with inflated zeros

I can suggest auto.arima function from forecast library, if you are R user, if you are Python user then follow this link. All you need is to write simple for loop, which allows you to built best ARIMA models in different geographic locations:

   for (my_time_series in set_of_all_time_series){
        model=auto.arima(my_time_series )
         }

You can cluster your time series by correlation (make sure that your time series are stationary to avoid spurious correlation). If this reduces the number of time series (which will depend on threshold on correlation), you can take any 1 member from each class, build any model (not only ARIMA) and apply model results on each member of that class.
Contract VAR model


Is there a relationship between learning rate and training set size?

I wonder why would you tune the learning rate (too much) at all? On many platforms you can include something like a learning rate scheduler. It will decrease the learing rate when no more learning progress is made. So you "tune" the learning rate "on the fly" while training.
With Keras this could look like:
early_stopping = EarlyStopping(monitor='val_loss', patience=50, mode='auto', restore_best_weights=True)
reduce_on_plateau = ReduceLROnPlateau(monitor="val_loss", factor=0.9, patience=20, cooldown=5, verbose=0)
checkpoint = ModelCheckpoint("xy.hdf5", monitor='val_loss', verbose=0, save_best_only=True, save_weights_only=True, mode='min')
...
model.fit(...,callbacks=[early_stopping, reduce_on_plateau, checkpoint]

The callbacks above introduce "early stopping" (to stop if no more progress is made), "reduce on plateau" (decrease learning rate when required), and "checkpoint" (save model results on each epoch).
Start with some reasonable standard value for the learning rate and decrease if necessary.


Looking for spatial clusters and anomalies. Is DBSCAN the right tool?

Clustering in X,Y,Value
Clustering in 3d is great. But be careful with feature scaling in this case. Presumably X, Y have the same scales - so unless you want to treat the different directions differently, make sure not not apply any normalization. As that would distort the grid.
Your Value column on the order hand, might be on a very different scale from the X,Y values. If not corrected for this creates an implicit and usually quite arbitrary feature weighting, which is rarely desirable.
If you want to weight the Value column equally to X and Y, then scale the Values by the either mean,std or max,min of the X/Y columns.
If you do want to weight the individual attributes differently, then you can introduce additionally some weighting factors that scales them.
PS: remember that DBscan hyperparamters, such as epsilon is also given in the same unit as your feature space.
Anomalies in Value given X,Y location
Often an interesting question is "is this value anomalous compared to the normal for this location?". This can be done on X and Y axis individually, but often there are interactions between X,Y and one should consider the two dimensions together. Here are some approaches for that:
Binning the 2D (X,Y) space
Divide the X and Y axis into a grid. Typically with uniformly sized cells. Assign each grid cell an location identifier. Then use this identifier as a feature.
This approach is simple and with a lot of dense data it usually works pretty OK. But the grid can fit the data pretty poorly, datapoints very close to eachother end up in different cells, and then quite distant in feature space.
Clustering in the 2D (X,Y) space
An alternative location scheme is to use clustering. Cluster the values just based on X,Y - and then use the cluster number as the location identifier. You may also want to include distance from the cluster center as a feature - since some values might fall pretty far from a cluster.
Angle from the cluster can in some cases also be informative.
This has a much better shot at creating good location features for the data. But is more complex. One also should consider when one wants to re-cluster.
Aside
Make sure to plot your data and look at it! For example use a 2d-plot with X,Y as coordinates, and the values using color. You may also want to do the same kind of plot, but then to use Anomaly Score as the color. If you upload a plot here, it will also be much easier to give good advice.


Any guidance for new beginners interested in data science

In order to learn data science you should know about main tools and ideas in the data scientist's toolbox.
Two steps which most newbee's follow are:

The first is a conceptual introduction to the ideas behind turning data into actionable knowledge which involves following steps:

    Getting and Cleaning Data,Data Analysis,Reproducible Research

    Statistical Inference, using Regression Models and Machine Learning

The second is a practical introduction to the tools that will be used in the program like version control, markdown, git, GitHub, R, and RStudio.

I would advice you to complete a general course on data science from any website or tutorial you find easy for example I personally like this course: https://www.coursera.org/specializations/jhu-data-science

And then start building some projects of your own for example in my free time I build a lot of small data science projects playing with any available data like US census, petroleum data etc. For example see following link for one of my many data science projects:
https://arjun-chaudhary.shinyapps.io/Data_Analytics/


Scoring metric for recommendation system

Other ways to measure real-valued vector distances:

Euclidean distance
Manhattan distance
Higher order Minkowski distance
Chebyshev distance

Beyond just trying different distance metrics, you can try re-framing the problem or using A/B testing to see which recommendations are most empirically useful for users.


In this paper a state of the art method (unsupervised smoothed inverse frequency) is described, you can find an implementation of this method here.


Which term is correct Datafication or Datification?

It's a neologism, there's no standard spelling or even meaning.
For the record there are related terms which are more standard: "data representation" (usually low-level) or "knowledge representation".


How to mathematically explain the translational and rotational invariance of PCA

Let $e$ be the all-one column vector of length $n$.

Let $X\in \mathbb{R}^{n\times d}$ be the original matrix.

and $X'$ be the transformed matrix, that is we have $X'=XR^T+ev^T$.

Let's first compute the transformed mean assuming the mean of $X$ is $0$.

$$\frac{e^TX'}{n}=\frac{e^TXR'}{n}+\frac{e^Tev^T}{n}=v^T$$

To compute the singular vectors of the original matrix, theoretically, we can compute the eigenvalues of $$X^TX=UDU^T$$

And we can extract $W$ from $U$.

To compute the PCA for the transformed data

We have $$(X'-ev^T)^T(X'-ev^T)=(XR^T)^T(XR^T)=R^TX^TXR^T=RUDU^TR^T$$

Now, we can extract $W'$ from $RU$.

That is $$W'=RW.$$


Simple, optimization problem of SVM is of quadratic order.
Just check first line of documentation

  "The fit time scales at least quadratically with the number of samples and may be impractical beyond tens of thousands of samples."


How to change Linear model in SGDClassifier scikit learn?

The specific linear classifier can be defined with the loss function argument. The options are { ‘hinge’, ‘log’, ‘modified_huber’, ‘squared_hinge’, ‘perceptron’}. For example, hinge loss is equivalent to a linear SVM and log loss is equivalent to Logistic Regression.


Like the update rule for bias terms in dense layers, in convolutional nets the bias gradient is calculated using the sum of derivatives of Z terms:

$$ dJ / db = \sum_h \sum_w dZ_{hw} $$

which J is the cost function, w is the width of the activation after convolution and h is the height of the activation after convolution.

db is computed by summing  dZs. It means you are summing over all the gradients of the conv output (Z) with respect to the cost.

Calculating the error of the net depends on the cost function that you have used. Depending on using cross entropy or mean-squared-error or other justified cost functions, you may have different update rule for other parameters of your net. But if you use cross entropy which is common for variants of classification tasks, the above update rule is used for updating bias terms.


Gerrymandering - Geospatial optimization to maximize votes in R

I found a number of interesting articles on mathematical analyses of gerrymandering. Hopefully some of these might give you some ideas on detecting / optimising against gerrymander.

Gerrymandering and Legislator Efficiency
Gerrymander and Convexity
Recursive Algorithm with Examples
Mary Solbrig's Automated District Model


Model that predicts probability of correctness of another model

This is a problem known as quality estimation (QE) in the application of Machine Translation (there's a regular Shared Task about it). The goal is to train a model to predict the quality of the output of a ML system.
There is an important theoretical obstacle to this kind of task: if such a QE model was able to predict perfectly, then in theory it would be possible to create a perfect model for the original task by trying many different models/parameters until the QE system predicts a high level of quality. What this means practically is that a QE model can only be as good as the system it's trying to estimate since it relies mostly on the same information, otherwise the system is clearly sub-optimal. Essentially the best the QE system can do is to determine how hard it is to correctly predict a particular input image, but it cannot predict with any certainty whether the actual prediction is correct or not.


How to perform data scaling/standardization on dataset containing grouped values?

There's not one right answer to this question, because what scaler works best really depends on the data and the algorithm you use to make the prediction. You should try different scalers combined with different algorithms to decide which preprocessing is best by comparing the cross validation results of each pipeline.
Of course, you don't have unlimited time. I would:

First look at the distribution of cost globally and see if you think you need to transform the target to make it more normal
Consider a tree-based algorithm for a first-pass which won't require you to scale your Problem_Size
Scale globally first, and then if you have time experiment with scaling based on problem type

You'll eventually run out of time or patience, but I think going in this order will make the most efficient use of your time.
Remember, there's not 1 best way to do it for every problem.


Search "dataset" instead of "database".
see these:

A Neural Network from Scratch in just a few Lines of Python code
Multi Layer Neural Networks with Sigmoid Function Deep Learning for Rookies


If you are looking for only for identical copies for files, then you can calculate a hash like md5sum on each file and compare that, which is much faster than analysing the content.

If you’re looking for a similarity metric then I think you should be able to use a faster algorithm than the one that SequenceMatcher uses, which finds the longest exactly matching sequence, which takes quadratic time on average. I suggest you look at other similarity metrics, such as Levenshtein or Hamming distance, which will be faster to compute.


Using Transaction Amount to Guide Learning in an Fraud Detection Machine Learning Model

Since you are using boosting models you can add something named monotonic constraints:
It is often the case in a modeling problem or project that the functional form of an acceptable model is constrained in some way. This may happen due to business considerations, or because of the type of scientific question being investigated. In some cases, where there is a very strong prior belief that the true relationship has some quality, constraints can be used to improve the predictive performance of the model.
I also recommend to read this post


Yes, the full ImageNet is a superset of the 1k version (see Benchmarking Neural Network Robustness to Common Corruptions and Perturbations). The 1k synsets (a.k.a. classes) corresponding to the 1k version are also present in the full 22k version. So there are 21841 classes in total, 1000 of them are exactly the same as in the 1k version (and images are exactly the same), and the other 20841 can be treated as OOD to the 1k version dataset (at least they are treated so in one of the papers by Hendrycks et al, Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty.


How do I recommend items to out of training users based on its recent views?

Making predictions for new users is often called the cold start problem.
This problem is difficult to over come in a purely item-to-item recommendation system. The most common solution is to include more than just item interaction information. Examples include user profile information and item content information.


Why Gaussian latent variable (noise) for GAN?

Why people often choose the input to a GAN (z)
  to be samples from a Gaussian?

Generally, for two reasons: (1) mathematical simplicity, (2) working well enough in practice. However, as we explain, under additional assumptions the choice of Gaussian could be more justified.

Compare to uniform distribution. Gaussian distribution is not as simple as uniform distribution but it is not that far off either. It adds "concentration around the mean" assumption to uniformity, which gives us the benefits of parameter regularization in practical problems.

The least known. Use of Gaussian is best justified for continuous quantities that are the least known to us, e.g. noise $\epsilon$ or latent factor $z$. "The least known" could be formalized as "distribution that maximizes entropy for a given variance". The answer to this optimization is $N(\mu, \sigma^2)$ for arbitrary mean $\mu$. Therefore, in this sense, if we assume that a quantity is the least known to us, the best choice is Gaussian. Of course, if we acquire more knowledge about that quantity, we can do better than "the least known" assumption, as will be illustrated in the following examples.

Central limit theorem. Another commonly used justification is that since many observations are the result (average) of large number of [almost] independent processes, therefore CLT justifies the choice of Gaussian. This is not a good justification because there are also many real-world phenomena that do not obey Normality (e.g. Power-law distribution), and since the variable is the least known to us, we cannot decide which of these real-world analogies are more preferable.

This would be the answer to "why we assume a Gaussian noise in probabilistic regression or Kalman filter?" too.

  Are there also potential problems associated with this?

Yes. When we assume Gaussian, we are simplifying. If our simplification is unjustified, our model will under-perform. At this point, we should search for an alternative assumption. In practice, when we make a new assumption about the least known quantity (based on acquired knowledge or speculation), we could extract that assumption and introduce a new Gaussian one, instead of changing the Gaussian assumption. Here are two examples:

Example in regression (noise). Suppose we have no knowledge about observation $A$ (the least known), thus we assume $A \sim N(\mu, \sigma^2)$. After fitting the model, we may observe that the estimated variance $\hat{\sigma}^2$ is high. After some investigation, we  may assume that $A$ is a linear function of measurement $B$, thus we extract this assumption as $A = \color{blue}{b_1B +c} + \epsilon_1$, where $\epsilon_1 \sim N(0, \sigma_1^2)$ is the new "the least known". Later, we may find out that our linearity assumption is also weak since, after fitting the model, the observed $\hat{\epsilon}_1 = A - \hat{b}_1B -\hat{c}$ also has a high  $\hat{\sigma}_1^2$. Then, we may extract a new assumption as $A = b_1B + \color{blue}{b_2B^2} + c + \epsilon_2$, where $\epsilon_2 \sim N(0, \sigma_2^2)$ is the new "the least known", and so on.
Example in GAN (latent factor). Upon seeing unrealistic outputs from GAN (knowledge) we may add $\color{blue}{\text{more layers}}$ between $z$ and the output (extract assumption), in the hope that the new network (or function) with the new $z_2 \sim N(0, \sigma_2^2)$ would lead to more realistic outputs, and so on.


Potential problems with expanding training set

One problem is that you insert correlated data points. Different days from the same user (may) have some underlying correlations that the days from different users don't. These correlations might be relevant for the classification task, and the model might learn them, instead of learning the relevant features.

This doesn't mean it's a bad thing. Data augmentation is a thing. For example, a common practice when training models such as CNNs on images is to perform data augmentation by rotating/scaling the initial images.

However, make sure you don't touch the validation data. The data augmentation is performed after the train-validation split. The validation data should be as "raw" as possible. You can test it this way, and see how it impacts the performance of your model.


How to extract features from long chemical names?

You can train an RNN with character embeddings. This can be done by splitting the name into sequences of chars and vectorize them numerically. If you are working with Keras, you can feed them into an Embedding() layer that will learn how to represent characters. RNN layers will then process their sequence. At the output node, your Network will perform a classification ('good'/'bad').


Artificial neural network on ternary content addressable memory

It is mostly a performance benefit thing. You can run a neural network on a computer (slow or not). A TCAM is the only real hardware suited to the problem of seeing the cause and supplying the effect, TCAM can perform hopfield net operations very natively in pure hardware.

It puts the network in a hardware spreadsheet formfactor.  It's really simple, and can be bewilderingly so, you have a linear amount of patterns, (which I like to call bullets, because it's like a machine gun ready to fire off and enter in patterns really damn fast, faster than a minigun)    and requires a lot of miniaturization to have a large amount, say a billion bullets,  more so than just ordinary RAM.

If you ever wanted to see a "real" robot in action,  it's going to use hardware TCAM.


Taking the log doesn't result in a normally-distributed target; it would tend to if the target was log-normally distributed, and you have something normalish there, not quite. But, this distribution isn't actually what matters.

What taking the log does is change your model of how errors arise when fitting a regressor. You're now saying that the target values are $e^{P + \epsilon}$ where $P$ is your model's prediction and $\epsilon$ is Gaussian noise. Or: $e^P e^\epsilon$. That part directly interacts with the assumptions in your regressor.

So what you're finding is that on average the predictions are wrong by a factor of 7.39, not +/- $7.39.

What you really want to do is evaluate MAE on actual target values vs $e^P$ . You probably have a better model but not that much better.


I recommend trying both (more than once), and exploring any differences. In my experience, using the same set of folds for all models or using a new set of folds for each model doesn't make any material difference. Post if you find different!

Regarding "I'm performing first variable selection, then hyperparameters tuning through cv", maybe watch https://www.youtube.com/watch?reload=9&v=S06JpVoNaA0 to be sure you are not introducing any bias.


What kind of machine learning can predict a probability rather than classification?

Most of the classification algorithms can return a probability or a similar measure.

Usually, you don't get black or white, but nuances of grey, and then based on a threshold, you select one or the other answer.


LSTM layers work on 3D data with the following structure (nb_sequence, nb_timestep, nb_feature).

nb_sequence corresponds to the total number of sequences in your dataset (or to the batch size if you are using mini-batch learning).
nb_timestep corresponds to the size of your sequences.
nb_feature corresponds to number of features describing each of your timesteps.

Thus an LSTM layer will work like this :
Let $t_i$ be the $i^{th}$ timestep of sequence $seq_j$, with $i \in [0, nb\_timestep], \ j \in [0, nb\_sequence]$.
An LSTM layer will make a prediction $p_i$ according to the $nb\_feature$ descriptors of $t_i$ with respect to its hidden state which is a representation of the timesteps $t_0$ to $t_{i-1}$.

Now, let's see what this means for your two configurations. For the sake of the explanation, I will suppose that we have sequences of words

For (53394, 3, 1) the LSTM will work on 53394 different sequences. Each sequence is 3 words long and each word is described through one feature only. For the first word of each sequence, the LSTM will make a prediction on its sole descriptor. For the second word, the prediction will be done from the unique descriptor with respect to what the first word was. Finally, for the third and last word in the sequence, the LSTM will emit a prediction from the unique descriptor with respect to what the two previous words were. Then, the LSTM begins the process anew for the following sequence.
For (53394, 1, 3), your sequences contain only one word which is described through 3 features. Sequences of one word are not really sequences, so the LSTM layer will not be useful in this case.

Hope it clears up how data are fed to an LSTM !

NB. Not related to the question but it may help : from your original shape, it seems your dataset contains 53394 words described with 3 features. If I am right, you would need a 3D shape like (53394, nb_timestep, 3) but with $nb\_timestep \neq 1 $. What you need then, is to define some window instead of reshaping your data.


Extracting Useful features from large convolutional layers

Applying a pooling layer following a convolution is a standard way to reduce the size of the input matrix and get the invariant features. You might also want to consider adding a dense layer with a smaller number of output neurons.


Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

You are correct. For $n > 1$, the multiplication of derivatives does not necessarily go to zero,  because each derivative could be potentially larger than one (up to $n$).

However, for practical purposes, we should ask ourselves how easy it is to maintain this situation (keeping the multiplication of derivatives away from zero)? Which turns out to be quite hard compared to ReLU, which gives derivative = 1, specially now, when there is also a chance of gradient explosion.

Introduction

Suppose we have $K$ derivatives (standing for depth $K$) multiplied together as follows
$$g=\left.\frac{\partial f(x)}{\partial x} \right|_{x=x_1}\cdots \left.\frac{\partial f(x)}{\partial x} \right|_{x=x_K}$$
each evaluated at different values $x_1$ to $x_K$. In a neural network, each $x_i$ is a weighted sum of outputs $\boldsymbol{h}$ from previous layer, e.g. $x = \boldsymbol{w}^t\boldsymbol{h}$.

As $K$ increases, we want to know what it takes to prevent the vanishing of $g$. For example, for the case of$$f(x)=tanh(x)$$we cannot prevent that because each derivative is smaller than one, except for $x=0$, i.e.
$$\frac{\partial f(x)}{\partial x}=\frac{\partial tanh(x)}{\partial x}=1-tanh^2(x) < 1 \text{  for } x \neq 0$$
However, there is a new hope based on your proposal. For $f(x)=tanh(nx)$, derivative could go up to $n > 1$, i.e.
$$\frac{\partial f(x)}{\partial x}=\frac{\partial tanh(nx)}{\partial x}=n\left(1-tanh^2(nx)\right) < n \text{ for } x \neq 0.$$

When are forces balanced?

Now, here is the core of my analysis:

  How far $x$ needs to move away from $0$ to have a derivative smaller than
  $\frac{1}{n}$ to cancel out $n$ which is the maximum possible derivative?

The farther $x$ needs to move away from $0$, the harder it is to produce a derivative below $\frac{1}{n}$, thus, the easier it is to prevent the multiplication from vanishing. This question tries to analyze the tension between good $x$'s close to zero, and bad $x$'s far from zero. For example, when good and bad $x$'s are balanced, they would create a situation like
$$g=n \times n \times \frac{1}{n} \times n \times \frac{1}{n} \times \frac{1}{n} = 1.$$
For now, I try to be optimistic by not considering arbitrarily large $x_i$'s, since even one of them can bring $g$ arbitrarily close to zero.

For the special case of $n=1$, any $|x| > 0$ results in a derivative $< 1/1 = 1$, therefore, it is almost impossible to keep the balance (prevent $g$ from vanishing) as depth $K$ increases, e.g.
$$g=0.99 \times 0.9 \times 0.1 \times 0.995 \cdots \rightarrow 0.$$

For the general case of $n > 1$, we proceed as follows
$$\begin{align*}
&\frac{\partial tanh(nx)}{\partial x} < \frac{1}{n}\\
&\Rightarrow n\left(1-tanh^2(nx)\right) < \frac{1}{n}\\
&\Rightarrow 1 - \frac{1}{n^2} < tanh^2(nx) \\
&\Rightarrow \sqrt{1 - \frac{1}{n^2}} < |tanh(nx)| \\
&\Rightarrow x > t_1(n):=\frac{1}{n}tanh^{-1}\left(\sqrt{1 - \frac{1}{n^2}}\right) \\
&\text{or } x < t_2(n):=-t_1(n)=\frac{1}{n}tanh^{-1}\left(-\sqrt{1 - \frac{1}{n^2}}\right)
\end{align*}$$
So for $|x| > t_1(n)$, the derivative will be smaller than $\frac{1}{n}$. Therefore, in terms of being smaller than one, multiplication of two derivatives at $x_1 \in {\Bbb R}$ and $|x_2| > t_1(n)$ for $n > 1$ is equivalent to an arbitrary derivative for $n=1$, i.e.
$$\left(\left.\frac{\partial tanh(nx)}{\partial x}\right|_{x=x_1\in {\Bbb R}} \times \left.\frac{\partial tanh(nx)}{\partial x}\right|_{x = x_2,|x_2| > t_1(n)}\right) \equiv \left.\frac{\partial tanh(x)}{\partial x}\right|_{x = z}, z \in {\Bbb R}\setminus\{0\}.$$
In other words,

  $K$ mentioned pairs of derivatives for $n > 1$ is as problematic as $K$
  derivatives for $n=1$.

Now, to see how easy (or hard) it is to have $|x| > t_1(n)$, let's plot $t_1(n)$ and $t_2(n)$ (thresholds are plotted for a continuous $n$).

As you can see, to have a derivative $\geq 1/n$, the largest interval is achieved at $n=2$, which is still narrow! This interval is $[-0.658, 0.658]$, meaning for $|x| > 0.658$, the derivative will be smaller than $1/2$. Note: a slightly larger interval is achievable if $n$ is allowed to be continuous.

Based on this analysis, we can now reach a conclusion:

  To prevent $g$ from vanishing, around half or more of $x_i$'s need to be
  inside an interval like $[-0.658, 0.658]$

so when their derivatives are paired with the other half, multiplication of each pair would be above one at best (required that no $x$ is far into large values), i.e.
$$\left(\left.\frac{\partial f(x)}{\partial x} \right|_{x=x_1 \in {\Bbb R}}\left. \times \frac{\partial f(x)}{\partial x} \right|_{x=x_2\in [-0.658, 0.658]}\right) > 1$$
However, in practice, it is likely to have more than half of $x$'s outside of $[-0.658, 0.658]$ or a couple of $x$'s with large values, causing $g$ to vanish to zero. Also, there is a problem with too many $x$'s close to zero which is

  For $n > 1$, too many $x$'s close to zero could lead to a large gradient $g \gg 1$ (potentially up to $n^K$) which moves (explodes) the
  weights into larger values ($w_{t+1} = w_{t} + \lambda g$), which
  further moves the $x$'s into larger values ($x_{t+1} = \boldsymbol{w}^t_{t+1}\boldsymbol{h}_{t+1}$) converting the good $x$'s into (very) bad ones.

How large is too large?

Here, I carry out a similar analysis to see

  How far $x$ needs to move away from $0$ to have a derivative smaller than
  $\frac{1}{n^{K-1}}$ to cancel out the other $K-1$  $x$'s assuming they are very close to zero and acquired the maximum possible gradient?

To answer this question, we derive the below inequality

$$\begin{align*}
&\frac{\partial tanh(nx)}{\partial x} < \frac{1}{n^{K-1}} \Rightarrow |x| > \frac{1}{n}tanh^{-1}\left(\sqrt{1 - \frac{1}{n^K}}\right)\\
\end{align*}$$

that shows, for example, for depth $K = 50$ and $n=2$, a value outside $[-9.0, 9.0]$ produces a derivative $< 1/2^{49}$. This result gives an intuition about how easy it is for a couple of $x$'s around 5-10 to cancel out the majority of good $x$'s.

One way street analogy

Based on the previous analyses, I could provide a qualitative analogy using a Markov Chain of two states $[g \gg 0]$ and $[g \sim 0]$ that crudely models the dynamical behavior of gradient $g$ as follows

When system goes to state $[g \sim 0]$, there is not much gradient to bring (change) the values back to state $[g \gg 0]$. This is similar to  a one way street that will be passed eventually if we give it enough time (large enough epochs) given convergence of training does not happen (otherwise, we have found a solution before experiencing a vanishing gradient).

A more advanced analysis of dynamical behavior of gradient would be possible by carrying out a simulation on actual neural networks (which potentially depends on many parameters such as loss function, width and depth of network, and data distribution) and come up with

A probabilistic model that tells how often vanishing happens based on a distribution of gradient $g$ or a joint distribution ($\boldsymbol{x}$, $g$) or ($\boldsymbol{w}$, $g$), or
A deterministic model (map) that tells which initial points (initial values of weights) lead to gradient vanishing; possibly accompanied by trajectories from initial to final values.

Exploding gradient problem

We have covered the "vanishing gradient" aspect of $tanh(nx)$. On the contrary, for the "exploding gradient" aspect, we should be worried about having too many $x$'s close to zero, which could potentially produce a gradient around $n^K$, causing numerical instability. For this case, a similar analysis based on inequality
$$\begin{align*}
&\frac{\partial tanh(nx)}{\partial x} > 1 \Rightarrow |x| < \frac{1}{n}tanh^{-1}\left(\sqrt{1 - \frac{1}{n}}\right)\\
\end{align*}$$
shows that for $n=2$, around half or more of $x_i$'s should be outside of $[-0.441, 0.441]$ to have $g$ around $O(1)$ away from $O(n^K)$. This leaves an even smaller region on ${\Bbb R}^K$ in which $K$ $tanh(nx)$ functions would work well together (neither vanished, nor exploded); reminding that $tanh(x)$ does not have the exploding gradient problem.


How do I convert a summation equation to a vector equation (backpropagation)?

$a_j^l$ is the $j$-th component of $a^l$. Hence, the first equation is $1$ dimensional, the second equation is a vector equation.

Similarly for notaiton $b_j^l$ and $w_{jk}$.

$\sigma$ means sigmoid function. If the input is a vector, then it evaluated the vector componentwise.

To answer your main quesiton, recall that from definition of  matrix vector product, where $A$ is a matrix and $x$ is a vector, $Ax$ produces a vector $y$ where the $j$-th component is $y_j= \sum_k A_{jk}x_k$.

That is element of $$y=Ax$$

can be written as $$y_j = \sum_k A_{jk}x_k$$

elementwise.

Hence that is how we can "remove" the summation sign.