Different Values in Training and Testing set

On principle, you should try to achieve a fairly balanced distribution between your training and testing sets. Otherwise your model might not make appropriate generalizations.  Additionally, you should use some form of normalization such as feature scaling.

It is difficult (or even impossible) to understand your data distribution by simply looking at the range.  It would be better to look at the mean/median or even a histogram/normal curve.

Let's say, for example, that our ordered column A in your training/testing sets look as such (before normalization):

\begin{array}{|c|}
\hline
&column \ A \ split\\
\hline
training && testing \\
0 && 0\\
4 && 2\\
5 && 3\\
7 && 3\\
9 && 8\\
15 && 11\\
23 && 15\\
30 && 80\\
\hline
\end{array}

\begin{array}{|c|}
\hline
&set\ metrics\\
\hline
& training & testing\\
range & [0,30] & [0,80]\\
mean & 11.625 & 15.250\\
median & 8.0 & 5.5\\
stddev & 10.309 & 26.650\\
\hline
\end{array}

\begin{align}
\large\frac{1}{\sqrt{2\pi}\cdot\sigma}e^{\frac{-(x-\mu)^2}{2\sigma^2}}
\end{align}

(red=training, blue=testing)

Here we can see that despite the large range discrepancy, our sets are relatively balanced.

We can observe a problematic training/testing split in this example where we shift the training mean from 11.625 to 100:

Moreover, we could generate another hypothetical dataset with a bad training/testing split where our set ranges are identical.

TL;DR:

Do not evaluate the set distribution by range; instead use mean/median or histogram/normal curve.  Also normalize your data.


The LIME framework can probably be used to do this as well.

Outlier detection sets a specific label to outliers (say 1), and another one to inliers (say 0). From then on, you can train interpretable models (decision trees for instance) to predict the labels set by your unsupervised model.

I don't know much about SHAP values, but I guess, with this approach, you could do the same.


r - rpart in text mining for classification of documents

rpart is a decision tree model and as such is very much interpretable.
You should visualize your decision tree. See the examples here, one of them is

Then for each given document you can follow the tree path and see

what were the features on the way responsible for the classification
what leaf your document arrived at and what is the percentage of correctly classified documents at these leaf.


How can a ML algorithm learn to classify fake news?

Following discussion with Erwan: one of his previous answer partially has answered my question. However I would like to understand the following. One needs to have a corpus, then label news/tweets in fake/not fake, then run the model. But how the algorithm works on texts and takes relevant words or features for detecting fake news?

First, let me emphasize that the concept of "fake news" is very vague and subjective. This fact alone is a red flag for any rational data scientist, since if humans don't always agree what is the correct answer it's going to be difficult to know whether a program gives the correct answer or not.
Now let's assume one has some news data annotated as fake or not. We can represent every news item as a vector of features (for example as a simple one-hot encoding vector of words), and for every document provide these features together with the "fake or not" boolean label to a supervised ML algorithm.
The supervised ML algorithm doesn't know and doesn't care about the semantic task, that is it doesn't have any idea what the label represents. It's job is only to find the best way to predict a given label (whatever this represents) from the given features. Typically it does this by measuring statistical links between features and labels, for instance using the fact that a particular word is often associated with a particular label in the training set.
It can do this part very well, but the rest is up to humans: are the features really good indications for the label? are the labels really representative of the task? Is the amount of training data sufficient? Is the training data diverse enough? Is the ML algorithm the right choice for this task? How to evaluate and interpret the results? Etc.
The point is: no matter how complex it is, the algorithm doesn't care about the meaning, it just uses the relations that it finds in the data. If for some reason the word "rhinoceros" always appears with the label "fake" in the training data, then when applying the system any news containing the word "rhinoceros" is likely to be predicted as fake. It's easy to see that this can lead to serious errors, depending on the quality/diversity of the data used for training. It's also easy to see that, with a very "sensitive" task such as fake news detection, a good evaluation result with the same kind of data as the training data is not a proof that the system can actually detect fake news in any context, if only because it's impossible to have a fully representative sample of any kind of fake news past, present and future.

So my question is: If I have a corpus on Trump, would the algorithm be able to detect fake news on Vitamin C, without any words (verb, adj, noun,..) in common between the two dataset, except stopwords?

As you can guess from my explanation above, it's not sure at all that the algorithm would easily adapt to a new type of data. That being said, normally a good algorithm on this task would probably focus on the subtle cues rather than the topic words, for instance the terms which are meant to trigger an emotional response with the reader. To some extent, these indications might work across topics. But again there's no guarantee that the system works in general, especially since the people who write and disseminate fake news can easily change their techniques if these become easily detected.


Is a dense layer required for implementing Bahdanau attention?

A Dense layer inside the Attention layer logic is a must. This is typically a single unit, single layer MLP (though in case of Additive Attention, there is a provision to use more units). This layer helps us derive the attention weights during training
This Dense layer transforms the input (each of the encoder LSTM layer output) keeping the dimensions the same. This is then multiplied individually with the last decoder hidden state & softmaxed to generate the attention weights.
Multiplying the attention weights with the encoder hidden states help to magnify or diminish the relevant input information and this context can then be used for Predictions. (In Additive layer addition is used instead of multiplication)
Without the Dense layer in the Attention layer, we don't get the Attention weights. Without Attention weights, there is no context. Without context there is no Attention.
So in order to incorporate Attention, a single layer Dense MLP is mandatory. The model learns the weights of this layer during training. These layer weights are then used to derive the attention weights.


what do we actually study in machine learning?

I want to point out one aspect which is not adequately reflected in the previous answers. When you start with linear regression, you basically I) propose some model (a way how to represent the relation between $y$ and $X$), e.g. a linear function, and II) you specify a loss function (the sum of squared residuals). The aim is to find coefficients (or weights), which map $X$ to $y$ in a way so that there is no “better way” to predict $y$ (viz. minimise the sum of square residuals).

In the case of linear regression, you have two options: solve the problem analytically or by gradient decent. Optimal coefficients/weights (vector beta) are given by:

$$ (X'X)^{-1} X'y=\beta. $$

So there is no need for gradient decent in this case. It is just simple matrix algebra.

Here is a minimal example in R, demonstrating how it works:

Source: https://www.r-bloggers.com/regression-via-gradient-descent-in-r/

# Input some data
x0 <- c(1,1,1,1,1)
x1 <- c(1,2,3,4,5)
x2 <- c(8,4,3,1,8)
x <- as.matrix(cbind(x0,x1,x2))
y <- as.matrix(c(3,7,5,11,14))

# (X'X)^-1 X'y
beta1 = solve(t(x)%*%x) %*% t(x)%*%y

# R's regression command
beta2 = summary(lm(y ~ x[, 2:3]))

# Gradient decent
m <- nrow(y)
grad <- function(x, y, theta) {
  gradient <- (1/m)* (t(x) %*% ((x %*% t(theta)) - y))
  return(t(gradient))
}

# Define gradient descent update algorithm
grad.descent <- function(x, maxit){
  theta <- matrix(c(0, 0, 0), nrow=1) # Initialize the parameters

  alpha = .05 # Learning rate
  for (i in 1:maxit) {
    theta <- theta - alpha  * grad(x, y, theta)
  }
  return(theta)
}

# Results
print(grad.descent(x,2000))
print(beta1)
print(beta2)

Results:

> print(grad.descent(x,2000))
            x0     x1        x2
[1,] -0.672238 2.6459 0.1530299
> print(beta1)
         [,1]
x0 -0.6722955
x1  2.6459103
x2  0.1530343
> print(beta2)

Call:
lm(formula = y ~ x[, 2:3])

Residuals:
      1       2       3       4       5
-0.1979  1.7683 -2.7245  0.9356  0.2185

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -0.6723     3.3565  -0.200   0.8598
x[, 2:3]x1    2.6459     0.7677   3.447   0.0748 .
x[, 2:3]x2    0.1530     0.3897   0.393   0.7325
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2.399 on 2 degrees of freedom
Multiple R-squared:  0.8561,    Adjusted R-squared:  0.7122
F-statistic: 5.949 on 2 and 2 DF,  p-value: 0.1439

So as you can see, there is not much of a machine involved in solving the problem (apart of doing the math of course). So why machine learning? Increasing computational power has made it possible to solve very complex problems using gradient decent. The idea is that you depart from simple solutions and look for better, more complex ways to map $X$ to $y$.

You can think about changing the base procedure, e.g. use decision trees instead of a linear function. Since one tree may be only a weak learner, you can combine (or ensemble) trees to a random forest. Or you go even further and use boosting, in which you repeatedly use information contained in the residual to improve your fit. Usually very small trees are used to do this, but you can boost on linear regression as well.

Here is a little example for L2 boosting:

(from: Boosting Algorithms: Regularization, Prediction and Model Fitting)

library(glmnet)

# Simple boosting example from:
# P. Bühlmann, T. Hothorn (2007), "Boosting Algorithms: Regularization, Prediction and Model Fitting", Statistical Science 22(4), p. 477-505.
# L2-Boosting from Section 3.3 (p. 483)

#######################
# Data
x = matrix(c(130,165,150,150,140,198,307,350,318,304,302,429),nrow=6,ncol=2,byrow = TRUE)
y = c(18,15,18,16,17,15)

# Parameter
# This works like a learning rate
nu = 0.1
# Early stopping (SSR value to stop)
es = 7.58765583743327
# Max. iterations (of boosting)
maxiter = 10000

#######################
# OLS benchmark model
olsdata = data.frame(cbind(y,x))
ols = lm(y~.,data=olsdata)
olspreds = predict(ols,newdata=olsdata)
olsssr = sum((y-olspreds)^2)
print(paste0("OLS SSE: ",olsssr))

#######################
# Boosting

# Initialize f0
f0 = mean(y)
f0ssr = sum((y-f0)^2)

# Lists to store results per boosting iteration
ssrlist = list()
bstep = list()

# Boosting (p. 483, Sec. 3.3, L2-Boosting)
for (n in seq(1:maxiter)){
  # I) Get residual
  if(n==1){f=f0} # initialize in first step
  u=y-f          # get residual from estimation
  # II) Fit residual / predict, I use ridge (alpha=0)
  reg = glmnet(x,u,alpha=0, lambda=2)
  g = predict(reg, newx=x, type="link")
  # III) Update f
  f=f+nu*g
  # Print feedback
  cat(paste0("Step: ", n," SSR: ", sum(u^2), "\n" ))
  # Save SSR/iter to list
  ssrlist[[n]] <- sum(u^2)
  bstep[[n]] <- n
  # Early stopping rule
  if(sum(u^2)<es){break}
}

#######################
# End statement
cat("\n ~~~ RESULTS ~~~ \n")
cat(paste0("OLS SSR:           ", olsssr, "\n"))
cat(paste0("Initialised SSR:   ", f0ssr, "\n"))
cat(paste0("Last SSR boosting: ", ssrlist[length(ssrlist)], " [after ",bstep[length(bstep)], " iterations] \n"))

The problem here is that you do not only need to solve one regression, but 10000 (in the example here). In real world applications, the amount of math that has to be done is immense (counts for boosting and neural nets as well).

So what is machine learning? You want to let “the machine learn things” (recognise patterns in data, recognise images, make prediction based on that etc) and you will likely be unable to do this learning task on your own using pen and paper. Improved computing capacity over time has enabled us to solve very complex problems. And because of this cheap resource, new tools and methods develop. You can do things today on your laptop which literally seemed impossible in the late 1990s.

Of course you need to tell “the machine” how to learn. This is a lot of statistics but also computer science. Neural nets for example have mostly been advanced by computer scientists. Ultimately, the machine is merely a tool, but a very useful one.


How the term "R-squared" in VIF(variance inflation factor) is different from normal R-squared calculation?

It is calculated in the same way. Instead of the response variable $y$ of the original model with numeric regressors $x_1, \dots, x_p$, the response of the VIF-model is $x_i$ and the regressors all other $x_j, j \ne i$.


PCA doesn't really make sense for a time series if it is left as a time series:

There are times when its useful to aggregate time series data into a rectangular representation in order to gain insight into interesting aspects of the data.  This is a feature engineering where the time series provides lots of the features.  For instance, you can create features for 'minute of hour', 'hour of day', 'morning', 'afternoon','evening', 'day of week', 'day of month', 'week of month', 'week of year', 'season', ...  You can even join this data with other data, like historical weather data or economic index, to see how daily temperature or economics affect your phenomenon. In these cases you could do all of this feature engineeting and then use PCA to find the most important features (radar plot) or simply to select the most important orthogonal features from the post PCA mashup (linear transformation).

Time Series Smothing:

There are a bunch of different methods to smooth and down-sample time series data. In this case, you are turning the quantized stochastic events and either turning them into regularly spaced data, or taking your already regularly spaced data and downsampling and smoothing it. Smoothing algorithms include Exponentially weighted moving average (EWMA) or Holt-Winters smoothing.  Connor Johnson has a nice blog write-up on the these.

ARIMA is a more complete algorithm that includes smoothing, de-seasoning, and forecasting. It is very powerful, but takes some time to master.

Hope this helps!


There is an online data science "game" that takes you from learning how to use Python for loading a csv and using scikit to machine learning algorithms such as support vector machines. Here is a blog post with a demo video and the actual site is Explore Data Science. Personally, I think its genius.


These are the equation of Adam [Ref - Dive Into Deep Learning]
\begin{aligned}
       \mathbf{v}_t & \leftarrow \beta_1 \mathbf{v}_{t-1} + (1 - \beta_1) \mathbf{g}_t \\
       \mathbf{s}_t & \leftarrow \beta_2 \mathbf{s}_{t-1} + (1 - \beta_2) \mathbf{g}_t^2
\end{aligned}
\begin{aligned}
\hat{\mathbf{v}}_t = \frac{\mathbf{v}_t}{1 - \beta_1^t} \text{ and } \hat{\mathbf{s}}_t = \frac{\mathbf{s}_t}{1 - \beta_2^t}
\end{aligned}
\begin{aligned}
\mathbf{g}_t' = \frac{\eta \hat{\mathbf{v}}_t}{\sqrt{\hat{\mathbf{s}}_t} + \epsilon}
\end{aligned}
\begin{aligned}
\mathbf{x}_t \leftarrow \mathbf{x}_{t-1} - \mathbf{g}_t'
\end{aligned}

The first two are the accumulation of momentum and the second moment of the gradient
The second set is for correction of initial bias
Last two are parameter update

Initial values are - [Ref - Arxiv Paper]
\begin{aligned}
       \mathbf{v} = \mathbf{s} = 0; \mathbf{t} = 1^{**} ; \beta_1=0.9 ; \beta_2=0.999 ;  \epsilon = 10^{-8}
\end{aligned}
Note - ** - It is initialized with 0 but incremeted in the loop before any other operation
These defaults will make,
\begin{aligned}
\mathbf{g}_0' = \eta \text{  (approximated for  } \epsilon\text{ )}
\end{aligned}
So, the initial movement will not be proportional to the Gradient.


Significance shifts when cases are excluded

Sounds like you came across Simpson's Paradox. Roughly speaking you have a result in one group of data but not when you look at different groups or subsets. This often happens when the variables depend on hidden dimensions in the data.

The wikipedia page explains it a bit better and in more detail.


What regressors are recommended with text modeling?

In sklearn Anything that will take sparse data can take output from TFIDF.

In sklearn basically all data any model can take is either dense (normal array) or sparse (that only stores the location of values != 0). You can convert from sparse to dense, but chances are you'll run out of memory if you try.

I'm quite a big fan of using linear algorithms on text data (sgdregressor for example have a ton of different options you can play with). But other algorithms like randomforest that Sergey mentioned, and naive bayes models can also work with that kind of data. Basically what you are looking for is anything that can take sparse input data.

(One thing I've done in the past when working with text+other data is taking output from an sgd's analysis of the data and feeding that + the other data to another algorithm like randomforest. It's a simple and quite powerful method)


As I have mentioned in the comment, there is not rule of thumb. One has to do a lot experimentation.

Based on how much data you have, bringing minority class to be be equal (50-50) to the majority class can be misleading the model.

You can also try ADASYN during your experimentation.

ADASYN is a improved version of SMOTE. What it does is same as SMOTE just with a minor improvement. After creating those sample it adds a random small values to the points thus making it more realistic. In other words instead of all the sample being linearly correlated to the parent they have a little more variance in them i.e they are bit scattered.

You can read about the implementation here


Method for solving problem with variable number of predictors

I don't see the problem. All you need is a learner to map a bit string as long as the total number of contestants, representing the subset who are taking part, to another bit string (with only one bit set) representing the winner, or a ranked list, if you want them all (assuming you have the whole list in your training data). In the latter case you would have a learning-to-rank problem.

If the contestant landscape can change it would help to find a vector space embedding for them so you can use the previous embeddings as an initial guess and rank anyone, even hypothetical, given their vector representation. As the number of users increases the embedding should stabilize and retraining should become less costly. The question is how to find the embedding, of course. If you have a lot of training data, you could probably find a randomized one along with the ranking function. If you don't, you would have to generate the embedding by some algorithm and estimate only the ranking function. I have not faced your problem before so I can't direct you to a particular paper, but the recent NLP literature should give you some inspiration, e.g. this. I still think it is feasible.


Calibrating probability thresholds for multiclass classification

Disclaimer: This answer describes the thoughts I had about this problem, I don't offer any guarantee about their validity so use at your own risks ;)
There are two distinct parts in this problem:

Finding an optimal threshold over three classes
Improving performance with respect to the minority class

First part: I might be wrong but as far as I know there's no way to select a particular probability threshold when there are three classes. And if there were a way, it probably wouldn't be from the ROC plot since the curves for the different plots are independent, so picking a point on the curve would correspond to different thresholds for different classes and I don't see how this would be manageable. The only way that I know is what you did: label with the class which has the maximum probability.
Second part: if you want to force the model to take care of all the classes, you could also try optimizing with macro-AUC, but that would probably lead to the same problem of decreasing the micro-AUC since there would be more errors of true neutral or high predicted as low.
The way I see it, the three way model doesn't work well: the imbalance of the minority class is not that bad, it's only 2 or 3 times less than the other classes. So the fact that the model almost completely dismisses this class is a bit strange. I also see that instances of the true class low are almost as often predicted as class high than class neutral, even though I would expect that the vast majority of errors on class low to be predicted as neutral.
So my vague intuition is that maybe the system could be designed in a way which avoids the three-way problem for the model. I can think of two options in this perspective:

A two-steps system where the first model classifies between low and neutral+high then the second one between neutral and high (or first with low+neutral vs. high and then low vs. neutral). This way each model is binary classification so you have more control over the thresholds at each step. Normally this is not recommended in classification, but here I assume that the classes are not truly categorical.
Pushing the same idea further: there seems to be an order between the classes low < neutral < high, so maybe it would be possible to treat the task as a regression problem. This might help the model avoiding these "big" errors between low and high. Importantly you would also have flexibility with the predicted values: there would be two thresholds to determine and these could be found to optimize any appropriate evaluation measure.


What are the current killing machine learning methods?

The question is very general. However, there are some studies being conducted to test which algorithms perform relatively well in a broad range of problems (I'll add link to papers later), concerning regression and classification.

Lately Random Decision Forests, Support Vector Machines and certain variations of Neural Networks are being said to achieve the best results for very broad variety of problems.

This does not mean that these are "the best algorithms" for any problem, that does not exist, and actually is not very realistic to pursue. Also it must be observed that both RDF and SVM are rather-easy methods to initially grasp and obtain good results, so they are becoming really popular. NN have been used intensively since couple of decades (after they revived), so they appear often in implementations.

If you are interested in learning further you should look for an specific area and deal with a problem that can be solved nicely by machine learning to understand the main idea (and why is impossible to find  the method).

You will find common the task to try to predict the expected behavior of something given some known or observable characteristics (to learn the function that models the problem given input data), the issues related to dealing with data in high-dimensional spaces, the need for good quality data, the notable improvements that can give data pre-processing, and many others.


What are similarity and distance metrics in classification?

The choice of the metric is somewhat dependent on how you are representing the text.

A common metric for vector space models would be cosine similarity.

Cosine similarity: K(X, Y) = < X, Y > / (||X||*||Y||)

  Blockquote My initial thought was that, for example, in logistic regression, the normalisation used would be L1 (Manhattan) or L2 (Euclidean). Would this be correct?

In vector space models for text representation, the vectors are from a high dimension and sparse. When using L2 or L1 norm, there will be a term for every dimension which has a term in either vector. However, for consine similarity, since it is a dot product, there will only be a term when both elements in either vector is non-zero. The sparsity renders the consine similarity a better choice for such comparisons.


How to delete a row if a values in a column is not NaN

If I am correct, you are looking to keep only the NaN values. You can use pd.isnull() to check which rows are NaN and then only keep those. Try this:

df1 = df1[pd.isnull(df1['column_name'])]


Is there any point in learning Hadoop in 2018?

At this point of time, if I had to start a project from scratch and I had to choose between Hadoop and Spark, I would certainly choose Spark over Hadoop. There are several reasons for this:

Spark is more efficient than Hadoop given the fact that processing in Spark is in-memory whereas Hadoop requires to store intermediate results in disk.
Spark provides dozens of different operations and it is not constrained to just Hadoop's map-reduce.|

However, I still think that there is value in learning Hadoop before trying to learn Spark, even if it is at a high level. These are some reasons:

You may came across some legacy applications/systems based on Hadoop technologies
It provides a gentle introduction to some of the concepts used in Spark
Spark is sometimes used in combination with some technologies in the Hadoop ecosystem, like Hive and HDFS (Spark does not incorporate a way of storing data, but it can fetch data from multiple sources, including HDFS, and a Spark cluster with HDFS based data storage is a usual combination).


Convert a pdf into a conditional pdf such that mean increases and std dev falls

Just apply an optimization to search for the X values that satisfy the criteria you're looking for. Here's a simple demo:

set.seed(123)
mu_x_true = 1e4
mu_y_true = 3.75
n = 1e2

x <- rpois(n, mu_x_true)
y <- rnorm(n, sqrt(mu_y_true))^2

plot(x, y)

# conditions:
# E[Y|X] > E[Y]
# std(Y|x) < std(Y)

mu_y_emp = mean(y)
sd_y_emp = sd(y)

objective <- function(par, alpha=0.5){
    if (par[1]>par[2]) par = rev(par)
    ix <- which((par[1] < x) & (x < par[2]))
    k <- length(ix)
    if (k==0) return(1e12)
    mu_yx <- mean(y[ix])
    sd_yx <- sd(y[ix])

    alpha*(mu_y_emp - mu_yx) + (1-alpha)*(sd_yx - sd_y_emp)
}

init <- mean(x) + c(-sd(x), sd(x))
test <- optim(objective, par=init)

ix <- which((par[1] < x) & (x < par[2]))

mean(y[ix]) > mean(y)
# TRUE

sd(y) > sd(y[ix])
# TRUE


Reinforcement Learning in 2018, best tips and tricks?

Have you looked at the Rainbow RL? It combines all improvements of RL. Apart from structural changes other improvements come from Reward Shaping, Curiosity-driven Learning, RL with Unsupervised Auxiliary tasks and RL with external memory (e.g. Neural Episodic Control, Model-free episodic Control). It's pitty that you leave out from your question Policy Gradients, they are lots of fun :) Happy reading!


Is the Cross Entropy Loss important at all, because at Backpropagation only the Softmax probability and the one hot vector are relevant?

During backpropagation we take the derivative of loss function with respect to all the weight parameters. For cross entropy loss this derivative is simply the difference between output probability and predicted probability. Upto this you're correct.

But you missed what loss function represents. The loss function represents the surface manifold of the network. The purpose of learning is to find the minima of this loss function.

We use CE loss because the surface define by it is smooth and continuous, hence we can find the derivative easily.

Mathematically that derivative comes out to be a simple difference, but theoretically we're differentiating the surface defined by the loss function, which is important.


Which open-source sgdb for kind of large data

analyzing 8 billion lines on a 4gb computer is pretty silly, but you can try

http://www.asdfree.com/2013/03/column-store-r-or-how-i-learned-to-stop.html


What is the best alternative for Fisher's Exact test for contigency tables that are NOT 2x2?

Fisher's exact test has a ready generalization to tables of an arbitary dimension and it is applicable here (Metha and Pathel, 2012).
You can for example use the fisher.test built-in function to compute this in R.


To answer the first question about why we need the learning rate even if we have momentum, let's consider an example in which we are not using the momentum term.  The weight update is therefore:

$ \Delta w_{ij} = \frac{\partial E}{\partial w_{ij}} \cdot l $

where:

$ \Delta w_{ij} $ is the weight update
$ \frac{\partial E}{\partial w_{ij}} $ is the gradient of the error with respect to the weight
$ l \space $ is the learning rate coefficient

Our weight update is determined by the gradient of our current error with respect to the weight at node $ ij $.  Therefore, our prior weight deltas are not factored into our weight update equation.  If we were to eliminate the learning rate, our weights would not update.

Now let's consider an example using the momentum term in its derived form:

$ \Delta w_{ij} = (\frac{\partial E}{\partial w_{ij}} \cdot l) + (\mu \cdot \Delta w^{t-1}_{ij}) $

where:

$ \mu $ is the momentum coefficient
$ \Delta w^{t-1}_{ij} $ is the weight update of node $ ij $ from the previous epoch

Now we are factoring the previous weight delta in our weight update equation.  In this form, it is easier to see that the learning rate and momentum are effectively independent terms.  However, without a learning rate, our weight delta would still be zero.

Now you might ask: what if we remove the learning rate after getting an initial momentum value so that momentum is the sole influence of the weight delta?

This destroys the backpropagation algorithm.

The objective of backprop is to optimize the weights to minimize error.  We achieve this minimization by adjusting the weights according to the error gradient.

Momentum, on the other hand, aims to improve the rate of convergence and to avoid local minimas.  The momentum term does not explicitly include the error gradient in its formula.  Therefore, momentum by itself does not enable learning.

If you were to only use momentum after establishing an initial weight delta, the weight update equation would look as such:

$ \Delta w_{ij} = (\mu \cdot \Delta w^{t-1}_{ij}) $

and:

$ \lim_{t \to \infty} \Delta w^t_{ij} =
\begin{cases}
0 & | \space \mu < 1 \space \lor \space (\mu = 1 \space \land \space \Delta w^{t=0}_{ij} < 1) \\
1 & | \space \mu = 1 \land \space \Delta w^{t=0}_{ij} = 1\\
\infty & | \space otherwise
\end{cases}
$

Although there exists a scenario where the weight delta approaches zero, this descent is not based on the error gradient and is in fact predetermined by the momentum coefficient and the initial weight delta: this weight delta does not achieve our objective to minimize the error and is therefore useless.

TL;DR:

The learning rate is critical for updating the weights to minimize error.  Momentum is used to help the learning rate, but not replace it.


What is Happening in the training process when we are fitting a model to the data

Options 2 and 4 are correct.
During fitting a model the training process Makes use of all observations available to create a model, then using distance or other measures to assess how well the model characterizes or "fits" its data. Also, Starting with a linear model and then regressing the shape of the line until it fits the data.


Depends on what is your goal with the data.

If you are trying to model your variable "outcome" as a classification problem (I am supposing this) then it depends on the modelling technique you are expecting to use.

Some techniques use scalled data as input because they have a functional form which allows them to perform better if the scale is the same for the variables (the canonic example could be Neural Networks).

Some techniques do not need to have scalled inputs, mostly because their functional form does not depend on relationships (interactions) with other variables just as random trees.

There are other techniques like logistic regression where scalling is optional because they work fine without scalling, the result is the same but sometimes you need to compare the resulting $\beta$'s and the only way to do this is if the $X$s have the same scale.

The process you are doing is ok, but make sure you actually need to do it.


The iris dataset is meant to be used for classification. You have 3 separate classes of irises and attempting to solve it as a regression problem would be a mistake.

Think about your proposed solution, you want the output to be -1,0 or +1 (for classes a,b and c). But this implies that class a is more similar to class b than to c, and by the same principal that class c resembles class b more than a. You are adding a prior to the model that was not there before, and you should not do that (unless you are an iris specialist).

you need to take your class output labels and convert them to a one-hot-encoding representation:
class a = [1,0,0]
class b = [0,1,0]
class c = [0,0,1]

Then use categorical cross entropy for your loss function.


TF-IDF not a strong measure in this senario?

Weighting scheme 2 in table Recommended TF-IDF weighting schemes in tf–idf | Wikipedia should solve your problem.


Training GANs becomes much more difficult with increasing input dimensionality. I encountered that sometimes even small changes to your network can decide about convergence/divergence of your network (like initialization of your variables).

I recommend following the tipps from https://github.com/soumith/ganhacks.

Another thing I personally recognized is that you should not use as many Conv2DTranspose layers as you do in your current network. Try instead increasing the latent space.


Predicting parameters of simple configured trajectories using RNN

If you know that your trajectory has a certain parametric form then you can use methods that explore the parameter space for that form. Examples of such methods are Hough transform and custom-built moments.
Hough transform maps a point in a real space into a manifold in the parameter space, and vice-versa, it maps a point in the parameter space into a line in the real space. If you have several points in the real space, they are transformed into the same amount of manifolds in the parameter space. These manifolds intersect in a point whose coordinates are the parameters of the line that goes through your data.
The parameter space for the ellipse is five-dimensional, so a naive application of the Hough transform is inefficient. Smarter implementations are known, e.g. [1].
The method of custom-built moments described in [2] allows to find the parameters of a curve expressible as
$$G(x,y) = \sum\limits_np_nz_n(x,y)=0$$
In the case of an ellipse, we have
$$G(x,y)=p_1+p_2x+p_3y+p_4x^2+p_5y^2+p_6xy=0$$
that is $z_1=1$, $z_2=x$, $z_3=y$, $z_4=x^2$, $z_5=y^2$, $z_6=xy$. You want to determine $p_1,...,p_6$ based on your data.
Enclose the portion of the curve where you have your data in a rectangle, and make a coordinate transformation so that your rectangle is now $ S = [-1 \le x \le 1, -1 \le y \le 1]$. The rectangle is separated by the curve into two regions. Assign the value $1$ to all points from one of the regions, and $0$ to all points in the other region. For example, in this Figure, all black points are $1$, and white points are $0$.

The data points are on the boundary. They are not explicitly shown in the figure because their exact position is not important anymore. It is also not important, how many points you have (as long as they clearly determine the shape of the curve segment) and in which order you collect them.
So, now you have a function defined on the rectangle:
$$I(x,y)=\begin{cases}
      1 & (x,y) \in Region~1 \\
      0 & (x,y) \in Region~2
\end{cases}$$
The parameters $p_1,...,p_6$ are found as the solution of a system of equations
$$\sum\limits_nu_np_n=0$$
where $u_n$ are custom moments
$$u_n=\iint_S I(x,y)U_n(x,y) \,dx\,dy$$
The functions $U_n$ are defined as $U_n(x,y)=\Upsilon(F,s,z_n)$ where
$$\Upsilon(F,s,z)=\left([sz_y-zs_y]F\right)_x-\left([sz_x-zs_x]F\right)_y$$
with the $x$ and $y$ subscripts indicating partial differentiation. The functions $F(x,y)$ and $s(x,y)$ are arbitrary. They can be anything you like as long as $F$ is  continuously differentiable, $s$ is twice continuously differentiable on an open set containing the rectangle $S$, and $F=0$ on the boundaries of $S$. More details in [2]. If you don't understand that paper during the first reading, read [3] first, and if you don't understand [3], read [4], then read [3], after which [2] should become clearer.
For ellipses, the authors of [2] recommend to generate 18 arrays of functions $U_n = (U_1,...,U_6)$ from combinations of three functions $F_1 = m_0n_0$, $F_2=xF_1$, $F_3=yF_1$ with six functions $s_n=z_n,\, n=1,...,6$, where $m_0=1-x^2$, $n_0=1-y^2$. With the additional notations $m_1=1-2x^2$, $n_1=1-2y^2$, $m_2=1-3x^2$, $n_2=1-3y^2$, these 18 arrays are found to be

$F=F_1,~s=1\Rightarrow % 1
U_1=0,~
U_2=2ym_0,~
U_3=-2xn_0,~
U_4=4xym_0,~
U_5=-4xyn_0,~
U_6=2(y^2-x^2)
$
$F=F_1,~s=x\Rightarrow % 2
U_1=-2ym_0,~
U_2=0,~
U_3=2y^2(3x^2-2)+2m_1,~
U_4=2x^2ym_0,~
U_5=2y(y^2(5x^2-3)+2m_1),~
U_6=2xm_1n_0
$
$F=F_1,~s=y\Rightarrow % 3
U_1=2xn_0,~
U_2=2(y^2(2-3x^2)-m_1),~
U_3=0,~
U_4=2x(y^2(4-5x^2)+3x^2-2),~
U_5=-2xy^2n_0,~
U_6=-2ym_0n_1
$
$F=F_1,~s=x^2\Rightarrow % 4
U_1=-4xym_0,~
U_2=-2x^2ym_0,~
U_3=2x(y^2(5x^2-4)-3x^2+2),~
U_4=0,~
U_5=4xy(y^2(4x^2-3)+2-3x^2),~
U_6=2x^2(y^2(4x^2-3)+2-3x^2)
$
$F=F_1,~s=y^2\Rightarrow % 5
U_1=4xyn_0,~
U_2=2y(y^2(3-5x^2)-2m_1),~
U_3=2xy^2n_0,~
U_4=4xy(y^2(3-4x^2)+3x^2-2),~
U_5=0,~
U_6=2y^2(y^2(3-4x^2)+3x^2-2)
$
$F=F_1,~s=xy\Rightarrow % 6
U_1=2(x^2-y^2),~
U_2=-2xm_1n_0,~
U_3=2ym_0n_1,~
U_4=2x^2(y^2(3-4x^2)+3x^2-2),~
U_5=2y^2(y^2(4x^2-3)+2-3x^2),~
U_6=0
$
$F=F_2,~s=1\Rightarrow % 7
U_1=0,~
U_2=2xym_0,~
U_3=m_2n_0,~
U_4=4x^2ym_0,~
U_5=2ym_2n_0,~
U_6=x(y^2(x^2+1)+m_2)
$
$F=F_2,~s=x\Rightarrow % 8
U_1=-2xym_0,~
U_2=0,~
U_3=x(y^2(7x^2-5)-5x^2+3),~
U_4=2x^3ym_0,~
U_5=2xy(2y^2(3x^2-2)+3-5x^2),~
U_6=x^2(3-5x^2)n_0
$
$F=F_2,~s=y\Rightarrow % 9
U_1=-m_2n_0,~
U_2=x(y^2(5-7x^2)+5x^2-3),~
U_3=0,~
U_4=x^2(y^2(9-11x^2)+7x^2-5),~
U_5=y^2m_2n_0,~
U_6=-2xym_0n_1
$
$F=F_2,~s=x^2\Rightarrow % 10
U_1=-4x^2ym_0,~
U_2=-2x^3ym_0,~
U_3=x^2(y^2(11x^2-9)-7x^2+5),~
U_4=0,~
U_5=2x^2y(y^2(9x^2-7)+5-7x^2),~
U_6=x^3(y^2(9x^2-7)-7x^2+5)
$
$F=F_2,~s=y^2\Rightarrow % 11
U_1=-2ym_2n_0,~
U_2=2xy(2y^2(2-3x^2)+5x^2-3),~
U_3=-y^2m_2n_0,~
U_4=2x^2y(y^2(7-9x^2)+7x^2-5),~
U_5=0,~
U_6=xy^2(y^2(7-9x^2)+7x^2-5)
$
$F=F_2,~s=xy\Rightarrow % 12
U_1=-x(m_2+y^2(x^2+1)),~
U_2=x^2(5x^2-3)n_0,~
U_3=2xym_0n_1,~
U_4=x^3(y^2(7-9x^2)+7x^2-5),~
U_5=xy^2(y^2(9x^2-7)+5-7x^2),~
U_6=0
$
$F=F_3,~s=1\Rightarrow % 13
U_1=0,~
U_2=-m_0n_2,~
U_3=-2xyn_0,~
U_4=-2xm_0n_2,~
U_5=-4xy^2n_0,~
U_6=y(y^2(3-x^2)-x^2-1)
$
$F=F_3,~s=x\Rightarrow % 14
U_1=m_0n_2,~
U_2=0,~
U_3=y(y^2(7x^2-5)+3-5x^2),~
U_4=-x^2m_0n_2,~
U_5=y^2(y^2(11x^2-7)+5-9x^2),~
U_6=2xym_1n_0
$
$F=F_3,~s=y\Rightarrow % 15
U_1=2xyn_0,~
U_2=y((5-7x^2)y^2+5x^2-3),~
U_3=0,~
U_4=2xy(y^2(5-6x^2)+4x^2-3),~
U_5=-2xy^3n_0,~
U_6=y^2m_0(5y^2-3)
$
$F=F_3,~s=x^2\Rightarrow % 16
U_1=2xm_0n_2,~
U_2=x^2m_0n_2,~
U_3=2xy((6x^2-5)y^2+3-4x^2),~
U_4=0,~
U_5=2xy^2(y^2(9x^2-7)+5-7x^2),~
U_6=x^2y(y^2(9x^2-7)+5-7x^2)
$
$F=F_3,~s=y^2\Rightarrow % 17
U_1=4xy^2n_0,~
U_2=y^2(y^2(7-11x^2)+9x^2-5),~
U_3=2xy^3n_0,~
U_4=2xy^2((7-9x^2)y^2+7x^2-5),~
U_5=0,~
U_6=y^3(y^2(7-9x^2)+7x^2-5)
$
$F=F_3,~s=xy\Rightarrow % 18
U_1=y((x^2-3)y^2+x^2+1),~
U_2=-2xym_1n_0,~
U_3=y^2m_0(3-5y^2),~
U_4=x^2y((7-9x^2)y^2+7x^2-5),~
U_5=y^3((9x^2-7)y^2+5-7x^2),~
U_6=0
$

After taking the integrals of these $U_n$ with $I(x,y)$ you get a $18 \times 6$ matrix $U$ of the moments $u_n$. Taking the eigenvector associated with the smallest eigenvalue of $U^TU$ as described here, you will get the parameters $p_1, ... p_6$.
I tested them on the curve segment in the figure above. The segment is from an ellipse
$$\frac{((x-1)\cos(\frac{\pi}{6})+(y+1)\sin(\frac{\pi}{6}))^2}{2^2} + \frac{((x-1)\sin(\frac{\pi}{6})-(y+1)\cos(\frac{\pi}{6}))^2}{1^2} = 1$$
I got the following result
[ 1.         -1.70012217  2.53636068  0.49314339  0.91927365 -0.74693632]

which is not very accurate but I used a 2D-trapezoid rule for integration as described here and took a uniform $600 \times 600$ grid. You will definitely get better results with a more sophisticated grid and integration rule.
References

D.H. Ballard, Generalizing the Hough transform to detect arbitrary shapes, Pattern Recognition, Volume 13, Issue 2, 1981, Pages 111-122, ISSN 0031-3203,
https://doi.org/10.1016/0031-3203(81)90009-1
Popovici, I., & Withers, W. D. (2009). Curve Parametrization by Moments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 15–26. https://doi.org/10.1109/TPAMI.2008.54
Popovici, I., & Withers, W. D. (2006). Locating Thin Lines and Roof Edges by Custom-Built Moments. 2006 International Conference on Image Processing. https://doi.org/10.1109/ICIP.2006.312501
Popovici, I., & Withers, W. D. (2006). Custom-built moments for edge location. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4), 637–642. https://doi.org/10.1109/TPAMI.2006.75


Using pandas get_dummies() on real world unseen data

Yes, the encoding would be lost. You should instead use sklearn OneHotEncoder and save the corresponding encoder instance so that you can re-load it on unseen data.

One can do something along these lines:

import pandas as pd
import pickle
from sklearn.preprocessing import OneHotEncoder

def get_encoder_inst(feature_col):
    """
    returns: an instance of sklearn OneHotEncoder fit against a (training) column feature;
    such instance is saved and can then be loaded to transform unseen data
    """
    assert isinstance(feature_col, pd.Series)
    feature_vec = feature_col.sort_values().values.reshape(-1, 1)
    enc = OneHotEncoder(handle_unknown='ignore')
    enc.fit(feature_vec)
    with open(file_name, 'wb') as output_file:
            pickle.dump(enc, output_file)
    return enc

which can then be loaded and applied as

def get_one_hot_enc(feature_col, enc):
    """
    maps an unseen column feature using one-hot-encoding previously fit against training data
    returns: a pd.DataFrame of newly one-hot-encoded feature
    """
    assert isinstance(feature_col, pd.Series)
    assert isinstance(enc, OneHotEncoder)
    unseen_vec = feature_col.values.reshape(-1, 1)
    encoded_vec = enc.transform(unseen_vec).toarray()
    encoded_df = pd.DataFrame(encoded_vec)
    return encoded_df

where the argument enc in the latter function is an instance of OneHotEncoder that you load through pickle.load. Of course the above is just a pseudo-code example, take care that all the objects that you are using keep the initial shapes and so on.

The problem with using pd.get_dummies is that it has no memory of the previously mapped encoding: it basically turns a column into factors, whereas OneHotEncoder actually maps categorical variables to a fixed-length vector representation that is stored and kept.


XGBoost most important features appear in multiple trees multiple times

To your questions:

It can be normal.
Features itself are not responsible for overfitting. It can be noise from them or just correlation without causation. I.e. Feature correlates in the training set, but not in the test set. Overfitting is related to models.
Maybe you still don't have a feature that would improve your model significantly. It's hard to say with such limited information. You need to try some experiments with feature selection.


After some researches and trial, the dimensions need to be normalised.

In this case, I consider 0 as normal and 1 be outliers.

I normalise the dimensions by take the 10% and 90% of the samples.

clf = svm.SVC(max_iter=1, kernel='rbf')


Text similarity with sentence embeddings

One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector.

  The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

Source: "From Word Embeddings To Document Distances" Paper

WMD can be modified to Sentence Mover’s Distance, comparing how far apart different sentence embeddings are to each other.


Which policy gradient method is used for continuous action spaces?

You can use  Soft Actor Critic or PPO from the same website. Most PG methods can be used in discrete and/or continuous action spaces.


Is Nvidia Jetson product family also suitable for machine learing model training?

Training a neural network involves a lot more computation than inferencing from a pre-trained one. During training, a neural network must go through both forward propagation and back propagation steps. The back propagation step is computationally expensive, since it requires many gradient computations and updates to weight variables.

During inference, the only computation needed is forward propagation, where the input is multiplied to the weights to produce a prediction. Boards like the NVIDIA Jetson are optimized for fast matrix multiplication, hence why they are effective during inference but not as much for training.

Another reason for the Jetson's popularity is its small form factor. Architects choose it because it can fit within the size and price requirements of the product while also offering decent performance. If you had unlimited budget and can fit a GTX Titan in your product for inferencing, you would do that, right? I worked in the autonomous vehicle sector for a while and that there is no way we could fit a machine with any full-size GPU in a car. There are no size limitations for training models, so naturally you would go after the machines with more powerful and efficient GPUs.


Training a sound localization neural network

Change your activation layers, use sigmoid or Tanh for your final layer.

I would try CNN again but with different strides, filter sizes, and number of filters. The thing about CNN is that because you have fewer features per layer you will be able to have more layers.

Here is an example of a Convolutional layer used for audio:

import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D
from keras.optimizers import SGD
from keras.layers.advanced_activations import LeakyReLU
model = Sequential()
model.add(Conv2D(16, (3,3), padding='same', input_shape=(513, 25, 1)))
model.add(LeakyReLU())
model.add(Conv2D(16, (3,3), padding='same'))
model.add(LeakyReLU())
model.add(MaxPooling2D(pool_size=(3,3)))
model.add(Dropout(0.25))
model.add(Conv2D(16, (3,3), padding='same'))
model.add(LeakyReLU())
model.add(Conv2D(16, (3,3), padding='same'))
model.add(LeakyReLU())
model.add(MaxPooling2D(pool_size=(3,3)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(64))
model.add(LeakyReLU())
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))

If you have time, try training an LSTM layer.


Does keras categorical_cross_entropy loss take incorrect classification into account

Does this mean the errors for $y_i=0$ do not contribute to the loss?

That is correct.
However, the respective weights that connect to wrong neurons will still have gradients due to the error, and those gradients will be influenced by the size of each incorrect classification. That is due to how softmax works:
$$\hat{y}_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$$
(where $z_i$ is the pre-softmax value of each neuron, a.k.a. the logit) . . . weights that affect one neuron's pre-transform value affect the post-transform value of all neurons. So those weights will still be adjusted to produce a lower $z_j$ value for the incorrect neurons during weight updates.

Why isn't the formula
$$J_{y'}(y) = - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$$
used?

It is not clear why when selecting a single class, that you would care how probability estimates were distributed amongst incorrect classes, or what the benefit would be to drive the incorrect values to be equal. For instance if $y' = [1, 0, 0, 0]$ then using the suggested formula for $J_{y'}(y)$ gives ~ 0.67 for $y = [0.7, 0.1, 0.1, 0.1]$ and ~0.72 for  $y = [0.73, 0.26, 0.05, 0.05]$, yet arguably the second result is better.
However, you would use this loss when dealing with non-exclusive classes (where the outputs would use sigmoid as opposed to softmax activation).


Colouring points based on cluster on matplotlib

The sklearn documentation shows you how:

colors = np.array([x for x in 'bgrcmykbgrcmykbgrcmykbgrcmyk'])
colors = np.hstack([colors] * 20)
...
if hasattr(algorithm, 'cluster_centers_'):
        centers = algorithm.cluster_centers_
        center_colors = colors[:len(centers)]
        plt.scatter(centers[:, 0], centers[:, 1], s=100, c=center_colors)


Supervised ML works on the assumption that the test data follows the same distribution as the training data. This is not always the case in real-world use cases, but it's at least necessary that the test data is "mostly similar" to the training data.
As a consequence, a BoW model can only be applied to data which uses mostly the same vocabulary, with a mostly similar distribution over the words. It is true that out of vocabulary (OOV) words frequently appear in the test data, because words in natural languages follow a Zipf distribution so there are many words which occur rarely. The general assumption in BoW ML models is that since these words occur rarely, they can reasonably be ignored.

During test time, while creating the term-frequency matrix, what if none of the words from my training BoW are found in some of my test emails/smses. Then, wouldn't the document vectors be zero vectors for those datapoints. How should I tackle this?

This event is supposed to be unlikely: a sentence usually contains at least a few common words (again due to the Zipf distribution). However this could happen with a very short text message. Anyway there is nothing special to do about it: all the words are ignored, the vector contains only zeros indeed, the model gives a prediction for this vector of features.

What if a new word from my test email/sms doesn't exist in BoW?

This is the traditional case of OOV words mentioned above. The most simple (and probably most common) option is to completely ignore the unknown word. With some probabilistic models smoothing can be used to account for the existence of OOV words, but as far as I know this is used only with n-grams.

How do I choose my BoW so as to improve my prediction accuracy?

Experimentally: use a validation set in order to evaluate several methods, then select the one which performs best and apply only this one on the test set.


So the problem is how to visualise your box plots so that they appear in the same plot (axes).
To do this, it is simply a minor alteration to your code.
f, axes = plt.subplots(1, 2)
sns.boxplot(x="status",y="assets" ,data=df1, palette="Set3",ax=axes[0])
sns.boxplot(x="status",y="assets" ,data=df2, palette="Set3",ax=axes[0])
fig.tight_layout()
plt.show()

The alteration is on the data=df2 line.


Alternative methods of population-level classification

I think the second approach is much superior to the first approach for two main reasons.

Reason #1: The first approach forces you to set a parameter (the threshold) somewhat arbitrarily.  When reporting your results, I think you would need to justify your choice of threshold.  Sure, 50% seems reasonable prima facie, but what if your models happen to underestimate the probability of diabetes?  Then a lower threshold might be appropriate.
This is particularly concerning if you are doing academic work, since it would allow you to "fudge" results by nudging the parameter in one direction or another.  It's best to avoid arbitrary parameters when possible.

Reason #2: You've already noted that, for some distributions, the choice of method will yield different estimates.  The second method will give you a more robust estimate of the target statistic.

To illustrate, suppose you have a very accurate model. Consider a case where your "second population" has five members, and your model outputs the following probabilities of diabetes for each member:

[0.49, 0.49, 0.49, 0.49, 0.49]

If you use the first method to estimate the percentage of the population that has diabetes (with a threshold of 50%), then your estimate will be 0%.  This is clearly unrealistic - the probability of none of the patients having diabetes is only $0.51^5 = .0345 = 3.45\%$.

If you use the second method, then you will compute the expected number of diabetes cases as follows:

$0.49*1 + 0.49*1 + 0.49*1 + 0.49*1 + 0.49*1 = 2.45$ cases

If you express this estimate as a percentage of the population, then your estimate will (obviously) be 49%.

I suppose you could also consider this estimate flawed, since it is impossible to actually observe. This will become less of a problem for large populations (like your case with n=10,000), but if it bothers you then you could round the expected number of diabetes cases to the nearest whole number.  In the contrived example above, this would yield an estimate of 40%.  This is still a much better estimate than that obtained using the first method.

Hopefully that helps!


Depends on which Quiz you are talking about. But these might help.

This is for AWS Machine learning exam:

https://d1.awsstatic.com/training-and-certification/docs-ml/AWS%20Certified%20Machine%20Learning%20-%20Specialty_Sample%20Questions.pdf

And this is for Microsoft Azure Machine Learning exam:

https://www.itexams.com/exam/70-774

Take a look.


In a nutshell
If you have a look at what mnist.train is you'll find there are two numpy arrays in it:
mnist.train.images (shape (55000, 784))and mnist.train.labels (shape (55000, 10)).Then all you have to do is iterate on these.
Example
This example is voluntarily a bit complicated so that it is robust to whatever sizes you chose but keeps the idea of fixing a number of batches as primary information on how much you want to train on the dataset :
X_train = mnist.train.images
Y_train = mnist.train.labels
number_of_examples = X_train.shape[0]
batch_size = 1000
number_of_batches = 200
epoch = 0

for i in range(number_of_batches):

    j = (i - epoch) * batch_size % number_of_examples
    k = (i - epoch + 1) * batch_size % number_of_examples

    if (k < j):
        k = number_of_examples
        batch_x = X_train[j:number_of_examples, :]
        batch_y = Y_train[j:number_of_examples, :]

        print('Shuffling data to retrain on dataset')
        data = numpy.concatenate((X_train, Y_train), axis=1)
        np.random.shuffle(data)

        # redefine X_train and Y_train from shuffled dataset
        X_train = data[:, :784]
        Y_train = data[:, 784:794]

        epoch = i + 1
    else:
        batch_x = X_train[j:k, :]
        batch_y = Y_train[j:k, :]

    # [...]
    train_step.run(feed_dict={x: batch_x, y_: batch_y, keep_prob: 0.5})

j is the row of the dataset which will be the batch's first row
k is the last one, so j-k=batch_size examples per batch, as expected.
Except if you've reached the end of the dataset (k<j)
in which case you just go to its end and then shuffle the
dataset so that batches won't be the same.
You count i - epoch and not just i so that you're sure you'll always
start at the 1st row of the dataset even when you've already gone
completely through it.
For instance if number_of_examples = 55555, batch_size = 1000 and
number_of_batches = 200:  i = 56 is the first time you go again
through the (shuffled) dataset but 56 * 1000 % 55555 = 445 so you'd
start at row 445 and not 0.
Also you HAVE to concatenate, otherwise the pairs (example, label) will be lost.

i, j ,k % Whaaat?
Run the toy code below to see how the iteration index behaves:
number_of_examples = 55555
batch_size = 1000
number_of_batches = 200
epoch = 0

for i in range(number_of_batches):

    j = (i - epoch) * batch_size % number_of_examples
    k = (i - epoch + 1) * batch_size % number_of_examples

    if (k < j):
        print('Overlap : ', j, number_of_examples)
        epoch = i + 1
    else:
        print(i, '   ', j, k)


What makes a problem good for an evolutionary strategy vs a genetic algorithm vs particle swarm optimization?

Both ES and PSO are algorithms that have canonically been described as inherently real-valued optimization algorithms. You can (and lots of people have) formulate combinatorial variants of those algorithms, but the variants that people learn first include operators that assume floating point numbers as parameters. I suspect that's part of the answer you're looking for. If you were taught that ES was for real-valued problems, you naturally tend to think of something else as being better at combinatorial problems.

A short answer that's likely accurate is "one really isn't better than the other". We have No Free Lunch results that show this to be true over a sufficiently broad class of problems, but even in practice, GAs, ES, PSO, these are sort of fuzzy concepts that bleed into one another without sharp boundaries. I could take a GA, modify it to work on real-valued problems, and if I keep tweaking it, end up with an ES, with no clear defining boundary of when it stopped being a GA and started being an ES. And at any point along that journey, it may or may not be performing better on a specific problem. The same is true if I start from ES or PSO and make combinatorial versions. Those versions will just be better sometimes on some problems.

The deeper answer is really that we just don't know why that is. There aren't good theoretical models of why one algorithm outperforms another on a specific problem. I've not been involved in this research field for a few years now, so I'm going to be unaware of some developments I'm sure, but generally, I think this is still an open question. You can sometimes take a specific problem and try to analyze what one particularly successful algorithm is doing, but often the answer is less than completely satisfying (e.g., it turns out that my particular mutation operator is biased to jump just the right distance to escape a local optimum on this specific problem, or whatever).

I'm also aware of work done maybe 6-8 years ago to attempt to build up a dataset of problem/algorithm performance data that one could feed to a machine learning algorithm to hope for insights, but I don't think that ever reached a point where a general question like "why is this better for combinatorial problems" could really be answered.


Ok, I was looking for an answer and now I have it clearer:
Scipy documentation does not elaborate too much on the explanation, but wikipedia article is much more clear. For those who are looking for an answer, there are two major groups of sparse matrices:

a) Sparse types used to construct the matrices:

DOK (Dictionary Of Keys): a dictionary that maps (row, column) to the value of the elements. It uses a hash table so it's efficient to set elements.
LIL (LIst of Lists): LIL stores one list per row. The lil_matrix format is row-based, so if we want to use it then in other operations, conversion to CSR is efficient, whereas conversion to CSC is less so.
COO (COOrdinate list): stores a list of (row, column, value) tuples.

b) Sparse types that support efficient access, arithmetic operations, column or row slicing, and matrix-vector products:

CSR (Compressed Sparse Row): similar to COO, but compresses the row indices.  Holds all the nonzero entries of M in left-to-right top-to-bottom ("row-major") order (all elements in the first row, all elements in the second row, and so). More efficient in row indexing and row slicing, because elements in the same row are stored contiguously in the memory.
CSC (Compressed Sparse Column): similar to CSR except that values are read first by column. More efficient in a column indexing and column slicing.

Once the matrices are build using one of the a) types, to perform manipulations such as multiplication or inversion, we should convert the matrix to either CSC or CSR format.


I think here you must maintain the actual tf-idf and create corpus over it.. Assuming you already have lables for documents available. You can rum classification over it.

Best classification I am anticipating for this problem would be naive bayes..


How to determine individual contribution when data is aggregated by team

Two alternatives

TrueSkill:

You might want to try a variant of TrueSkill. As Wikipedia says:

  Factor graphs are used to "pack up" each team into $(\mu ,\sigma )$
  pairs on which the update formulas are run; the skill updates are then
  correctly distributed to each player.

Here's the TrueSkill paper with more details. From the abstract:

  The new system tracks the uncertainty about player skills, explicitly
  models draws, can deal with any number of competing entities and can
  infer individual skills from team results. Inference is performed by
  approximate message passing on a factor graph representation of the
  model.

Simple Regression:

You can try to set this up as a multivariate regression problem. Inputs are tuples representing who was involved in the sale, outputs are # of sales. So something like this, assuming tuples are (Fred, Susan, Tom, John, Wayne, Conner, Winona):

(1, 1, 1, 0, 0, 0, 0), 25
(0, 0, 0, 1, 1, 1, 0), 12
(1, 0, 0, 1, 1, 0, 0), 16
etc...

Now each person's coefficient should be proportional to their contribution, assuming no interaction effects.


How Ipsative Standardization actually works for time series?

The pdf describes the following way:

Let us call the variable as var
where

m = mean(var)

sd  = sd(var)

x = (var-m)/sd

This will create a list of values which are Ipsative standardized, which is called Zi score


Decision tree on big categorical dataset

You can see this dataset as a regression problem and not a classification one. The label is ordered, so predicting 2.8 is not a mistake, it is just that it is close to 3 (satisfactory but not there).

I can suggest to start with a regression tree and then round the results to have integers.

Also you can consider this a multiclassification task, but as a starter I would go with regression.


Don't understand why I get an inverse ROC curve for SVM (Python)

For any classification problem if AUC<0.5, you are not performing better than random(0.5).
Reason could be:

Your classifier is over-fitted on the training set and performs very poorly on the test set.
Your test sample might be very small.
Your classifier is giving you the probability that the class is -1. Thus, you get a prediction (close to) 0 for a class 1, and 1 for a class -1 prediction. If your ROC method expects positive (+1) predictions to be higher than negative (-1) ones, you get a reversed curve.

A valid strategy is to simply invert the predictions as:
invert_prob=1-prob

Reference: ROC


How much training data does Word2Vec need?

It sounds like Doc2Vec (or paragraph/context vectors) might be the right fit for this problem.
In a nutshell, in addition to the word vectors, you add a "context vector" (in your case, an embedding for the author) that is used to predict the center or context words.
This means that you would benefit from all the data about "democracy" but also extract an embedding for that author, which combined should allow you to analyze the bias of each author with limited data about each author.
You can use gensim's implementation. The doc includes links to the source papers.


Devices behavior in one continuous variable vs events rate

Done with K-means clustering with descriptive statistics as features:

In short, I've tried the idea described in the question, even if I was thinking it won't work. Let the experience talk...

I initially had a list of devices data. Each element of the list were 2 columns, R rows matrix, and R was different for each device. So, per device:

[
    [mesureValue, timestamp],
    ...,
    [mesureValue, timestamp],
]

Since I'm only interested in the measureValue distribution, I've transformed the inital data to a 8 columns, N rows matrix, where N = number of devices.

The columns are, computed on the correponding device's measure value:

Arithmetic mean
Median
First quartille
Third quartille
Minimum
Maximum
Range
Standard deviation

With this matrix, I've applied K-means clustering using scikit learn (python).

I made the link between the matrix line and the physical device by using pandas Data Frames (python) who's line index are in fact the serial number of the device.

I've tried with 5 clusters, and it works.

Just in case of, if I need improvements in the future, I'm planning to add other statistics in the columns, especially for deviation vs. normality. So, for example Kurstosis and normal q-q plot p value.

Best regards.


The use of feature scaling in scikit learn

Short answer: if you apply the same scaling method as what the algorithm does, it won't change anything. If you choose a different scaling method, then you'll end up performing two transforms on the original data, which may not be a bad thing depending on the context.

This implies that you have to know what kind of scaling the algorithm performs.

As a takeaway, it is good practice to scale the data beforehand yourself, so that you are sure what you input the algorithm with. An scaling rarely arms your model results.

As a note (I know this is not your question, but it may help), you have to understand why you scale the data. Taking the simple case of a linear regression, if your input features are house size and number of rooms and say you want to build a model of the house price from those features, number of rooms and size will have very different ranges. Scaling will help to:

make the algorithm converge more rapidly
allow you to answer questions such as "what is the main contributor to the price" by comparing the coefficients (this would be incorrect w/o scaling beforehand)


How are new neural network architectures 'discovered'

Think of neural networks as math, and ask yourself how is new math discovered? New math comes from a combination of theory, experimentation and feedback from community.

Instead of thinking about it terms of creativity vs randomness to see what new ideas stick; I prefer to think in terms of theory and experimentation.

The current methodology for new neural net discovery is no different than any other applied field of math/engineering.


What you are looking for is performed exactly by sklearn.LabelEncoder, an example of which can be found here.

In your case:

from sklearn.preprocessing import LabelEncoder

encoder = LabelEncoder()
encoder.fit(["tt0984332", "tt0984332", "tt0847742", "ttnanana1"])

# Show the unique classes
print(encoder.classes_)
# out: array(['tt0847742', 'tt0984332', 'ttnanana1'], dtype='<U9')

# Convert labels to integers
encoder.transform(["tt0984332", "tt0984332", "tt0847742", "ttnanana1"])
# out: array([1, 1, 0, 2])


How is the Fully Connected Layer of a cnn network trained?

Their weights are also trained as well as the weights of the CNN filters.
So, the gradient of the loss is computed with respect to these weights also, and at each gradient step, both of the dense (fully connected) and CNN layers' weights are updated.


What Unsupervised Machine Learning algorithms can be used to detect Fraud in insurance?

Fraud, by definition, is a supervised concept.

No unsupervised algorithm will reliably detect this. At best you'll get some suspicious cases with unsupervised approaches such as anomaly detection.


Transitioning to Data Science from Actuarial Science

Since you have both degrees in pure math, most likely you won't need to do another Masters or similar. For now, focus on making your CV look more like DS/AI/ML/DL. Keep in mind online courses at present do not increase the value of your CV too much. So this is not an easy task, but you have a good start with pytorch. What I'd do if I were you: assuming, you are interested in applying DL in computer vision. Here are some links:

kaggle.com (competitions)
paperswithcode.com
labelled datasets: Pascal VOC, MS COCO,
torchvision library (model zoo) for pytorch (simply import torchvision to get started)

What you should do, is 1) read through some papers on state-of-the-art OS solutions for CV, like Faster R-CNN, Mask R-CNN, 2a) use them in pytorch, they are available in torchvision and/or github, 2b) same with the datasets, 3) hack into them to understand them at a deeper level, luckily they are all opensource, 4) write your own functionality for them, 5) contribute to contests/OS packages, put up your own models on github
You can put all of the above, especially 4) and 5) on your CV. Good luck!


Convolutional Neural Network: learning capacity and image coverage

You can look at nolearn/lasagne/util.py to see how learning capacity and image coverage are computed for each layer:

real_filters = get_real_filter(layers, img_size)
receptive_fields = get_receptive_field(layers, img_size)
capacity = 100. * real_filters / receptive_fields
capacity[np.logical_not(np.isfinite(capacity))] = 1
img_coverage = 100. * receptive_fields / img_size

Xudong Cao. "A practical theory for designing very deep convolutional neural networks". 2015. https://www.kaggle.com/c/datasciencebowl/forums/t/13166/happy-lantern-festival-report-and-code explains how to compute the capacity of a layer:

  To quantitatively measure the learning capacity of a convolutional
  layer we define the c-value of a convolutional layer as follows.

  c-value = Real Filter Size / Receptive Field Size

  where the real filter size of a k-by-k convolutional layer is k if
  there is no down sampling, it doubles after each down sampling i.e. 2k
  after one down sampling and 4k after two down sampling etc. The
  receptive field size is defined as the maximum size of a neuron can
  see on the raw image. It grows proportionally as the convolutional
  neural network goes deep. Figure 3 shows how the receptive fields
  grows in an exemplar convolutional neural network.

This is why the coverage increases as you go deeper in the network, e.g.:

# Neural Network with 122154 learnable parameters

## Layer information

name        size        total    cap.Y    cap.X    cov.Y    cov.X
----------  --------  -------  -------  -------  -------  -------
input0      1x28x28       784   100.00   100.00   100.00   100.00
conv2d1     32x26x26    21632   100.00   100.00    10.71    10.71
maxpool2d2  32x13x13     5408   100.00   100.00    10.71    10.71
conv2d3     64x11x11     7744    85.71    85.71    25.00    25.00
conv2d4     64x9x9       5184    54.55    54.55    39.29    39.29
maxpool2d5  64x4x4       1024    54.55    54.55    39.29    39.29
conv2d6     96x2x2        384    63.16    63.16    67.86    67.86
maxpool2d7  96x1x1         96    63.16    63.16    67.86    67.86
dense8      64             64   100.00   100.00   100.00   100.00
dropout9    64             64   100.00   100.00   100.00   100.00
dense10     64             64   100.00   100.00   100.00   100.00
dense11     10             10   100.00   100.00   100.00   100.00

Explanation
    X, Y:    image dimensions
    cap.:    learning capacity
    cov.:    coverage of image
    magenta: capacity too low (<1/6)
    cyan:    image coverage too high (>100%)
    red:     capacity too low and coverage too high


Problem in constructing co-occurence matrix

Whether to window only over sentences or over the whole corpus depends on the context of your problem and on the structure of your corpus. In this corpus the order of your sentences does not matter, if they were switched around it would still be the same corpus. This means that you should stick to only the lines and not see it as one line. If your corpus would be a book for example, spilling over sentences makes a lot more sense. If you are unsure, try both, it is just a hyperparameter of your model.


Using TensorFlow's preexistent Inception labeler

Yes, but the labels are limited to the ones from ImageNet.


How to begin understanding of audio and music analysis

To learn about the basics and a bunch of advanced topics, check out Meinard Müller's "Fundamentals of Music Processing" (FMP) (Amazon/accompanying website). There's also a website with many  Jupyter Notebooks demonstrating the book's content very well. FMP does not use librosa, but teaches you all the concepts you need to understand librosa. Most demonstrated approaches are signal-processing oriented. For work that relies on machine learning, you probably need to go through the recent research literature—ISMIR papers are a great place to start.

About the points you've raised:

Just like images (think JPEG, PNG, etc.) audio is stored in some format. OGG is just another format, like WAVE or MP3.
After audio has been decoded from some format like OGG, you get the raw samples, pretty much like points in a bitmap for images. One usually uses PCM for this (librosa does).
Most of the time we don't care about stereo, which is why librosa simply mixes stereo channels into one, mono channel by default. Also, most of the time we don't require CD quality, i.e., a sampling frequency of 44.1 kHz, so librosa downsamples the audio to 22.05 kHz by default. In a way, it's similar to reducing the size of an image by reducing its resolution.

Good luck!


Evaluating metrics F1, F2, Mean Average Precision for object detection

AP is more accurate than the F scores because it considers the PR relation globally. Articles adopt mAP on VOC because it is the official metric and they have to do comparison with other methods which also adopt this metric. Other competetitons such as some text detections also adopt P R and F score as the default metrics.


What is the novelty in AlphaGo, Google Deepmind's Go playing system?

The existing programs before AlphaGo were based on:

1) Convolutional neural networks (CNN), trained on a database of existing games.

OR

2) Monte Carlo tree search (MCTS)

AlphaGo is based on a combination of:

A) reinforcement learning: train networks by letting versions of CNN's (see above) play against eachother.

AND

B) MCTS using moves generated by step A)

On top of that, the performance was improved even further by using distributed computing with large amounts of CPU's and GPU's.

So the novelty was the combination of the above techniques A) and B).


Why there are no tutorials on calculating Gini impurity in a three-valued categorical feature variables?

There are no categories, you encode these columns as integer ones and proceede as usual.
Gini is there just to help you make decision on how and where you split. Gini,Entropy,variance are all replaceable components.


How to prepare training data for deep learning models

You can create a mask image with the background having value 0 and give all pixels the value of its class.

I am not sure what approach you are taking, but you can for example use Convolutional Neural Networks. They will learn from the mask image what features contribute to a pixel class and can therefore, at inference time, output a per pixel class probability.


Train, test split of unbalanced dataset classification

Best way is to collect more data, if you can.

Sampling should always be done on train dataset. If you are using python, scikit-learn has some really cool packages to help you with this. Random sampling is a very bad option for splitting. Try stratified sampling. This splits your class proportionally between training and test set.

Run oversampling, undersampling or hybrid techniques on training set. Again, if you are using scikit-learn and logistic regression, there's a parameter called class-weight. Set this to balanced.

Selection of evaluation metric also plays a very important role in model selection. Accuracy never helps in imbalanced dataset. Try, Area under ROC or precision and recall depending on your need. Do you want to give more weightage to false positive rate or false negative rate?


How do I perform weighted loss in multiple outputs on a same model in Tensorflow?

As I understand it, you want to have a model trained on multiple "tasks". If so, this is what it could look like:

input_data = <your input sequence input>
output_data = <array of size (N, 3)>

out1_weight, out2_weight, out3_weight = <your weights for each output to adjust loss contribution>

input = Input()
simple_rnn = SimpleRNN()(input)

# Define the three outputs
out1 = Dense(1)(simple_rnn)
out2 = Dense(1)(simple_rnn)
out3 = Dense(1)(simple_rnn)

model = Model(inputs=[input], outputs=[out1, out2, out3], loss_weights=[out1_weight, out2_weight, out3_weight])

model.compile(<whatever params>)
model.fit(input_data, [output_data[:, 0], output_data[:, 1], output_data[:, 2]])


I would assume that the physical input parameters occur in some sort of sequence & the Neural Network can easily identify patterns from it.

I would suggest you do some scaling on the output parameters. Or use non-linear activation functions so that your output ranges from upper bound & lower bound of that function.

Also i would like to suggest that you use LSTM. Recent time-series application using LSTM have promising outputs. Their application ranges from text generation to pixel filling in images & videos. They can be used to get bounded outputs & using torch neural networks allow you to implement them as per your requirements.

You can find implementation guide for Neural Network using torch library here


LSTM Time-series classification - derived feature

Add two new features, dummy variables, that represent the presence or absence of weekend and weekdays. You can create the feature (weekend/weekday) and use one hot encoding to create the two new.


What is feature channels mentioned in U-Net?

The feature channels simply mean the number of channels at a given point in the network. In the U-Net architecture, the number of channels doubles after the max pooling layers, see for example the first max pooling layer. The input is of size 568 by 568 pixels with 64 (feature) channels, after max pooling this becomes an array of size 284 by 284 pixels with the same 64 channels. The next convolutional layer then keeps the same size, so 284 by 284 pixels, but doubles the number of channels to 128.


Using Scorer Object for Classifier Score Method for scikit-learn

Why should it work? scoring is an argument of cross_val_score, not score. The documentation says score "returns the mean accuracy on the given test data and labels." This behavior is not specific to DecisionTreeClassifier, but common to classifiers in sklearn.

Perhaps you want sklearn.metrics?


best way to regularize gradient boosting regressor?

The hyper parameters that you could tune in any boosting technique are:

Depth of each tree: As you rightly pointed out this is very important because each tree in boosting technique learns from the errors of the previous trees. Hence underfitting the initial trees ensure that the later trees learn actual patterns and not noise.
Number of trees: this is kind of intuitive from previous point as the number of trees increase the learnable signal decreases and hence the ideal number of trees is more than underfitting trees and less than overfitted trees.
Learning rate: this parameter gives weights to previous trees according to a value between 0 and 1. Lower learning rates give lesser importance to previous trees. Higher weights lead to faster steps towards optimization.  Lower weights typically lead to global optimum. But lower learning rates need more trees to learn the function.

4.Sub sample: if the value is less than 1 a subset of variables is used to build the tree making it robust and learn signal from more variables. This variable reduces overfitting by not fitting only 1 variable but a group of variables.

These  variables if tuned correctly are sufficient to reduce overfitting.


Derivation of CNN math equations in Matrix format

I can't explain to you exactly what you want. However, if you got in this link and work on Q4, maybe this is what you are looking for.


What is the best technique/algorithm to compare trees changes?

Most of the machine learning algorithms are designed to work with data in a tabular format. That mean, each data instance is contained in a single row, and the values from each column are the observed values for a specific instance for a given variable. There are few reasons why the most ML algorithms are designed to work on this kind of data. An important factor is that the structure is very simple and various operations can be done with ease. A second reason is that even if looks like a inflexible structure, some sort of additional structure in your data still can be represented on tabular format (using redundancy). Another reason would be that an algorithm designed to work for a specific structure of the data will be constraint to work on a much smaller set of problems.

So, the main point is that "If the mountain won't come to Muhammad then Muhammad must go to the mountain" (note there's nothing religion related here). So what you have to do would be to fabricate the features yourself in a tabular format.

I will give you an example on how I see a starting point. Consider an instance a row in a table. Each row will be a change. A change has a label, it is good or bad. So, you can add a feature used as target feature called class. We go further by noting that a change is an insertion of a node. If your changes are of multiple type you can add a feature called operation-type having values: insert, delete, change, etc. Now, a node has a type also. You can add a new feature called node-type, which could be A, B, etc. What you have to do would be to invent those features by noting what is important for you or your business and eventually select only those features which would be relevant enough. I really hope it was clear enough.


Python plot for confusion matrix similar to confusion wheel?

I would strongly recommend against using a confusion wheel to visualize your confusion matrix. As impressive and fancy as they look, confusion wheels are visually complicated and unintuitive to read. Good data visualizations summarize information in a way that is simple, clear, and intuitive. Confusion wheels have none of those properties.

Unfortunately, there really isn't a great way to visualize a large confusion matrix. The best ones I've seen are a grid-based plot with either color-coding or size-coding representing the value in each table cell. Funnily enough, the visualization in Fig. 1(a) of the link you included is significantly easier to read and more informative than the confusion wheel in Fig 1(b). A grid graph like the one in the link's Fig. 1(a) is what I would recommend.


It does not represent anything of significance and is probably a bug (see the links below). It is caused due to the use of Sequential API as it omits the input layer and directly takes the embedding layer as input. It can be removed by the use of functional API or by commenting out this in keras/engine/sequential.py :

@property
def layers(self):
    # Historically, `sequential.layers` only returns layers that were added
    # via `add`, and omits the auto-generated `InputLayer`
    # that comes at the bottom of the stack.
    if self._layers and isinstance(self._layers[0], InputLayer):
        return self._layers[1:]
    return self._layers

You can take a look at issues raised on github regarding this here and here.


How to estimate the not available observation in time series data?

One of the most common ways for this purpose is to generate artificial data for that intervals. One of the most novel and powerful algorithms for this purpose is Generative Query Network (GQN).

In this algorithm the basic application is to create a 3D model of an object by observing a few 2D samples of it. So, the network by receiving the angle of the camera beside the 2D image, try to construct the 3D model.

But in this case for time series data, time can play the role of camera angle. So, you can generate high precision artificial data in the case of data lost.

Here is a brief description of this algorithm.

And also here is an implementation of it using Pytorch.

For more information you can read it's respective paper.


I'm assuming y_true and y_pred both are vectors. You are trying to apply a conditional on the value of a tensor. The library is telling you that it is not possible (tf.Tensor as Python bool).

You may want to check tensorflow control flow operations like tensorflow.equal


read csv file directly from URL / How to Fix a 403 Forbidden Error

The problem is that the url you have doesn't accept "non-browser" requests. The default header of Python requests is

'User-Agent': 'python-requests/2.13.0'

You can pass your own headers as an argument like that

from io import StringIO
import pandas as pd
import requests

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36'}

url="https://fineli.fi/fineli/en/elintarvikkeet/resultset.csv"
s=requests.get(url, headers= headers).text

c=pd.read_csv(StringIO(s), sep=";")
c


Are mainstream pre-trained models useful as discriminators?

Using a standard network architecture is perfectly reasonable. Most discriminator architectures are trivially different variants of well-known architectures anyway.
Depending on the GAN loss, starting with a pretrained network as a discriminator may be dangerous. For instance, the classical GAN loss minimizes the Jensen-Shannon divergence, so having a powerful discriminator right from the start will mean little support overlap and no useful gradients from the discriminator. This is a major reason for training instability in GANs. Yet, more recent losses (e.g., the various WGAN losses) do not have this shared support problem. However, losses like WGAN assume certain properties of the discriminator (i.e., Lipschitz-$k$), which pretrained networks are quite unlikely to satisfy out of the box. So from a purely theoretical perspective, I can see why it's not very appealing.
Another reason, I suspect, is that the discriminator's job is quite specialized and requires it to be dynamic; i.e., it has to be able to "change its mind" quickly as the generator adapts. It's not clear to me that a pretrained network, which people have worked to place in a high-quality and robust part of weight space, is good for this. Furthermore, since a discriminator's job is a little easier than e.g. ImageNet classification I suspect that the massive deep networks often used for transfer learning are simply unnecessarily large for the task (the backward or even forward passes being unnecessarily costly, I mean; GANs already take enough time to train).
That being said, it is actually common to use pretrained networks for the discriminator for perceptual feature matching losses. (You may be familiar with the even more common "perceptual losses" commonly employed in autoencoders).
The idea is to match feature statistics, rather than optimize a scalar probability (this idea comes from Salimans et al, I believe).
Note that the pretrained networks used for this (usually VGG) are not trained or even fine-tuned most of the time though; they're usually left alone.
Some relevant papers: [1], [2], [3].
I suspect that you can always use a pretrained network in a setup similar to these or in the manner employed by McGAN, and it will be helpful.
Despite all this, I am fairly sure that using a pretrained network as a starting point will still work fine in many cases, though I'd suggest starting with a WGAN variant for it to work. It may even do very well, depending on the data and the other training details. Let me know if you try it out =)


If I understand correctly, a single patient would be one feature, so one column, and they had interations with hospitals over short periods; followed by no interations (and therefore no data) over longer periods.

Due to the sparsity of data along each single time-series, perhaps you could look into ways to encode the information a little more consicely. I think this approach may lead to fairly good results because you are only aiming to categorise the patients into one of two groups.

This could perhaps be done by converting your base data into more insightful statistics. For example, for each patient, you could compute some standard values:

Number of time periods with(-out) hospital interaction
Number of different codes (assuming the code refers to the ailment or something useful?)
Average length of gap between periods of activity
Average length of periods of activity

For the above, you'd need to probably heuristcally define what a "period of activity" is, e.g. 5 data points within 100 timesteps. Otherwise you're in a "gap" between such periods.

This would dramitically reduce the amount of datapoints and lend itself quite well and allow you to use simpler classification methods, such as GLMs, logistic regression (as you only have two classes) or perhaps a simple feed-forward neural network.

The idea is to extract as much of the temporal aspect from the data as possible during this feature engineering/pre-processing, such that formal time-series modelling would become unnecessary.

If you want to try normal time-series, such as regression models, you need to remember that they return a numerical value by default (regression problem versus classification), so you would need to decide on a threshold to put patients in one of the two groups.

Try trying to encode the missing data simply as a count of timesteps between datapoints would not work because of the fact that each patient has different timelines, so the resulting data would have strongly varying dimensions, making many models unusable.


Supervised Learning with Necessarily Missing Data

I solved a similar problem not so long ago. Let X be a numerical variable that has missing values. First, we assign the value 0 to those instances in which the value is missing. Then, we add a new categorical variable to our model, X_missing, which domain is {True,False}. We obtain a data model with mixed numerical/categorical variables. You can then apply gradient descent to train a regression model from these variables.


Graph & Network Mining: clustering/community detection/ classification

Well ... Some points. Networked data is modeled with graphs. When you have different attributes you have Property Graph.

For clustering, you can extract the topology of the subgraph based on desired attributes and then use any Modularity-based algorithm (most recommended is Blondel algorithm). In Blondel algorithm you don't need to know the number of communities in advance.

Have a look at Network Science book by Barabasi to get more insight to networks.

For classification you may extract features from graphs and use common classification algorithms or use graph kernels and feed it to kernel methods such as SVM. Follow this.

Page rank is one of the methods for ranking but you have simpler choices according to your problem. See Centrality measures from the book above. There you can also see details of different ranking algorithms.

If you need more info you may drop a comment here.

Hope it helped. Good Luck!


Why Root Finding is important in Logistic Regression? (i.e. Newton Raphson)

This blogpost gives a broad answer to your question. In short, Newton's method is not used to find a root of the loss, but a root of the gradient. If you find a root of the gradient, then you are either in a maximum, a minimum, or a saddle point (the three of them are critical points). When using the cross-entropy loss function in logistic regression, it can be proved that this loss function has only one critical point, and this critical point corresponds to the global minimum. For this reason, finding a root of the gradient is equivalent to finding the only critical point, therefore the global minimum of the cross-entropy loss function.


The zero time delay seems wrong there. Suppose you have an autoregressive model. A general formula could be:
$$y_t = \alpha + \sum_{i}\beta_i y_{t-i} + w_t$$
Now, if $i$ is zero then you will have
$$y_t = \alpha +\beta_0 y_t + w_t$$
which is really weird. It looks like you predict values at time $t$ from the values at time $t$. In plain English is like you predict the future values from the real values. In that case the error is zero, since it is not very hard to predict something from the real values. So, I think the offsets should start at $1$.


Got the very same error recently.
Your network is usually defined as a class (here class EfficientNet(nn.Module). It seems when we load a model, it needs the class to be defined so it can instantiate it.
In my case, the class was defined in the training .py file. So what I did to fix that error was just copy-paste (it seems importing it didn't work for me, so I had to copy paste) the whole class definition to the file where you load your model.
In my case the class definition looked something like that and needed to be pasted before using the torch.load function:
class FCNClass(nn.Module):
    def __init__(self, in_feat=1000, nb_classes=3, nb_hid=1, hidden_size=500, act=nn.ReLU):
        super(FCNClass, self).__init__()

        self.act = act()
        self.flat = nn.Flatten()

        self.fc1 = nn.Linear(in_feat, hidden_size)
        self.fcs = nn.ModuleList()
        for nb in range(nb_hid):
            self.fcs.add_module("hid" + str(nb), nn.Linear(hidden_size, hidden_size))
        self.out = nn.Linear(hidden_size, nb_classes)

    def forward(self, x):
        x = self.flat(x)
        x = self.fc1(x)
        x = self.act(x)
        for lay in self.fcs:
            x = lay(x)
            x = self.act(x)
        x = self.out(x)
        x = self.act(x)
        return x

torch.load('Model.h5')

The class basically defines the architecture of the model, if it is not defined, then the load function doesn't know where to put the weights. Mine was done with torch.load(path) and not model.load_state_dict(torch.load(path)) but it should not make a difference


R formulas are just a common grammar for practical model specifications.

When you input a dataset and a formula like : cancer_probability ~ age + height to a model fitting function. The function will know that it has to work on age and height predictors to provide a model to give a cancer probability output.

As mentionned in the Spark link you give at the bottom, this grammar rely on some convention, as introducing an intercept by default.


My Neural Network is not learning anything

Make sure that while training, you are passing the data in a random order. It looks like you are passing all images of one class at a time then the next class etc. Shuffle the images and pass it in a random manner.


BERT training on two tasks: what is the order of tasks?

BERT is trained on both task at the same time.

MLM is done using the token representations.
NSP is done using the CLS representation.


Keras custom loss function as True Negatives by (True Negatives plus False Positives)

The Above Solution does not work when you use it during training it throws an error because of this line:

TN = np.logical_and(K.eval(y_true) == 0, K.eval(y_pred) == 0)

Maybe this could be a better solution to this problem you can calculate specificity after every epoch and also plot!

def specificity(y_pred, y_true):
    """
    param:
    y_pred - Predicted labels
    y_true - True labels
    Returns:
    Specificity score
    """
    neg_y_true = 1 - y_true
    neg_y_pred = 1 - y_pred
    fp = K.sum(neg_y_true * y_pred)
    tn = K.sum(neg_y_true * neg_y_pred)
    specificity = tn / (tn + fp + K.epsilon())
    return specificity


How to properly set up neural network training for stable accuracy and loss

The learning rate is one of those first and most important parameters of a model, and one that you need to start thinking about pretty much immediately upon starting to build a model. It controls how big the jumps your model makes, and from there, how quickly it learns.
There are learning rate technique called Cyclical Learning Rates.
Training with cyclical learning rates instead of fixed values achieves improved classification accuracy without a need to tune and often in fewer iterations. Check this out : Cyclical Learning Rates for Training Neural Networks. And implementation in pytorch and details blog : Adaptive - and Cyclical Learning Rates using PyTorch.
By this small trick, you can build a stable version of your model. Best of luck.


How does the cost function of LSTM works?

It is my understanding that there are no differences for loss function optimization in RNN's as compared to any other neural network models.  The loss is calculated at the very end of your network output.

Say you have this network architecture for predicting a categorical variable:

Layer Type, Output Size

Input: (None, 10, 6) = (batch_size,seq len,num_features)

LSTM: (none, 10,100) =(batch_size,sequence_len,hidden_unit_size)

TimeDistributed Dense: (None, 10, 10) = (batch_size,sequence_len,num_categories)

Softmax Activation: (none, 10, 10) = (batch_size, sequence_len, num_categories)

Here, you have 10 time-steps for the sequence and the final softmax output is giving a prediction for the next value of the categorical variable at every timestep (Assuming 1-hot representation of the categorical feature).  The Y value for each time step for training is simply the 1-hot representation of the categorical value at t+1.  The LSTM layer output gives (batch_size, 10, 100).  So, you now have a 100 dimensional representation of every time-step.  You pass this to the TimeDistributed Dense and then to softmax.  So, at the very end, at the softmax output layer, you have a Y vector of size (batch_size, seq_len, 1) that contains the true output for all 10 time-steps, for every sample in the batch.  You then would use this the same exact way as any other NN model using mini-batches to calculate loss and backprop errors to network weights.


Document similarity matching between Doc2Vec documents

Based on what you have shared, it seems like Doc2Vec should be suited to your objective.

That said, I think the Doc in the package name can lead people astray.  It gives the impression that you can feed in any documents and it will find the similarities among them. And while you certainly can feed whole documents, the reality is that the similarity / dissimilarity may not be aligned in a useful way to your task.  I have found that training the model on smaller chunks of the document, like sentences or paragraphs, allows for the model to identify subtle differences that can then be aggregated to the whole document.

In your case, are the 17 documents you mentioned, exemplars of 17 different types of documents whose label you want to assign to the entire corpus? If so, Doc2Vec may be overkill.  Have you looked into using Tf-Idf? Sometimes this approach works well if you don't necessarily need the word embeddings.  Just some food for thought.