There are some problems with your description:

During training, the decoder receives all the shifted target tokens, prepending the BOS token. You removed sole. The actual input would be: [<bos>, io, amo, il, sole]. Note that the output at the position of sole would be the end-of-sequence token <eos>.

During training, there is a single forward pass (not one per token), and all the output tokens are predicted at once. Therefore, only the last one of your attention masks is used.

During inference, we don't have the target tokens (because that is what we are trying to predict). In this case, we have one pass per generated token, starting with <bos>. This way, the decoder input in the first step would just be the sequence [<bos>], and we would predict the first token: io. Then, we would prepare the input for the next timestep as [<bos>, io], and then we would obtain the prediction for the second token. And so on. Note that, at each timestep, we are repeating the computations for the past positions; in real implementations, these states are cached instead of re-computed each timestep.

About some piece of Python code illustrating how the Transformer works, I suggest The annotated Transformer, which is a nice guide through a real implementation. You may be most interested in the function run_epoch for the training and in the function greedy_decode for the inference.


how to see decision tree when running in anaconda?

there is a simpler way but sometimes it doesn't work
 first, you should call your method like this

tree_clf=tree.DecisionTree()

then you can choose a name and path for it, let others remain the same as I did
so overall, it would look like:

from sklearn.tree import export_graphviz
export_graphviz(
tree_clf,
out_file=image_path("iris_tree.dot"),
feature_names=iris.feature_names[2:],
class_names=iris.target_names,
rounded=True,
filled=True
)

hope it works for you


What is the relationship between hard-sigmoid function and vanishing gradient descent problem?

The vanishing gradient problem appears with activation functions that squash their input in a very small output range, like sigmoids and tanh. The input range is large and therefore the probability for it to fall in the squashing part of the sigmoid is very high. Chain several sigmoids and there you have vanishing gradients,

Typical examples of such a type of activations are sigmoid and hyperbolic tangent. Hard sigmoids are no exception. Their main difference with normal sigmoids is that they are computationally cheaper.

ReLU's are different. Half of their range is linear. That's why they don't suffer from vanishing gradients. However, they have their own problem: dying ReLU. This happens when the input falls far in the left side and cannot escape.


Should one-hot encoded categorical features needs to be scaled when used along with text feature while deriving semantic similarity?

No, do not scale the one hot encoded vector with min-max scaling. That will lose the meaning of the data. One hot encoding means a data point is completely on a dimension or not. There is no meaning for those data points that are only fractional part of a dimension.
A better option to derive textual similarity using multiple features is to embedded all features in the same embedding space (including categorical features). StarSpace is one such embedding method.


When you use a 2D-convolutional layer, you assume that there is some 2D topological order in your data. E.g. images have 2D topological order: it's important that some pixel is at position (x, y) - if we move that pixel to some other position, the image would be wrong. The number of channels can be arbitrary in images (1 for grayscale, 3 for RGB), but we still consider this as a 2D topological order.

Check the Table 9.1. at chapter 9.7. Data Types (Deep Learning book by Ian Goodfellow and Yoshua Bengio and Aaron Courville).

So, since your data doesn't have 2D topological order, but 1D, you should use 1D-convolutional layer with multiple channels.

You could have 4 channels: 1 for time, and 3 other for space coordinates.


By default, GridSearchCV provides a score of nan when fitting the model fails.  You can change that behavior and raise an error by setting the parameter error_score="raise", or you can try fitting a single model to get the error.  You can then use the traceback to help figure out where the problem is.
For the LogisticRegression, I can identify the likely culprit: the default solver is lbfgs, which cannot handle L1 or ElasticNet penalty.  Use saga.
I don't immediately see an issue with the XGBoost model or parameters.  Get the error traceback using the first paragraph, and search/ask that as a separate question if needed.


The use is the same as always, without the non-linear activation, no matter how many layers you use, the result will still be a linear model.

The non-linear activation is performed on every single output "pixel" indevidualy:


Calculating Median and Mode of numerical variables for different subgroups in R

assuming your data is stored in an object named df you can do:

tapply(df$S_Calls, df$Emp_Stat, median)

As for the mode, oddly enough R does not have a built in function for that. You could define one yourself using:

mode_stat <- function(x) {
  ux <- na.omit(unique(x))
  ux[which.max(tabulate(match(x, ux)))]
}

and then do in a similar fashion:

tapply(df$S_Calls, df$Emp_Stat, mode_stat)


One option is the time command on the command line.


differentiating vector with different operation on each elements

Different operations on different elements don't prevent differentiation in any way.
Lets, say we call your above Loss function:
$$\mathcal L=L_1(\mathbf w) + L_2(\mathbf w)$$,
where $\mathbf w$ represents the weights of your model, $L_1$ and $L_2$ are the two loss functions you have defined using the different outputs of your model. The key point is that mathematically, I don't care which output you used to find $L_1$ or $L_2$, just the fact they both somehow depend on the weights of the model.
Thus, if our aim is to differentiate, we simply get (assuming everything in our model is differentiable):
$$\frac{\partial\mathcal L}{\partial\mathbf w} = \frac{\partial L_1}{\partial \mathbf w}+\frac{\partial L_2}{\partial \mathbf w}$$


Got some troubles with using OneHotEncoder to multiple categories

df.columns is incorrect because after One-hot-encoding you have more columns with different names. In your example instead of Sex you have two columns for the actual values and instead of Embarked you have three.
The whole point of the pipeline is to create those additional columns so why do you want to drop them? I do not know where you define df.columns but it seems to me the error is happening in this line:
df_total = pd.DataFrame(df_prepared, columns=df.columns)


Very low accuracy of new data compared to validation data

You are experiencing Data Leakage. In a comment, you explained that you shuffle your data before splitting into train/validation. For each validation point, you likely are showing the model data that is nearby temporally both before and after the time of the validation point. This is information that the model can not possibly hope to have when running in real time.

To alleviate this, I would keep the data in the correct time order, and instead take the validation data as a contiguous chunk of time. If you want to be the most careful, you could throw data away in a small window around the validation data, so that the training data won’t contain information about the edges of the validation data.

This appears to be a good resource, but I have not read it too thoroughly: https://www.kaggle.com/dansbecker/data-leakage


How is calculated the error with multiple output neurons in neural network?

You are mixing various concepts:

$L = \frac{1}{2N}\sum^{N}_{i=1}(e_i)^2$ is used only for regression problem and not for binary classification because MSE fits very well when your target distribution is normal
You can use the latter formula for binary classification but will works really bad because your target data distribution is a Bernoulli, not Normal. Remember that the choice of the right imply a prior assumption on the target data distribution. For this reason the right formula is binary crossentropy (aka negative log likelihood of a Bernoulli) $$ L = - \sum_i y_i \log \hat{y_i} (1 - y_i) \log(1 - \hat{y_i})   $$
For multi classification problem there is a generalized formula of binary crossentropy which is called categorical crossentropy. If $\hat{y}$ is a vector of C element, one for each class and the true class $y$ is encoded as integer (e.g 0, 1, 2 ...) then the loss is $$ L = - \sum_i \log(\hat{y_i}[y]) $$


After consulting with someone I found out that the  corresponds to a model that would include all the variables, in other words none of the variables were removed. So consider the line in the output for the variable aaa. The listed RSS and AIC are the values for a model that would include all variables but aaa and we see an increase in the RSS and AIC. The other listed results can be considered in the same fashion. The best model is then the one where none are removed since this has the smallest AIC.


See also this StackExchange post. The Schwarz criterion, or BIC, is very general, so the question is, "how does this formula apply the BIC to $k$-means?"

The Wikipedia page on Bayesian Information Criterion (BIC) uses this definition of BIC:

$$BIC = -2 \ln \hat{L} + \kappa \ln {n}$$

where:

$n$ is the #observations
$\kappa$ is the #free parameters (I use Greek $\kappa$ to distinguish from your $k$)
$\hat{L}$ is the maximized likelihood after fitting parameters

Pelleg and Moore (2000) use $R$ for the #observations.  Assuming your professor does to, that means:

$\kappa = \lambda m k$, the # free parameters.
$W(C) = -2 \ln \hat{L}$

So, what are $\lambda$ and $m$?  I can't tell which is which, but it has to come out something like this:

$k$ is the number of clusters [as you say]
$\lambda$ is the number of dimensions (columns) in the data
$m$ is the number of parameters to estimate for each dimension (2 for the usual Gaussian: mean and variance).

It makes sense that $W(C)$, the "within-class scatter" would be measured by log likelihood.  Note, $W(C)$ has to sum the log likelihood for all clusters.


How to interpret a drastic accuracy loss while training a neuronal net (CNN)?

There could be several reasons:

Numeric stability issues + overfitting. Some neural activation, or some weights, continue to increase/decrease monotonically until some function (e.g. softmax) encounters a numerical stability issue. This could happen when your network is memorizing instead of properly learning.
Improper learning rate. A well defined learning algorithm usually gradually decrease its learning rate as training proceeds. If that didn't work properly, or you are using a too large constant learning rate, some activation or weights could be exploding.

For the above two reasons, you may want to print the min and max values for the activations and weights in each layer during your training. If something looks like exploding (i.e. reaching something like 10E+38) you need to check on that. Also monitor if any values become NaN or infinity.

A bug in your code for calculating accuracy or plotting. You may want to manually inspect the predicted labels and the ground truth labels to confirm whether your algorithm is performing poorly after epoch 38.


Difference between tf-idf and tf with Random Forests

Decision trees (and hence Random Forests) are insensitive to monotone transformations of input features.

Since multiplying by the same factor is a monotone transformation, I'd assume that for Random Forests there indeed is no difference.

However, you eventually may consider using other classifiers that do not have this property, so it may still make sense to use the entire TF * IDF.


Why don't we use Manhattan distance instead of euclidean distance in linear regression?

The main reason for that may be typically use Euclidean metric; Manhattan may be appropriate if different dimensions are not comparable.. While using regression-based methods you may have noticed that you usually have features of real values. You usually normalise the features and feed them to your model. The act of normalising features somehow means your features are comparable. In cases where you have categorical features, you may want to use decision trees, but I've never seen people have interest in Manhattan distance but based on answers [2, 3] there are some use cases for Manhattan too. You can also consider that they are comparable.


Combining data sets without using row.name


Is there a maximum limit to the number of features in a Neural Network?

It highly depends on your data. If it's image, I guess it is somehow logical but if not I recommend you constructing covariance matrix and tracking whether features have correlation or not. If you see many features are correlated, it is better to discard correlated features. You also can employ PCA to do this. Correlated features cause larger number of parameters for neural network.

Also I have to say that maybe you can reduce the number of parameters if your inputs are images by resizing them. In popular nets the length and height of input images are usually less than three hundred which makes the number of input features 90000. Also you can employ max-pooling after some convolution layers, if you are using convolutional nets, to reduce the number of parameters. Refer here which maybe helpful.


I would start off with All Of Statistics by Larry Wasserman. This quickly gets you upto speed with statistics assuming you have some mathematical background. I think all it needs is introductory calculus and linear algebra.

R is pretty straightforward to pick up and there are a number of resources you can use. The R Programming course on coursera is an excellent short course to get you familiarized with R. Besides that there are a number of books and tutorials on the subject.

I would recommend you start working through All of Statistics and start playing around in R along with that book.

If you are at a SaaS company, there are a number of data sciency roles and responsibilities that should be available. Most SaaS companies will have analytical tools to provide basic insight. Once you start learning statistics and data science, you will be able to identify the gaps in the pre baked tools and ways to improve them. I would also read The Field Guide to Data Science, it's a short book that will give you a high level idea of data science and its utility.


Welcome to the forum. I don‘t know how well you understand statistical models. However, as you want to evaluate some (exogenous and controlled) change, you could start off with „difference in difference“ https://en.wikipedia.org/wiki/Difference_in_differences.

This is the standard method for ex post evaluation and it is very flexible in controlling other influential factors.


How to delete entire row if values in a column are NaN

Well if the dataset is not too large I would suggest using pandas to clean the data. So you would need to first do

Python2

python2 -m pip install pandas

Python3

python3 -m pip install pandas

If you already have anaconda installed you can skip the above step. Next you could go through an IDE (like jupyter) or through the shell type the following commands

import pandas as pd
df = pd.read_csv("filename", dtype=str)
#or if excel file
#df = pd.read_excel("filename", dtype=str)

df = df[pd.notnull(df['Gender'])]

Then you would want to save your result in a file with

df.to_csv("newfile");


What algorithm can help me discover synonyms?

word2vec is probably the way to go. It maps words to a point in n-dimensional space. You can use Euclidean (or whatever distance) to find the nearest points to a given word. If training went well, the closest points should be a synonym.


What's wrong with my implementation of the Adaline algorithm?

First of all, let me add this schema which I think is quite nice to understand the transition and improvement from the initial Rosenblatt's perceptron and the Adaline algorithm:

In Adaline, provided that the cost function (your y(t)-s(t)) is differentiable, the weights can be updated and there is no restriction of y and s having the same sign: the objective is to minimize the cost y-s.
Below you can find the code provided in the excellent book by Sebastian Raschka:
class AdalineSGD(object):
"""ADAptive LInear NEuron classifier.
    Parameters
    ------------
    eta : float
    Learning rate (between 0.0 and 1.0)
    n_iter : int
    Passes over the training dataset.
    shuffle : bool (default: True)
    Shuffles training data every epoch if True
    to prevent cycles.
    random_state : int
    Random number generator seed for random weight
    initialization.
    Attributes
    -----------
    w_ : 1d-array
    Weights after fitting.
    cost_ : list
    Sum-of-squares cost function value averaged over all
    training samples in each epoch.
"""
def __init__(self, eta=0.01, n_iter=10,
                shuffle=True, random_state=None):
    self.eta = eta
    self.n_iter = n_iter
    self.w_initialized = False
    self.shuffle = shuffle
    self.random_state = random_state

def fit(self, X, y):
    """ Fit training data.
    Parameters
    ----------
    X : {array-like}, shape = [n_samples, n_features]
    Training vectors, where n_samples is the number
    of samples and
    n_features is the number of features.
    y : array-like, shape = [n_samples]
    Target values.
    Returns
    -------
    self : object
    """
    self._initialize_weights(X.shape[1])
    self.cost_ = []
    for i in range(self.n_iter):
        if self.shuffle:
            X, y = self._shuffle(X, y)
        cost = []
        for xi, target in zip(X, y):
            cost.append(self._update_weights(xi, target))
        avg_cost = sum(cost) / len(y)
        self.cost_.append(avg_cost)

    return self

def partial_fit(self, X, y):
    """Fit training data without reinitializing the weights"""
    if not self.w_initialized:
        self._initialize_weights(X.shape[1])
    if y.ravel().shape[0] > 1: #if we have more than one sample
        for xi, target in zip(X, y):
            self._update_weights(xi, target)
    else:
        self._update_weights(X, y)

    return self

def _shuffle(self, X, y):
    """Shuffle training data"""
    r = self.rgen.permutation(len(y))

    return X[r], y[r]

def _initialize_weights(self, m):
    """Initialize weights to small random numbers"""
    import numpy as np

    self.rgen = np.random.RandomState(self.random_state)
    self.w_ = self.rgen.normal(loc=0.0, scale=0.01,
                               size=1 + m)

    self.w_initialized = True

def _update_weights(self, xi, target):
    """Apply Adaline learning rule to update the weights"""
    output = self.activation(self.net_input(xi))
    error = (target - output)
    self.w_[1:] += self.eta * xi.dot(error)
    self.w_[0] += self.eta * error
    cost = 0.5 * error**2

    return cost

def net_input(self, X):
    """Calculate net input"""

    return np.dot(X, self.w_[1:]) + self.w_[0]

def activation(self, X):
    """Compute linear activation"""
    return X

def predict(self, X):
    """Return class label after unit step"""

    return np.where(self.activation(self.net_input(X))
                    >= 0.0, 1, -1)


Are cluster feature and micro-cluster good summary statistics for outlier detection in high dimensional data streams?

No.

Because assignment to microclusters is distance-based, and distances do not work in high-dimensional data anymore. Most likely one mucrocluster will become most central by chance and collect all the samples.


For starters, I am not confident that combining the results will give you what you expect. Have you checked if the true negative remain the same in both occurrences?

Moreover, have you tried to alternate the hyper parameters on the XGBoost model responsible for class balancing like max delta step or scale pos weight.

You can also try to use Sampling techniques for Over Sampling and Under Sampling. Or try possibly using many trees on  XGBoost on a different random Under Sample.

P.S. I am interested on the anomaly detection paper you mentioned, could you provide a link, thnx.


sckit-learn Cross validation and model retrain

"I want to train a model and also perform cross-validation in scikit-learn"

By this, I assume you meant saying, "I want to assess different model performace using K-fold Cross-validation approach and based on the performance, I want to select a model fitting"

Primarily, K-fold CV will divide the training data into a number of folds k, and then will fit and evaluate the model. Reference from chapter 5 in ISLR:

  Cross-validation can be used to estimate the test error associated
  with a given statistical learning method in order to evaluate its
  performance, or to select the appropriate level of flexibility.

I try to explain with a sample code:

from sklearn import svm
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score

# Training Data
feature_cols=['X1', 'x2', 'x3']
X = train_data[feature_cols]
y = train_data.target

#Build a svm classifier model
model_svm = svm.SVC(C=100, gamma=1)

#Initialise the number of folds k for doing CV
kfold = KFold(10, False, 2)

#Evaluate the model using k-fold CV
cross_val_scores = cross_val_score(model_svm, X, y, cv=kfold, scoring='accuracy')

#Get the model performance metrics
print("Mean: " + str(cross_val_scores.mean()))

You can do k-fold CV for different model parameters say (different C and gamma values in SVM) or even different models altogether (say Logistic regression) and choose the best model.

Once you chose the best performing model you then can proceed to fit the model for making the prediction.

#Fit the model
model_svm.fit(X,y)

#Prediction on test set
model_svm.predict(test_data)

"I believe and according to my understanding if the CV is training the model for performing k-fold validation, do we need to train again as any of k-models are not trained on the whole training data."

In the above sample code, the number of folds k=10. So CV will be done 10 iterations with each training set containing (k − 1)n/k = 9(n/10) observations and n/10 observations for testing. So no, you don't need to train again as K-fold CV does this procedure for you. From ISLR:

  K-fold CV approach involves randomly dividing the set of
  observations into k groups, or folds, of approximately equal size. The
  first fold is treated as a validation set, and the method is fit on
  the remaining k − 1 folds. This procedure is
  repeated k times; each time, a different group of observations is
  treated as a validation set.

"My main confusion is that for hyper parameter tuning and in grid search we can have best paramters,but in cross validation only do we need to call the fit method after CV step"

Again, the k-fold CV method is used for model assessment. What you can do is to use the grid search for selecting the best parameters and say the GridSearch method to use K-fold Cross-validation to selecting the best parameters. Also in a way Yes, you will call the fit method after CV because you would have chosen the best performing method and be proceeding to fit the model.

    from sklearn import svm
    from sklearn.model_selection import KFold
    from sklearn.model_selection import GridSearchCV

    # Training Data
    feature_cols=['X1', 'x2', 'x3']
    X = train_data[feature_cols]
    y = train_data.target

    #Build a svm classifier model
    model_svm = svm.SVC(C=100, gamma=1)

    #Initialise the number of folds k for doing CV
    kfold = KFold(10, False, 2)

    Cs = [0.001, 0.01, 0.1, 0.3, 1, 3, 10, 100]
    gammas = [0.001, 0.01, 0.1, 0.3, 1, 3, 10, 100]

    param_grid = {'C': Cs, 'gamma' : gammas}
    grid_search = GridSearchCV(model_svm, param_grid, cv=kfold, scoring='accuracy')
    grid_search.fit(X, y)
    grid_search.best_params_

I hope this clears thing up.


Predicting a Continuous output in a dataset with categories

A random forest will work, however standard regression will also work with categorical variables as predictors. You will have to "one-hot" encode your categorical predictors into 6 "dummy" variables (classes-1 = 7-1 = 6). The first dummy variable will encode 0/1 for whether or not the observation is class A, second dummy variable as 0/1 for class B, etc. You only need 6 dummy variables because if all of them are 0 for a given observation, that means the observation is in group 7 (G).

In some languages, such as R, the regression command will automatically do this one-hot conversion for you. For python, the pandas package can do this for you with pd.get_dummies(data).


Adding some extra general points to the previous answer:

As a decision tree algorithm, Random Forests are less influenced by outliers than
other algorithms. A good discussion about it is here.
They also do not make any assumptions about the underlying distribution of your data,
and can implicitly handle collinearity in features, because if you have two
highly similar features, the information gain from splitting on one
of the features will also use up the predictive power of the other
feature. Read about it here.
Random Forests can be used for feature selection because if you fit
the algorithm with features that are not useful, the algorithm simply
won't use them to split on the data. It's possible to extract the 'best'
features (which could be the total number of
times a feature was used to split on the data, or the mean decrease
in impurity etc). However, as with my point above, you cannot read too much into the
relative importance of the features, especially if you have ones that are highly correlated.
I think the biggest deciding factor of whether to use a RF or another algorithm is probably if you want to understand more about the relationship that
features have with the target and the degree of influence they have.
If this is important for your use case (for example, you want to know if a feature has a positive or negative relationship
with the target and the degree to which it affects the outcome) then others like Logistic Regression and Lasso are better choices.
Also, if you want your model to extrapolate to predictions for data that is
outside of the bounds of your original training data, a Random Forest
will not be a good choice.

Hope this helps!


GAN's and traditional augmentation techniques are fundamentally different in a way: A GAN produces (and combines) patterns previously seen in a dataset, data augmentation adds patterns to the data.

Well thought out data augmentation tries to add variations that could exist in the data. For instance: In arial photography rotation around the z-axis is very trivial and it could be smart to add that generously, on the other hand of the spectrum: In computer generated cartoons for instance, you'd reasonably do not expect much gaussian noise, and you might want to use that sparsely.

GAN's don't add information, they mostly add the same localised patterns that your (I'm assuming) ConvNet to train also has. The benefit a GAN offers is that you can use unlabelled data to train your convolutional layers with concepts you'd might expect in the domain.

Other techniques that go in that same direction, and you might want to check out, are weakly supervised learning and auto-encoding (which is imho very close to your original GAN idea).


Batch Normalization and Dropout together causing incorrect segmentation results

The differences between UNET and the deeper networks like RDRCNN and Tiramisu is that the deeper networks have Batch Normalization layers.  They also include dropout.

I thought it might be the bias terms in the convolutional layers, but the bias for all those layers was set to 'False'.  I tried regularization, changing the momentum value of the Batch Norm layers, and removing the Batch Norm layers closest to the output.  However, the only thing that worked was removing all the Batch Norm layers entirely or removing all the dropout entirely.

It seems this is a data issue on my end.  Certain images are exacerbating a common problem with using batch normalization and dropout together.  For some, this may introduce too much noise into the network, leading to incorrect predictions.


What if the training data and the prediction data are the same?

Yes, it will impact the results.
In the best case your model will not overfit at all so you could expect the same performance between the training set and a previously unseen set (= normally the testing set). In case your model overfits even a little bit, then your results on the training set will be higher (better) than on unseen data.
The impact depends on the classifier/hyperparameters you are testing. For instance, if you are using a KNearestNeighbors classifer with k=1, then the impact will be extreme as you will have 100% accuracy on whatever training set.


I haven't seen the specifical cases you mention , but I know of machine learning within insurance, and the types of models I've seen being used always depended on the criticality of the data and use of the results.

So a more general answer: if they need to justify all modelling decisions (as is often the case in insurance), they stick to more traditional models, such as logistic regression, tree-based methods and the like. More cutting-edge models are considered black-boxes.

If the project if more about analysing ad-hoc topics and non-core businesses, but which could perhaps support core business decisions (or otherwise), then anything is game: from stacked bidirectional LSTMs on word embeddings of their contracts to capsule networks on images scraped from social media websites.

I suppose you could model Moral Hazard similarly to fraud, in which case there are plenty of examples to be found online. I think there was even a Kaggle competition on that.


The relevant terminology is "multi-output" or "multi-target" regression/classification.  A survey article from a few years ago has some good discussion. This functionality is implemented in many packages (for instance, in sklearn (where it extends the functionality of other regressors, rather than being an independent implementation) and in R.


What is an object detection problem with only one class called?

If you are talking about "two-stage" obejct detectors like Faster R-CNN, note that the second phase is not only for classification, but to obtain more accurate results (https://stackoverflow.com/a/61965140/4762996).

in addition, I guess training a detector with many classes acts like a regularizer, which results in much better accuracies.

The only benefit of explicitly training with one class could be the reduzed model size (and a corresponding speedup).

Note also that there are one stage object detectors (CornerNet: https://arxiv.org/abs/1808.01244, YOLOv3: https://pjreddie.com/media/files/papers/YOLOv3.pdf, DETR: https://arxiv.org/abs/2005.12872, and many more). I just wanted to stress that just because you only have one class, it does not make sense to take a two-stage detection architecture and skip the second stage.

Finally, have a look at: https://stats.stackexchange.com/q/351079

An interesting direction would be to incorporate one-class classification approaches, e.g. as mentioned here: https://stackoverflow.com/a/61965358/4762996.


Different significant variables but same Adjusted R-squared value

It seems that removing features was not really helpful in bringing up the model fit. Differences in significance of features may be due to exclusion.

One thing you should try is regulation by lasso/ridge regression. In R, this can be easily implemented by the glmnet package. Here is a tutorial. This is the best feature selection method imo since there is a mathematical rational behind (see Introduction to Statistical learning for more background).

Lasso can shrink features to zero (viz drop them). Ridge cannot shrink features to zero. Just give it a try.

Hint: In linear regression with continuous features you can also add polynomials to your regression to increase fit using the poly function. You could also see if regression splines help you to deal with hidden non-linearity. The gam packages is a really good start. Here are the docs.

The book Introduction to Statistical Learning covers these topics in a very good way and comes with useful R examples. Here is the code.


Why is averaging the vectors required in word2vec?

The reason to average the embedded vectors of the words in a paragraph or document is to obtain a single fixed-size vector that represents the whole text. Then, the document-level vector can be used as input to a document classification model or any other document-level model.
If you explicitly want to compute word-level representations and then combine them into a document/paragraph-level representation, then averaging is the standard approach.
On the other hand, to obtain document/paragraph/sentence-level representations in general, there are many alternatives to combining word-level vectors. Some remarkable examples include doc2vec for paragraph/document-level, or LASER or BERT for sentence-level representations.


How to utilize measurement accuracy metadata in classifier

You could use a multi label classifier so that your model could use the extra feature to decrease its confidence score. If there is indeed a correlation with inaccuracy of the measurement and "acc" it should be able to account for that in its prediction through its probability output


Is it possible to build a regression model for predicting movie gross using sections on their wikipedia pages?

If your models suffers from multicollinearity problems, then yes, I'd say go for dimensionality reduction (possibly non-linear, such as t-SNE or Autoencoders). Moreover, 100 variables for 650 observations is a lot, I would apply a much more drastic reduction, but that's up to your preferences.

If your model is overfitting excessively, then I suggest you to try Neural Networks with high dropout probability. That helps a lot in preventing overfitting. Also, try other models, the more you try, the more confident you'll be about results.

Please keep in mind that normally Neural Networks require a lot of data to perform very good. If you have just 650 observations, tree-based models (Random Forest Regressions or XGBoost Regressions) could work even better than Deep Learning. They are ensembles of trees, i.e. desgined to combat overfitting, and could capture non-linear patterns of your dataset.


That's about expected. If you divide the number of seconds by the number of images you processed, you get 33 milliseconds per image, which seems about right for such a small network. Larger networks usually take in the ballpark of 50 to 200 milliseconds per image.

Yes, a large dense layer is likely to hurt your performance, since that's a huge matrix (256 by 4096) and a large matrix multiplication to go along with it every time you run the network.


So, you want to make embeddings for sequences of items, and order doesn't make much sense. You don't have a specific objective, just want to retrieve embeddings with some conventional properties, such as natural clustering.
If you can train word2vec and take mean of item embeddings in a cart, you'll probably get noisy vectors. It doesn't work as well as expected, from my experience. I suggest that you try dimensionality reduction methods, such as NMF. Fit it with a sparse binary matrix of shape (num_samples, num_items). You'll get transformed vectors of desired dimensionality.
Then you might want to evaluate embeddings somehow. If don't have any quality metrics, you can either perform clustering to see how your embeddings form groups, or make nearest neighbour search queries by hand.
Good luck!


Does gradient descent always converge to an optimum?

Gradient Descent is an algorithm which is designed to find the optimal points, but these optimal points are not necessarily global. And yes if it happens that it diverges from a local location it may converge to another optimal point but its probability is not too much. The reason is that the step size might be too large that prompts it recede one optimal point and the probability that it oscillates is much more than convergence.

About gradient descent there are two main perspectives, machine learning era and deep learning era. During machine learning era it was considered that gradient descent will find the local/global optimum but in deep learning era where the dimension of input features are too much it is shown in practice that the probability that all of the features be located in there optimal value at a single point is not too much and rather seeing to have optimal locations in cost functions, most of the time saddle points are observed. This is one of the reasons that training with lots of data and training epochs cause the deep learning models outperform other algorithms. So if you train your model, it will find a detour or will find its way to go downhill and do not stuck in saddle points, but you have to have appropriate step sizes.

For more intuitions I suggest you referring here and here.


Handwritting Recognition moving from character level to word level

I do not think that you would need a recurrent neural net here. This will be much slower to train than a ConvNet. Additionally, your data is images, correct? I think you could use a deep convolutional net to a fully-connected net. This should be fast and perform well. Please look at the following archived blogs from keras.
Building powerful image classification models using very little data
How convolutional neural networks see the world


What are the pros and cons of Keras and TFLearn?

TFlearn is a modular and transparent deep learning library built on top of Tensorflow. It was designed to provide a higher-level API to TensorFlow in order to facilitate and speed-up experimentations, while remaining fully transparent and compatible with it. Even with TensorFlow, however, we face a choice of which “front-end” framework to use. Should we use straight TensorFlow, or TF Learn, or Keras, or the new TF-Slim library that Google released within TensorFlow.

Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.

Straight TensorFlow is really verbose while Keras and TfLearn both seem solid, but the TfLearn syntax seems a little cleaner. One drawback to Tflearn is the lack of easily integrated pre-trained models.

Actually there are so many answers for your question here and here and I quote some of them here.

TensorFlow is currently the mainstream of deep learning framework, they are all the wrapper of TF. Whereas, Keras was released at the age of Theano, and therefore having a good support from Theano’s users. While TensorLayer and TFLearn are both released after TensorFlow. A good reason to choose Keras is that you could use TensorFlow backend without actually learning it. Plus Keras tends to wrap up the model deeply, so you don't necessarily need to consider the backend to be Theano or TF, which is a big advantage of Keras.

It depend on what do you want to do, fast prototyping or something else?

Keras : Many people are using it, easy to find examples on github. Suitable for beginner. Capable of running on top of either TensorFlow or Theano. Tflearn : Why no one discuss it? It is also a famous library, transparent over TensorFlow. High running speed. TensorLayer: Just release (Sep 2016), transparent over TensorFlow. High running speed. Easy to extend, suitable for professional, its tutorial include all modularized implementation of Google TensorFlow Deep Learning tutorial. TF-Silm: Just release (Aug 2016) similar with Tflearn, but no RNN layer at the moment (Sep 2016).

The best deep learning framework is the one you know best.


Q-Learning: Target Network vs Double DQN

Ok, it's simple!

Just the "Target network approach":

  Select an item from Memory Bank
  Using Target Network, from $S_{t+1}$ determine the index of the best action $A_{t+1}$ and its Q-value
  Do corrections as usual

The "double DQN approach"":

  Select an item from Memory Bank
  Using Online Network, from $S_{t+1}$ determine the index of the best action $A_{t+1}$.
  Using Target Network, from $S_{t+1}$ get the Q-value of that action.
  Do corrections as usual using that Q-value

As seen from 3, the Online Network gives us the action that it thinks is highest.  However, we let the Target Network have the final say on the score of such an action.

Were we to let Target Network select a best action, it could have very well selected some other index.


Track underlying observation when using GridSearchCV and make_scorer

Firstly; this is a really clear, well written question. Kudos!

I think the answer is to take the folding out of the CV and do this manually. You can generate the indices of the training and testing data using KFold().split(), and iterate over them in this manner:

from sklearn.model_selection import KFold, GridSearchCV
import pandas as pd
from sklearn.ensemble import RandomForestClassifier

iris = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv')

kf = KFold(n_splits=3)

for train_idx, test_idx in kf.split(iris):
    print(train_idx, test_idx)

And what you'll get is three sets of 2 arrays, the first being the indices of the training samples for this fold and the second being the indices of the testing samples for this fold. Using that, you could manually cross-validate like this:

kf = KFold(n_splits=3)

x = iris.drop('species', axis=1)
y = iris.species

max_depths = [5, 10, 15]

scores = []

for i in range(len(max_depths)):
    rfc = RandomForestClassifier(max_depth=max_depths[i])
    scores.append({'max_depth':max_depths[i], 'scores':[]})
    for train_idx, test_idx in kf.split(iris):
        rfc.fit(x.iloc[train_idx], y.iloc[train_idx])
        scores[i]['scores'].append(custom_scorer(y.iloc[test_idx], rfc.predict(x.iloc[test_idx]), train_idx, test_idx)

So that's running once per value in max_depths, setting that parameter to the appropriate value in a RandomForestClassifier. It's then fitting 3 times, once per fold defined in KFold() and passing several things to the call to custom_scorer()...

y.iloc[test_idx] which is our y_true
rfc.predict(x.iloc[test_idx]) which is our y_pred
train_idx which is the indices of the samples of our training data
test_idx which is the indices of the samples of our testing data

Hope that helps. Out of interest: why do you need to know which observations are left out?


How to handle unseen labels in test data?

It depends on the problem.There is no single answer to it.
Things you can do:

Delete these features and focus on features that appear in both train and test set.
Semi-supervised learning.
You can also turn these unusual cases into a new unique category.
Find nearest neighbour and use that value for that feature.


How can I draw bar graph in python on aggregated data?

import pandas as pd

df = pd.Series(data=[11, 8, 28, 1, 70, 13, 45],
               index=[22, 23, 27, 28, 29, 30, 31],
               name="age").rename_axis("age", axis=0)

x = df.index
y = df.values

plt.bar(x, y)


Train a model using multiple data sources

Add all data together, and make sure you have features representing all possible insights. In your case one feature with age/maturity (young, adults...). Lets say you fit a decision tree (or Random Forest, gradient boosting...) the model will decide whether to do a split or not on this feature if it contains meaningful information.
If you combine you should be able to have more data and the models will work better.
In case you have different columns in data1 than in data2, consider leaving them as NaN, and then treating them properly.


Multivariate, Multi-step LSTM time series forecast

I know this tutorial, it's a good start for RNNs but it contains a lot of passages and transformations that could have been kept shorter.

First, he defines a function called series_to_supervised() to process data to be fed into an RNN. Paragraph 3, line 37:

reframed = series_to_supervised(scaled, 1, 1)

This reframed dataframe contains all data, either y columns and all the X variables to make a prediction.

In the following code block it is turned into a numpy array at line 2:

values = reframed.values

Ok, so now all our information is store in values. Now it's time to separate it in train and test:

train = values[:n_train_hours, :]
test = values[n_train_hours:, :]

And again, each train and test is separated in x and y pieces:

# split into input and outputs
train_X, train_y = train[:, :-1], train[:, -1]
test_X, test_y = test[:, :-1], test[:, -1]

Line 7-8. This is where the dependent variable is separated from the rest. Knowing it was the last column, it was extracted with index -1 (i.e. the last element).

As I said, this tutorial is a good start to learn time series prediction with RNNs. However, I find that sometimes he tried to simplify the steps so much... that he ended up with some messy parts. All those objects: reframed, values, train, test, ... there was no need to make so many of them.

That apart, I'm a fan of the blog. It provided a lot of useful tips on RNNs.


Do batch GD and stochastic GD give the same results?

They are not the same; the batch gradient descent averages the gradients of samples 1 through M. This is seen in the first equation, denoted by the summation over M elements and divided by M total elements (hence the average).
Stochastic gradient descent, as illustrated in the second equation, updates via a single, randomly selected instance.
The confusion I believe stems from thinking about loops over 1 through M. The batch gradient descent summation (loop from 1 to M) is performed per iteration, whereas the stochastic gradient descent loop is performed over all epochs.
Thus the intermediate weight updates will be different, and these results compound. So even if both methods process every instance over the entire training course, the act of updating only after a single instance will inherently cause a different trajectory than if one were to average all the gradients per iteration.
SGD can be faster simply because it only processes a single instance per weight update. In fact, purely from a hardware efficiency perspective, SGD is suboptimal due to its intrinsic serialized nature over the course of training. Batch gradient descent can easily be parallelized and take advantage of GPUs by processing multiple instances simultaneously.
Because SGD is, well, stochastic, the perturbations this causes in the surrogate model development can help alleviate bias in the training set. This is why SGD can sometimes help avoid local minima, albeit using a rather crude optimization strategy.
Edit to answer the comment asking about the relationship between training set bias and local minima:
Picture an example where a model has 2 parameters that define the xy plane and the corresponding error is plotted as the z component. This defines a function mapping R² to R. This function is only an approximation of the real-world function because you are not testing on every possible training instance. A biased training set poorly approximates the real-world function. Thus its local minima are different from the real-world local minima. When using batch GD, you will likely target the nearest local minima, without really any change of target over the course of training. This is good if your training set is relatively unbiased, but otherwise not ideal as you would be over fitting to a poor approximation. SGD can improve generalization by forcing a redirect of target in each iteration. Every iteration, if you sample a different function out of the M training samples, then you will likely change target minima multiple times. If your training set is relatively biased, then this can help not overfit to a poor approximation. If your training was a perfect approximation, then SGD would be pointless. Some may argue, given a perfect approximation, that the random sampling helps to avoid getting trapped in the starting basin, but at that point it's just luck.


Need help with python code as part of a data analysis project

All of your plots are appearing on top of each other.  You need to invoke plt.subplot(xxx) before you create each plot.  For info on how the xxx command works, go to the MATLAB documentation.

You might end up with multiple figures - see this page for info about that.


$ R^2 $ shows what variation of your purpose variable is described by independent variables. So their synergetic effect could give you good better answer than their correlation. Better use $ R^2 adjusted $. Look at p-values of your variables and think about their real correlation. Are they important in real life. If they have any adequate relationship, so your regression is right.

Hope it will help.


Metrics/Methods for deciding duration of video retention for on-demand websites

Clustering can be an ideal choice here. From the question, seems the data will most probably be in continuous format. Essentially, clustering is a method of finding groups of similar objects. The similarity between the objects is determined by the type of distance measure being used.
With this background, you can start by finding the videos that are most liked or disliked by people. Measurements like mutual information gain can be helpful.


Reason for square images in deep learning

There is no requirement for specific pixel dimensions for convolutional neural networks to function normally. It is likely the values have been chosen for pragmatic reasons - such as a compromise between using image details vs number of parameters and training set size required.

In addition, if source data has a range of different aspect ratios, some portrait, some landscape, with the target object usually in the centre, then taking a square crop from the middle could be a reasonable compromise.

When you increase the input image size, you will also increase the amount of noise and variance that the network will need to deal with in order to process that input. That could mean more layers - both convolutional and pooling. It could also mean that you need more training examples, and of course each training example will be larger. Together, these increase the computation resources you need to complete training. However, if you can overcome this requirement, it is possible that you will end up with a more accurate model, for any task where the extra pixels could make a difference.

One possible rule of thumb for whether you would want higher resolution is if, for goal of your network, a human expert could make use of the extra resolution and perform better at the task. This might be the case in regression systems, where the network is deriving some numerical quantities from the image - e.g. for face recognition extracting biometrics such as distance between facial features. It might also be desirable for image-processing tasks such as automated masking - state of the art results for these tasks may still be lower resolution than the commercial images where we would like to apply them in practice.


In elbow curve how to find the point from where the curve starts to rise?

TL;DR

Use the two functions from below to get the index of the elbow:

elbow_index = find_elbow(data, get_data_radiant(data))

Edit: I put all of the code below into a python package called kneebow. Now, you can simply do it like this:

from kneebow.rotor import Rotor

rotor = Rotor()
rotor.fit_rotate(data)
elbow_index = rotor.get_elbow_index()

Long Answer

If this curve is representative for all of the curves (e.g. unimodal and continuous) then a quick and dirty method is to rotate it to some degree and simply take the minimum value.

The rotation can be done by multiplication with the rotation matrix

$$\left(
\begin{array}{cc}
\cos\theta&-\sin\theta\\
\sin\theta&\cos\theta
\end{array}
\right)$$

where $\theta$ is the desired angle in radians.

In python, you can do this with the following function:

def find_elbow(data, theta):

    # make rotation matrix
    co = np.cos(theta)
    si = np.sin(theta)
    rotation_matrix = np.array(((co, -si), (si, co)))

    # rotate data vector
    rotated_vector = data.dot(rotation_matrix)

    # return index of elbow
    return np.where(rotated_vector == rotated_vector.min())[0][0]

Note that theta is the angle in radians. You can caculate that by np.radians(angle).

Important: One thing to remember is that the x- and the y-axes may have different scales. So on your plot, it may look like a 45° rotation would be enough, while actually, it is not. Therefore, you can use the following function to calculate which radiant you should use. It takes the slope from the minimum to the maximum values in your data and converts it to radians:

def get_data_radiant(data):
  return np.arctan2(data[:, 1].max() - data[:, 1].min(),
                    data[:, 0].max() - data[:, 0].min())

If you want to get the angle, run np.rad2deg(get_data_radiant(data)).

Example

How to use

Let's test the approach on a sample data similar to yours:

# Let's define our sample data:
data = np.array([
    [1, 1],
    [2, 2],
    [3, 3],
    [4, 4],
    [5, 5],
    [6, 6],
    [7, 7],
    [8, 8],
    [9, 16],
    [10, 32],
    [11, 64],
    [12, 128],
    [13, 256],
    [14, 512]
])
# y is linear until (8,8) and increases exponentially afterwards

plt.scatter(data[:, 0], data[:, 1])

Plotting the data gives us the following figure:

Now, let's try to find the elbow by combining the functions from above:

elbow_index = find_elbow(data, get_data_radiant(data))

print(elbow_index)        # 10
print(data[elbow_index])  # array([11, 64])

In Detail

To sum it all up, we just calculated the slope from the min to the max value and then rotated the plot in such a way that the slope was zero. Subsequently, we took the min value of the data to get the elbow.

We can get the angle of rotation the following way:

angle = np.rad2deg(get_data_radiant(data))
print(angle)  # 88.543

The plot on the left has the slope included as an orange line. The scales of the axes make it look like it would have an angle of 45° while in reality, it has an angle of 88.5°! After vector rotation, the data looks like the plot on the right. From this data, we took the minimum value which was the 11th data point.

Drawbacks

Note that this method has a drawback: The more inequal the scales of your axes are, the more it will choose points in favor of the larger axis.

You can try to use the scikit-learn MinMaxScaler to scale your data before you use this method in order to reduce this effect. If you use the kneebow package, the data gets scaled by default.


You will want to implement this yourself.

Encoding the data as 5 dimensional vectors per pixel crates a substantial memory overhead. But if you can try that first, to see if the results are as good as you believe then to be...
Pay attention to scaling!


What does the embedding mean in the FaceNet?

Assume you have features wich lie in a $R^n$ space, e.g. your input is a picture with $28 \times 28$ pixels, then $n$ would be $28 \times 28 = 784$.

Now you can "embedd" your features into another $R^d$ space, where often $d < n$. This way you learn a rich representation of your input. When you compress your $784$ input-pixels to, lets say $64$ you compressed your input by more than a factor $10$ and can elimnate redundant/useless features.

This embedding learning is of course done in such a way, that you could fully restore your original $784$ pixels out of your $64$ compressed pixels.


In clustering, sequence number such as customer ID and dates such as purchase date should be dropped?

I think you should drop the customer ID since it's a unique identifier and won't give you any additional value.
Instead, I would be rather cautious with the date. The date can be split into year, month and day. Depending on the analysis you are doing, it might be interesting to have K-means running also with those features.
For instance, suppose that you want to check whether you have some blobs of data in a specific period of the year, you should use the month feature.


Crop all written letters from image to form a website

This can be done with OpenCV.

Code needs to :

Prep-process image to enhance contrast
Use OpenCV to identify 'bounding box' for each character
Save each character's image as a new file

Code example : http://cvisiondemy.com/extract-roi-from-image-with-python-and-opencv/


Model accuracy changes when target attribute change

There is not reason to believe that the accuracy wouldn't change as you change the target. After all, the features that is able to explain a target distribution could be very different from the features that is able to explain another target distribution.

It is similar to how human learn. We can't expect a human who is just trained to speak English to suddenly speak fluently in another language.


Is the way to combine weak learners in AdaBoost for regression arbitrary?

The ISL description is of gradient boosting (regression, with mse as the loss function), not of AdaBoost.  There, $\lambda$ is constant, not weights for each tree.  Since each tree is fitted to the residuals, we need to add the results to better approximate the true values, not average.
However, the title question is still an interesting one.  It does seem probably mostly arbitrary, but at least some testing has been done, see e.g. "Experiments with AdaBoost.RT, an Improved Boosting Scheme for Regression" by Shrestha and Solomatine.


Difference between Validation data and Testing data?

There are two uses for the validation set:

1) Knowing when to stop training

Some models are trained iteratively - like neural nets. Sooner or later the model might start to overfit the training data. That's why you repeatedly measure the model's score on the validation set (like after each epoch) and you stop training once the score on the validation set starts degrading again.

From Wikipedia on Overfitting:

"Training error is shown in blue, validation error in red, both as a function of the number of training cycles. If the validation error increases (positive slope) while the training error steadily decreases (negative slope) then a situation of overfitting may have occurred. The best predictive and fitted model would be where the validation error has its global minimum."

2) Parameter selection

Your model needs some hyper-parameters to be set, like the learning rate, what optimizer to use, number and type of layers / neurons, activation functions, or even different algorithms like neural net vs SVM... you'll have to fiddle with these parameters, trying to find the ones that work best.

To do that you train a model with each set of parameters and then evaluate each model using the validation set. Finally you select the model / the set of parameters that yielded the best score on the validation set.

In both of the above cases the model might have fit the data in the validation set, resulting in a biased (slightly too optimistic) score - which is why you evaluate the final model on the test-set before publishing its score.


Predicting Soccer: guessing which matches a model will predict correctly

This sounds like an interesting project. I recently worked on an almost case study. In order to get only 3 most accurate predictions, I think you may wanna sort those correctly predicted 5 matches by probability of event (win, draw or loose) and then select the first three matches.I hope your models are able to give you probabilities of events.
I hope this helps :-)


Creating similarity metric with Doc2Vec and additional features

You could think of your similarity measure as a search problem if you consider one record a query, and the "near" records as search results.

I've had some good results following this paper: https://arxiv.org/pdf/1602.01137.pdf

As I understand it, the document vectors used in the paper were only good for improving the relevance of search results for results that were already decent (the top N results).

That to me suggests you might try developing a similarity score that works with your other attributes first, and then do something like a weighted average, where the significance of the doc2vec score decays quickly based on the first metric.


Association Rule Learning for Home Electricity or Water Data?

In order to draw conclusions from this one building's data on other buildings on the campus, you would need a set of parameters shared between the building you know about and the buildings you want to predict for. For example, if the monitored building is of a male dormitory type, perhaps you can infer something for male dormitories; if the building is at location X, perhaps you can infer something for buildings at or close to location X. You'd need as many as possible of such additional columns to extrapolate knowledge to other buildings.

As for visualizing/predicting resource usage of that single building, I suggest you try data mining tool Orange, which I find phenomenal for prototyping machine learning workflows. It also has a time series add-on, which I can't vouch for, but it does seem to have some forecast capabilities.


Input for individual perceptron in input layer in MLP

With MLPs you can follow a set of simple rules:

The number of neurons in your input layer equals the number of features / data instances you have.
A single neuron takes only one input (one feature)
The output from one neuron can go to multiple neurons in the next layer which may or may not be fully connected.

May the force be with you ;)


Why does changing the cluster number change the plot in Kmeans?

plotcluster changes the projection based on the result that you give. See the documentation for more details.

Because of that you cannot compare the two plots.


Prediction model for marketing to prospective customers (using pandas)

First, I would ask the company if there is more information about the customer. You mentioned you have 10 original columns, which might not be enough to make a good prediction. Same goes for number of rows. Usually, more data, better the model, up to a certain limit.

Second, encode categorical features (demographical data in your case) is good thing to do. The increased number of columns dont have to bother you in your case.

For the task itself, yes, it is doable. Start easy, simply check importance of each feature (I would let PCA for later), pick few models and test them.

Also consider train simple decision tree. Your results can be easily visualized in way the business people understand. As oposed to black-box methods as K-Means.


Use cases for graph algorithms and graph data structures in finance and banking

There are many use cases of graph theory in Finance industry and it is a very broad question. As Emre said can be used for Fraud Detection, Risk Modelling, Economic Networks etc.

These below links can give you better understanding of different application, please go through for better understanding:

Applications of Graph Theory In Finance
 Graph Theory for Systemic Risk Models
Graph theory: connections in the market
Analysis of Equity Markets: A Graph Theory Approach
Portfolio Diversification
From Graph Theory to Models of Economic Networks. A Tutorial

Do let me know if you need any additional information.


Why does increasing the training set size not improve the results?

The model that you are generating is most probably under-fitting the model. Imagine, you are fitting a straight line to a non-linear data. Certainly even if you are increasing your sample size, the model is still not getting any better.

Study about learning curves and how they can be used to solve the under-fitting!


Jupyter Notebook not importing pandas module

I had to start the virtual environment, then install pandas module

source virtual_enviornment_folder/bin/activate
pip install pandas


How to cluster some text using TensorFlow

I do not see many benefits of using TensorFlow here. TensorFlow (and PyTorch) are great tools if you need to compute gradients of a loss function and use it in an optimizer. Clustering does not give the kind of training that would allow you to train an RNN or a Transformer that would give you a reasonable representation.

In your case, I would try:

represent the by average word embedding of content words
represent the text using BERT

And then do use a clustering algorithm of your choice. You can indeed implement k-means or hierarchical clustering in TensorFlow.


This is an open feature request (at time of writing):
https://github.com/dmlc/xgboost/issues/2175
https://github.com/dmlc/xgboost/issues/3439
There, a very wasteful but working solution is mentioned: predict using ntree_limit for each number of trees of interest.  I've put together a quick demonstration Colab notebook here.
It also has been asked several times over at SO, see e.g.
https://stackoverflow.com/questions/51681714/extract-trees-and-weights-from-trained-xgboost-model
https://stackoverflow.com/questions/37677496/how-to-get-access-of-individual-trees-of-a-xgboost-model-in-python-r
and their Related questions.
In the first link, another workaround is mentioned: by dumping to text/PMML, you should be able to reload each individual tree (or subsets thereof) and make the predictions.  It's not clear how to make this work though: XGB itself doesn't have an easy way to load a model except from its own binary format.  You might be able to do it by parsing the output (JSON seems most promising) into another library with tree models.


Feeding large dataset to neural network using tf

You can use generator to read the data if your model uses "Mini-Batch SGD"-like optimization method which just uses a small batch of samples each step. For example,

df_iterator = pandas.read_csv(your_data,chunksize=batch_size,iterator=True)
for small_batch in df_iterator:
#feed it to your model's input


I have recently started polylearn, a scikit-learn-contrib package for efficient implementation of factorization machines and polynomial networks, in Python with Cython. Polylearn currently coordinate descent solvers for 2nd and 3rd order factorization machines, and for arbitrary order polynomial networks, using the approach described in the papers cited on the polylearn website.

I'm currently working on an implementation of arbitrary higher order FMs using SG and CD style algorithms, using the approach from this recent paper, I should have it ready within a week.

Polylearn has not had a release yet, so I would love to hear any feedback on the API and functionality, in order to round things off before the 0.1 release!


How to extract particular key from the dict()?

You could iterate over all the keys and apply function1 to them and them append to the full_data_set. At the end you have all them concatenated.

However, Python lists are unordered, so you don't control the order which keys are chosen.

data = {
    'aaa': [1, 0, 6, 8, 0, 5, 9, 1, 1, 0],
    'abc': [1, 1, 1, 2, 4, 0, 0, 0, 9, 8],
    'cfg': [0, 0, 0, 4, 3, 1, 0, 0, 0, 1],
    'cghjj': [7, 8, 9, 2, 3, 0, 0, 0, 0, 0]
}

def function1(l: list):
    # do something
    return l

full_data_set = []

for key in data.keys():
    full_data_set += function1(data[key])

If you do want to control the order of the keys, you need to make the ordered list of the keys somehow.

key_order = ['aaa', 'abc', 'cfg', 'cghjj']

data = {
    'aaa': [1, 0, 6, 8, 0, 5, 9, 1, 1, 0],
    'abc': [1, 1, 1, 2, 4, 0, 0, 0, 9, 8],
    'cfg': [0, 0, 0, 4, 3, 1, 0, 0, 0, 1],
    'cghjj': [7, 8, 9, 2, 3, 0, 0, 0, 0, 0]
}

def function1(l: list):
    # do something
    return l

full_data_set = []

for key in key_order:
    full_data_set += function1(data[key])


Apart from having an average accuracy of the trained model, also you can use cross-validation to approximate the optimal value for the k parameter and the optimal metric.


Is time series multi-step ahead forecasting a sequence to sequence problem?

Seq2Seq architecture can definitely be used for time series problem. The only twist is that you will need a linear layer on top of your decoder to project the outputs to the required size (for example, 1 for univariate).

The stepwise forecast approach can be used for short sequences but because any biases are compounded using this approach, it is not good for longer sequences.

For example, if you have a sequence where the value is constant at each time step $x_{i+1} = x_i$, but you model learned to do $x_{i+1}=1.01x_i$ instead (which is highly likely with gradient descent algorithm). For $t=10$, the target value will be $1^{50} = 1$, however you model will predict $1.01^{50}=1.64$.

Thus a 1% single step error results in a 64% difference in 50 steps.


How do you do 1-vs-rest classifiers in XGBoost Library (Not Sklearn)?

Try to give sklearn wrapper of xgboost to sklearn OneVsRestClassifier.
https://xgboost.readthedocs.io/en/latest/python/python_api.html#module-xgboost.sklearn
If it doesn't work you can create four different label arrays in which samples belong to corresponding class labeled as 1s, and others as 0. Then training with each of the label arrays you can get results for each class.


Imagining a Linear Regression model with more than 3 dimensions

Humans live in a 3 dimensional world. Consequently it can be hard for us to visualise anything beyond 3 dimensions. For 4 dimensions, we can perhaps think about time as the next dimension, so we might think about how a 3 dimensional image changes over time. Even that is quite tricky, so for 5 dimensions and above it is basically impossible.
Rather than trying to visualise a model in terms of geometry, it may be better to understand how models work by using linear algebra (ie matrix algebra)


Do word embeddings help with out of vocab tokens?

By definition an out of vocabulary word (OOV) is a word which haven't been seen in the training data, so it's virtually impossible to know which other word it is similar to since this would have to be determined with some training data. In the example that you mention (synonym), what you could use is a resource such as WordNet which tells you which word it's similar to, then you can use the other word embedding.

Cases of misspelled words is a different story: you could pass your OOV words through string matching techniques, or maybe use a character-based NN which would recognize the word despite the misspelling.

Both cases could be part of a pre-processing stage.


BLEU_SCORE gives bad scores - what am I doing wrong?

You must be getting a warning with this output. Warning pretty much tells you the reason why the scores are 0. Because there ARE NO 2-gram , 3 -grams in you example which are overlapping.

Here is the detailed explanation, I couldn't explain it better-
https://github.com/nltk/nltk/issues/1838

EDIT-
Solution-

Although the warning tells the reason, here is how you can fix this-

  Notice the ref and pre,

from nltk.translate.bleu_score import corpus_bleu
ref =[[['electron', 'and', 'a', 'proton']]]
pre =[['electron', 'and', 'a', 'proton']]

print("Individual n-gram")
print("Individual 1-gram")
print('BLEU-1: %f' % corpus_bleu(ref, pre, weights=(1.0, 0, 0, 0)))
print("Individual 2-gram")
print('BLEU-2: %f' % corpus_bleu(ref, pre, weights=(0, 1.0, 0, 0)))
print("Individual 3-gram")
print('BLEU-3: %f' % corpus_bleu(ref, pre, weights=(0, 0, 1.0, 0)))
print("Individual 4-gram")
print('BLEU-4: %f' % corpus_bleu(ref, pre, weights=(0, 0, 0, 1.0)))

You can refer help of python-


How to determine Number of neuron in hidden layer for classification

There is not a clear a answer for this question. You should try and evaluated how the models performs testing diffferents configurations. But, basically, the next two premises are followed:

More neurons per layer --> more complex model, and probably you will
obtain better accuracy.
More hidden layers --> more complex model, and again, probably you
will obtain better accuracy.
WARNING Both approaches increase the chance of overfitting by increasing the complexity of model. You need to carefully evaluate them through statistically robust model selection procedures e.g. cross-validation

Be aware that incrementing the complexity of the model will increase the computational process and , as mentioned, be careful with overfitting.
I recommend try a small number of neurons first, and then, try to increase step by step seeing if some improvement is achieved in the model.


Data sets for evaluating text retrieval quality

TF IDF will give you the degree of measure of how relevant a document is to your query. owever, to evaluate your IR system you need to use metrics such as - Precision, Recall and F score.

Precision: Out of all the documents that your system retrieves, which ones are relevant? This measures how much noise there is in the output of your IR system.

Recall: Out of all the documents that are relevant, which ones did your system retrieve? This measures how much coverage does your IR system have?

It is possible to get 100% recall all the time by basically retrieving ALL documents from a collection for any query. However, the precision in this case will be very low.

It is possible to get a very high precision by hand modeling an IR system ti produce very accurate results. However, it would produce a very bad recall as there will not be coverage over all the documents.

So we need to measure F score- which is the harmonic mean between Precision and Recall
Check out Chapter 8 of the Stanford IR book.

If you are looking for datasets only here are a few that are relevant:

http://zitnik.si/mediawiki/index.php?title=Datasets
http://www.daviddlewis.com/resources/testcollections/
http://boston.lti.cs.cmu.edu/callan/Data/


Collinearity and Multicollinearity in the features?

One way to measure multicollinearity is the Variance Inflation Factor (VIF), which assesses how much the variance of an estimated regression coefficient increases if your predictors are correlated.

If no factors are correlated, the VIFs will all be 1.

If the VIF is greater than 1, the predictors may be moderately correlated.

A VIF between 5 and 10 indicates high correlation that may be problematic.

And if the VIF goes above 10, you can assume that the regression coefficients are poorly estimated due to multicollinearity

If multicollinearity is a problem in your model the solution may be relatively simple. Try one of these:

Remove highly correlated predictors from the model.  If you have two or more factors with a high VIF, remove one from the model. Because they supply redundant information, removing one of the correlated factors usually doesn't drastically reduce the R-squared.  Consider using stepwise regression, best subsets regression, or specialized knowledge of the data set to remove these variables. Select the model that has the highest R-squared value.

Use Partial Least Squares Regression (PLS) or Principal Components Analysis, regression methods that cut the number of predictors to a smaller set of uncorrelated components.


train_test_split is just a utility function around ShuffleSplit, which on its turn just randomly assigns each sample to either train or test, taking the desired probability into account.
You can do that however you'd like, and there's no real reason to use that specific function.

Its not too hard to come up with some code that does that for three values or N values, if you rather avoid calling train_test_split twice.


According to the paper A Comparison of CNN-based Face and Head Detectors for
Real-Time Video Surveillance Applications CNN based neural networks are significantly better than Viola-Jones in terms of precision but need more computational power to calculate the results. The mean average precision error for CNN based networks is 5 times less than for VJ for FDDB data. Here is the precision comparison table from the paper:

But VJ allows real time recognition with 60 frames per second. Here is the speed comparison from the same paper:


Kernels are considered valid if they are positive-definite, so we must have
$$
\sum_{i=1}^n \sum_{j=1}^n c_i c_j K(x_i, x_j) \ge 0
$$
for all $n \in \mathbb{N}$ and $c_1, \dots, c_n \in \mathbb{R}$ and (in your case) all $x^1, \dots, x^n \in \mathbb{R}^2.$

Letting $n=1$ and $c_1 = 1$ and $x^1 = (0, 1)$ shows that $K$ is not positive-definite, since
$$
K\big((0,1), (0,1)\big) = 0\times 0 - 1\times 1 = -1.
$$

The problem with your approach is that you define $\Phi$ on $\mathbb{C}$ but write $K$ as the dot product for real-valued vectors.
The feature map argument for positive definite vectors only works if the product is an inner product on $\Phi$'s codomain.
The dot product for real-valued vectors is not an inner product on $\mathbb{C}^2$ (because $i\cdot i = -1 < 0$).
You would have to use the generalized dot product $x \cdot y = \sum_i x_i \overline{y_i}$, or another inner product on $\mathbb{C}^2$.


The accepted answer is correct, but it may also be helpful to think of a batch from a classification standpoint.

Suppose you have a binary classification problem that you are trying to solve using a multilayer perceptron, with 1000 examples of each class.

When training the model, you don't want to have to wait until the model has seen all of the data before a weight update is performed. That's computationally inefficient. Instead, you take, for example, 100 random examples of each class and call it a 'batch'. You train the model on that batch, perform a weight update, and move to the next batch, until you have seen all of the examples in the training set. One pass through the training set in this manner is called an 'epoch'.


Unable to use readline() for function() input

Try

if (DistriName == 1)

(two == means "is equal“ in a logical way)


How to change parameters in LSTM for multivariate binary classification time series?

The sigmoid activation gives you a number between 0 and 1 (you can consider it as the probability of item i to be of class 1). To get hard predictions (0s and 1s) you just use a rule that says "I consider that the item is predicted positive if the probability is above a certain threshold". For example, for a threshold of 0.5, you just map every predicted value above 0.5 to 1 and the rest to 0.


How to understand ANOVA-F for feature selection in Python. Sklearn SelectKBest with f_classif

Intuition
We have two classes and we want to find a score for each feature saying "how well this feature discriminates between two classes". Now look at the figure bellow. There are two classes red and blue and two features on $x$ and $y$ axes.

$x$ feature is a better separator than $y$ because if we project data on $x$ axis we get two completely separated classes but if we project data onto $y$, two classes have overlap in the middle of axis (comment if we need more clarification).
What makes $x$ better than $y$? As you see in the figure above:

According to $x$, two classes are far from each other.

Math Translation: The distance between means of class distributions on $x$ is more than $y$.

According to $x$, the scatter of classes do not fall on each other but according to $y$ they do. It means that according to $x$, classes are more compact so more probable to not have an overlap with another class.

Math Translation: The variance of each single class according to $x$ is less than those of $y$.

Now we can easily say $\frac{distance\_between\_classes}{compactness\_of\_classes}$ is a good score! Higher this score is, better the feature discriminates between classes.
Now we know, according to this definition, what $good$ and $bad$ features mean. Let's find a math formulation to quantize it.
Mathematics (to do on the paper)
Let's formulate our two criteria:

Distance between means of class distributions is the numerator. Population is taken into account, I assume for statistical significance (needs a reference from a statistician!).

A concept similar to sample variances of classes is the denominator. Here instead of dividing sum of squares by $(sample\_population -1)$, we sum up all $(sample\_population -1)$s and divide the final value by them.

Now Back To Your Data
To calculate the above you calculate sum of between-class distances and sum of within-class variations for each feature according to different classes. I do it for only one feature. Let's choose Loan.
Class 1: [5000, 18000]
Class 2: [47500, 45600, 49500]

Mean of all points: (47500 + 45600 + 49500 + 5000 + 18000) / 5 = 33120
Mean 1: (5000 + 18000) / 2 = 11500
Mean 2: (47500 + 45600 + 49500) / 3 = 47533
Numerator: 2 x (11500 - 33120)^2 + 3 x (47533 - 33120)^2 = 1,558,052,507

For denominator we go with Sum of Squares Within class (it is simply the numerator in formulation of sample variance):
SSW 1: (5000 - 11500)^2 + (18000 - 11500)^2 = 84,500,000
SSW 2: (47500 - 47533)^2 + (45600 - 47533)^2 + (49500 - 47533)^2 = 7,606,667
Na = 2, Nb = 3 --> (Na - 1) + (Nb - 1) = 1 + 2 = 3
Denominator: (84,500,000 + 7,606,667)/3 = 30,702,222

Now the F-Score for feature Loan is:
F-Score: 1,558,052,507 / 30,702,222 = 50.74

as you see with your calculation in Python.
Note

I tried to explain in a simple way. For example the denominator of sample variance is called degree of freedom but I skipped those terms for simplicity.
Just understand the main idea. Further the means and smaller the within variances, better the feature is. You can formulate it yourself as well (however you will not have p-values anymore ;) )
Finding P-values and understanding what it means, is another story which I skipped.

Hope it helped.
Good Luck!


General equation for getting an idea of the scale of a machine learning project

Interesting but difficult question! It depends on the efficiency of your algorithm, both in terms of training and scoring/predicting, but to get a first idea I would go by the amount of data that we're talking about.

256 colors are 8 bits per pixel, times 300x300 pixels. Uncompressed, you have 720 kB per image. 10 images per location: 7.2 MB data per location. 100k locations: 720 GB of uncompressed input data, 15M locations: 108 TB. If you compress your data, meaning you store them a JPG or something, I don't know, but I would expect you need about a factor 10 less storage (it depends how easily the images can be compressed and how well JPG compression works).

Given unlimited time (and storage), any amount of data can be crunched on a laptop, although my laptop doesn't have 10 TB storage. But I would expect 720 GB worth of uncompressed data to be impractical on a single computer, unless you have an exceptionally efficient algorithm. That would be an algorithm that might be specifically designed so that it can be trained on all the data on a single computer, not a CNN with a bunch of layers that I expect you have in mind.

Using distributed computing in the cloud or on-premise has its cost too, in terms of infrastructure cost, complexity to debug, etc. But I would expect that with this kind of dataset, it's worth it.

That's something different from "the level of infrastructure that Google uses", which is millions of servers. I would start processing with, for example, 100 cores, and see how it goes. Don't forget to turn them off when you're done!


What is affine transformation in regards to Neural Networks?

It's a linear transformation, for example lines that were parallel before the transformation are still parallel. Scaling, rotation, reflection etcetera. With regards to neural networks it's usually just the input matrix multiplied by the weight matrix.


What are the pros and cons of zero padding in a convolution layer?

I summarized from some other web pages:

Pros: Deep network because the output dimension could be constant after convolution

Saving image border information

Cons:

Waste of computational resources


Outlier detection with non normal distribution

I would suggest a nearest neighbors approach. This technique is non-parametric, such that it does not assume your features follow any given distribution. The degree from which a novel instance can be classified as anomalous can set through some p-value estimation. These techniques are computationally expensive however due to your small dataset this may be well suited.

Check out:

Learning Minimum Volume Sets
http://www.stat.rice.edu/~cscott/pubs/minvol06jmlr.pdf

Anomaly Detection with Score functions based on Nearest Neighbor Graphs
https://arxiv.org/abs/0910.5461

New statistic in P-value estimation for anomaly detection
http://ieeexplore.ieee.org/document/6319713/

You can also use more rudimentary anomaly detection techniques such as a generalized likelihood ratio test. But, this is kind of old-school.


Predicting the likelihood that a prediction from a linear regression model is accurate

In case you are talking about providing certain interval to your predictions, what you might need is adding some confidence interval to your linear regression predictor, something which you can make via a resampling method like bootstrapping as a robust way to find predictions intervals. One key advantage is that it does not assume any kind of distribution, being a distribution-free method to find your predictions and, if needed, to your regression coefficients estimates.
The steps would be:

Draw n random samples (with replacement) from your dataset, where n is the bootstrap sample size
Fit a linear regression on the bootstrap sample from step 1 and predict a value
Take a single residual at random from the original regression fit, add it to the predicted value and save the result.
Repeat 1 to 3 steps several times (1k times for instance)
Find the desired percentiles of your interval (2.5th to 97.5th for instance)
Source of info in this book

On the other hand, if you mean providing a generic confidence metric value for your model, you should find for instance a MSE or MSA on a test set.


Dividing the weights obtained on an already standardized data set by the standard deviation of the features? (Ridge regression)

I found my answer myself.. I'm not sure if I should delete the question, so I'll just share:

Say for a given data point before normalising we have
a*x + b*z =  y Normalising makes this a * x/std(x) + b * z/std(z) = y This means that for standard deviations greater than 1 the features are shrunk and therefore the weights are blown up. In the code the y_hat values are obtained using the original data, which means the coefficients have to be shrunk to reflect the relationship between the output and the original feature values.


There's two common approaches here, depending on how much data you have.

If you have plenty of data, you'll probably be fine to:

set part of the training set aside, and call it your validation set;
start with a set of hyperparameters, train a model on your training set, evaluate performance on the validation set;
repeat step 2 with different hyperparameters;
pick the hyperparameters which give you the best score on the validation set;
train your model on the training set and the validation set;
Test your model ONCE on your test set.

Otherwise, you can do the following:

start with a set of hyperparameters, evaluate your model's performance on unseen data via cross-validation on the training set;
repeat step 2 with different hyperparameters;
pick the hyperparameters which give you the best score on the validation set;
train your model on the entire training set;
Test your model ONCE on your test set.


Resize and then normalize, that's the only pipeline that makes sense.

If you resize after normalization, depending on the resize algorithm, you may end up with values that are outside of the normalized range.


When you go backward for max-pooling you keep track of the position of the maximum:

$$ X = \begin{bmatrix}
1 && 2 \\
3 && 4
\end{bmatrix} \quad \rightarrow \quad dZ *\begin{bmatrix}
0 && 0 \\
0 && 1
\end{bmatrix}$$

This gives you the position of the input value that ultimately influenced the cost/output. The gradient will then be "propagated" back to this value.

For average pooling, all values influence the cost equally and therefore it should look like this:

$$ dZ = z \quad \rightarrow  \quad \begin{bmatrix}
z/n && z/n \\
z/n && z/n
\end{bmatrix}$$

where $n$ depends on the filter size, here $n=4$.

The gradient gets distributed evenly back to all input values.


Inference from text data without label or Target

Is it possible to cluster TEXT OF non English language using python libraries?
Sure! classic approaches based on Bag-of-Words are language independent. For modern approaches based on DNNs, mostly pre-trained models, you just need to find a model in your language or train one model from scratch (for this you need lots of text in that language). For example in case of using AWS infrastructure, check Object2Vec algorithm.
Can Clustering of text help?
Can help. For instance for an initial labeling you can cluster data into similar texts and labels each according to overal concept. More sophisticated solution (easily implemented in python) is topic modeling e.g. LDA algorithm.
More sophisticated solution is, again, pre-trained models like S-BERT.
In this direction, I also recommend having an analysis on keywords for algorithms like RAKE or YAKE.
Hope it helps!


Using the GA R package to optimize the weights of a MLP neural network

I finally got the results I was looking for.

Apparently a single hidden layer neural network cannot learn the cubic function. Instead of just evolving the weights, I included the topology of the neural network into the chromosome. My neural network can have maximum 5 hidden layers with maximum 5 neurons in each layer and the chromosome contains information for the structure of the network and the selection of the weights.

When I ran the GA with population of 200 and for 100 iterations I got a full topology of:

1(input) - 4-2-3-2-2 - 1(output), a mean squared error of 0.0051 and the following plot

I would be glad if anyone has any comments or questions.


How would one impute missing values for a Discrete variable?

It depends, if you have the distribution of that feature, you can take the marginal distribution over that feature which its interpretation is to use the expected value of that feature. If you don't have the distribution you can take the mean of the sample in hand of those samples which have value for that feature and add the mean for those which don't have. Another solution is to separate the data of each class and find the mean of the feature of those data samples having the value and putting the mean for each entry of the corresponding class which does not have value in that entry.


Predictive modeling based on RFM scoring indicators

A Google search leads to many relevant resources that answer your question:

USING RFM DATA TO OPTIMIZE DIRECT MARKETING CAMPAIGNS: A LINEAR PROGRAMMING APPROACH http://www.thefreelibrary.com/Using+RFM+data+to+optimize+direct+marketing+campaigns%3A+a+linear...-a0272246211
Data Mining using RFM Analysis http://cdn.intechweb.org/pdfs/13162.pdf
Libby on Recency, Frequency, Monetary Value (book) http://www.amazon.com/Recency-Frequency-Monetary-Century-Library/dp/1882222067

From a data science point of view, there is nothing very special or unique about this problem. You have three independent variables and one dependent variable. Regression, clustering, and classification methods can be applied.


[x[1,2,3,4], x2[1,2,2,3], x[2,3,2,1]]
The data above has 4 features.  We can gives those features labels with a header.  We'll just call them feature 1, feature 2, feature 3.  For the first entry, feature 1 has a value of 1 and feature 2 has a value of 2 and so on.

A sample, is a subset of data taken from your dataset.  x[1,2,3,4] is a single sample of the dataset.

Whatever you are trying to do with Scikit-learn wants to know how many features you have, my example has 4 features (or columns).


Optimizing decision threshold on model with oversampled/imbalanced data

I am not sure if in the last point, you meant the validation set instead of the testing set.

Here is my advice:
1- understand the impact of having data imbalance. Let start with understanding the difference between overall accuracy and average class accuracy. If you only care about overall accuracy, then data imbalance is not a problem, else you need  to handle the data imbalance problem.

2- the data distribution of training set can be changed by using oversampling. Undersampling, synthetic sampling, data augmentation... etc. BUT you should NOT change the data distribution of the validation and the testing sets.

3- use the training set for training,  the validation set for tuning  the hyper parameters , BUT do not touch the testing set

4- use the testing set for testing only

5- you can control the behavior the model by controlling the data distribution, you do not need to have fully balanced data, you can control the oversampling process in a way to control the behavior of the model without using a threshold.


How to train LSTM with daily timeseries?

Do you mean you mave multiple sensors? Ir one single sensor that record one measurement every day?

In short, you will need to simply match the dimensions of your data to the LSTM. To do that, you design your LSTM to consume tensors of the correct shape.

Let's make the following assumptions:

You have 2 sensors
Each sensor records one measurement per hour (so 24 per day)
You want to consider blocks of 5 consecutive hours to predict the 6th hour

Your LSTM shape would have the following input_shape shape:

(5, 2)

You can pass any number of these sample blocks as a batch, using a batch_size parameter.

So it summarises to this format:

(batch_size, num_time_steps, num_features)

This is a brief introduction that hopefully generalises the idea for you. I would suggest looking at more examples to start getting a feeling for the required dimensions and the terminology that people generally use.

Have a look at this answer, where I explained relevant points for a similar question.
If you have multiple sensors, you need a multivariate LSTM - check out this answer
Another Great Source that explains LSTM use in TimeSeries Jason Brownlee - LSTM TimeSeries Prediction in Keras


Covariates in machine learning classificatoin

As a classification task, you should add age and sex as features to your classifier in addition to your body measures. This will incorporate to be considered when learning or making a prediction.
So your first option would be reasonable.


Dimension of weight matrix in neural network

There seems to be an error in the screenshot. The weight, $W$ should be transposed, please correct me if I am wrong.

You are wrong.

Matrix multiplication works so that if you multiply two matrices together, $C = AB$, where $A$ is an $i \times j$ matrix and $B$ is a $j \times k$ matrix, then C will be a $i \times k$ matrix. Note that $A$'s column count must equal $B$'s row count ($j$).

In the neural network, $a^{[1]}$ is a $n^{[1]} \times 1$ matrix (column vector), and $z^{[2]}$ needs to be a $n^{[2]} \times 1$ matrix, to match number of neurons.

Therefore $W^{[2]}$ has to have dimensions $n^{[2]} \times n^{[1]}$ in order to generate an $n^{[2]} \times 1$ matrix from $W^{[2]}a^{[1]}$


It simply depends what you count as your object of interest: from your description the unit can be either document or annotation. Your method describes using the document as unit, it's fine as long as the tests you plan to do are compatible with this.
Another option is to use the annotation as unit: in this case you would pick 60% of the 44k annotations every time, so you would have a mix of annotations from multiple documents. Depending what you test exactly, this might be an issue, in particular I don't see how you would count False Negative cases in this way.
Since you have text documents of varying size (I assume), you could also consider different options: sentence, paragraph, block of N sentences, etc.


Why isn't dimension sampling used with gradient boosting machines (GBM)?

sklearn's GradientBoostingClassifier / GradientBoostingRegressor have a max_features parameter and XGBoost has colsample_bylevel and colsample_bytree parameters that control how many features are sampled for each tree / split.


One hot encoding alternatives for large categorical values

One option is to map rare values to 'other'.  This is commonly done in e.g. natural language processing - the intuition being that very rare labels don't carry much statistical power.

I have also seen people map 1-hot categorical values to lower-dimensional vectors, where each 1-hot vector is re-represented as a draw from a multivariate Gaussian.  See e.g. the paper Deep Knowledge Tracing, which says this approach is motivated by the idea of compressed sensing:

BARANIUK, R. Compressive sensing. IEEE signal processing magazine 24, 4 (2007).

Specifically, they map each vector of length N to a shorter vector of length log2(N).  I have not done this myself but I think it would be worth trying.


Nice question.

Short answer: the distinction between independent and dependent variables may not matter to your loss/error function, but it might matter for your modeling function.

Think about fitting a parabola with orthogonal regression.  Here, it's really important which variable is dependent and independent, because the function is not one-to-one.  So I'd say the distinction matters more for "modeling."

In a simple case, such as fitting a line, then any distinction is probably more philosophical than practical.


It seems that problem was anyhow with conda installation.
I tried pip installation for jupyter and it worked fine.

python -m pip install jupyter


Why should I use data augmentation as Keras layer

Data Augmentation is basically a technique to increase the diversity of your training set by applying random (but realistic) transformations such as image rotation. Training deep learning neural network models on more data can result in more skillful models, and the augmentation techniques can create variations of the images that can improve the ability of the fit models to generalize what they have learned to new images. So you can decrease the rate of overfitting or take some precautions for it.
The Keras deep learning neural network library provides the capability to fit models using image data augmentation via the ImageDataGenerator class. By using this class you are able to modify the dataset by using its a lot of functionality.