Efficient use of resources
It is a balancing game with the learning rate, and one reason you don't normally see people do this is that you want to utilise as much of the GPU as possible.
It is commonly preferred to start with the maximum batch size you can fit in memory, then increase the learning rate accordingly. This applies to "effective batch sizes" e.g. when you have 4 GPUs, each running with batch_size=10, then you might have a global learning rate of nb_gpu * initial_lr (used with sum or average of all 4 GPUs).
The final "best approach" is usually problem specific - small batch sizes might not work for GAN type models, and large batch sizes might be slow and sub-optimal for certain vision based tasks.
Friends don't let friends use large batch sizes
There is literature to support usage of small batch sizes at almost all times. Even though this idea was supported by Yann Lecun, there are differences of opinion.
Super convergence
There are also other tricks that you might consider, if you are interested in faster convergence, playing with learning rate cycling.


How do you choose the min and max for the min-max normalization on a histogram classifier?

First build the min-max normalization on the entire data set. Then apply different workloads to your train and test split(s) separately. If you believe your data set will change over time you could consider estimating the extreme min and max if you are still interested in this min-max normalization.


How to detect time for the future events in time series data?

Try boosting and random forests. You would need a long enough time series. A good reference (with examples in R) is
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2017). An Introduction to Statistical Learning: with Applications in R. Springer.
Usefulness of particular features can be assessed via variable importance plots.


Clustering is the wrong tool for this. And so is cosine similarity, bag of words model etc.

Obviously, standard scaling on house numbers and zip codes is a svery bad idea, don't you think?

There are special techniques developed for this kind of data with a reason. For example, it's unlikely that the customer entered house number 100 but lives in 5. That single difference is a very clear indication that it probably is not the same customer. On the other hand, "Park Rd." And "Parkrd." almost certainly is the same, but in cosine similarity that is three tokens difference, pretty big.

Clustering is an explorative technique, not one to achieve automatic matching. Choose other tools instead.


Apache Spark alternatives for local compute

You're right, Spark is intended to scale in a distributed computing environment, but it is absolutely performs well locally.

When running locally Spark will still 'distribute' processing across local Executors, you have options to control the number of CPU cores the Spark job can use, and how many cores each Executor can use. You can get a lot out of using Spark locally, and Spark makes it very easy to scale up to a cluster of machines (e.g. on AWS or Google Cloud)  when your single machine can't handle the tasks and it becomes necessary.

I'd also check out SMILE as an alternative to Spark MLlib in Scala.


Solution for in Time/Space Complexity challenge in Recommendation System?

Apache Spark is a great solution for such problems.
But, first let's be clear about the definition of real time processing. It's the type of processing that must guarantee response within specified time which on an interactive business site is actually very low. You can read about those kind of specifications in this answer.
Spark doesn't provide such luxury in predicting under 0.1 sec and I'm citing

Excerpt from Chapter 5 in my book Usability Engineering, from 1993:

0.1 second is about the limit for having the user feel that the system is reacting instantaneously, meaning that no special feedback is necessary except to display the result.

Having an interactive business site on which you'd want to display predictions doesn't mean that your predictions has to be in real time.
So the obvious is, actually, the following :
Q: Now that I have computed recommendations for my users, what should I do ?
A: Let's define a serving layer for our systems fast enough to query when recommendations are needed.
It can be anything fast enough to answer your calls e.g Elasticsearch, Solr, HBase, Redis. Whatever flavor suits you.
On other hands, well
Q: I don't want my system to be static, I need to recompute my predictions every T hours/days/etc
A: Spark can do a scheduled job perfectly here. (a simple cron would do)
Q: But when do I retrain my recommender system ?
A: I would say it actually depends on so many stuff a bit too broad to discuss here. You can read about the topic here if you wish.
Ok, so we defined now our batch layer.
Q: And what about data coming in real time, through Kafka, Rabbit, etc. ?
A: This is actually when it can get more complicated, because the method that you'll use to compute distances, approximations, new recommendations will depends on what type of recommender systems you are building and what technologies you are using.
Spark streaming can fit very well to apply "simple" computations on "window" based micro batches. This can be our speed layer.
To conclude, all of the above defines what is called a lambda architecture. And one of the best framework that follows this design is Oryx (personal opinion).
It's quite interesting, you ought taking a look at it.
I also believe that it's quite possible to have a RT set-up for a recommendation system without the speed layer.
I hope that this answers your question.


two autoencdoers learned by two similar vectors (each one with its own). Similarity of hidden layer vectors will be the same?

I'd say if you are training with a single vector, then no they wouldn't be the same.

It seems unintuitive at first but one has to realize it doesn't make sense to learn the features of a single vector, only the features of a set of vectors. The goal of an auto-encoder it to use its parameters to compress its dataset into a lower-dimension representation (A.K.A latent space) as efficiently as possible.

By efficiently, I mean that the most amount of useful information is encoded into the latent space. This means the vectors close in input space will also be close in the latent space.  The auto-encoder achieves this by learning to utilize groups of commonly occurring pixels (e.g. features). For images, an A.E. might learn to describe a picture in terms of these-

Now instead of describing an image as a with all its pixels (i.e. the original image) we can instead describe it with lines of blobs of colors, effectively reducing the amount of information needed. Lines and blobs are learned instead of random pixels because lines and lines and blobs can describe any picture, but a random group of pixels can not.

As you can see, it is necessary when we have multiple vectors that we learn features that can approximate all the vectors in the dataset. And hence when we have only one vector we can use any combination of features because any combination of features will be able to approximate the original vector.


Computer Vision: Handling dataset(3D data or scan) with different timesteps

I would use an LSTM-RNN to encode sequences of arbitrary length (length meaning the number of slices) into fixed-size output. It should be very easy to implement it in Keras :)

Then feed the sequences in your CNN.

Check sequence-to-sequence encoder-decoder scheme, like this one.

The same concept is applied with state-of-the-art algorithms that require fixed size inputs, like this one.


Using Machine Learning to Predict Musical Scales

From a very high level -- You can convert the song to a spectrogram, there are a large number of implementations to do this. From there you can analyze the sound waves. In the case of the key, for instance, the note A is equal to 440 hz. Look into FFT as well.  Hope this helps get you started. I know spotify trains neural networks on spectrograms of songs to find similar songs based on "sound".


Training LSTM for time series prediction with nan labels

I suggest implementing it this way :

Set the nan value to 0 or any other value
when compiling keras model use parameter sample_weight_mode='temporal'
You can use masking on top of this by supplying the weight as the mask (sequence of values 1 if not nan 0 otherwise).

The steps above should give you the desired result.


Giving you some links which have worked:

deploying-keras-deep-learning-models-with-java :  this uses DeepLearning4J, ND4J which supposedly you have tried already.

tensorflow-keras-java

And same question is asked here - converting-keras-in-python-to-java . It uses KerasModelImport.importKerasSequentialModelAndWeight

You should give HDF5 file operation java examples: hdf-java . It would be helpful if you want to change the weights yourself

Please comment if you have already tried all these already.


what is the difference between positional vector and attention vector used in transformer model?

So the question asks between the difference between an attention vector and a positional vector.
To answer this question, will give some context into how the transformer differs from a sequential model, such as RNNs and LSTMs.
In the case of RNNs and LSTMs, data is fed sequentially "one-by-one" into the model to predict the output (whether that is at each time step or after observing the whole sequence. This is irrelevant though in the context of the question.)
In a transformer model, the whole sequence is fed into the model, just like as you would with a conventional neural network. However, the problem is that, unlike with RNNs/LSTMs, there is no way for the transformer model to understand the ordering of the instances in the sequence as the whole sequence is fed into the model. Therefore, we need positional embeddings (positional vector, in your terminology) to add information to the individual instances which tells the model the ordering of the instances in the sequence.
Attention, in the context of transformers, works on the basis that it assigns higher coefficients to instances in the sequences which are most relevant to decoding the hidden representation from the encoder. Unlike a basic encoder-decoder model, with attention, we are able to flexibly assign which input instances in the sequences have the most "say" in predicting the next output instance in the output sequence.
I hope this clarifies some understanding. If not, there is a great article on transformers here: http://www.peterbloem.nl/blog/transformers


Why do good word embeddings come out of maximizing cosine similarity?

Training word embeddings does not rely on optimizing the cosine similarity of words. It usually relies on prediction problems. Take for instance the skipgram model: you are predicting the context of a word, given the word. Such models, project words in geometric spaces (for example the commonly used ~300 dimensions). In other words, a word is associated with 300-dimensional dense vectors. Due to the way that these vectors are learned, they capture the semantics of the words and hence similar words are close to the induced space.

Cosine similarity (or dot product) captures exactly this semantic closeness. Intuitively, we want similar words to be close, because they are similar and we hope that the space models some of the word properties and meaning.


Gini index as a labeling strategy for leaf nodes

Short answer: No

Long answer:

What do you mean by 'assign a class to a leaf node'?  The question itself is strange. Gini index is used as splitting criteria in the building process of decision tree and the classes in leaf nodes are the final result of a building process.

The building of a decision tree is an automatic process based on the principle of greedy maximization of some quality criteria/splitting rule (Gini uncertainty (Gini impurity/Gini index), information gain, Misclassification error): at each step, the algorithm chooses the variable that gives the greatest value upon splitting. Then the procedure is repeated recursively until the entropy is zero (or some small value to account for overfitting).

Maximizing the Gini uncertainty can be interpreted as the maximization of the number of pairs of objects of the same class that are in the same subtree.

I will leave an example of tree below.

Also, if you want, you can check out this wonderful notebook on the theme that gives a full understanding of decision trees and the theory behind them:

https://mlcourse.ai/articles/topic3-dt-knn/#2.-Decision-Tree


Does K - Means clustering on data reduced using PCA and the original data make any difference?

But can I assign the cluster labels from the pca reduced data to the original data ? would it be a right approach ? I guess not.

Yes, that is totally the right approach. Principal components are just the linear combinations of your original features that explain the most variance, so you can definitely use them for clustering. Moreover, since you only kept 2 of them you also removed a lot of noise from your data - since K-means is based on distances, the fact of having 70 features can be problematic, as you're equally weighting distances on useless features and distances on important ones.
Summing up - yes, assign the cluster based on the results of Kmeans you obtained on the Principal Components.
Unfortunately I cannot help you with the implementations, but remember then when mixing categorical and continuous variables you need to define an appropriate distance metric between your points. And any good implementation should allow you to do so for categorical only or continuous only data. Not sure about the mix.


What is the problem with: the condition has length > 1 and only the first element will be used?

The reason you receive a warning is that Age is a vector, but you are comparing it with a scalar. R automatically takes the first element of Age. You can use "ifelse" and convert the class if needed.

If you want to limit yourself to base R, you can use the following:

age <- function(Age){
    sapply(Age, function(x) if(x < 15) "under 15" else if (x <= 50) "15 to 50" else "over 50")
}


Using Tensorflow object detection API vs Keras

Keras provides you high level api or can say wrapper written on top of multiple backends.
These back ends have the core implementation of DNN.
List of Keras supported backends are:

Tensorflow
Theano
CNTK

**Source: Keras documentation for supported backends

  Keras hides a bit complexity of DNN implementation, but again restrict
  your freedom. In case if you write a code in Tensorflow, you have
  explicitly specifies and calculate optimizer, cost function and other
  things, but it provides you flexibility.

So for me writing in Keras just a convenience.

As far my knowledge is concern, so far we dont have any fix formula to identify number of layers sufficient for Object detection :).


How to evaluate the "importance" of a variable in a function

The way to start would be to understand intuitively what makes some variable important.
For example, the output value of the function might depend to a large extend on the value on that variable (so correlation would be quite high between output and that variable).
Another example would be to make that variable zero and measure the correlation between the original function and the new function (eg it would exhibit low correlation).
Still another way would be to compute the rate of change (derivative) of the function with respect to the rates of change (derivatives) of each variable and see how much a small perturbation of that variable affects the output of the function.
Similar other approaches are possible as well. Since you have the formula of the function wrt the variables one of the above approaches is fine.
There is no agreed upon universal metric for variable/feature importance but there are various approaches one can take depending on problem at hand.
Further reading:

Variable importance analysis: A comprehensive review

Assessing Variable Importance for Predictive Models of Arbitrary Type

Variable-importance Measures

Methods to quantify variable importance: implications for the analysis of noisy ecological data


Can you give a bit more information on the size of the data you're training on (and if it's really 6 parameters you're basing the predictions off of)?  If it's really 6 questions with binary answers (1, 0 as you suggest), then there are 2^6 (i.e. 64) unique answer combinations, and to determine a probability for them you'll want a multiple entries per combination.  Standard error scales like 1/sqrt(n) so for 10% accuracy you'll need roughly 6,400 inputs which given your description, sounds like more data than you may have.  You may want to invest time into automating data collection.

If on the other hand, you have a reasonably large data set and are hoping for some alternative models, both boosted decision trees and random forest models sound like good candidates for this problem.


Training a function that maps n-dim to n-dim

I would call a mapping between N dimensional input and N dimensional output a regression problem.

If you add more constraints about the relation between the input and output it might be called different names: linear filtering, nonlinear filtering, etc...

some examples on common techniques for that would be: neural networks, regression trees, regularised regressions...


In a discrete probability space, the expectation of a random variable (RV) is a sum over all possible values multiplied by their individual probabilities. Here, your RV is $q_\pi(s,a)$, with $s$ being fixed. Therefore you need to take the expecation with respect to the actions only:

$v_{\pi}(s) \stackrel{\text{1. task}}{=} \mathbb{E}_{a\sim \pi(s, a)}[q_{\pi}(s, a)] \stackrel{\text{2. task}}{=} \sum_\limits{a}\pi(s, a) q_{\pi}(s, a)$.

$\pi(s, a)$ represents the "individual probabilities".


How do we get the baseline of each fold in cross-validation?

I think you look for: sklearn.model_selection.cross_validate, see the docs here:

  Returns:
  scores : dict of float arrays of shape=(n_splits,)
  Array of scores of the estimator for each run of the cross validation.
  A dict of arrays containing the score/time arrays for each scorer is
  returned. The possible keys for this dict are:

  test_score The score array for test scores on each cv split.

  train_score The score array for train scores on each cv split.
  [...]

Minimal example:

from sklearn import datasets, linear_model
from sklearn.model_selection import cross_validate

# Data
diabetes = datasets.load_diabetes()
X = diabetes.data[:150]
y = diabetes.target[:150]

# Model
lasso = linear_model.Lasso()
cv_results = cross_validate(lasso, X, y, cv=10)

# Print keys
print(sorted(cv_results.keys()))
# Print test_scores
print(cv_results['test_score'])

Returns:

['fit_time', 'score_time', 'test_score', 'train_score']
[ 0.34557351  0.34848715  0.26654262 -0.01126674  0.24875619  0.08731544
  0.13386583  0.14000888  0.2873109   0.00960079]


Score for Ranked plots using scatter plot widget

Hm, what was the meaning of this score?

See. This is why it was removed.

(As I recall, it was the average probability assigned to the correct class by the k-nearest neighbour classifier on the projection -- or something similar. The number is useful for ranking projections, but doesn't have any meaningfyl interpretation or absolute scale, like >0.70 is good, >0.90 is excellent. Hence there's no point in showing it.)


What Are Stats Metrics To Analyze How Well One Column Predicts Another Column

Other possible evaluation measures are:

Mean Absolute Error (MAE): $\frac{\sum_{i=1}^n{\| y_i - x_i\|}}{n}$
R^2: also known as coefficient of determination, the formula is rather lengthy, but a detailed definition, as well as a motivation, can be found here
Root Mean Squared Logarithmic Error (RMSLE): $\sqrt[]{\frac{1}{n}\sum_{i=1}^n{(\log(y_i + 1) - \log(x_i + 1))^2}}$
Explained Variance Score: $1 - \frac{Var(y - x)}{Var \ y}$ more here and here.
Mean absolute percentage error (MAPE): see the wikipedia for the formula and major drawbacks of this measure.
Median Absolute Error (MAE2): $Median(| y_1 - x_1 | \dots | y_n - x_n |)$

These measures, as well as others, can be found here together with some Python code.


How to find and use the top features for XGBoost?

SKLearn is friendly on this. Simply with:

from sklearn.feature_selection import SelectFromModel
selection = SelectFromModel(gbm, threshold=0.03, prefit=True)
selected_dataset = selection.transform(X_test)

you will get a dataset with only the features of which the importance pass the threshold, as Numpy array. Point that the threshold is relative to the total importance, so it goes from 0 to 1.

If you want to visualize the importance, maybe to manually select the features you want, you can do like this:

xgb.plot_importance(booster=gbm ); plt.show()


There are many supervised machine learning algorithms for classfication.
Here is a list to get you started:

Tree-based

Decision trees
Random Forest
Boosted Trees

Naive Bayes (NB)
K-nearest neighbors algorithm (k-NN)
Support vector machines (SVM)
Neural Networks / Deep Learning (DL)


Neural Network unable to track training data

Ok I have found out atleast one thing why it's happening that way after asking my friends,

We can't use sigmoid as the last layer since it will always output something between 0 and 1 and hence that's why your predictions are not varying at all...
i.e we can't at all use sigmoid as the last layer until it's a classification of let's say images and all...


Suggestions for good data science certified online courses

There are many but some of the resources are awesome. One such resource is Udacity. This is probably the best resource I have encountered yet.


Suitable instance counting CNN for training on polygonal masks

Which CNN should I use for instance counting given that my dataset consists of labeled polygons?

CNNs for instance segmentation. To start with you can try Mask-RCNN. Here are all state of the art CNNs for instance segmentation. You will also find code for most of them.


The model is overfitting right from epoch 10, the validation loss is increasing while the training loss is decreasing.
Dealing with such a Model:

Data Preprocessing: Standardizing and Normalizing the data.
Model compelxity: Check if the model is too complex. Add dropout, reduce number of layers or number of neurons in each layer.
Learning Rate and Decay Rate: Reduce the learning rate, a good starting value is usually between 0.0005 to 0.001. Also consider a decay rate of 1e-6.

There are many other options as well to reduce overfitting, assuming you are using Keras, visit this link.


Logistic regression is a standard method of performing binary classification, which matches your task here. Categorical variables can be dealt with, depending on the model you choose.

You can see from the Scikit-Learn documentation on logistic regression, that your data only really needs to be of a certain shape: (num_samples, num_features). It might ignore the columns that are non-numerical, so you should convert e.g. strings to class IDs (e.g. integers) - see below.

Computing the correlation can make sense for categorical values, but to compute these, you need to provide numerical values; strings like "bbc.com" or "US" won't work.

You can map each of the values to a numerical value and make a new column with that data using pd.factorize like this:

df["Country_id"] = pd.factorize(df.Country)[0]   # taking the first return element: the ID values
df["Name_id"] = pd.factorize(df.Name)[0]

You don't need to do it really for the approved and political columns, because they hold boolean values, which are seen by Python as 0 and 1 for False and True, respectively.

Now you can do something like this to see a correlation plot:

import matplotlib.pyplot as plt    # plotting library: pip install matplotlib

# compute the correlation matrix
corr_mat = df[["Name_id", "Country_id", "approved", "political"]].corr()

# plot it
plt.matshow(corr_mat)
plt.show()


Difference between indicator column and categorical identity column in tensorflow

indicator_column encodes the input to a multi-hot representation, not one-hot encoding.
The example clarifies more:
name = indicator_column(categorical_column_with_vocabulary_list(
    'name', ['bob', 'george', 'wanda'])
columns = [name, ...]
features = tf.parse_example(..., features=make_parse_example_spec(columns))
dense_tensor = input_layer(features, columns)

dense_tensor == [[1, 0, 0]]  # If "name" bytes_list is ["bob"]
dense_tensor == [[1, 0, 1]]  # If "name" bytes_list is ["bob", "wanda"]
dense_tensor == [[2, 0, 0]]  # If "name" bytes_list is ["bob", "bob"]

The last two examples describe what is meant by multi-hot encoding. for example if the input be ["bob", "wanda"] the encoding will be [[1, 0, 1]].


How does class_weight work in Decision Tree?

It is used, for example, when classes are imbalanced, so different weights are assigned to different classes, instead of equal ones.
Another case is when some class is more significant than others, so loss wrt this class counts more.
The class_weight parameter (eg for decision tress) is used by giving different weight to different class samples (doc: https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html, src: https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/utils/class_weight.py#L11), which is then used to position the sample accordingly.
Note that class_weight can be used in different ways depending on algorithm and model used. What makes sense for each algorithm. Mathematicaly, usually, is a simple multiplication of the sample value in some loss function
Related: How does the class_weight parameter in scikit-learn work?.
For CARTs with Gini Criterion:
The original gini impurity is defined as:
$${\displaystyle \operatorname {I} _{G}(p)=\sum _{i=1}^{J}\left(p_{i}\sum _{k\neq i}p_{k}\right)=\sum _{i=1}^{J}p_{i}(1-p_{i})=1-\sum _{i=1}^{J}{p_{i}}^{2}}$$
If classes are assigned weights $w_i$ then the weighted gini impurity is computed as follows:
the weight of all the observations in a potential child node, $c$, is
$$t_c = \sum_i w_i * n_i$$
where $n_i$ is the number of observations of class $i$ in $c$, and $w_i$ is the weight assigned to class $i$.
The impurity of child node $c$ is then
$$i_c = 1 - \sum_i (\frac{w_i * n_i}{t_c})^2$$
where $n_i$ is again the number of observations of class $i$ in the node, $w_i$ is the weight assigned to the class and $t_c$ is as calculated previously. The impurity of the entire potential split is then
$$\sum_c \frac{t_c}{t_p} * i_c$$
where $t_c$ and $i_c$ are as calculated previously, and $t_p$ is the total weight of all observations in the parent node that is being split.
Reference: How is the Weighted Gini Criterion defined?


How to make smaller categories with factor character variables

While you are trying this in R, I have a solution in Pyhton. You could use a similar logic in R:

# Say this is your list:
countries = ['IN', 'DE', 'US', 'UK']

# Create a list of countries in different continents like this:
asia = ['IN', 'NP']
america = ['US', 'CD']
europe = ['DE', 'UK', 'FR']

# You can then map your list with these values using a definition:
for i in range(len(countries)):
    if countries[i] in asia:
        countries[i] = 'Asia'
    elif countries[i] in america:
        countries[i] = 'America'
    elif countries[i] in europe:
        countries[i] = 'Europe'
    else:
        countries[i] = 'Others'

# Now check your list
countries
['Asia', 'Europe', 'America', 'Europe']

Once this list is ready, you could create your boxplot.

Hope this helps.


How do I know that model performance improvement is significant?

Your last paragraph is actually the answer but there is no need to train your model several times. You actually need to "validate/evaluate" your model several times which of course can be run on smaller sets.

Have several validation set. Quantity is not the main major but make sure they cover the distribution of data specially all classes.
run both models on those sets and keep results in two lists/arrays
simple/practical solution: If the difference between their means is larger than their individual standard deviations, then you are good to go
a bit more confident: You have two sets of numbers (performance of two models). use a simple anova and monitor the p-value to see if their differences is significant.


What about a model stating that you have two vectors of size N (where N is the total number of players, maybe unknown to us) where each element belongs to a space of {B,S} sequences, maybe empty. If you then define a distance function between two arbitrary sequences (say, normalized Levenshtein distance for two non-empty ones and some fixed cost when one is missing), you can define cosine similarity between the vectors.

(Obviously, you just consider your sequence a compact representation of sparse vector in this case.)


Can LSTM be used for non time series data?

If the packets are sorted in time and you think there might be some correlation between packets that help identify TOR/nonTOR, then you may try an LSTM.
I suggest you add a new column with the delta time since the previous packet.


What to do when seed has a big impact on model performance?

It's just a simple idea that I often recommend to understand what happens in cases like this:
You could try to do an ablation study, where you train the model with varying sizes of training data (e.g. 10%, 20%,..., 100% of your full training data). Given that your problem is instability, it would be even better to do every size several times with different random subsets. Then you plot the performance as a function of the data size, for example with boxplots in order to visualize the variance. The evolution of the performance (and its variance) would show you whether there is enough training data given the complexity of the model: if yes, the curve increases and then starts stabilizing around its maximum. If not, the curve is still in a phase of increase when it reaches 100%, meaning that more data would be needed to allow the model to stabilize.
Of course the disadvantage is that you need to run the training/testing many times, so this might not be practical.


Is there standard parameters that characterize how fast neural network learns to achieve specific error rate?

Sometimes you can estimate that by smoothing out your val_loss curve and hoping that your training is repeatable and stable, but seems that word "unpredictable" was invented specifically to describe loss curves. :) Especially if you're using some sort of dynamic data generation or custom cost function.

Take a look at few species at http://lossfunctions.tumblr.com/


Assistance needed on what machine learning approach to use

Just a few observations from my side:

I think you should clarify your goal for yourself. Writing a program isn't a goal in itself. I'm missing the top level picture.
The second part: 'The goal is to have the program automatically tag similar sounds with the correct label by finding related sounds using machine learning.' is more like a goal, but where is the subjective part in it, and how does this answer questions in your thesis?
I'm not sure about the meaning of the output of Algorithm 1. Is this supervised or unsupervised classification? If it would be supervised, the 'supervision' is not subjective (until Algorithm 2). If is it unsupervised, the classes will find similarities, but the similarities are unlikely to correspond to the labels you intend to use.
However, if you take e.g. $k=9$ classes with k-means (you will have to optimize the number) the similarities in the classes will increase, and you could 'learn' the matching of labels and the classes. The $k$ classes should map to one of 3 labels.
Don't try to fit all these steps into one model. Create a few and connect them properly. Measure, interpret and report intermediate steps in your thesis.


LSTM training/prediction with no starting sequence

You might choose to demand predictions only after N steps of your sequence have elapsed. Then, predictions are trust-worthy.  You've got to give your LSTM something to begin from, some context so to speak.

Usually you sum the errors your network produced across all timesteps, but in such a case you ignore its outputs until the N'th timestep onwards.
As follows, any prediction it makes shouldn't contribute to the sum of your errors: TotalErrorOfTheSequence

Notice, in such a case, when backpropagating, your gradient won't flow into the network before N'th timestep.

You can also learn the initial state directly, by backpropagating with respect to the actual cell_t0. Traditionally you would initialize cell_t0 with zeros, but because you can compute a gradient towards it, you can pull its starting value closer and closer to what's usually needed.

It's possible to improve the best starting-state of LSTM with noise, which does actually improve the quality of predictions. It mitigates network overfitting on early states of your sequence where the error is usually the largest due to small context available, and where corrections are strong. "Forecasting with Recurrent Neural Networks: 12 Tricks". - read the 3rd trick

Here is my understanding, please correct it if it's wrong:

1) train your Cell state as a variable (as described above), to get a good value.

2) Construct a random variable, which is centered around your cell_t0.

3) Collect a buffer of several hundred G-values, then start sampling randomly and use each one as a maximum possible deviation from your learned 'cell_t0'. So you randomly deviate away from the learned 'cell_t0' by some sampled G-Value

Each of such G values is the value of the gradient towards the actual already learned cell_0

I think it will make sense to keep track of which such G values are the oldest, and slowly remove them from the buffer when they become obsolete

The noise is to be disabled during testing.

Once again, not sure if I understood that approach correctly

Also have a look at these posts:

https://datascience.stackexchange.com/a/33994/43077
https://stats.stackexchange.com/a/319854/187816


implementing conv2d in fourier domain using np.einsum --> ValueError: einstein sum subscripts string contains too many subscripts for operand 0

import tensorflow as tf
output  = tf.einsum('ijkl,jklo->ijko', fft_x, fft_kernel)

I used tf.einsum instead of np.einsum, and it worked.
Also as seen in above, I removed the loop and changed the equation in einsum.


Which classification algorithms are negatively affected by class imbalances?

[This answer is based on my limited knowledge, please don't hesitate to edit or propose improvements in the comments]

Actually I think it's a bit misleading to say that algorithms can be affected by class imbalance, because it's not exactly the algorithm which is affected it's the evaluation method (I mean "evaluation" in a broad sense including the loss function used by the algorithm during training). Some algorithms may be closely related to a particular loss function or internal optimization strategy, so by association such algorithms have the same weaknesses.

A simple way to see that class imbalance issues completely depend on the evaluation method is to compare micro-average performance with macro-average over the classes in a case where 99% of the instances belong to the same class:

micro-average gives the same weight to every instance, so a model which assigns the majority class will look as if it performs very well.
macro-average gives the same weight to every class, so assigning the majority class  won't work better than random.

So technically the problem of class imbalance could (should?) be seen as a design choice between maximizing the number of instances correctly classified (default evaluation) and any other alternative, for example giving equal weight to every class.  But of course it's not practical nor common to design a specific evaluation measure or loss function specific to every problem.


Seaborn Pairplot not showin colors of the labels

So, the question is asking about why the legend of the plot displayed here is not displaying any colours.

Without looking at the code you have used to generate the pair plot, I can see that, for example, the column name which inputted as the "hue" to the pair plot function should be used as the legend title. However, the title of this legend on the plot is not the same as the given column name. So, my first intuition leads to me to believe that you might have overwritten the legend attributes after this line sns.pairplot() line (maybe using plt.legend() after the pairplot line). Hence, maybe why the colours do not appear as well.


Error while trying glmnet() in R: "Error in storage.mode(xd) <- "double" : 'list' object cannot be coerced to type 'double'"

Glmnet requires a matrix as input for both, $X$ and $y$. So you need to define as.matrix() on all model inputs.
For further examples also see the Glmnet Vignette by Trevor Hastie and Junyang Qian.


How to display graph with MathJax in jupyter notebook?

I ended up creating a new magic, based on the %dot magic, called %dotMJ.  It works by looking for <text> objects in the SVG (corresponding to labels in GV), checking if they are LaTeX code (start/end with $), and if so converting them to a <div> and wrapping with a <foreignObject>.  This allows MathJax to "see" the LaTeX code and convert it appropriately for display.

Source is available at: https://gitlab.com/jello/ML-Notebooks/blob/master/dotMJ_magic.py


How to feed data to Machine Learning Model?

you can feed data as SVM(train_X,train_Y)
before feeding the data to model,do required proprocessing of data like having dummies or one-hot encoding for categorical variables and scaling numerical variables when mean of numerical column values are 10fold or more apart


How to plot using facet_wrap, over multiple pages as a .pdf files in r cran

Depending on the shape of your plots, it might make sense to create four separate plots. I suggest this just because it makes sense to show the axes and labels on each individual page, otherwise it could be that the reader would have to always flips back to e.g. page 1 to see what the scales are.

I would probably do something like a 6 (rows) by 5 (columns) on each page, assuming portrait page orientation. Then you need to just adjust your code to do something like this (as pseudo-code):

page1 <- ggplot(steps[0:30]), aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,)) + facet_wrap(~id_study,)

page2 <- ggplot(steps[30:60]), aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,)) + facet_wrap(~id_study,)

page3 <- ggplot(steps[60:90]), aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,)) + facet_wrap(~id_study,)

page4 <- ggplot(steps[90:114]), aes(x=factor(edu))) + geom_bar(aes(y = (..count..), group = id_study,)) + facet_wrap(~id_study,)

Then continue to export the images as you require, for example:

ggsave('./first_page_plots.pdf', page1, dpi=600)

check out more save options here.


Is there a case that random forest data (a bunch of trees) consumes MB or GB of memory?

In theory, it could consume any amount of memory.
You could use an infinite amount of trees.


Cross-Validation in Anomaly Detection with Labelled Data

After thinking thoroughly and having worked on the problem I mentioned, I will respond to my own question.
In supervised learning, we divide the data into three parts, namely train, dev and test sets. We use dev/validation test to see how our fitted model does in unseen data. What is unseen in this case, is the target variable, y_test. During training, y_train is seen by the model while y_test is kept unseen. In order to use every part of train and dev data, and eliminate the effect of chance, we do it K-Fold.
In case of unsupervised learning, the model does not see any y_train because only X_train is fitted. Since the model is not exposed to the labels, we can simply go with train and test set, rather than train, dev and test. In this scheme no K-Fold CV is required either because we can simply predict for all of the train + dev data at once.
Hence, it makes sense to use the labels of X_train, y_train, for validation during model development, and it is not required to use K-Fold CV.


This is not necessary related to xts. You are comparing to floating point values. As pointer out in this excellent resource these type of comparisons are dangerous. Best way to code it if you do not have Inf, -Inf or NaN is by doing,

(abs(rts.fut$Ret - rts.fut$Ret2) / pmin(abs(rts.fut$Ret) + abs(rts.fut$Ret2),
  .Machine$double.eps)) < .Machine$double.eps

Hope this helps.


I solved the problem following the advices in the comments of this discussion. I paste here my code:

dizionario = dict({'1': 0,
 '10': 1,
 '11': 2,
 '12': 3,
 '13': 4,
 '14': 5,
 '15': 6,
 '16': 7,
 '17': 8,
 '18': 9,
 '19': 10,
 '2': 11,
 '20': 12,
 '21': 13,
 '22': 14,
 '23': 15,
 '24': 16,
 '25': 17,
 '26': 18,
 '27': 19,
 '28': 20,
 '29': 21,
 '3': 22,
 '4': 23,
 '5': 24,
 '6': 25,
 '7': 26,
 '8': 27,
 '9': 28})

as you can see I created a dictionary to map classes label to index . I used the output of final_train_generator.class_indices to create this dictionary.
After predict_generator I created the csv of prediction with following code:

predicted_class_indices = np.argmax(predictions, axis = 1)
final_predictions = []
for element in predicted_class_indices:
  final_predictions.append(list(dizionario.keys())[list(dizionario.values()).index(element)])
df = pd.DataFrame()
df['class'] = final_predictions
df['imnames'] = imNames
df.to_csv('predictions_xception_all_data_bon.csv', sep=',')

I paste here also the code that I changed following the discussion link in the comments:

test_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)
test_generator = test_datagen.flow_from_directory(
    directory="./TEST/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=20,
    shuffle = False,
    class_mode = "categorical",
    )
test_generator.reset()
imNames = test_generator.filenames
predictions = model_xcpetion.predict_generator(test_generator, steps=len(test_generator), verbose = 1 )


illumination problem for face landmark detection

Maybe you might try out the brightness data augmentation; in Keras you have something like:

ImageDataGenerator(brightness_range=[0.2,1.0])

in case you want to generate different brightness levels between 20% and 100% based on your train images. For example (taken from Deep learning for computer vision book by Jason Brownlee):


The last number 40 has nothing to do with the sequence length, its a hyper-parameter setting that is basically the length of the vector 'representation' of each token in the sequence. In this case , its 40 length. If you set it to 40 and use input embeddings of input 300-dimension (common if Glove), then the 300-dimensional word gets mapped to a 40-dimensional word that goes through the LSTM permutations.
The idea is very similar to the number of 'kernel maps' in a CNN, if you're familiar with those. Its just a way to tell your network how many internal features you want your model to generate as your LSTM does its thing. More features means stronger representational power and better information flow, but at the possible cost of overfitting.


Using strong predictor in Model training?

I can think of two way inducing priors into supervised machine learning training process.
1) Using a Bayesian framework(PYMC for example) where you can define you random variables with whatever prior you like.
2) Engineering features that describe the needed relation. For example, in your instances I would create a new Boolean feature named 'is_strong_predictor_by_prior', and would assign true if x=s1 and y=d1 and false else wise. This new feature can hold multiple priors(not only the s1>>d1 relation).


How to represent an image as state in a Q-table

The Atari games are some of the best examples in which you would need a function approximator (e.g. Neural Network) in order to solve them. This is because the state space explodes and thus you would need something that will generalize quite well in unseen states.

I am not quite sure what you mean by

  I want to use the image as the state of my algorithm, but I came up with a doubt: Is the state all the possibilities for all the pixels of the image?

The state at a specific timestep represents the state of the game and will be the input to your algorithm. Why you would need all the possible combinations of the pixels in that image (most of these combinations won't make any sense and the agent will never encounter them. What you need though is your agent to encounter enough game states that will enable it to generalize well in unseen states.

If you were trying to solve Atari games with a look up table then you would have to store every single image, that your agent encounters at every single time step, in a table (more precisely a reference of that image which will correspond to a $Q$ value). In other words you are trying to store all the states that your agent might encounter in the game (impossible in Atari games - unless you do some very sloppy assumptions about the nature of the states).


I see three main ways of comparing columns: automated comparisons of column names (e.g. regex,  Levenshtein distance), comparing content (e.g. compare mean and standard deviation of the data for the column; if the mean value of a column is 10,000 then it probably isn't recording Front Axial Temperature), and manual comparison. You can combine these, for instance clustering on column names and content, then manually looking at the contents of each cluster. The smaller the number of different column names, the more you can rely on manual examination. You may also be able to get other metadata sources, such as looking for documentation for whatever process generated the files.


Do we need to use off-policy methods for policy shaping?

You can use Policy Gradients for this which is on-policy. One of the most straightforward ways to do that is to provide the teacher/expert's probabilities over actions at each time-step. Then you can use the usual policy gradient loss plus the cross entropy between expert's actions distribution and agent's action distribution:

$$\mathcal{L(\boldsymbol{\theta})} =(R_t-V_{\boldsymbol{\theta}}(\boldsymbol{s}_t)) \log \pi_{\boldsymbol{\theta}}(a_t|\boldsymbol{s}_t) + \lambda H(\pi_{expert},\pi_{\boldsymbol{\theta}})$$

You could get a look at this paper for basic ways to affect on-policy learning in PG methods.

On the why this works, I will give you a descriptive explanation: Cross entropy is related to KL divergence which is a "measure" of the distance between two distributions. With the extra term in the Loss function we are trying to minimize the distance between the expert's distribution and agent's distribution (i.e. making the agent's one more similar to the expert's one).


How to implement a Restricted Boltzmann Machine manually?

ML From Scratch has an implementation in Python


Can I consider this pattern of data as a linear and use parametric multiple linear regression?

I don't think "can" is the right question to ask; it's not going to give you a syntax error. The right question is "what could go wrong?". Any modeling technique will have assumptions that may be broken, and knowing how those assumptions impact the results will help you know what to look for (and how much to care when those assumptions are broken).

The best test of whether or not linearity is appropriate is whether the residuals are white or structured. For example, it looks like X9 might have a nonlinear relationship with Y. But that might be an artifact of the interaction between X9 and other variables, especially categorical variables. Fit your full model, then plot the residuals against X9 and see what it looks like.
Treating it as continuous won't cause serious problems, but you might want to think about what this implies. Is the relationship between 1 and 2 in the same direction and half the strength as the relationship between 2 and 4? If not, you might want to transform this to a scale where you do think the differences are linear.
Same as 2, except it's even more reasonable to see time as linear.
Standardization is not necessary for most linear regression techniques, as they contain their own standardization. The primary exception is techniques that use regularization, where the scale of the parameters is relevant.

It's also worth pointing out that multivariate linear relationships, while they can capture general trends well, are very poor at capturing logical trends. For example, looking at X3 and X4, it could very well be that there are rules like Y>X3 and Y>X4 in place, which is hinted at but not captured by linear regression.


How does the time taken for backpropagation change with the increase in neurons?

The general idea is that time complexity is related to the number of operations needed to perform all the steps of backpropagation. You can find an example workthrough here:
https://medium.com/swlh/computational-complexity-of-neural-networks-38c01e7e566a
For a 4 layer NN it seems the complexity is O(n⁵), where n is the number of nodes of each layer (also n is the same for every layer in that example).


Don't know if this is what you need but I know of the Ampligraph library:

Python library for Representation Learning on Knowledge Graphs


How do I set/get heap size for Spark (via Python notebook)

You can manage Spark memory limits programmatically (by the API).

As SparkContext is already available in your Notebook:

sc._conf.get('spark.driver.memory')

You can set as well, but you have to shutdown the existing SparkContext first:

conf = SparkConf().setAppName("App")
conf = (conf.setMaster('local[*]')
        .set('spark.executor.memory', '4G')
        .set('spark.driver.memory', '45G')
        .set('spark.driver.maxResultSize', '10G'))
sc = SparkContext(conf=conf)

If your workload is the same for all analysis, then editing spark-defaults.conf as cited above is the way to go.


CNNs - Hyperparameter tuning with different training sizes of the same data set

First suggestion: you should first find a CNN architecture that satisfies you, and then stick with it.

Second suggestion: be careful with cross validation. CNNs are extremely "heavy" models, they can easily take hundreds of thousands or millions of trainable parameters. Running cross validation on those could be computationally impossible.

In other words, keep the two problems separated.

I suggest you to take a look at common CNN architectures. There's plenty of them on Kaggle competitions (for MNIST, Fashion MNIST or any other dataset). You can choose one that works fine, and then use it to show how much the classification performance (test accuracy) of CNNs changes depending on the size of the data set.


What are some good models to complement XGBOOST in stacking?

It really depends on how other models behave on your particular dataset, but generally, you should check 2 things:

Predictions from ensembled models are not highly correlated and other models also perform well on your dataset.
It's better to use a model from another family. So if you are using XGboost, its better to add some model like SVM than a catboost (because you want a model with different approach in solving the data, rather then a model with similar approach).

I think this article may give you additional insights about the topic
https://machinelearningmastery.com/stacking-ensemble-machine-learning-with-python/


How to download dynamic files created during work on Google Colab?

Based on what I've seen and experienced, the best way is to store and retrieve your data from your drive account. Actually your question is a bit unclear but first I say, try to use the following command to see the current files in your directory, although I guess each 12 hours they all would be deleted automatically.

!ls

Anyway I recommend the following instructions:

Use the following code for having permission to access to your drive account:

!pip install -U -q PyDrive

import tensorflow as tf
import timeit

config = tf.ConfigProto()
config.gpu_options.allow_growth = True

from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

# Authenticate and create the PyDrive client.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

Use the following code to get the id of contents in your drive:

file_list = drive.ListFile({'q': "'root' in parents and trashed=false"}).GetList()
for file1 in file_list:
  print('title: %s, id: %s' % (file1['title'], file1['id']))

Put the id of the desired file, e.g. a typical text file, in the content of the following dictionary with id key:

downloaded = drive.CreateFile({'id': 'the id of typical text file'})
file = downloaded.GetContentString()
print('Downloaded content "{}"'.format(len(file)))

Till now you have copied the text file, then you have to write it in your Colab disk using the following code:

text_file = open("your desired name.txt", "w")
text_file.write(file)
text_file.close()

Create & upload a file.

uploaded = drive.CreateFile({'title': 'filename.csv'})
uploaded.SetContentFile('filename.csv')
uploaded.Upload()
print('Uploaded file with ID {}'.format(uploaded.get('id')))

Downloading from Colab without Uploading to drive

from google.colab import files
files.download('your typical h5 file or what ever.h5')

For more information about transferring different data formats there are more explanations in the notebook provided with Colab.


Purpose of test data in binary classification

One of the well known problems of machine learning is overfitting. The main reason for having a test data set is to

Obtain a realistic measure of how good your model is doing.
Prevent overfiting the validation set.

Now think about the second point, quite often it might occur that you tune up the hyper parameters so that your validation set provides a good measure, but this means you might be overfitting your validation set and when you deploy your shiny model in the real world, boom! poor results are returned.

As per the amount of data to reserve for testing, it depends on how big is your data set, normally the rule of 60% for training, 20% for validation and 20% for test has been used, but if your data set has 5 millions of points, you probably just need 1% for testing (as that will provide you with 5000 examples for testing anyway).

So, yes, your last question makes sense, you will probably be fine taking a small portion of the original data set (again, say 1%) in order to get good results.


Data mining: Clique based clustering to make comparison in social network analysis

There are over 300 social networks captured in the Index of Complex Networks.  Thats usually the best place to start.
While your quantitative method is not very clear, it sounds like you could start with networkx and use something like find_cliques.  There is also a community detection method built for all types of subgraphs, not just cliques Benson et al. (2018).


Correctly interpreting Cosine Angular Distance Similarity & Euclidean Distance Similarity

If you look at the definitions of the two distances, cosine distance is the normalized dot product of the two vectors and euclidian is the square root of the sum of the squared elements of the difference vector.

The cosine distance between M and J is smaller than between M and G because the normalization factor of M's vector still includes the numbers for which J didn't have any ratings. Even if you make J's vector more similar, like you did, the remaining numbers of M (2 and 5) get you the number you get.
The number for M and G is this high because they both have non-zeroes for all the books. Even though they seem quite different, the normalization factors in the cosine are more "neutralized" by the non-zeroes for corresponding entries in the dot product. Maths don't lie.

The books J didn't rate will be ignored if you make their numbers zero in the computation of the normalization factor for M. Maybe the fault in your thinking is that the books J didn't rate should be 0 while they shouldn't be any number.

Finally, for recommendation systems, I would like to refer to matrix factorization.


Forecasting sales of next year using sales of past years?

If you have daily data you could create a dummy time calendar, i.e. you create a dummy variable for each day of the week and include your company's promotions for each product, Christmas, Easter, public holidays.
Then use autoarima() or nnetar() (or combine them) to forecast the time horizon you want. This link is a good example.

Another possible way to do that is to use the method of following paper: "Hsiang-Fu Yu, Nikhil Rao, and Inderjit S. Dhillon. 2015. Temporal Regularized
Matrix Factorization. CoRR abs/1509.08333 (2015)" This paper can help you with with forecasting the new products too.

Or for forecasting new products you could find a comparable products from your historical data then use those comparables' sales data to forecast performance of the new ones.

If your data really looks like above I think you could use Hsiang-fu's paper. this is a good approach.

For autoregressive forecasting you could read from Rob Hyndman's blog.


The main problem in your code is that you are passing the whole data at once ,instead of that you should use a batch size of 500 or some other value .There is a parameter of batch size in model.fit()
as well. And you should also reduce the number of epochs.


How to choose the best hyper-parameter when it is directly influenced by the random_state?

I am assuming, you are taking about the random_state of the Model, not the GridSearch (as it doesn't have a random_state) RandomizedSearchCV do need one.

A random state defines a starting point for an underlying random process.

With an ideal model, the difference should be very small. Though it will be.
If it is high, it means there is some Outlier data point/pattern due to which certain value is favoured.

Ignore - if small
 If it is large for a specific value
- Check the data
 - Increase K for the CrossVal

From another Stackexchange Question

  random state values which performed well in the validation set do not correspond to those which would perform well in a new, unseen test set.

From MachineLearning Mastery

  We can increase k and build even more models, as long as the data within each fold remains representative of the problem

Links
Stochastic Process
Randomness in ML
Is-random-state-a-parameter-to-tune


Time series analysis model evaluation performance metrics integration in time series application

I am assuming you have a model running in production and it retrains periodically (example: every month) and it forecasts for the next X days (example: 30 days) and you are trying to evaluate the model using RMSE and MAPE (If this assumption is wrong, please clarify it in your question)

If I use past 1 year data for training and forecast for next 30 days but we will get actual data after 30days in real time. How could we integrate these RMSE and MAPE with time-series application?

you will have to back test your model, i.e. predict with your model on a day in the past from which you will have 30 forecast-ed values and 30 actual values. Then the aggregation happens in mean part of the Root Mean Square Error or Mean Average Percentage Error. Ideally, your model should not have seen the evaluation data during the training phase.

As I heard from some data scientist if I skip first training cycle and calculate these RMSE and MAPE in next retraining cycle for past training cycle where I have actual data and forecast data, Is this approach useful ?

Yes, this is one of the ways to evaluate the model.

Could the evaluation metrics calculation every time on retraining useful from production point of view?

It is a good practice (unless it costs too much) to monitor the model in production regularly, for this you will have to estimate the evaluation metric in the frequency that is most suitable for you.


How to perform a reggression on 3 functions using a Neural Network

Should my output layer have 1 or 3 neurons?

The easiest thing to do is to create 3 separate networks, one for each function you want to approximate.

While it may certainly be beneficial if you combine all 3 outputs into the same model via a multi-task framework, I'd suggest starting off with separate networks, which is more intuitive. Then, if you want to see if combining them under one network improves your performance, you could check out how to create models with multiple outputs in keras.

  Also, should I be using a relu or linear activation function for the output layer?

Linear, definitely.

  For my metrics, should I use 'accuracy' or 'r2'?

Accuracy is a metric for classification, not regression. $r^2$ can be used, but for certain problems it isn't very reliable. You could try $r^2$, as well as MSE, MAE, MSLE (MSE of the $log$s of the predicted/actual values), etc.


How do I build a permutation invariance neural network in keras?

Concatenate was fine.

By concatenating, you give to your network access to all information it needs to predict the output. The order does not matter.

By averaging, you crunch your data into something smaller : something that represent your data in a less informative way. That's why your loose accuracy : you are giving to your network less information to learn from.

What you need to do is get more data. Your initial approach is fine, you just need enough data for your network to learn that when the data represent a 90 degree-rotation, the output should be the same.


Principal Component Analysis and abnormal data

The first eigenvector of the PCA have the direction of the maximum variance of the data. Then the second follows with the direction of the next highest variance and so it continues....

The easiest way to think about this, by my experience, is with 2D data and transform it to 1D. Visualization can be found here: PCA Visualization

It does not maximize the separation between the regular and irregular data points. It does dimension reduction, by trying to explain as much of the variance as possible by creating a new basis, eigenvectors(directions).
However, it is not very robust to outliers. Imagine all your datapoints are basically on a line, now suddenly you receive an abnormal value. As this abnormal value will have an impact on the variance, the eigenvalue directions will change, and with that the values of the eigenvectors!

For anomaly detection:
You can use the PCA eigenvectors of "normal data" against thos of new data. Simply do distance calculations between the eigenvectors. If you have a big distance => Anomaly

There are methods such as robust PCA that better cope with noisy data and can find the outliers.

Hope this helps.


How to determine key frames in a video for video classification?

Most video formats should already mark such frames marked as "Key Frames" (Frames 1 in this example).

It is possible to get identifiers of such frames (or split video around such frame). For example, mkvmerge tool.

https://mkvtoolnix.download/doc/mkvmerge.html

http://nickyguides.digital-digest.com/keyframes.htm


Export dataset with predicted target - Python

If you are just looking for generating Confusion Matrix, then you can try using this command:

#for generating confusion matrix
sklearn.metrics.confusion_matrix(y_true, y_pred, labels=None, sample_weight=None)

If you are just looking for binding predicted and actual values, you can use this command

#For Arrays you can use this
numpy.concatenate()

#for dataframe you can use this
DataFrame.append(other, ignore_index=False, verify_integrity=False)[source]

for better understaning you can go through this Link-1,Link-2,Link-3


Instance segmentation performance depending on the number of classes

The process you have used it spot on. Your problem: Give more importance to one class than the rest will have to be hard-coded in. There will be no pre-trained networks for that class especially.
One thing to note could be that assuming you have to segment roads in a road scene problem. You can create a function which can bias the weights towards the road specifically. This is just a thought, I haven't implemented it myself!


This question has no general answer; it depends on the specifics of both the statistical method in question and the implementation of that method in whatever language you used.

For example if you used a logistic GLM, then a missing value is an error since data records must be complete. Presumably you would have included a default value, and you will have to manually apply that if you want those records to be scored.

Alternatively, if you were using gradient boosted trees (gbm), then they can cope with missing values, but do so in an implementation-defined way. The answers would be different in the gbm and xgboost R packages, for example.


Time Series Machine Learning Feature Selection Problem

I will give you some other idea of it, but without mathematical.

Like horaceT's mentioned, it's all about feature construction and feature selection, then you have many machine learning method which can do the regression.

So, just look it as regression model, not just the time series model. Then you will get the following model:

y(Dependent variable 'Spending') = X1(moving average term)...X2(lastquarter moving average term) + X3(country id) + X4(Fourier term) ... + Error term

The moving average can catch the dependent variable's trend, Fourier term can catch the dependent variable's cycle. Maybe you can construct other feature.

Finally, you can selection the features using tree-based model, such as xgboost.


Re-estimating parameters with additional data is called online machine learning.

Scikit-learn is not designed for online learning. Scikit-multiflow is designed for online learning.


Large categorical dataset for regression

I would like to recommend to check the following open data repositories and meta-repositories (they are not focused on categorical data, but I'm sure that many data sets, listed there, contain such data):

http://www.kdnuggets.com/datasets
http://www.data.gov
http://www.datasciencecentral.com/profiles/blogs/big-data-sets-available-for-free

Also check built-in data sets in the open source software Parallel Sets, which is focused on the categorical data visualization: https://eagereyes.org/parallel-sets.


How do i create a classifier on sensor data?

By looking at your datasets, it resembles a multivariate time series problem , not sure why you are opting for Random Forest classifier ?

I would suggest you to start implementing simple statistical
algorithm first  here is the link

Then explore on complex Deep Learning algorithm based on the data
set size , please find the referenced link

Thanks,
Durga


Multivariate Polynomial Feature generation

General Formula is as follow:
 \begin{equation} N(n,d)=C(n+d,d) \end{equation} where n is the number of the features, $d$ is the degree of the polynomial, $C$ is binomial coefficient(combination).

Example with vector (2,3) to 3rd degree :

x = np.array[[2,3]]
pf = PolynomialFeatures(degree=3, include_bias=True)
pf.fit_transform(x)

returns :

array([[ 1.,  2.,  3.,  4.,  6.,  9.,  8., 12., 18., 27.]])

wich is : [1, $\:$ $x_1$, $\:$ $x_2$,
 $\:$ $x_1^2$,$\:$ $x_1x_2$, $\:$ $x_2^2$,
 $\:$ $x_1^3$, $\:$ $x_2x_1^2$, $\:$ $x_2^2 x_1$, $\:$ $x_2^3$]

You can see this as expanding this equation and throwing out the coefficients :

\begin{equation}1 + (x_1 + x_2) +  (x_1 + x_2)^2 +  (x_1 + x_2)^3 \end{equation}


What does Conv1d do in a sentiment analysis?

Just think of it as another feature transformation that grasp a set of sequential data group into a convoluted, compressed data. The text is a sequence of words(tokens), and each focus word would have different relationship to another words back and front of it. Conv1D help us to get nice and neat set of features that represent such relationship.

https://realpython.com/python-keras-text-classification/


Looking for help calculating a probability formula

Ok, I am not sure if I understood it correctly. But if it is only the right-side figure equation it would be:

=1/(1+EXP(3.1058))

(I have tested on LibreOffice only, but it should work on excel as well)

Now, if you want to do something more complex using the prior knowledge of event A, it should be more clear what is exactly this event and how it is related to the equation.

(I am relly not sure it this is what you want, so if it is not, I apologize)


I have 50 videos. I ask a customer 10 questions. Based on their answers, I send them a set of videos. How do I do it?

Frame this as a classification problem and learn a decision tree to map question responses to video selections.

EDIT: Fleshing this out a bit more:

Collect appropriate data. Get members of your target population to complete the survey and also indicate which videos they think would be appropriate to them, or alternatively have subject matter experts associate surveys with appropriate videos. However you do it, you need to acquire a dataset pairing surveys with "ground truth" video suggestions.
For each video, train a classifier with the survey responses as input to predict the binary target variable of whether or not that video was suggested for that survey.
To construct a suggested set for a new survey, use all your classifiers to score it. Any classifier that outputs a 1 is a video you should send in response to that survey.


How to work with large amount of data overcoming RAM issues in python

Dask is designed to manage these types of workloads. It provides an interface like NumPy, Pandas, or Python iterators for larger-than-memory operations. An example of using Dask with TensorFlow can be found here.


Time Series Data Multi-Class Classification

Your time intervals seem irregular. For most examples of time series, we have observations at fixed intervals, such as GDP every quarter. You can instead just create a regular machine learning model. If you think there is some sort of time component where you would not want January data to make predictions for November data, then you can manually create your cross validation partitions instead of creating them randomly.

However, you are predicting 1, 2, or 3 based on low, medium, or high DaysInDuration. A classifier would not know that low and high are farther apart than low and medium. You can instead create a regression model that predicts DaysInDuration as your target feature. Once you have your prediction (e.g. 31), you can act accordingly whether the predictions is low, medium, or high.

Finally, you wouldn't be able to use DaysInDuration as a feature to predict the Classification_Label because it would give you the perfect answer every time. That would be target leakage.


ValueError: shapes (1,10) and (2,) not aligned: 10 (dim 1) != 2 (dim 0)

X_ne1 = X_test[:,3]
y_pred2 = regressor_OLS.predict(X_ne1)


No. Loss measures the error between your predicted values and true values in a given train set. Whereas variance mesures how your model performs (i.e. how the loss changes) on different training sets. Also, variance is an atribute of a given model whereas loss depends on your dataset.

In other words, we can also say that loss on different sets is used to measure variance of a given model.


Training a model that has both 2D and 1D features using a CNN

I do not see how each datapoint would have a different dimension (you have simply 2 extra scalars, always). Could you clarify that?

Two solutions come to my head.

The first one would be to incorporate the scalars into the original picture - augment the picture with it. So that you would have, for example, an extra row for each scalar and end up with a picture of size 66*64. How to do it reasonably is another question - I do not have experience with any problem exactly like this, but using simply one pixel for the value seems pretty marginal. Maybe having the whole row of pixels be the same, representing the value... Or maybe add another dimension and filling all the points in that dimension with the representation of the scalar.

Another solution would be to create an extra branch in the model for the scalar features (with their own classifier) and then merge them with another classifier. I personally have never done it before, but this article https://www.pyimagesearch.com/2019/02/04/keras-multiple-inputs-and-mixed-data/ seems to explain it quite well. At this site, https://stackoverflow.com/questions/47818968/adding-an-additional-value-to-a-convolutional-neural-network-input/47819063 you can also find another brief example of the code (and the guy is basically proposing both ways that I mentioned above :) )

I hope this gives you at least an idea. You will need to play with it a bit to choose the best alternative.


How is that possible that a reward function depends both on the next state and an action from current state?

The equations you have given are consistent with each other. The outer sum over $a$ provides the value of $a$. The "link" between $s, a$ and $s'$ that gives the correct associations is the transition function $\mathcal{P}^a_{ss'}$ which will be zero for any state/action/next-state triplet that does not make sense for the MDP. By "not make sense", I mean not allowed by the dynamics of the problem - such as an action that would successfully move too many spaces or through a wall for a maze problem.

It is normal for the reward to not actually depend on $a$, but only on the successor state. This might happen in e.g. a dynamic or adversarial environment where the desired goal is to achieve a specific end state. However, it is also possible for the reward to depend on action taken in a given state, such as activating some object in a particular location. It depends on how you have set up the MDP. Another example of where $a$ might be important is if there is a cost associated with each action that varies, for example an energy requirement, so that the total reward might be separate contribution from $s'$ (for reaching a goal) and $a$ (for spending energy to reach any next state).

Using an expected reward $\mathcal{R}_{ss'}^a$ that depends on all three terms $s, a, s'$ gives you a generic version of the Bellman equation that can be applied to the widest range of MDP designs. It doesn't matter if, in a specific case, the only thing you care about is $s'$ which is what the example reward function shows. Once you decide that the expected reward is dependent on $s'$, then the Bellman equation has to have that expected reward term inside the inner sum (the only place where $s'$ is defined), and there is no additional structure in the equation needed to have this term dependent on $s$ and $a$ as well.

You could legitimately use a variant $\mathcal{R}_{s'}$ in a Bellman equation that is otherwise identical to the equation you give in the question, to describe the value function for a MDP where reward only depends on the state transitioned to. However, you will not find that in the literature because it is less generic.

One common variant that you will see is $\mathcal{R}_{s}^{a}$ i.e. the expected reward for taking action $a$ in state $s$. You might use this for cost-based systems, where the goal is to complete a task efficiently and taking actions expends some resource such as time. Using your notation, the Bellman equation for that design looks like this:

\begin{align}
V^\pi(s) &= \sum_{a \in \mathcal{A}(s)} \pi(s, a)( \mathcal{R}_{s}^a + \gamma \sum_{s' \in \mathcal{S^+}} \mathcal{P}^a_{ss'} V^\pi(s'))
\end{align}


Re: size of data

The short answer

Scala works for both small and large data, but its creation and development is motivated by needing something scalable.  Scala is an acronym for “Scalable Language”.

The long answer

Scala is a functional programming language that runs on the jvm.  The 'functional' part of this is a fundamental difference in the language that makes you think differently about programming.  If you like that way of thinking, it lets you quickly work with small data.  Whether you like it or not, functional languages are fundamentally easier to massively scale.  The jvm piece is also important because the jvm is basically everywhere and, thus, Scala code can run basically everywhere.  (Note there are plenty of other languages written on the jvm and plenty of other functional programming languages, and languages beyond Scala do appear in both lists.)

This talk give a good overview of the motivation behind Scala.

Re: other tools that have good Scala support:

As you mentioned, Spark (distributable batch processing better at iteratative algorithms than its counterpart) is a big one.  With Spark comes its libraries Mllib for machine learning and GraphX for graphs.  As mentioned by  Erik Allik and Tris Nefzger, Akka and Factorie exist.  There is also Play.

Generally,  I can't tell if there is a specific use case you're digging for (if so, make that a part of your question), or just want a survey of big data tools and happen to know Scala a bit and want to start there.


Just by eyeballing the data i would think the number of visitors for store A on jan 4th is around 110. Basically it is always approximately half of Store B on any given day.
It seems like the number of visitors across stores are correlated, so you could potentially use simple linear regression to get a reasonable estimate for any given day.


This is supervised learning problem. Type is a predictor. #serviced classifier is target variable. Model is trained on samples set you already have. Best guess is that any model will not have substantual predictive ability. Type is not enough.

Try including more factors (predictors) into the model. Like years_being_in_usage, equipment_model, have_been_in_service_before and so on. The more you get, the better model you can train.


Suggest text classifier training datasets

Some standard datasets for text classification are the 20-News group, Reuters (with 8 and 52 classes) and WebKb. You can find all of them here.


Can we change the structure of the feature-extraction layers of deep networks architectures?

You can change whatever you like!

The benefits will depend on your data and what exactly you are comparing to. As you didn't say exactly what sequence you have as your starting point, I can't compare anything exactly, but provide a simple outline.

Generally, you follow the following recipe:

Input layer: perform any reshaping or normalisation of your data
Multiplication layer: either fully-connected layer, a convolutional layer or something else
Non-linearity: relu, tanh - gives the network its power to fit non-linear functions
regularisation layer: batchnorm, dropout - help models converge and prevent overfitting

This scheme is then repeated as many times as you feel is necessary.

Doing something like applying several ReLU layers, one after another, definitely doesn't make sense, as it would actually change anything! A ReLU simply filters out negative values, setting them to zero. Applying it a second time would therefore have no effect.


On which algorithm for boosting is the method xgbLinear of the xgboost/caret- package based on?

I am not sure if this is helpful or not but here is a link to an academic paper detailing both the algorithm behind XGBoost and some use cases.

http://dmlc.cs.washington.edu/data/pdf/XGBoostArxiv.pdf


I'm the fellow Stackexchange user, hi! I wrote the function that iterates the one-hot encoding on all your categorical_columns:

def serial_OHE(df, categorical_columns):

    # iterate on each categorical column
    for col in categorical_columns:

        # take one-hot encoding
        OHE_sdf = pd.get_dummies(df[col])

        # drop the old categorical column from original df
        df.drop(col, axis=1, inplace=True)

        # attach one-hot encoded columns to original dataframe
        df = pd.concat([df, OHE_sdf], axis = 1, ignore_index = True)

    return df

So you can call it like this:

df = serial_OHE(df, categorical_columns)

Let me know it there are any problems.


How would knowing spammers email address improve spam detection algorithms?

Please accept my apology as it is not really a data science answer.

A lot of spammers are using spoofed emails address as sender, so that field alone might not be very helpful as a dimension. The number of possible values also could make it impractical to be used (i.e., 1 hot-encoding) as a feature unless you preprocess it somehow. Sender domain, sender server reputation scores, etc. would probably be more helpful than the email address by itself.

How spammers spoof your email


Deriving Confidences from Distribution of Class Probabilities for a Prediction

If I have Classes [C1, C2, C3, C4, C5] and my Probabilities are {C1:
  50, C2: 12, C3: 13, C4: 12, C5:13} my confidence in predicting C1
  should be higher than if I had Probabilities {C1: 50, C2: 45, C3: 2,
  C4: 1, C5: 2}.

Assuming that those probabilities are accurate, this isn't true. In your second case you can be a lot more confident that the ground truth is one of C1 or C2, but in terms of absolute confidence about C1 the probability is the same across both examples. To illustrate this with a more clear example, if you had a 100 sided die that had 50 sides labeled with "C1" then the labels on the the other 50 sides are irrelevant to the likelihood that you would roll a "C1".

Now with that said, your probabilities from your model are most certainly not perfect so there may be a way to use the intra-class correlations to improve them. Can you provide some more details about your specific problem and modeling workflow that you have used to get your probabilities?


Can xgboost (or any other algorithm) give bad results with some bad features?

Yes it can for sure, some algorithms are more robust to this than others but doing proper feature selection is adviced. This is due to the curse of dimensionality. It's not only the algorithm but also has a lot to do with the amount of data points you have compared to the number of features. If you have 10,000,000 data points 150 features is not an issue, if you have 400 data points 150 features is way too much. How can you learn something when there is so little to learn from? Overfitting is a real issue for almost any algorithm, and noisy features make it more easy to learn things that are not actually there.


"Change the features of a CNN into a grid to fed into RNN Encoder?" What is meant by that?

It seems they use a shared RNN which process each row sequentially on the sequence of concatenated channels of individual pixels. From the paper

Implementation with channels last
Let the output of the ConvNet be of size (batch_size, height, width, channels). The RNN expects an input of size (batch_size, sequence_length, input_size)`. So you have to reshape it with the following correspondence.
batch_size*height -> batch_size
channels -> input_size
width -> sequence_length

And process each row (along height dimension) with the same RNN and concatenate the result.
To do that, we simply reshape to merge the batch and height axis into one dimension so that our RNN will process columns independantly.
rnn_input = keras.layers.Reshape((batch_size*height, width, channels))(convnet_output)
rnn_output = keras.layers.RNN(hidden_dim, return_sequences=True)(rnn_input)

rnn_output will have shape (batch_size*height, width, hidden_dim). You can then combine this tensor into a context vector using a dense layer with tanh activation, as it is written in the paper.
The paper also use trainable initial state for the RNN, you might be interested in this library to implement it.
Implementation with channels first
If you set your Conv2D layer with "channels_first", the output convnet_output will be of size (batch_size, channels, height, width). Therefore you need first to permute the dimensions before reshaping.
convnet_output = keras.layers.Permute((0, 2, 3, 1))(convnet_output)

After this step, convnet_output has dimension (batch_size, height, width, channels). You can then proceed as previously, reshaping and feeding to the RNN.
rnn_input = keras.layers.Reshape((batch_size*height, width, channels))(convnet_output)
rnn_output = keras.layers.RNN(hidden_dim, return_sequences=True)(rnn_input)


How to predict whether or not a customer will renew

What you are trying to do is called Churn Prediction. Unfortunately, the dataset you have is not enough to train a model. You need a variety of different features for a proper prediction model.

For example, you need demographic data for the customers, but most important, data related to customer's actions. For example, if you have a mobile network company, you might need features as:

Age
Sex
Country
Daily usage (number of call minutes, SMS or mobile data)
Previous day usage
Mobile package (details about the contract etc)

In general, the more data you have around your problem, the easier would be to build such a model.


Kmeans only works on numerical data, so

throw you categorical data out,
standardize your numerical data,
calculate a distance matrix for a selected metric (for ex. dist() with euclidean),
run kmeans(distance_matrix,k=3) (try k=3 to say k=7)
look at model$cluster to get the data point classes


Should scaling be done for mixed data (categorical and numerical)?

Ans 1: get_dummies() or (label encoder + one-hot encoder) would do the trick.

Ans 2: Scaling categorical dummy data does not make sense. It also loses out on interpretability.

Ans 3: Logistic regression might tend to overfit since you only have 180 observations. KNN might perform well with a small number of observations but it doesn't handle categorical variables well. Random Forest or Extreme Random Forest might be your best bet. However, there's no "one fits all" concept in ML. You'll have to try a variety of algorithms and see which fits your dataset best. Though there are Linear SVM and other algorithms, I'd suggest that you try simpler ones as otherwise it might be an overkill.

Good luck!


How do you apply hypothesis testing to your features?

welcome to StackOverflow.

I will try to summarize as much as possible, but we have to cover a lot of concepts to properly answer your question, if something is not clear, please let a comment and I can change/correct the answer.

First of all, I will assume from the tone of your question that you know what a hypothesis test is.

Second, in the context of Linear Regression, you usually assume that your features are normally distributed, iid (identical independent distributed), constant error variance (that is, they do not vary with X attributes) and so on. These are strong assumptions, and it is important to have it in mind since we are calculating our regression coefficients using data that have these properties.

Put in simple terms, you have your set of features X, in this case A, B, and C, and you want to predict cost, therefore you have the following function:

Thus, you use above equation to minimize the RSS (Residual Sum of Squares) and find out what are your coefficients.

Now, in the context of hypothesis test, you want to verify if your coefficients are statistically relevant, that is, saying it on simple words, you want to check if they are far enough from zero in order you can say they are relevant, not a statistical incident. How exactly you do that? Creating a Null Hypothesis that you coefficient is zero, and an Alternative Hypothesis saying otherwise.

(Example to calculating Hypothesis Test for Beta_1 coefficient)

Then you calculate t-test for your coefficient:

So you are assuming your coefficient has a t-distribution and you want to test if it is far enough from zero to determine its relevancy.

You then calculate your t-test value and estimate a p-value. Therefore, in the end, you will ask the following question: What is the probability that my coefficient comes from a distribution centered around zero? If you have a high p-value, it shows you that probably your coefficient is zero, or close to it and not significant. Otherwise, if you have a low p-value, then it is not likely that it comes from a distribution around zero, then you should take this coefficient (and feature) in your analysis.

In the example from the video, he uses a rule that all coefficients above 0.05 are not statistically relevant. Here, 0.05 is widely used as a cutoff point, but you must be aware that there are a lot of issues regarding this choice and the debate inside scientific community is not settled.

If you really want to understanding all nuances involving Linear Regression, Statistical Tests and problems when modelling data using this technique, I highly recommend that you read the Introduction to Statistical Learning Book - Chapter 3. All images and insights were taken from there, and for me it is my reference book for Machine Learning techniques.

I hope this helps to clarify your question.


Training set of a NN that classify Cat or Not Cat

It is possible to build that kind of a CNN. It is important to maintain uniform distribution for both the classes ('cat' and 'not cat'). That is you should have an almost equal number of samples for each of these classes to avoid biasing your model to the 'non-cat' class just because it has huge number of examples.
The number of non-cat examples can be potentially trillion because it's everything in the world apart from cat. You don't want to include those many examples in your training set just because trillions of examples are available. As I told, it's important to match the class distributions. You can choose any animal or object you want for the 'non-cat' class. You can even mix multiple classes for 'non-cat'.


How do you work with Latent Dirichlet Allocation in practice

There are many methods of performing this optimization- namely, choosing the optimal number of topics to supply for LDA and many papers have been authored on the topic.

Several of note, which each define metrics by which to evaluate LDA models for quality of topics are:

Rajkumar Arun, V. Suresh, C. E. Veni Madhavan, and M. N. Narasimha Murthy. 2010. On finding the natural number of topics with latent dirichlet allocation: Some observations. In Advances in knowledge discovery and data mining, Mohammed J. Zaki, Jeffrey Xu Yu, Balaraman Ravindran and Vikram Pudi (eds.). Springer Berlin Heidelberg, 391–402. http://doi.org/10.1007/978-3-642-13657-3_43
Cao Juan, Xia Tian, Li Jintao, Zhang Yongdong, and Tang Sheng. 2009. A density-based method for adaptive lDA model selection. Neurocomputing — 16th European Symposium on Artificial Neural Networks 2008 72, 7–9: 1775–1781. http://doi.org/10.1016/j.neucom.2008.06.011
Romain Deveaud, Éric SanJuan, and Patrice Bellot. 2014. Accurate and effective latent concept modeling for ad hoc information retrieval. Document numérique 17, 1: 61–84. http://doi.org/10.3166/dn.17.1.61-84
Thomas L. Griffiths and Mark Steyvers. 2004. Finding scientific topics. Proceedings of the National Academy of Sciences 101, suppl 1: 5228–5235. http://doi.org/10.1073/pnas.0307752101

As luck would have it, if you're using R, these metrics have already been compiled for you in a convenient package called ldatuning which provides a  set of utilities and metrics to help tune the correct number of topics within LDA models.

Alternatively, if you're using Python, the gensim package can provide you with a lot of utilities to assist. For example, the package implements a metric that they call "topic coherence" that they claim corresponds roughly to clarity of  topic distinction for a human among many other utilities for tuning.


Punctuation removal effect on n gram detection for keyword generation

The n-gram model is often built after segmenting into words and sentences. If the data is segmented by sentence it's easy to avoid any overlap between sentences: one can simply extract n-grams sentence by sentence independently. In case it's more convenient to extract all the n-grams at once, padding can be used to mark the beginning/end of sentences like this:

  The sign was red #SENT# Balls are the toys of children #SENT#

Dealing with other punctuation signs which don't mark the end of a sentence can be a bit more tricky, especially if you want to keep the possibility of a keyword which spans over some punctuation signs (for instance in "red-handed" or "tl;dr").


NER vs Text classification for very short sentences

Using NER (more generally sequence labeling) means classifying every token in the sentence, so if the goal is only to label every sentence there's no strong need for it in your case.

However NER might be more appropriate in case the order of the words is important, because sequence labeling models take it into account whereas traditional text classification methods often use a "bag of words" representation (order doesn't matter).

To some extent it also depends whether the labels are always related to a particular term in the sentence: if yes, then NER might be better at locating these terms (this is related to the point about order). If no, then classifying at the level of sentences is likely to perform better.


In the paper of Hinton - Distilling the knowledge of Neural Networks, the following is mentioned (Section 5) when defining specialist models:

When the number of classes is very large, it makes sense for the cumbersome model to be an ensemble that contains one generalist model trained on all the data and many “specialist” models, each of which is trained on data that is highly enriched in examples from a very confusable subset of the
classes (like different types of mushroom).

What they do is they use a general model first and then a specialist model each to focus on a different subset of the classes.
You could consider your problem something similar, instead of a specialist to classes, and specialist in countries. This way you could build a country(cluster of countries) specific ensemble of models.


What exactly is BatchNormalization() in Keras?

Batch Normalization is a layer that is put in between convolution and activation layers or sometimes after activation layers. It is used to normalize layer’s input to reduce the internal covariate shift problem.
This problem occurs by changing in distribution of the input data in the early layers, and because every layer depends on the input of the former layers, it becomes a problem for this layer since it requires repeatedly adjusting to new input distributions.
Batch Normalization is a technique for training very deep neural networks that standardizes the inputs to a layer for each mini-batch. This has the effect of stabilizing the learning process and dramatically reducing the number of training epochs required to train deep networks.
It can be implemented during training by calculating the mean and standard deviation of each input variable to a layer per mini-batch and using these statistics to perform the standardization.
You read more here, here or here.
Also, I can tell you that this layer has a good impact on the performance and the achieved scores of your model.
Hope this helps you understand the importance of BN.


find value which occurred more times in a group from data frame column R

I think below code may help you.
df <-data.frame(place=c('hsy','hsy','hsy','hsy','bls','bls','bls','bls','bls','bls'),
                lat = c(7.343,7.343,7.567,6.934,2.67,3.345,2.987,2.987,1.876,2.987),
                long=c(32.849,32.122,31.567,32.122,6.2,5.9,6.321,4.56,6.321,6.321) )

library(dplyr)

Mode function which will fetch number which has highest frequency
mode_fun<-function(x){
  tbl=table(x)
  tbl=sort(tbl,decreasing = T)
  return(as.numeric(names(tbl[1])))
}

df1  = df %>% group_by(place) %>% summarise(lat1=mode_fun(lat),long1=mode_fun(long))


Some algorithms and approach for identification of specific patterns?

I suggest , go for Anomaly detection:

Anomaly Detection is done assuming our data has a probability distribution(gaussian). We can plot data to see if thats the case, if not we can make it gaussian using log transforms. Gaussian distribution specifies the regions and probabilities of our data lying in those regions.

For example : replace original feature x -> Log(x) or feature x -> (x)^4/3 etc..

Also regarding the threshold value which decides outliers you can play with it and see that with Higher threshold you will be rejecting more entries and this might be required where doctors are trying to isolate cancer patients amongst many normal ones without taking any risk/chances. Again outliers here doesn't mean cancer patient but definitely worth a medical test. And you can set it to lower value if you are getting too many normal data flagged as outliers.

We have skewed data sets since we have more examples of one kind than the other. For example when we get air craft engine data we might just have data for few bad ones and mostly for good ones.Use of cross validation data is suggested.F1-Score is a pretty good metric to evaluate the performance of the algorithm.

To get a proper hold on this topic , I also suggest go through  anomaly detection course videos by Andrew NG in machine learning on Coursera. Free course and very nicely made.

The well known techniques for the same are:
Density-based techniques (k-nearest neighbor),Support vector Machines,Cluster Analysis Anomaly detection. People also use Fuzzy logic based techniques.
And for python I suggest use libraries like scikit-learn and TensorFlow , they have superb implementations of usable machine learning algorithms.Heres a video link from a googler,giving general overview for the same :https://www.youtube.com/channel/UCwUtgW0JCPFjLbKx4VVr3bw