As a first start I would recommend to use Precision and Recall.

You can also produce stats on the different types of errors:

entity not found (false negative, or Type I error). Also you can get the false negative rate by dividing by the total number of true entities. This would be 1-recall.
entity found where it shouldn't be (false positive or Type II error). Also you can get the false positive rate by dividing by the total number of returned entities. This would be 1-precision.
entity found but system got wrong number of tokens (e.g. York instead of New York). You can also get the "token boundary error rate" by dividing by the total number of true entities.

So the first two error rates I mentioned don't give any extra info other than precision and recall but the third allows you to check how accurately you are getting entity boundaries.

If you are doing entity disambiguation too, you can produce additional metrics by only counting a success if the entity was correctly resolved.

You can also use the F-score (thanks Erwan) which combines precision and recall. However you need to be careful because you have a large class imbalance of O-tokens vs entity tokens, and the F-score gives equal importance to precision and recall. You also have the options of weighted F-scores such as F2 score.

Other option: ROC curve and AUC

Finally I would add that if your model outputs a score or probability rather than a binary classification, it means you can tweak its sensitivity by changing the threshold (do you want to count all entities with score>0.5 as entities, or score>0.75, for example). This naturally leads to the possibility of calculating the false positive and false negative rates for every possible threshold choice. You can plot it on a graph and you have a ROC curve. This is a nice graphical way to see the performance of your model and exactly how it misclassifies instances. The area under the ROC curve is itself a nice metric called AUC. A classifier that is as good as random has AUC=0.5, a classifier that classifies everything wrong has AUC=0, and a perfect classifier has AUC=1. I normally plot the ROC curve as well as generating metrics, Sklearn and most stats packages have a function to do it.


Relation between Cross Validation and Confidence Intervals

It depends how on the confidence interval (CI) is generated. The most common method is on a sample mean with the assumption that the samples are drawn from a normal distribution . However, a CI can be generated from any statistic from observed data. An alternative method would be through bootstrapping, resampling the statistic, which does not require the normality assumption.


How can I handle a column with list data?

You basically want to create a column for each product bought, as the presence or absence of each in the list is a feature in itself. See Hadley Wickham’s definition of tidy data.

That being said, you seem to have a large number of products. To avoid the course of dimensionality, what I would do is to take your binary bought/not features (or count values might be even more effective if you have that data) and do dimensionality reduction to get a reasonable set of features. Latent Dirichlet Allocation (which comes from topic modelling), PCA, t-SNE, our UMAP are all easy to implement and worth trying. PCA is the least sophisticated and the fastest to run and would be a good baseline.

When you have your smaller list of features, you might want to try using a classifier that’s further selects the most relevant features, like gradient boosted trees.


Is it wrong to use Glorot Initialization with ReLu Activation?

As a general answer for hyperparameter tuning, you have to try both and see what works better for your problem. I suspect that some (if not most) of general tuning rule have been observed on a given problem / with a given architecture. (for exemple the He paper is about vision, including convolutional layers).

As for keras choice, sometimes, for practical reasons, it is easier to implement an unique default option than to adapt the default option to the activation. Given your Hends-on machine learning citation, it's not hard to see why they would implement Glorot as the default.


How does CNN doing bounding box regression and what do features and weights represent for?

It depends on the concrete model, but we can consider the most popular single-stage model for object detection: SSD.

It has a set of default bounding boxes (prior boxes), for each of them it predicts:

Probability distribution over the set of classes (this is a classification problem, solved with crossentropy loss)
Offsets for the default location of the prior box (x, y coordinates for the center) and its height/width. This is a regression problem optimized with smoothed L1 loss (localization loss):

The exact definition is a bit more complicated, as it includes variance and exponents, but the general idea is like this. Also, depending on the bounding box encoding type, we may predict offsets not for the (cx, cy, w, h), but for (xmin, ymin, xmax, ymax).

The predicted shape offsets for prior boxes are not absolute values, of course. These values are relative to the default size of the prior box. The exact formulas for decoding are:


Being itself a non-linearity, the very advantage of ReLUs is their linear regime, that prevents it from saturating too early.

Having said that, a couple of alternatives for your case are to either clip the output of the ReLUs or to place sigmoid (or tanh, adjusting the output range) after them (or instead of them).


Cross Validation for Different Metrics - Sklearn

It only really matters if you want to shuffle your data in the cross-validation. The default for both cross_val_score and KFold is to NOT shuffle.

If you do want to shuffle your second option is best if you want to make sure that you are comparing scores across the same splits on the data.

Keep in mind that in the default KFold(shuffle=False), so if you want to shuffle your data make sure that you set Kfold to:

kf = KFold(n_splits=10, shuffle=True, random_state=seed)


Use of cross validation for Polynomial Regression

I think that you want this:

K-fold

If you want say MSE of each check out section 3.1.1 here:

cross validated metrics


Policy Gradient custom loss function not working

There could be many reasons. Custom loss functions are difficult to get right.

One conceptual problem is that the policy of the agent should not be the label. The label should be the reward signal from the environment. If the label is the reward signal from the environment, the agent would learn which policy would maxim rewards. If the policy of the agent is the reward, the agent would "chase its own tail" and not learn environmental rewards


Highly accurate convnets appear to have random-looking visualized weights?

The weights learned have no straightforward interpretability and i would not interpret those to be "random". What exactly do you mean by "random"? They seem to be learning very basic patterns that might reasonably extract and combine into more complex patterns as you deeper in the network, which is exactly how the networks are supposed to work.

from the Keras blog post you linked,

      A remarkable observation: a lot of these filters are identical, but rotated by some non-random factor (typically 90 degrees). This means that we could potentially compress the number of filters used in a convnet by a large factor by finding a way to make the convolution filters rotation-invariant. I can see a few ways this could be achieved --it's an interesting research direction.

The stanford page shows a great bit of what the network seems to be learning at an early layer -- combinations of edges and color blobs.

Perhaps this is more philosophical in nature, but why would we expect the layers to be meaningful or easy to interpret for humans? The network's huge number of weights are learned by optimizing a loss function via gradient descent. It can be difficult to interpret how far less complicated non-linear models "learn" or weight their coefficients; I'd expect these models would be (and are!) wildly difficult to interpret. The fact that we can see any structure in the various visualizations is cool, but they probably won't help us understand why a model isn't working all that well.

Regarding your problem, how many classes do you have and what type of accuracy are you hoping for? MNIST is an almost trivial dataset since the classes are very well separated. Some of the real-life machine vision tasks I deal with at work with good somewhat clean classes can get 80-90%~ accuracy, but errors are made in photos that humans might call something else (e.g., a photo of a studio apartment's living room in which the kitchen is clearly visible might be labeled as a kitchen and not a living room, but this is mostly okay). Defining what is a "good enough" accuracy is an issue on its own -- that's more than enough for what we're doing at the moment.

If I missed the point of your question, please let me know.


Retrain image classifier using MobileNet v2

I do not see any overfitting there. Overfitting is when your validation loss becomes worse with time, while training loss improves. Just because your training error is less than the validation (which is expected) does not mean it overfits.

But you can try to freeze the first layers (up to the fully connected one) to ensure the network does not lose generalization.


My go to library would be matplotlib, with which it is relatively easy to generate something similar.

I don't have the correct font family to render the exact output as above, but this hopefully illustrates the point

Source Code

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Create the data to plot on
# create a 2d array of evenly-spaced points on a grid
xy_range = np.arange(10.)
data = pd.DataFrame([[x, y] for x in xy_range for y in xy_range], columns=['X', 'Y'])

# color the points
# all green first and then with exceptions
data['color'] = 'green'
data.loc[(data['X']>=0)&(data['Y']==1), 'color'] = 'yellow'
data.loc[(data['X']>=8)&(data['Y']==2), 'color'] = 'yellow'
data.loc[(data['X']>=6)&(data['Y']==0), 'color'] = 'blue'

# We'll use this to calculate axis fractions
max_x = data['X'].max()
max_y = data['Y'].max()

fig, currAX = plt.subplots(figsize=(5,5), facecolor='lightgray')

# plot/mark each point as an annotation
# we do this so we can get a custom emoji instead a marker
for x0, y0, color in zip(data['X'], data['Y'], data['color']):
    plt.annotate(s=u'\u263B', xy=(x0/max_x, y0/max_y), fontname='STIXGeneral', color=color, size=15, ha='center', va='center')

# clear the axis lables, ticks, and lines
currAX.get_xaxis().set_visible(False)
currAX.get_yaxis().set_visible(False)
plt.axis('off')

plt.show();


Suspiciously low False Positive rate with Naive Bayes Classifier?

I find the easiest way for people to understand this is to think of the confusion matrix. Accuracy score is just one measure of a confusion matrix, namely all the correct classifications over all the prediction data at large:
$$\frac{True Positives + True Negatives}{True Positives + True Negatives + False Positives + False Negatives}$$
Your False Negative Rate is calculated by:
$$\frac{False Negatives}{False Negatives + True Positives}$$
One model may turn out to have a worse accuracy, but a better False Negative Rate. For example, your model with worse accuracy may in fact have many False Positives but few False Negatives, leading to a lower False Negative Rate. You need to choose the model which produces the most value for your specific use case.
Why do some classifier perform poorly? While an experience practitioner might surmise what could be a good modeling approach for a dataset, the truth is that for all datasets, there is no free lunch... also known as "The Lack of A Priori Distinctions Between Learning Algorithms" You don't know ahead of time if the best approach will be deep learning, gradient boosting, linear approaches, or any other number of models you could build.


How to get predictions with predict_generator on streaming test data in Keras?

To get a confusion matrix from the test data you should go througt two steps:

Make predictions for the test data

For example, use model.predict_generator to predict the first 2000 probabilities from the test generator.

generator = datagen.flow_from_directory(
        'data/test',
        target_size=(150, 150),
        batch_size=16,
        class_mode=None,  # only data, no labels
        shuffle=False)  # keep data in same order as labels

probabilities = model.predict_generator(generator, 2000)

Compute the confusion matrix based on the label predictions

For example, compare the probabilities with the case that there are 1000 cats and 1000 dogs respectively.

from sklearn.metrics import confusion_matrix

y_true = np.array([0] * 1000 + [1] * 1000)
y_pred = probabilities > 0.5

confusion_matrix(y_true, y_pred)

Additional note on test and validation data

The Keras documentation uses three different sets of data: training data, validation data and test data. Training data is used to optimize the model parameters. The validation data is used to make choices about the meta-parameters, e.g. the number of epochs. After optimizing a model with optimal meta-parameters the test data is used to get a fair estimate of the model performance.


How to determine if a company decision was successful or not?

You could use time series approach to model the "what if not" scenario and compare it with your values after the new program introduction. Check the causal_impact packet from Google and you might also find helpful this tutorial for probabilistic programming. http://nbviewer.jupyter.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Prologue/Prologue.ipynb


What are the differences between Apache Spark and Apache Flink?

Flink is the Apache renaming of the Stratosphere project from several universities in Berlin. It doesn't have the same industrial foothold and momentum that the Spark project has, but it seems nice, and more mature than, say, Dryad. I'd say it's worth investigating, at least for personal or academic use, but for industrial deployment I'd still prefer Spark, which at this point is battle tested. For a more technical discussion, see this Quora post by committers on both projects.


Word2Vec how to choose the embedding size parameter

I have checked four well-cited papers related to word embedding: 2013 Word2Vec, 2014 GloVe, 2018 BERT, and 2018 ELMo. Only GloVe has experimented on the embedding dimension for the analogy task (answering "a" is to "b" as "c" is to ?).

Other papers did not report an experiment on embedding dimension size. They are all using an arbitrary dimension on the order of hundreds (100 and 300 are used more frequently). The lack of experiments for embedding size implies that the performance is not very sensitive to this parameter and only the order of magnitude matters, and also other aspects of the model architecture are more important to investigate.

You can carry out a similar experiment using values from three orders of magnitude 10, 100, and 1000. The required dimension size will depend on the task too. I speculate that a classification (discriminative) task would require fewer dimensions than a sentence generation task.

Generally, for hyper parameter optimization, methods like Bayesian Optimization can be used to find the best hyper parameter (here, embedding dimension) with as few (costly) training-evaluations as possible. These techniques are more useful in competitive settings where every improvement counts.


loss/val_loss are decreasing but accuracies are the same in LSTM!

I think this is because your targets y are continuous instead of binary. Therefore, either ignore the accuracy report, or binarize your targets if applicable.

I assumed you are using Keras.
When you use metrics=['accuracy'], this is what happens under the hood:

if metric in ('accuracy', 'acc'):
   metric_fn = metrics_module.binary_accuracy

where

def binary_accuracy(y_true, y_pred):
    return K.mean(K.equal(y_true, K.round(y_pred)), axis=-1)

In the case of continuous targets, only those y_true that are exactly 0 or exactly 1 will be equal to model prediction K.round(y_pred)). Therefore, accuracy cannot be used for continuous targets.

Here is a code that demonstrates this issue:

from keras import Sequential
from keras.layers import LSTM, Dense
import numpy as np

# Parameters
N = 1000
halfN = int(N/2)
seq_len = 10
dimension = 2
lstm_units = 3

# Data
np.random.seed(123)  # to generate the same numbers

X_zero = np.random.normal(0, 1, size=(halfN, seq_len, dimension))
y_binary_zero = np.zeros((halfN, 1))  # output is only 0
y_continuous_zero = np.random.randint(0, 50, (halfN, 1)) / 100  # output is in [0, 0.5]

X_one = np.random.normal(1, 1, size=(halfN, seq_len, dimension))
y_binary_one = np.ones((halfN, 1))  # output is only 1
y_continuous_one = 0.5 + np.random.randint(0, 50, (halfN, 1)) / 100  # output is in [0.5, 1.0]

p = np.random.permutation(N)  # to shuffle zero and one classes
X = np.concatenate((X_zero, X_one))[p]
y_binary = np.concatenate((y_binary_zero, y_binary_one))[p]
y_continuous = np.concatenate((y_continuous_zero, y_continuous_one))[p]

# Build model
model = Sequential()
model.add(LSTM(lstm_units, input_shape=(None, dimension)))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
print(model.summary())

# Fit model
# fit using binary outputs
print('-----------------------Binary---------------------')
model.fit(X, y_binary, batch_size=32, epochs=10)
# fit using continuous outputs
print('-----------------------Continuous---------------------')
model.fit(X, y_continuous, batch_size=32, epochs=10)

which outputs

...
-----------------------Binary---------------------
...
1000/1000 [==============================] - 0s 122us/step - loss: 0.3989 - acc: 0.9500
-----------------------Continuous---------------------
...
1000/1000 [==============================] - 0s 135us/step - loss: 0.5759 - acc: 0.0050


Which non-training classification methods are available?

What you are asking about is Instance-Based Learning. k-Nearest Neighbors (kNN) appears to be the most popular of these methods and is applicable to a wide variety of problem domains. Another general type of instance-based learning is Analogical Modeling, which uses instances as exemplars for comparison with new data.

You referred to kNN as an application that uses training but that is not correct (the Wikipedia entry you linked is somewhat misleading in that regard). Yes, there are "training examples" (labeled instances) but the classifier doesn't learn/train from these data. Rather, they are only used whenever you actually want to classify a new instance, which is why it is considered a "lazy" learner.

Note that the Nearest Template Prediction method you mention effectively is a form of kNN with k=1 and cosine distance as the distance measure.


Calculating cosine similarity between 3D arrays using Python

Your input matrices (with 3 rows and multiple columns) are saying that there are 3 samples, with multiple attributes. So the output you will get will be a 3x3 matrix, where each value is the similarity to one other sample (there are 3 x 3 = 9 such combinations)

If you were to print out the pairwise similarities in sparse format, then it might look closer to what you are after.

I have created two example matrices of random numbers that fits your description:

from sklearn.metrics.pairwise import cosine_similarity
from scipy import sparse

a = np.random.random((3, 10))
b = np.random.random((3, 10))

# Create sparse matrices, which compute faster and give more understandable output
a_sparse, b_sparse = sparse.csr_matrix(a), sparse.csr_matrix(b)

sim_sparse = cosine_similarity(a_sparse, b_sparse, dense_output=False)
print(sim_sparse)

Output:

  (0, 2)    0.7938732813430508
  (0, 1)    0.7575978172453429
  (0, 0)    0.7897664361147338
  (1, 2)    0.740418315571796
  (1, 1)    0.833981672896221
  (1, 0)    0.7184526671218405
  (2, 2)    0.8746293481677073
  (2, 1)    0.6456666045233884
  (2, 0)    0.7925289217609924

Hopefully this output makes it clearer, what you are actually getting as output.

Have a look here for a few more details on performance aspects, and the documentation on sparse matrices is here.


Found a fantastic answer here that avoids high-level black box libraries.

Essentially, numerical floats are categorized into bins based on boundary values which are determined by their distribution. Text tokens are hashed with column ID, and then concatenated with other columns' hashed tokens via an interaction array. All hashed tokens, whether or not they have in interaction, are fed as inputs into the model.

That's the gist of it.


Personally , at the first time, I used this link to do predictions in deep learning:

https://machinelearningmastery.com/how-to-make-classification-and-regression-predictions-for-deep-learning-models-in-keras/

if you're using another algorithm (not neural networks), which one?


how to interpret a hierarchical clustering in the heatmap in the picture bellow?

This type of plot is refereed as Double Dendogram or Coloured Heatmap.
As seen in the code you have used Single Linkage Method for clustering.It yields clusters in which individuals are added sequentially to a single group.
From the example we can see that label dia2,ht and ob belong to one group but ht and ob are more correlated with each other.

I am not sure what exactly the heatmap does

The color of a cell is proportional to its position along a color gradient.
Heatmap is helpful in finding the correlation coefficient. It can helps us to visualize the difference in value within clusters. It's more like a color representation of value.


grid search result max_features = 'sqrt' in random forest - how to understand

The way to understand Max features is "Number of features allowed to make the best split while building the tree". The reason to use this hyperparameter is, if you allow all the features for each split you are going to end up exactly the same trees in the entire random forest which might not be useful. To overcome this we let the model select a fixed number of features randomly, in this case, the no of features allowed = Square root of total no of features in your dataset.
Hope this clears up !!


Mean Absolute Error in Random Forest Regression

Let me clarify few fundamental things:

In sklearn, RandomForrest Regressor criterion is:

  The function to measure the quality of a split

It's a performance measure (by default, MSE) which helps the algorithm to decide on a rule for an optimum split on a node in a tree.
Kaggle is giving you a metric, i.e. MAE (again a performance/ quality measure) but to evaluate the performance of your ML model, once finalized.

To come back to your question: while both MAE/ MSE are performance measures, they are being used at two different stages of a modeling process and might not be related. So, while it makes sense to evaluate your final model on MAE as you would be judged on it, you can choose any of MAE/ MSE for criterion (i.e. for RandomForest) depending on performance at validation stage.

While the above being said, keep in mind that you might want to evaluate the validation errors (i.e. for finalizing a model) on the same metric (i.e. MAE in this case), to keep error measure consistent with the test set evaluation.


Which is better: Out of Bag (OOB) or Cross-Validation (CV) error estimates?

OOB samples are a very efficient way to obtain error estimates for random forests. From a computational perspective, OOB are definitely preferred over CV.

Also, it holds that if the number of bootstrap samples is large enough, CV and OOB samples will produce the same (or very similar) error estimates. Thus, if you perform many bootstrap samples, I would recommend performing cross-validation along the way with OOB samples until the OOB error converges.


Fraud risk propagation in large scale network

You could try applying a graph convolutional network to do some semi-supervised learning. See Kipf and Welling's paper "Semi-Supervised Classification with Graph Convolutional Networks". It probably depends on how unbalanced your dataset is though. If the dataset is too large, you could train a sample of it, and train the GCN on that subset. I'd try to find some exemplar data points and create a train set from that.


Which are the worse machine learning models for text classifications?

First, the question is too broad because there are many different kinds of text classification tasks. For example one wouldn't use the same approach for say spam detection and author profiling (e.g. predict the gender of the author), two tasks which are technically text classification but have little in common (and there are many others).
Second, even with a more specific kind of problem, the question of the type of model is misleading because a lot of what makes a ML system perform better than another in text classification is due to other things: the type and amount of training data of course, but also crucially the features being used. There are many options in terms of representing text as features, and these different options usually have a massive impact on performance. I even think that most of the time the choice of a type of classification model does not matter as much as the design of the features.
Finally I'm actually going to answer the question but probably not in the way OP expects: the worst model in any classification task is exactly like the best model, but it swaps the answers in order to have as many wrong predictions as possible (e.g. class 1 -> class 2, class 2 -> class 3, .., class N -> class 1).
Since it's a lot of work to implement the best classifier just to obtain the worst one, a close to worst one can be done with a minority baseline classifier: just predict every instance as the least frequent class in the training data.
I hope a few of the things I said will be helpful, even though it's probably not what OP wished for! :)


Identity covariance matrix, decorrelated data?

Having only taken a few courses on this stuff, I'm going to offer my understanding on this.

The main reason seems to be simplifying, as you assume. It manifests in different areas though. One is mathematical - if your co-variance matrix is identity matrix then the math becomes much easier. The one directly following from this is computational - inverting a matrix is very costly, but identity matrix is its own inverse, so that's perfect.

Another part of the simplicity is definitely the ease of interpretation. Independent variables are easy to explain to a non-technical person, but as soon as you get into covariances, that goes out the window.


I would suggest you to use libsvm, which already has adjustable class weights implemented in it. Rather than replicating the training samples, one modifies the C parameter for different classes in the SVM optimization. For example if your data has 2 classes, and the first class is only 10% of the data, you would choose class weights to be 10 and 1 for class 1 and 2 respectively. Therefore, margin violations of the first class would cost 10 times more than the margin violations for second class, and per-class accuracies would be more balanced.


outlier detection in time serie without using windows

You can try to remove trend and seasonality and check whether the time series is stationary. If it is, you can calculate mean ($\mu$) and standard deviation ($\sigma$). Now for each data point, you calculate the distance from the mean ($\|y-\mu\|$) and check if it exceeds some threshold. The default value for threshold is $3 \sigma$ what comes from three-sigma rule motivated by the assumption about the normality of the data (residuals after time series decomposition in our case). You can choose any threshold which is reasonable for you. One solution is to find the distribution of the data and derive a rule for this particular distribution.


It's called Label Encoding

In python with the help of scikit-learn you can do following:

  le = preprocessing.LabelEncoder() le.fit(["paris", "paris", "tokyo",
  "amsterdam"])

  list(le.classes_)

  le.transform(["tokyo", "tokyo", "paris"])

  list(le.inverse_transform([2, 2, 1]))

Documentation for more info: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html

PS: Label Encoding might be responsible for unintentional ordering of possible values. I.E. model might think that nominal value with associated numerical value 2 is more important than the nominal value with associated numerical value 1. The solution is One Hot Encoding.


Regression Model for explained model(Details inside)

AFAIK if you want to predict the value of one variable, you need to have one or more variables as predictors; i.e.: you assume the behaviour of one variable can be explained by the behaviour of other variables.
In your case you have three independent variables whose value you want to predict, and since you don't mention any other variables, I assume that each variable depends on the others. In that case you could fit three models (for instance, regression models), each of which would predict the value of one variable, based on the others. As an example, to predict x:

x_prediction=int+cy*y_test+cz*z_test

, where int is the intercept and cy, cz, the coefficients of the linear regression.
Likewise, in order to predict y and z:

y_prediction=int+cx*x_test+cx*z_test
z_prediction=int+cx*x_test+cy*y_test


Dealing with an apparently inseparable dataset

You might have a domain adaptation problem.
The samples are actually taken from two different sources and have different rules governing their behavior.

I suggest that you'll try the following method.

Split the data into two data set using the target concept.
Build a model for each data set. In case your performance doesn't improve it is likely that the domain adaptation assumption is wrong.
Otherwise, we gain an improvement. Now we should find out which model to apply. Use the full dataset and use the target in order to create a new concept indicating to which dataset the sample belongs. Since you said that given the target you can deduce that I assume the two are different.
Build a model for the new dataset. If it performs well, the problem is solved. Run this model first and apply the proper model for each sample. Otherwise, it means that the two data set are indistinguishable (at least with respect to the methods you applied), weakening the assumption the the cause of the problem is domain adaptation.


How can I build a seq2seq model , which is topic aware

Try this approach (not embedded but filtering your model) with Topic modeling, that could be used to filter the results of your model by topic:

Separate each doc in several sentences
Obtain the best topics ( occur > 3, chars > 3 ) from sentences if you do not have an idea of topics to declare (LatentDirichlet Allocation)
Discard some topics out of interest

Finally assign a dominant topic to every sentence and filter results by topic.

I have found a reference from steps 1-3:
https://medium.com/@actsusanli/when-topic-modeling-is-part-of-the-text-pre-processing-294b58d35514


Number of epochs in Gensim Word2Vec implementation

Increasing the number of epochs usually benefits the quality of the word representations. In experiments I have performed where the goal was to use the word embeddings as features for text classification setting the epochs to 15 instead of 5, increased the performance.


Sequential pattern mining in SPMF: Giving words as input

Thanks for using SPMF. I'm the founder of that library. Currently, SPMF does not support text files containing some text as input. Thus, users need to do some preprocessing to convert the text file to the format used by SPMF.  But in the future, it is my plan to add the feature of handling text files natively in SPMF.  It will likely be added in the next few weeks, depending on my schedule.

By the way, if you have any question about SPMF, you can also directly ask them on the forum on the SPMF website. Then, I will answer you more quickly.


The goal is to identify at least 98% of the customers, that do not repay their debt. So the bank can "accept a maximum number of 'good' customers, that can be granted loans" Here the goal is focused on the bad customers.

There should be at least 85% good customers accepted while the side focus is to reject as many bad customers as possible.

I think the difference between 1) and 2) is identifying bad customers in 1) and good customers in 2)


Original Meaning of "Intelligence" in "Business Intelligence"

Howard Dresner, in 1989, is believed to have coined the term "business intelligence", to describe "concepts and methods to improve business decision making by using fact-based support systems.". When he was at Gartner Group. This is a common mantra, spread over the Web. I have not been able to trace the exact source for this origin yet. Many insist on he was not at Gartner group in 1989, which is confirmed in the following interview. In his 2008 book, Performance Management Revolution: Improving Results Through Visibility and Actionable Insight, the termed is defined as:

  BI is knowledge gained through the access and analysis of business
  information.

He says, at the beginning, that

  In 1989, for example, I started-some might say incited-the BI
  revolution with the premise that all users have a fundamental right to
  access information without the help of IT.

No apparent claim of the invention of the term on his side. Indeed, one can find older roots in H. P. Luhn, A Business Intelligence System, IBM Journal of Research and Development, 1958, Vol. 2, Issue 4, p. 314--319.

  Abstract: An automatic system is being developed to disseminate information to the various sections of any industrial, scientific or government organization. This intelligence system will utilize data-processing machines for auto-abstracting and auto-encoding of documents and for creating interest profiles for each of the "action points" in an organization. Both incoming and internally generated documents are automatically abstracted, characterized by a word pattern, and sent automatically to appropriate action points. This paper shows the flexibility of such a system in identifying known information, in finding who needs to know it and in disseminating it efficiently either in abstract form or as a complete document.

The author claims that:

  The techniques proposed here to make these things possible are:

  Auto-abstracting of documents;
  Auto-encoding of documents;
  Automatic creation and updating of action-point profiles.

  All of these techniques are based on statistical procedures which can
  be performed on present-day data processing machines. Together with
  proper communication facilities and input-output equipment a
  comprehensive system may be assembled to accommodate all information
  problems of an organization. We call this a Business Intelligence
  System.

He also gives the explanation of the terms "business" and "intelligence":

  In this paper, business is a collection of activities carried on for
  whatever purpose, be it science, technology, commerce, industry, law,
  government, defense, et cetera. The communication facility serving the
  conduct of a business (in the broad sense) may be referred to as an
  intelligence system. The notion of intelligence is also defined here,
  in a more general sense, as "the ability to apprehend the
  interrelationships of presented facts in such a way as to guide action
  towards a desired goal."

So the idea of "linking the facts" is already present in H. P. Luhn paper. To many sources, Howard Dresner has re-invented "Business Intelligence" to re-brand decision support system (DSS) and executive information system (EIS) when at DEC, and the term became famous throught the influence of the Gartner group.

Apparently, the term has already been used way before, as in the book Wholesale Business Intelligence and Southern and Western Merchants' Pocket Directory to the Principal Mercantile Houses in the City of Philadelphia, for the Year 1839.

As I could not fetch this source, I will stick to the Luhn/Dresner acception. It relates to the etymology of intelligence:

  late 14c., "faculty of understanding," from Old French intelligence
  (12c.), from Latin intelligentia, intellegentia "understanding, power
  of discerning; art, skill, taste," from intelligentem (nominative
  intelligens) "discerning," present participle of intelligere "to
  understand, comprehend," from inter- "between" (see inter-) + legere
  "choose, pick out

In Business Intelligence for Dummies (Scheps, 2008), the definition chapter plays on Military Intelligence:

  Business Intelligence Defined: No CIA Experience Required So what the
  heck is business intelligence, anyway? In essence, BI is any activity,
  tool, or process used to obtain the best information to support the
  process of making decisions.

  For our purposes, BI revolves around putting computing power (highly
  pecialized software in concert with other more common technology
  assets) to work, to help make the best choices for your organization.

  Business intelligence is essentially timely, accurate, high-value, and
  actionable business insights, and the work processes and technologies
  used to obtain them.

I would thus bend toward "Gathering data and information about the business", maybe more "to better conduct business". Additional historical comments can be found in Father of BI? Is he having a laugh?!


How to apply model to training data to identify mislabeled observations?

Sure, its called cross validation. Have a look


Rolling average: when is it possible to consider it?

You can! But the implications can be huge.
If you choose a big number, you automatically shorten the dataset from which you can train your time-series or forecast model. Also, if things start to "change" in recent time, your model might take too long to react if it weights those rolling averages heavily.
If you go too short, it might overreact to recent changes, or you're just missing that information from which your model might be able to learn on. I usually make that decision based on how I want to use the forecast on unseen data. This is not an easy decision for which there is a solid rule-of-thumb.
Welcome to the real-world complexities of a time-series problem!


Dimensionality reduction for feature extraction when missing some feature values

Denoising autoencoders will be your bet in this case. I don't have a book handy for this case. They are good at reconstruction and calculate a good latent representation.  Just replace your missing feature with the mean or a fixed value.


If the predicted outcome targets(i.e., probabilities in this example), are consistently higher than actual, it implies you have statistical bias model. Statistical bias is when there is a systematically different result from the population parameter being estimated.
There are many ways to reduce statistical bias:

Resample at more extreme ratio
Change the loss function to more heavily penalize mispredictions
Change algorithm


What does a negative coefficient of determination mean for evaluating ridge regression?

A negative value means you're getting a terrible fit - which makes sense if you create a test set that doesn't have the same distribution as the training set.

From the sklearn documentation:

  The coefficient $R^2$ is defined as (1 - u/v), where u is the residual sum of squares ((y_true - y_pred) ** 2).sum() and v is the total sum of squares ((y_true - y_true.mean()) ** 2).sum(). The best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of y, disregarding the input features, would get a $R^2$ score of 0.0.


The proper way to codify Na in a list in R

I made it work by coercing the timeserie to numerics

little_slice <- slice[c(750: 850)]
for (row in 1:nrow(little_slice)) {
  ts <- as.numeric(little_slice[row,])
  if(any(is.na(ts)))
     {
       imputed_ts <- na.kalman(ts, model = "auto.arima")
       little_slice[row,] <- imputed_ts
  }
}


What are the advantages of the rectified linear (relu) over the sigmoid activation function in networks with many layers?

The sigmoid function becomes asymptotically either zero or one which means that the gradients are near zero for inputs with a large absolute value.
This makes the sigmoid function prone to vanishing gradient issues which the ReLU does not suffer as much.

In addition, ReLU has an attribute which can be seen both as positive and negative depending on which angle you are approaching it. The fact that ReLU is effectively a function that is zero for negative inputs and identity for positive inputs means that it is easy to have zeros as outputs and this leads to dead neurons. However, dead neurons might sound bad but in many cases, it is not because it allows for sparsity. In a way, the ReLU does a similar job of what an L1 regularization would do which would bring some weights to zero which in turn means a sparse solution.
Sparsity is something that, lots of times, leads to a better generalization of the model but there are times which has a negative impact on performance so it depends.
A good practice when using ReLU is to initialize the bias to a small number rather than zero so that you avoid dead neurons at the beginning of the training of the neural network which might prevent training in general.


Data aggregation and split train test samples

If you are training the model based on the aggregated values anyway then it wouldn't make any difference on the final dataset that is fed to the model.

You might consider aggregating earlier in your pipeline as to reduce the amount of work that has to be done afterwards (whatever steps you go through).  Aggregating means less data points, so following steps will run faster.


1st approach: Use append=True in the call to to_hdf:

import numpy as np
import pandas as pd

#filename = '/tmp/test.hdf5'
filename = 'D:\test.hdf5'

df = pd.DataFrame(np.arange(10).reshape((4,2)), columns=['C1', 'C2'])
print(df)
#    C1  C2
# 0  0   1
# 1  2   3
# 2  4   5
# 3  6   7

# Save to HDF5
df.to_hdf(filename, 'data', mode='w', format='table')
del df    # allow df to be garbage collected

# Append more data
df2 = pd.DataFrame(np.arange(10).reshape((4,2))*10, columns=['C1', 'C2'])
df2.to_hdf(filename, 'data', append=True)

print(pd.read_hdf(filename, 'data'))

2nd approach:  you could append to a HDFStore instead of calling df.to_hdf:

import numpy as np
import pandas as pd

#filename = '/tmp/test.hdf5'
filename = 'D:\test.hdf5'
store = pd.HDFStore(filename)

for i in range(2):
    df = pd.DataFrame(np.arange(10).reshape((4,2)) * 10**i, columns=['C1', 'C2'])
    store.append('data', df)

store.close()

store = pd.HDFStore(filename)
data = store['data']
print(data)
store.close()

3rd approach:  using chunksize parameter and append each chunk to the HDF file which  was answered here.

Personally I like 1st and 2nd approach.


When to use model.train() vs model.eval() in Pytoch?

I can see arguments to have the self-play phase use both .train() and .eval(), so I had a look at the implementation of facebook's ELF OpenGo and saw that they have the model in eval mode during self-play (see selfplay.py). I would do as they do because their software seems to work.


As @spacedman correctly mentioned, this will answered quicker at StackOverflow. But you can use this to create a dictionary like this. There might be a better way but this is a quick work around.

# -*- coding: utf-8 -*-
import re
st = '''Pafient Name : Thomas Joseph MRNO : DQ026151?
Doctor : Haneef M An : 513! Gandar : Male
Admission Data : 19-Feb-2V'3‘¥T12:2'$ PM Bill No : IDOGIII.-H-17
Discharge Date : 22-Feb-20$? 1D:5‘F AM Bill Dale : E2-Feb-2017'''
st = st.decode('utf-8').replace('\n','')+'<eof>'
words = ['Pafient Name','MRNO','Doctor','An','Gandar','Admission Data','PM Bill No','Discharge Date','Bill Dale','<eof>']
print {words[i]:st[st.index(words[i])+len(words[i]):st.index(words[i+1])].replace(':','').strip() for i in range(len(words)-1)}


Why don't convolutional computer vision networks use horizontally - symmetric filters?

I'd like to implement this symmetry in the network, and by my understanding of CNNs that means i'll generally need all the filters to be horizontally symmetric, and then the network would both be more robust and would take almost 40% less time to train assuming filter size = 5.

If two mirrored images are in the same class - e.g. they both show a dog or a cat - that is not the same as having all the components in the image (lines, textures, shapes) responding well to symmetric filters. In general this is not the case. Even for symmetric looking shapes such as faces, it is only true at a certain scale and specific pose.

  what disadvantages does this architecture have that I can't find a single mention of such idea?

It will work poorly, because the lines, textures and shapes being detected as components of the image are rarely horizontally symmetrical.

CNNs cannot easily take advantage of mirroring or rotational invariance - where the class or detection does not vary under mirror or rotation transformations. One thing you can do to improve generalisation for transformations is data augmentation, i.e. performing the non-class-changing transformations of your training images, perhaps randomly on demand during the training process. This does work, but has the opposite impact that you were hoping for in terms of efficiency.

The recently-published CapsNet by Geoffrey Hinton's group may be able to take advantage of more types of transformation invariance. However, this is still in early stages of research, and not clear whether it offers a practical advantage in your case. There are implementations in various frameworks such as Keras, that you could try if you are interested.


How to construct a self learning process for an ensemble model?

It is not much more sofisticated than what you thought, but you can try training a linear regression (in the online setting, by stochastic gradient descent) whose inputs are the predictions spit by your models and the output is the real value of the predicted variable. In this case, you wouldn't need to retrain with the whole dataset.

In fact, I think that all online machine learning techniques can be applied in this setting.


Can we take of benefit of using transfer learning while training a word2vec models?

Yes, you can take benefit of pre-trained models. Most famous one being the GoogleNewsData trained model which you can find here.

Pre-trained word and phrase vectors https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing

You can then load the vectors in binary format in your model using gensim as shown below.

>>> model = Word2Vec.load_word2vec_format('/tmp/vectors.txt', binary=False)  # C text format
>>> model = Word2Vec.load_word2vec_format('/tmp/vectors.bin', binary=True)  # C binary format

Here is a different pre-built model for English Wikipedia:

https://github.com/idio/wiki2vec/raw/master/torrents/enwiki-gensim-word2vec-1000-nostem-10cbow.torrent

Source: https://github.com/idio/wiki2vec/

Using a prebuilt model

Get python 2.7
Install gensim: pip install gensim
uncompress downloaded model: tar -xvf model.tar.gz
Load model in gensim:
from gensim.models import Word2Vec
model = Word2Vec.load("path/to/word2vec/en.model")
model.similarity('woman', 'man')

You can also use Stanford NLP Glove

http://nlp.stanford.edu/projects/glove/
https://github.com/stanfordnlp/GloVe

Here is a great compilation of pre-trained word2vec models.

https://github.com/3Top/word2vec-api#where-to-get-a-pretrained-models

Some additional pre-trained models:

http://www.zuccon.net/ntlm.html
https://code.google.com/archive/p/word2vec/

More on gensim and code here:  https://radimrehurek.com/gensim/models/word2vec.html

Quora forum with a similar questions

https://www.quora.com/Where-can-I-find-pre-trained-models-of-word2vec-and-sentence2vec-trained-on-Wikipedia-or-other-large-text-corpus
https://www.quora.com/Where-can-I-find-some-pre-trained-word-vectors-for-natural-language-processing-understanding


resampling image takes a large amount of time

Although my money would be on the interpolation function taking a fair while, here are a few other ideas:

1. Is that resize_factor matrix going to be different for every single image? You could otherwise pre-compute it one for all images of a given size and then just use it, instead of re-computing it for every image.

2. One aspect, which may answer your other related question on DS), is memory usage, that can take time if the arrays are large enough. Those matrix divisions that you perform will return a new matrix, that means a new chunk of memory. If it turns out (from the profiling idea below), that those division operations are costly, you could instead try pre-allocating an array to hold all your results (assuming you know the relaevant shapes), then filling the array with the output.

3. ``scipy.interpolation.zoomreturns anndarray, so you don't need to callnp.arrayon it again in your return statement of theresample` function.

I would suggest profiling your script. This will allow you to see which function calls are called most often and which take the most time. You can then focus on making making changes that will have the biggest effect on the runtime. Here is a nice example tutorial if you're new to profiling in Python.

You can use cProfile, a built-in python module. You can do this in the terminal quite simply (if your script is designed to be run that way) by running:

python -m cProfile -o profiling_results.prof your_script.py

In words (flags shown in bold): python run the module cProfile, to run your_script.py, producing the output file profiling_results.prof.

The generic version of this command it:

python -m cProfile [-o output_file] [-s sort_order] (-m module | myscript.py)

You can open and read the output file in a normal text editor. The suffix is a common convention, because you can pass it to a tool like SnakeViz, which will allow you to visualise the results and introspect the method tree of your script; the order in which functions call each other.

There is a notebook on Kaggle, which you may find interesting, as it performs similar tasks on the same LUNA16 dataset.


What makes binary cross entropy a better choice for binary classification than other loss functions?

As mentioned in the blog, cross entropy is used because it is equivalent to fitting the model using maximum likelihood estimation. This on the other hand can be interpreted as minimizing the dissimilarity between the empirical distribution of training data and the the distribution induced by the model (measured as Kullback-Leibler divergence, hence the name cross-entropy).

Now you may ask why maximizing the likelihood. Maximum Likelihood estimators have nice asymptotical properties (they are the best estimators in terms of convergence rates), they are consistent and statistically efficient.

Another advantage is that you don't need to think much about how to the define a cost function. The ML framework gives you the cost function right away as long as you have a model that specifies $P(y|x)$.

Also, directly optimizing the cross-entropy leads naturally to well-calibrated probabilities compared to, say, a random forest. That means you can interpret the resulting probabilities as you would expect, namely that of all the cases with about x% predicted probability, x% should belong to the positive class.

There is another important point. Given the prevalence of gradient-based learning algorithms (especially in deep learning), the logarithm in the cross-entropy will undo any exponential behavior that is often given through popular activation/output units like the sigmoid/softmax. Thus, the logarithm will avoid that the gradient saturates for extreme values. Large gradients are important for gradient-descent algorithms to make sufficient progress in each iteration. (There are more details, for example, in the "deep learning book" in chapter 6.2)


Get Gate Activation of GRU-Cells in TensorFlow

I am trying to figure out the same problem as well. I don't have the full solution but maybe we can help each other out.

If you do: names =[n.name for n in tf.get_default_graph().as_graph_def().node]

This will give you a list of all the names of the nodes in the graph. For Rnn the out put for names should look some what like this

.
'RNN/Shape',
'RNN/strided_slice/stack',
'RNN/strided_slice/stack_1',
.
.
.
'RNN/transpose'
.

Then you can use get_operation_by_name to get the operation you want to look at.

 rnn_transpose=tf.get_default_graph().get_operation_by_name('RNN/transpose').outputs[0]

From that you can run the eval() with your regular sess and feed_dict

 rnn_transpose.eval(session=sess,feed_dict =feed_dict)

The problem here is now we have to find which operations in the graph correspond to the proper gate operations.


How to generate a sample from a generative model like a Restricted Boltzmann Machine?

For a RBM, you run the stochastic network - forward and back from the "input" and hidden layers - multiple times. After a few steps it will converge into sampling from the data population it has learned from. If you have learned from a whole picture and want to fill in a patch, then hold the input values that are not in the patch (don't allow them to change randomly).

Other generative models may have different approaches. GANs and VAEs typically have a simpler approach where you generate a random input vector and run the generator part of network forward from that input.


What is C in sklearn Logistic Regression?

Exactly as it says in the documentation; it's $1/\lambda$, where $\lambda$ is the regularization strength.


Do models learn a test set after multiple predictions upon it?

TL/DR: No.

The concept of prediction, here applied by your predict method, is to just try out the model on the test set. It does not imply any kind of adaptation to such set, thus your model doesn't directly learn anything from it.

You can generalize that to any other framework or discussion in data science as a whole - training and fitting are tasks performed on the train set, and they imply adapting to it, and actually learning its patterns. Predicting does not.


Since you are trying to compute $\partial loss \over \partial x$ you need to perform all operations that lead from $x$ to $loss$ inside GradientTape's scope, so that it can monitor them.

x = tf.Variable(2.)

with tf.GradientTape(persistent=True) as g:
    loss_ = x**2-2*x+1
    loss = loss_*1

print(g.gradient(loss,x))
# Will output: tf.Tensor(2.0, shape=(), dtype=float32)


Is it a acceptable way to write a loss function in this form?

It just means to sum over all $x_i$ in $M$. That is completely acceptable notation.


How to apply my deep learning model to a new dataset?

There is a difference between using pre trained weights and transfer learning.

When you use pre trained weights and re train all the layers in the entire model for the dataset, then you are just working towards converging faster in the direction of better results.
If you were to only train the upper layers and lock all the other layers from training, then you are implementing transfer learning.

For your problem at hand, I would use approach 1.

Would be more than happy to elaborate in case you want me to.

Here is a link that explains how to do transfer learning with Keras : https://towardsdatascience.com/transfer-learning-using-keras-d804b2e04ef8


Combining multiple neural networks with different activation functions

As you are trying to average out the values, and given the three have different domains, it makes sense to bring all in same domain before averaging out. You can normalize P2 (tanh) to 0-1 and then average the values.

If you want to try another way, you can combine these three networks and take the three outputs p1, p2, and p3 and input it to some more dense layers and finally predict the single output. This way instead of averaging out, you are bringing in some nonlinearity which will be helpful to learn the task.


I think post below is a good resource.

https://developers.googleblog.com/2017/11/introducing-tensorflow-feature-columns.html

Basically, all categorical variable is initially converted to one-hot encoding, then layer size defined by dimension argument is stacked on top of one-hot encoding; thus learning optimal representation of categorical variable based on specified dimension.

There is general rule in the blog post to take the 4th root of the number of categories.

Another approach is to perform MDS to inspect your categorical variables to decide dimensions.


SVM SVC: Metric for parameter optimization on imbalanced data

Using resampling methods to fix the imbalanced databases a good method. This can be completed by oversampling, downsampling or creating synthetic instances.

In this kind of problem, precision is not necessarily a good evaluation metric. Instead evaluating True Positives, True Negatives, False Positives and False Negatives is better way to evaluate this.

For this evaluation following techniques can be utilized from Sklearn library.

Recall score
Accuracy score
F1 score
AUC Score

In Sklearn the method caled "classification_report" gives a summary of those evaluations (except AUC Score from the list). That can be a good evaluation metric to check the performance of the model.


Here is a list of several important CNN architectures:

LeNet-5 the first successful application of CNNs to digit recognition,   developed by Yann LeCun in 1990. It consists of a sequence of Convolutional, ReLU, Max Pooling layers followed by a Fully Connected layer.
Reference: Y. LeCun et. al., "Gradient-Based Learning Applied to Document Recognition", Proceedings of the IEEE, 1998
AlexNet popularized CNNs in computer vision, did really well on the ImageNet ILSVRC challenge in 2012 showing significant gains in performance.
The network has similar architecture to LeNet but is deeper and bigger and features convolutional layers stacked on top of each other. Reference: A. Krizhevsky, et. al. "ImageNet Classification with Deep Convolutional
Neural Networks", NIPS, 2012
VGG16 demonstrated the importance of depth as a critical component to good performance, it was a runner-up in ILSVRC 2014. The architecture consists of a stacked convolutional and max pooling layers with increasing depth and
it uses a large number of parameters due to the final fully connected layers. Reference: K. Simonyan, "Very Deep Convolutional Networks for Large-Scale Image Recognition", ICLR, 2015
Inception original version called GoogLeNet from Google won the ILSVRC 2014 challenge the architecture consists of inception modules that dramatically reduce the number of parameters, it uses multi-scale 3x3, 5x5 convolutional filters including 1x1 convolutions for dimensionality reduction. Reference: C. Szegedy et al., "Going Deeper with Convolutions", CVPR, 2015
ResNet50 residual network was the winner of ILSVRC 2015, it introduces skip connections for easier training that enable very deep architectures
and makes use of batch normalization. Reference: K. He et al., "Deep Residual Learning for Image Recognition", CoRR, 2015


How to check for "statistical significance" of categorical feature in black box models

First, you have to encode the feature. Models only take numerical features.

Then assessing the solution:

You can either see the feature importance of the model
Or use an XAI tool that will help you to understand the predictions. I normally use SHAP (SHapley Additive exPlanations):is a game theoretic approach to explain the output of any machine learning model. It connects optimal credit allocation with local explanations using the classic Shapley values from game theory and their related extensions.

https://github.com/slundberg/shap


splitting of training examples into the mini batch: what to do with the rest tiny mini-batch?

A few basic approaches come to mind. It depends on your context and intention which might work best for you. Here are some ideas assuming that working with a partial batch is not an option:

Drop the remainder. This might make more sense when you have many complete batches, and you consider the set of examples to be enough.
Complement the final batch with duplicated samples from the other batches. E.g. randomly get one example from each batch.
If something like that is feasible at all, generate additional (possibly synthetic) examples to complete the final batch.


Single vs Multiple deep learning networks for multi-label classification?

This kind of setup is normally addressed with a single neural network that has an output sigmoid layer of $N$ units (where $N$ is the number of possible fixes).

Having combinations of fixes represented with their own label does not make much sense; instead, let the different labels take the expected value: if 2 fixes are fine, let both expected outputs be 1.

Having $N$ networks each one generating a single fix-nofix is equivalent but does not benefit from the potential gains of the multitask learning obtained when training everything together and obtaining a shared internal representation.


Would it be okay to stop training my neural network?

You should keep training.

In many scenarios ~1% decrease in validation loss a bid deal in itself. However, looking at the trend it looks like you validation loss is set to decrease by more than ~1%, if you let it train form, say, 20 more epochs. The decreases will get smaller and smaller, but they will accumulate.

Generally, you should continue training if your validation loss is decreasing.


What mu and sigma vector really mean in VAE?

In a variational antoencoder (VAE), the output of the encoder, $\mu$ and $\sigma$, is a deterministic function of input data $x$ (you put $x$ into your encoder neural network, and it generates $\mu$ and $\sigma$, there is nothing random here).
Then the hidden representaition $z$ (posterior) is sampled from the Guassian distribution parameterized by $\mu$ and $\sigma$. So notice that we don't talk about what 's the distribution of $\mu$ and $\sigma$ (since they are deterministic based on $x$, not random variables), but we do talk about the distribution of $z$.
"why do we want to feed forward latent matrix with Gaussian distribution?"
To be specific, we feed a Gaussian distributed variable $z$ into the decoder network. This is the basic model assumption of VAE regarding how samples are generated. Notice that one of the main difference between VAE and standard autoencoder is that VAE is a generative model, in the sence that you can randomly generate samples, instead of using a input $x$ to reconstruct something similar to $x$.
If we imagine a VAE trained on the MNIST hand written digits dataset, we want to randomly sample $z$~$p(z)=\mathcal{N}(0,1)$, and feed this $z$ into the decoder network which give us the output distribution (which is again a Gaussian), and sampling from that output distribution results in an image of a digit.
"Why compare a distribution of Mu and Sigma matrices? "
We do not compare the distribution of $\mu$ and $\sigma$. As explained above, we are talking about the Gaussian distributed hidden variable $z$ (posterior $q_{\theta}(z|x)$) parameterized by $\theta=(\mu,\sigma)$. We compare the posterior distribution of $z$ with its prior ($p(z)=\mathcal{N}(0,1)$) and the difference (measured by their KL-divergence) is used as a regularization term in the loss function.
The reason for that is we want the latent representation $z$ to be close to its prior (standard Gaussian). This is relevant to how we generate samples from the VAE (we first sample from the standard Gaussian prior). If we encode a $x$ to somewhere very far from this prior, it will be unlikely that we can generate such samples.
For example, a well trained posterior (first two dimensions) for MNIST looks like this (photo from here)

Notice that the posterior for each digit (e.g. dark blue for digit "0") is distributed like a Gaussian and not far from $\mathcal{N}(0,1)$, but difference digits do occupy different area.
Many online tutorials (e.g. here or here) have more detailed explanation about the concepts in VAE. I would suggest to read them.


How to scale numbers according to proportions

Seems like you want to show both percentage and magnitude at the same time.

I would then suggest that you plot the two values for comparison.

You could plot all in one chart with two axes:

or split it into two charts:

both visualizations make clear that Bananas have the largest increase, while Apples is the fruit that sells the most.


Keras loading images in incorrect format

This is caused due to the img_to_array method which converts the image to a float32 array.

x = np.asarray(img, dtype=K.floatx())

Matplotlib interprets NxMx3 uint8 array as a standard image (0..255 components) in which case there is no preprocessing. Otherwise the pixels are multiplied by 255(without checking the range) and then cast into uint8, which I guess leads to this behaviour.

Check this:https://stackoverflow.com/questions/39925420/bizzare-matplotlib-behaviour-in-displaying-images-cast-as-floats

To answer the second part of your question, I guess the imagenet competitors used OpenCV to load images in BGR format to train vgg16 and hence the pretrained weights work well with images opened in BGR format.


Is there any standard or normal range for the amount of LSTM loss function?

Although your loss function is an indication of how well the model is training, usually one uses other more intuitive metrics to assess how good the model is.

If you are looking to a classification problem, your loss function is most probably the cross entropy. In what regards the loss function what matters is to understand its behaviour during training, more than its value.

A loss function that reduces its value during training is an indication that the model is effectively training. The loss function will, at some point, start reducing its value, and that means that the model has arrived to a minimum. One need to understand also the interaction of the loss function in training and validation set, and how to detect things like overfitting. If you are not aware of that, there is plenty of literature about the topic.

To know how good a model is, I would use other metrics that give a better indication and intuition. For example, in a classification problem, pne can look to things like Precision, Recall, Accuracy (if the classes are not very unbalance) or even ROC AUC. If it is a regression problem, maybe you are more interested in Mean or Median Absolute Percentage Error (MdAPE or MAPE).


class_weight is used in the process of model-training to train(=fit) a better model (let us call it clf).

Your question is is about choosing the most sure predictions.
You just need to predict the probability (for binary classification this will be the probability of positive class)

y_test_predicted_probability = clf.predict_proba(X_test)

and then choose 10 points with the highest y_test_predicted_probability

#some code to do this
top_picks_indexes = y_test_predicted_probability[:,1].argsort()[-10:] # chose top 10 probabilities for class = 1

# create a vector, Y_top_picks, with all zeros except ones for the selected top probabilities
Y_top_picks = np.zeros(len(X_test))
Y_top_picks[top_picks_indexes] = 1


What is the immediate reward in value iteration?

what is $R_a(s,s')$ ?

In this case, it appears to represent the expected immediate reward received when taking action $a$ and transitioning from state $s$ to state $s'$. It is written this way so it could be implemented as a series of square matrices, one for each action. Those matrices might of course be very sparse if only a few transitions are possible, but it's a nice generic form for describing MDPs.

Notation varies between different RL tutorials, so do take care when looking at other sources.

  The problem I'm having is that terminal states have, by default, $V(s_T) = R(s_T)$ (some terminal reward).

No, $V(s_T) = 0$ by definition. The value of a state, is the expected discounted sum of future rewards. A terminal state has no future rewards, thus its value is always $0$.

The "terminal reward" in your system occurs on the transition $s \rightarrow s_T$, so its expected value should be represented by $R_a(s,s_T)$

If the terminal state is some goal state, or a bad exit from an episode, and it doesn't matter how you arrive there, then in the formulation you have given, you still represent it as $R_a(s,s_T)$, just that the values of $a$ and $s$ don't matter for your case. In general they might matter, for other MDPs.

  So the only conclusion I seem to be able to get is that in my case, $R_a(s,s') = R(s)$.. is this correct?

Not in general. However, in some problems this might be a reasonable simplification. If you make that simplification, then you should take care when having a value associated with a single state to be consistent about whether the reward is for entering a particular state, or exiting it. There cannot be a reward for "being in a state", the closest you can get to that is granting a consistent reward when entering a state (ignoring the previous state and action that caused the transition).

It appears here that you have suggested here that you want a reward for exiting a particular state - ignoring which action is taken in it, or what the next state is. I don't know the details of your MDP, so cannot say whether this would work for you. Maybe not, given the rest of your question.

If you are asking in general, then you have the answer $R_a(s,s') \ne R(s)$

If you are asking in order to work on a specific MDP, I suggest take another look at your original problem, with the extra information in this answer.


How can I dynamically distinguish between categorical data and numerical data?

I'm not aware of a foolproof way to do this. Here's one idea off the top of my head:

Treat values as categorical by default.
Check for various attributes of the data that would imply it is actually continuous. Weight these attributes based on how likely they are to correlate with continuous data. Here are some possible examples:

Values are integers: +.7
Values are floats: +.8
Values are normally distributed: +.3
Values contain a relatively small number of unique values: +.3
Values aren't all the same number of characters: +.1
Values don't contain leading zeros: +.1

Treat any columns that sum to greater than 1 as being numerical. Adjust the factors and weights based on testing against different data sets to suit your needs. You could even build and train a separate machine learning algorithm just to do this.


For binary classification, which is best Random Forest or Neural networks?

There is no best model for binary classification. Instead it is better to focus on what kind of data you have and let that guide your choice of model.

Decision tree based models usually perform very well when dealing with regular tabular data of categorical and numerical features. Mainly random forest and gradient boosting (LightGBM and XGBoost). If you have very few samples a SVM can also be a good option.

Neural networks shine on more unstructured data like images and text. Time series are a bit more nuanced and often multiple models can work well.


Equations in "Intoduction to RL": What is the meaning and difference between E, and E with subscript?

In general, the expectation is taken with respect to some random variable X. Often, when dealing with a single random variable, it can be implicitly inferred over which random variable it is being integrated and thus writing $\mathbb{E}$ suffices. However, when dealing with multiple random variables, this is not the case anymore. Then, a subscript denotes with respect to which random variable the expectation is taken.

However, the subscript can also denote on which random variable to condition on. This seems to me to be the case on the page you are referring to. Here, $\mathbb{E}_{\pi'}$ means that you are conditioning on actions which are distributed according to $\pi'$.

(Just as a side note: sometimes you might see something along these lines of: $\mathbb{E}_{a\sim \pi'}$, the subscript here is the actual notation for "$a$ distributed according to $\pi'$")

For a more technical answer, have a look at this question on Cross Validated.


try RSelenium. with phantomjs since the date is requested and filled in by ajax calls. so any static web scraping tools wont work.

I managed to get the list on the first page.

http://cran.r-project.org/web/packages/RSelenium/vignettes/RSelenium-headless.html

sample of what i managed to pull.

remove( mopub, m, run , rx, x , first1)
library(RSelenium)
pjs<- phantom( pjs_cmd="C:/Users/bhavin.patel/Downloads/phantomjs-2.0.0-    windows/bin/phantomjs.exe")
Sys.sleep(5)
remDr <- remoteDriver( browserName = 'PhantomJS')
dsurl <- "http://en.journals.sid.ir/SearchPaper.aspxstr=project%20management"
remDr$open()
remDr$navigate(dsurl)
allt3 <-remDr$findElements('id', 'Table3')
lapply( allt3 , FUN=function(dst){ dst$getElementText(); })

[[1]]
[[1]][[1]]
[1] " 1 :   EFFECTIVE FACTORS ON RURAL PEOPLE’S NON-PARTICIPATION OF     MAHABAD’S DAM CATCHMENT IN WATERSHED MANAGEMENT PROJECTS\nAuthor(s): RASOULIAZAR SOLEIMAN*,FEALY SAEID\nJournal: INTERNATIONAL JOURNAL OF AGRICULTURAL MANAGEMENT AND DEVELOPMENT (IJAMAD)\nNumber: MARCH 2015 , Volume  5 , Number  1 ; Page(s) 19 To 26.\nKeyword(s): NON-PARTICIPATION, CATCHMENT, WATERSHED MANAGEMENT, MAHABAD TOWNSHIP, IRAN\nReference(s):  (0)      Citation(s):  (0) FullText:"


Make a random forest estimator the exact same of a decision tree

You need to set bootstrap=False in the random forest to disable the subsampling.  (I originally commented because I expected there to be more impediments [in addition to your already-coded random_states and max_features=None], but I guess there aren't any!)

You probably don't want to do this in general; by stripping out all the randomness so that the first tree is the same as the DecisionTreeClassifier, you'll end up with all the trees being the same, and the random forest loses its usefulness.


How to make ML model publicly available?

The easier solution in my opinion consist to build your application using a framework like Streamlit and deploy it using docker.
Docker let you encapsulate your code and all needed package in a container, which you could think as an virtual machine (but its not); it will run on all OS since it use your configuration.
Of course there is plenty of other way but its my personnal favorite.


How to maximize recall score for specific label in multiclass classification?

I would highly suggest doing some deeper Analysis on why the ("-") class is beeing predicted. Meaning what do you predict for class ("-") when no the correct class. Analysing this you could see that you are ordering class ("-") as class "++". Than you can look at the Input data and feature engineer Features that discriminate between These two class. Forcing you model to do this indirectly is a fair try, but model cant learn whats not to be found in the data. With feature Engineering you can enhance the data so that discrimination is better.


Why do we need data scientists if there are websites like this?

Without having a complete knowledge of the features on that website I would say:

Data visualization is only one part on data scientist (ds) pipeline from data understanding thought model validation and model production as per CRISP-DM, and that site seems to be focused on that.

Introductory courses on data science most of the time work with "ready to use" data frames (Iris, Titanic) that do not reflect the real way in which data must be preprocessed and aggregated to the correct level (ex. you may have accounts level data that needs to be aggregated to user's level to predict user's default) also, in many cases you have multiple sources of data stored in a variety of places like relation tables or non structured information that will need to be queried and joined beforehand.

The latter role is commonly attributed as a "data engineer", however in small to medium projects the line between the two roles are less discernible. It is important to note, however, that this line should be fuzzy, as more often than not choices made by data scientists require work from a data engineering perspective, - as other answers below have touched more upon.
And this gives a wrong impression of simplicity on the task.
Besides, the above step would need for you to have at least basic understanding of the data you are working on in order to create "meaningful" features that otherwise will end just by aggregating by min, max and average (sometimes useful but not sufficient)

Finally, a trained model in a notebook would hardly being helpful if you cannot use that model in an completely deployed way (having a AWS lambda for example)


Getting Value Error while training a model for binary classification

The problem is the shape of y_train. You're defining an output layer of 2 neurons, and only passing labels with a shape of (n_samples, 1). You need to convert this to shape (n_samples, num_labels) and can use keras.utils.to_categorical to do so easily.

The number of neurons in your output layer has to match the second dimension of the label array, so you transform it from one dimensional like this:

[0,1,0,0,0,1]

To two dimensional like this:

[[1,0],
[0,1],
[1,0],
[1,0],
[1,0],
[0,1]]

When you make your predictions, softmax activation will give you a 'probability' for each class like so:

[[0.97,0.03],
[0.85,0.15],
...
]]


what is the one hot encoding for cancer data classification

Both ways would work equally, but the way you see in the github repo is more standard.

The standard way of converting a integer label $y_i$ (from 0 to K-1) into a one-hot vector encoding is by creating a all-zero vector of length K, and set the element indexed by $y_i$ to be 1, i.e.

label_one_hot = np.zeros(k)
label_one_hot[label] = 1


The most standard implementation uses PyTorch's LayerNorm which applies Layer Normalization over a mini-batch of inputs. The mean and standard-deviation are calculated separately over the last certain number dimensions which have to be of the shape specified by normalized_shape argument. Most often normalized_shape is the token embedding size.
The paper "On Layer Normalization in the Transformer Architecture" goes into great detail about the topic. The paper proposes "the layer normalization plays a crucial role in controlling the gradient scales." Better behaved gradients help with training.


It depends what it's going to be used for, but in general it can make sense to use TF-IDF with short sentence. The main difference with the more standard case of long sentences is that TF (Term Frequency) won't play any role since the frequency will almost always be 1. IDF can still be useful though, assuming it's relevant to assign more weight to rare words than to frequent words.

However the problem is that comparisons such as cosine are often going to be zero, since there will be little chance of words in common between two sentences.


How to compute the maximum likelihood hypothesis?

We can define a function $f(D,h) =P(D|h)$, that is it is a function of both $h$ and $D$ and we want to maximize it.

In the event that $D$ is already fixed to be $\hat{D}$, then the goal should be to maximize the function $$g(h) = f(\hat{D}, h) = P(\hat{D}|h).$$

We can plot the function $g$, here $g$ is a function of $h$ rather than $D$.


Modifying a distribution by adding in samples incrementally

Calculation of standard deviation on the fly is possible (turn to our brothers at math.stackexchange):

https://math.stackexchange.com/questions/198336/how-to-calculate-standard-deviation-with-streaming-inputs

It is easier to keep track of the variance and only take the square root to calculate the stdev when you really need it.

And the mean is even easier, don't overthink it. If you have 4 samples with a mean of 4 and 3 samples with a mean of 3, this totals up to 25 in 7 samples.

So

$$\mu = \frac{\mu_1 N_1 + \mu_2 N_2 + \mu_3 N_3}{N_1 + N_2 + N_3}$$

and

$$\sigma^2 = \frac{\sigma_1^2 N_1 + \sigma_2^2 N_2 + \sigma_3^2 N_3}{(N_1 + N_2 + N_3)^2}$$


Can I apply survival analysis to predict if a user will revisit the website?

If you want to use survival analysis (which can be more flexible and insightful), I'd recommend this package and this great tutorial. Speaking shortly, as a result, you'll get "probability of being alive" for each customer.

If you want to use logistic regression I think it's trickier. Why I think so - Like any other churn problem, it's hard to define it properly. The definition depends on your task and where the model outcome will be used. Let's say churn is a particular amount of inactivity, e.g. 30 days. You can do an initial analysis of how to find this number. Just pick a particular date (you can do it multiple times) and check % of people who made next transaction. Important thing is - your time period from both sides should be same for all users:

if it's a new users he's not able to be inactive for a long period, right?
if it's a last date in the dataset (e.g. yesterday) - not all users are not able to perform transaction within 1 day. And you'll get high churn rates. So be attentive to dates.

So you need to understand from your data - which inactivity is "normal" for the average user and define it as N. After that you can label users binary like "if inactivity > N then "churned (1)" else "not churned (0)"". And you can use this label with any classification model.


It makes no sense to re-order inputs in the general case because the order might matter. In your example it does not; you can shuffle the columns as long as the corresponding outputs remain the same.

I've seen the input reversed, which is a less arbitrary transformation than the one you cite, to improve prediction in sequence-to-sequence models, though that's not set in stone, either.


When we should use binning to redure noise? or How we find out we have noise?

You can plot your datapoints and see if there are many outliers to the dataset. Making frequncy plots and heatmaps helps a lot in this case.Binning helps to remove smaller erroneous data points. There are other methods of reducing noise in a dataset like applying IQR.

If you want to reduce noise, apply Inter Quartile Range after applying binning only if you have a large dataset. : https://www.wikihow.com/Find-the-IQR


No correlation found between dataset features

Think about the data generating process: For taxi fares, THE relevant thing is distance/time of the trip. This is by definition the case for most taxi services (Uber might be different). Maybe time of the day plays a role as well, e.g. when fares during day/night are different.

In a simple case you might not even need a statistic solution, viz. if the problem is deterministic, meaning fare is a simple linear function of distance.

You can use (for example) a linear regression to predict fare. In this case you could have a model:

fare = b0 + b1 * dist + error.

In this model, there is nothing wrong with having only one predictor (this is a univariate linear regression).

BTW: If you predict a continuous variable, you do regression. If you predict classes (like yes/no) you do classification.

If you have many variables with only weak correlation to y, you can also use regulation (by L1 norm), in linear regression this is called Lasso, to „shrink“ features which are irrelevant.

P.S. Plot price (y) against distance (x). You will likely find close alignment of both.


CNN for subsets of a dataset - how to tune hyperparameters

It depends on how you choose the subset of the dataset. One required assumption about data for any Machine Learning model to generalize well on unseen data is that the data must come from the same statistical distribution.

So, if you select the subset at (uniformly) random, it's quite sure that your model will continue to perform well. Otherwise, I think it's better to fine-tune the big model on the subset for better performance.


Stuck implementing k means for big and small dogs - dodgy results

The centroid is likely correct, you have a display error

The line

plt.scatter(centroid2[0][0], centroid1[0][1], color = "green")

should be

plt.scatter(centroid2[0][0], centroid2[0][1], color = "green")

That's just one of those things that happens when you implement an algorithm from scratch in order to learn it . . . I bet you've spent hours looking at the top part of your script :-)


Details of the k-means++ algorithm that is used to seed k-means

You want to spread out the initial clusters, the article argues that this gives (on average) faster convergence and lower error. This means that points with a higher distance to the closest center are likely to be better candidates for initial clusters. So the probability distribution for a new initial clustering point is:

$\mathbb{P}(C = c) = \frac{D(c)^2}{\sum_{\forall x\in X}D(x)^2}$

Where $X$ is the collection of all candidate points and $C$ the chosen point distribution.

This means the higher the distance, the higher the chance it will be picked as initial point.


How to code a simple forward propagation of recurrent neural networks?

In other words, what does the forward pass of a RNN look like. You read about using the inputs plus values from the previous node (here it will be prev_s)
First initialise the weights, than perform the foreward pass. I highlighted what you was looking for.

U = np.random.uniform(0, 1, (hidden_dim, T))
W = np.random.uniform(0, 1, (hidden_dim, hidden_dim))
V = np.random.uniform(0, 1, (output_dim, hidden_dim))

 for i in range(Y.shape[0]):
        x, y = X[i], Y[i]

        layers = []
        prev_s = np.zeros((hidden_dim, 1))
        dU = np.zeros(U.shape)
        dV = np.zeros(V.shape)
        dW = np.zeros(W.shape)

        dU_t = np.zeros(U.shape)
        dV_t = np.zeros(V.shape)
        dW_t = np.zeros(W.shape)

        dU_i = np.zeros(U.shape)
        dW_i = np.zeros(W.shape)

        # forward pass
        for t in range(T):
            new_input = np.zeros(x.shape)
            new_input[t] = x[t]
            mulu = np.dot(U, new_input)
            mulw = np.dot(W, prev_s)
            add = mulw + mulu
            s = sigmoid(add)
            ***mulv = np.dot(V, s)***
            layers.append({'s':s, 'prev_s':prev_s})
            prev_s = s

So the '*  *' area can be roughly translated: mulv = np.dot(V, s) are the weights multiplied with the current state. (same as before, s==input_vector) but the difference is that the s will be calculated with weights from previous output and current input i.e.

mulu = np.dot(U, new_input)
mulw = np.dot(W, prev_s)
add = mulw + mulu
s = sigmoid(add)

Thats why we have 3 initial weights in the first place.


Analyse this code

for value in data['instructions'].iteritems(): # Here we iter by line int the instruction colunm
    myList = list(value[1]); # Here you get 1 of the values and assing to your list

myList

opcodes = [instruction.split()[0] for instruction in myList]

len(opcodes)

your "myList" variable only holds 1 of the lines (the last one) of your data, which is probably the number of instructions in that line of your dataframe

While the 30k (I don't know how panda works here) is probably the number of lines in that column (or very unlikely the sum of all instructions in every line)


Intuitive explanation of Adversarial machine learning

Consider a game being played between two people, for simplicity, we'll assume this game is distinguishing a true picture of a panda vs. a fake picture of a panda. The first player will take the painting and show it to the second player, if the second player guesses whether it is a fake correctly, they receive a reward, if not they do not. Both players are playing the game with the goal of maximizing their reward.
To go a little deeper but make it slightly more relevant to the context of Adversarial ML. We can further assume that both players start from 0 knowledge of pandas. You might imagine that player 1 just throws random colors at a canvas and tries to convince player 2, and player 2 just randomly guesses, slowly building their intuition for what a Panda is.
After several hours/days/years, we might find that both players are extremely skilled and drawing pandas and identifying fake pandas respectively.
This is really what adversarial ML is about in a nutshell. The goal is to have two agents with competing rewards, where their optimal solution is at some form of a mixed strategy Nash equilibrium.


Where can I find an algorithm for human activity classification using thigh and shank sensors?

What you want to do can be called activity recognition or time-series classification. Almost all classification algorithms can be made to do this. A popular choice is to use a type of RNN (recurrent neural network) called a LSTM (long short-term memory). It is made to work with sequences and handles time series quite well.

Here are some relevant readings that might get you started:

Activity recognition tutorial using RNN models

Similar tutorial using various models


I've put together some example tensorflow code to help explain (the full, working code is in this gist). This code implements the capsule network from the first part of section 2 in the paper you linked:

N_REC_UNITS = 10
N_GEN_UNITS = 20
N_CAPSULES = 30

# input placeholders
img_input_flat = tf.placeholder(tf.float32, shape=(None, 784))
d_xy = tf.placeholder(tf.float32, shape=(None, 2))

# translate the image according to d_xy
img_input = tf.reshape(img_input_flat, (-1, 28, 28, 1))
trans_img = image.translate(img_input, d_xy)
flat_img = tf.layers.flatten(trans_img)

capsule_img_list = []

# build several capsules and store the generated output in a list
for i in range(N_CAPSULES):
    # hidden recognition layer
    h_rec = tf.layers.dense(flat_img, N_REC_UNITS, activation=tf.nn.relu)
    # inferred xy values
    xy = tf.layers.dense(h_rec, 2) + d_xy
    # inferred probability of feature
    p = tf.layers.dense(h_rec, 1, activation=tf.nn.sigmoid)
    # hidden generative layer
    h_gen = tf.layers.dense(xy, N_GEN_UNITS, activation=tf.nn.relu)
    # the flattened generated image
    cap_img = p*tf.layers.dense(h_gen, 784, activation=tf.nn.relu)

    capsule_img_list.append(cap_img)

# combine the generated images
gen_img_stack = tf.stack(capsule_img_list, axis=1)
gen_img = tf.reduce_sum(gen_img_stack, axis=1)

  Does anyone know how the mapping between input pixels to capsules should work?

This depends on the network structure. For the first experiment in that paper (and the code above), each capsule has a receptive field that includes the entire input image. That's the simplest arrangement. In that case, it's a fully-connected layer between the input image and the first hidden layer in each capsule.

Alternatively, the capsule receptive fields can be arranged more like CNN kernels with strides, as in the later experiments in that paper.

  What exactly should be happening in the recognition units?

The recognition units are an internal representation that each capsule has. Each capsule uses this internal representation to calculate p, the probability that the capsule's feature is present, and xy, the inferred translation values. Figure 2 in that paper is a check to make sure the network is learning to use xy correctly (it is).

  How it should be trained? Is it just standard back prop between every connection?

Specifically, you should train it as an autoencoder, using a loss that enforces similarity between the generated output and the original. Mean square error works well here. Aside from that, yes, you'll need to propagate the gradient descent with backprop.

loss = tf.losses.mean_squared_error(img_input_flat, gen_img)
train_step = tf.train.GradientDescentOptimizer(0.1).minimize(loss)


How to calculate the mini-batch memory impact when training deep learning models?

@StatsSorceress TL;DR:

I'm going through this activity to see if I can calculate the memory required myself:

Activations: 532,752 * 2 * 4 / (1024^2) = 4.06 MB

Parameters: 19,072,984 * 4 / (1024^2) * 3 = 218.27 MB

Miscellaneous: 128 * 9,000 * 4 / (1024^2) = 4.39 MB

Total Memory: (4.06 * 128) + 218.27 + 4.39 = 742.34 MB

(Someone please correct me on this if I'm wrong. FYI, you already multiplied miscellaneous by 128, so that's why I didn't multiply it by 128 above)

I would point you to this article and the corresponding video. They helped me to understand what is going on a lot better.

NOTE: The memory required to use a network for predictions is far less than that required for training for two reasons:

When predicting, we only send an image forward through the network and not backward (so we don't multiply memory X 3; see below)
There's one prediction per image (so we don't need to multiply the memory required for one image by a batch size because we don't use batches in prediction).

Process (Memory to Train)

Calculate the memory required to train on one image
Multiply this number by the number of images in your batch

(REMEMBER: Mini-batching says we take a subset of our data, compute the gradients and errors for each image in the subset, then average these and step forward in the direction of the average. For convnets, weights and biases are shared, but the number of activations is mutliplied by the number of images in the batch.).

STEP 1: Memory for 1 Image

To train one image, you must reserve memory for:

Model Parameters:

The weights and biases at each layer, their gradients, and their momentum variables (if Adam, Adagrad, RMSProp, etc., optimizers are used)

To approximate the memory for this, calculate the memory required to store the weights and biases and multiply that by 3 (i.e. "by 3" because we're saying the amount of memory needed to store the weights and biases is (roughly) equal to that needed for the gradients and for the momentum variables)

EQUATIONS:

Convolutions:

weights(n) = depth(n) * (kernel_width * kernel_height) * depth(n-1)

biases(n) = depth(n)

Fully Connected (Dense) Layers:

weights(n) = outputs(n) * inputs(n)

biases(n) = outputs(n)

where n is the current layer and n-1 is the previous layer, and outputs are the number of outputs from the FC layer and inputs are the number of inputs to the FC layer (if the previous layer is not a fully-connected layer, the number of inputs is equal to the size of that layer flattened).

NOTE: The memory for the weights and biases alone, plus the memory for the activations for one image (see below), is the total amount of memory you need for predictions (excluding some overhead for memory for convolutions and some other things).

Activations (these are "Blobs" in Caffe):

(I'm using terms loosely here, bear with me)

Each convolution in a convolution layer produces "number of pixels in image" activations (i.e. you pass an image through a single convolution, you get a single feature map consisting of "m" activations, where "m" is the number of pixels from your image/input).

For fully-connected layers, the number of activations you produce is equal to the size of your output.

Convolutions:

activations(n) = image_width * image_height * image_num_channels

Fully Connected (Dense) Layers:

activations(n) = outputs(n)

Note that your input is really only an image at the beginning of the network. After convolutions, it turns into something else (feature maps). So really replace "image_width", "image_height", and "image_num_channels" with "input_width", "input_height", and "layer_depth" to be more precise. (It's just easier for me to think of this concept in terms of images.)

Since we also need to store the error for the activations at each layer (used in the backward pass), we multiply the number of activations by 2 to get the total number of entities we need to make room for in our storage space. The number of activations increases with the number of images in the batch, so you multiply this number by the batch size.

STEP 2: Memory to Train Batch

Sum the number of weights and biases (times 3) and the number of activations (times 2 times the batch size). Multiply this by 4, and you get the number of bytes required to train the batch. You can divide by 1024^2 to get the answer in GB.


Is probabilistic machine learning just the mathematical background of machine learning?

Knowledge in Probability and Statistics is absolutely important. It might feel you can do quite a bit without that and can get things work using the high level ML libraries that we have today. However if you are a serious practitioner you will figure out that beyond a point your knowledge on how these techniques work and are tuned is very hollow and that will automatically lead you into the more mathematically inclined books that give you the foundation knowledge.


Which deep learning network would be suitable for classifying this kind of text data?

You need character embeddings. I assume you are already familiar with word2vec technology. Its goal it to make a model "learn" the relative meaning of words, placing them into a highly dimensional space.
The same can be done with single characters, instead of whole words. The preprocessing steps you need will be a little bit different, but the embedding technique is the same. In that way, you can generate representations of characters, feed their sequences into some RNN model, and perform the final classification task.
Therefore, RNNs are perfectly suitable for this task. If you are working with tensorflow.keras you can simply tokenize characters, and feed them through an Embedding() layer that will do the job for you. An alternative to RNNs is 1D conv layers, that can do the job as an alternative to recurrent cells. That's up to your preference.


Use embeddings to find similarity between documents

There are several ways you can obtain document embeddings. If you want to obtain a vector of a document that is not part of the trained doc2vec model, gensim provides a method called infer_vector which allows to you map embeddings.

You can also use bert-as-service to generate sentence level embeddings. I would recommend using Google's Universal Sentence Encoder (USE) model to generate sentence embeddings if your goal is to find some sort of similarity between sentences or documents. There are ways you can combine sentence level embeddings to document level, the first step to try would be to take the mean, or you could generate sentence embeddings for a sliding window over the document and take the mean of that.

The reason I recommend USE over BERT is that, USE was trained specially for sentence similarity tasks whereas BERT, even though can be applied to any NLP task was original trained to predict words in a sentence or complete a sentence. You might find this link helpful, it draws a great comparison between USE and BERT, and why it is important to choose a model based on task.