Disclaimer,

You can use PyCM which is a python module in order to save your confusion matrix and its evaluation parameters in a html report.

You can use the following commands:

from pycm import *
y_actu = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]
y_pred = [2, 0, 2, 2, 0, 2, 1, 2, 2, 0, 2, 2]
cm = ConfusionMatrix(y_actu, y_pred)
cm.save_html(filename,color=(100,50,250))

and the result will be as the following:


Best Way to combine multiple datasets into one model

Why don't you give Multi-label classification a try.you can actually train your model to predict multiple labels for one image by training the model on a dataset that has images with multiple labels the only problem you might face here is to where to get the data from you can actually use this dataset for practice:
https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/data.

Another better option that you have is as you said in your question you want to train your model on multiple classes you can actually use Imagenet dataset it mas 1000 classes and more than a million images.
Note:Use tranfer learning to build your model and do watch lesson 1,2,3 from fast.ai part1 2019 course you will have all your answers. Jeremy Howard is a god!!!!


How can I identify the most predictive factors?

It basically lies within the fitted object.

model.fit(X, y)
importances = model.feature_importances_

The importances are in the same order as the columns of features ($X$).


Transforming negative correlated non linear variable to linear positive correlated variable

What's a BU? Business Unit?

Can you tell anything about the variable and target variable? Can you include the scatter plot?

It is still not clear if the problem is not in the domain knowledge/assumptions.

A business example:
Profit depends on sales, which is generally assumed to have a positive correlation. But after a threshold marginal sales cost exceeds profits (very expensive to acquire new customers), we can still have a (local) negative correlation. If you only have data in the 'marginal' segment, you can draw incorrect conclusions.


How would you encode missing pixels in image data?

You'd substitute a statistical aggregator, such as the median or mean, for the missing data. Calculating the aggregator for a neighborhood region would "smooth out" the missing pixels.
You wouldn't set it to -1 because lost pixels are effectively noise, which you want to suppress, not exaggerate.


I tried to explore some of the best available resources, which includes online courses (Free/Paid), Books etc.
Books

Big Data and Analytics (WIND)
Hadoop for Dummies
Big Data for Dummies
Hadoop: The Definitive Guide
Learning Spark: Lightning-Fast Big Data Analysis
MapReduce Design Patterns

Online Courses
These are some best platforms that provide lots of courses with rich content and hands-on labs. You can go from beginner to expert level, followed by intermediate. (courses will be - Free/Paid  )

Udemy - Big Data Analytics Courses
Coursera - Big Data Analytics Courses
edX - Big Data Analytics Courses
Udacity - Data Analysis

Other References
For more thoughts on that you might like to explore these websites-

Quora
Big Data - Made Simple
From Dev
KDnuggets

Update [BigData in R]
You can go through these references-

Wikipedia - Programming with Big Data in R
RStudio - Working with BigData in R
InfoWorld - Learn To crunch BigData with R

That's all from my side. Hope it helps. Cheers!


How to identify if there is a relationship between 5 categorical independent variables to a binary dependent variable?

One simple, tricky but probably best approach would be using a Decision Tree. After building a model, you can analyze the structure of the tree to identify the relation. Afterward, you can list the feature importance to see the effect of an independent variable on the dependent variable.
In python, you can use sklearn to achieve it. feature_importances_ will give you Gini's importance of the features and the plot_tree() function to see the structure of the tree. This article might also help to see how to do that.


No need for algorithms, or recommendation systems.  You have:

  For each user a have a bunch of features.

As long as they're numeric, or can be made numeric (e.g. aggregating the values or one-hot-encoding them), you already have distances.  What you may not have is the proper variance across the feature space, i.e. features are scaled in different orders of magnitude.

If you know the exact weight of the features in relation to user similarity you may try to tune (scale) the features by hand. Otherwise you can simply make every feature have mean 0 and standard deviation 1.  In other words, per feature subtract the mean from all points and divide by the current standard deviation. (sklearn has a StandardScaler that does exactly that.)

In the scaled dataset, from any point (user), you can just calculate euclidean distance to any other point.  And the closest the points the more similar the pair of user will be.  i.e. top $N$ similar users to a user are just the $N$ closest points.

Plain euclidean distance works in many cases.  If euclidean distance does not work for the problem at hand, then you can explore more complex possibilities: starting from manhattan distance, through minkowski distance (combination of euclidean and manhattan distances).


Scaling of streaming data must be based on your training set parameters. Your model has learnt based on that scaling, therefore, in order to assess its generalization power, you have to repeat the same scaling that your model learned from.


How to convert trained data (feature extracted) into a prediction model?

This question is insanely broad… you should better go for an API documentation/tutorial... Python is usually the preferred porgramming language and two major API are used: scikit-learn for machine learning and Keras/tensorflow for Deep learning… (but there are quite a few more)

I'll give you some links... You should firstly consult these kind of sources of information and coming here later on with more specific and detailed issues.

Scikit-learn examples:

https://scikit-learn.org/0.20/auto_examples/plot_anomaly_comparison.html (for your problem)

https://towardsdatascience.com/5-powerful-scikit-learn-models-e9b12375320d (general)
Really good blog with Machine/Deep Learning examples:

https://machinelearningmastery.com/blog/

https://machinelearningmastery.com/introduction-python-deep-learning-library-keras/


A Derivation in Combinatory Categorial Grammer

I guess I got it. It is simply:

(X/Y)(Z\(X/Y)) which Z.

In the example,

X = VP/PP, Y = NP, and Z = VP.


LDA vs Word2Vec vs Others for predicting recipients of a message

Why the detour with the topic?

You can just learn a linear SVM to directly predict the recipient and avoid all the difficulties from topic modeling. What if a mail isn't about any of the previous topics? how many topics are there?


How to go about training a NER model to extract book citations in free-form?

Interesting task :)
I think even with a good amount of training data it will be difficult for a regular NER model to perform well with new books titles and authors:

The book may contain persons names which are not authors.
The book titles are difficult to identify as such in general. For example "the Republic" might or might not be about the book, and if the only indication the model can use is the capitalization it's probably going to make some errors.

To be clear, I think it could work to some extent but it would probably make quite a lot of errors.
On the other hand you could obtain a database of books, for instance from Wikipedia (there might be better resources), and you could use this in two ways:

Directly identify the books/authors in the documents by simple string matching. I would imagine that even if the coverage of the resource is not perfect, this method would easily catch a majority of occurrences.
In case the above method is not sufficient, it provides you with some good training data from which you could train a NER model in order collect titles which don't exist in the database. Note that there might be issues due to the unknown books being labelled as negative in the training data, so ideally you would have to go manually through the training data and annotate the remaining cases.


shapes (127,1) and (13,) not aligned: 1 (dim 1) != 13 (dim 0)

The score method of the classifier object does not work the way you are trying it to. You need to directly give x_test as input and that it will calculate y_pred on its own and give you the result with y_test. So, you do not need to reshape and the correct syntax would be:

y = clf.score(x_test, y_test)

If for any reason you need to calculate the score using y_pred then take a look at sklearn.metrics module.


How to normalize the data correctly in spam dataset

Normalizing so that "all the observations have the same importance" is kinda ambiguous and ill-defined. In any case, it would be strongly advised to avoid re-inventing the wheel, and use one of the several scalers available out there (e.g. in the sklearn.preprocessing module).
Here is an example using MinMaxScaler, which will re-scale your data in [0, 1] column-wise:
import pandas as pd
df = pd.read_csv("spambase.data", header=None)
print(df.head())
# result:
     0     1     2    3     4     5   ...     52     53     54   55    56  57
0  0.00  0.64  0.64  0.0  0.32  0.00  ...  0.000  0.000  3.756   61   278   1
1  0.21  0.28  0.50  0.0  0.14  0.28  ...  0.180  0.048  5.114  101  1028   1
2  0.06  0.00  0.71  0.0  1.23  0.19  ...  0.184  0.010  9.821  485  2259   1
3  0.00  0.00  0.00  0.0  0.63  0.00  ...  0.000  0.000  3.537   40   191   1
4  0.00  0.00  0.00  0.0  0.63  0.00  ...  0.000  0.000  3.537   40   191   1

[5 rows x 58 columns]

from sklearn.preprocessing import MinMaxScaler
sc = MinMaxScaler() # define the scaler
df_scaled = pd.DataFrame(sc.fit_transform(df)) # fit & transform the data
print(df_scaled.head())
# result:
         0         1         2    3   ...        54        55        56   57
0  0.000000  0.044818  0.125490  0.0  ...  0.002502  0.006007  0.017487  1.0
1  0.046256  0.019608  0.098039  0.0  ...  0.003735  0.010012  0.064836  1.0
2  0.013216  0.000000  0.139216  0.0  ...  0.008008  0.048458  0.142551  1.0
3  0.000000  0.000000  0.000000  0.0  ...  0.002303  0.003905  0.011995  1.0
4  0.000000  0.000000  0.000000  0.0  ...  0.002303  0.003905  0.011995  1.0

[5 rows x 58 columns]

Keep in mind that normalization depends also from your choice of a model to use: it is practically necessary for neural networks and k-nn (and for k-means clustering), but is is completely redundant for decision trees and tree-ensemble models (Random Forest, GBM etc).


Trying to figure out which book to tackle first

I assume it will probably be more beneficial for you to focus on getting a sample of the different techniques in use rather than learning the theoretical frameworks, probability bounds, and so on. In that case, I'd say the following sections are most useful to you:

Elements of Statistical Learning. Chapters 1–4 should give you a good flavour of 'traditional' classification and regression techniques such as nearest neighbours, linear regression, logistic regression, etc. Chapter 7 is also recommended by the authors as essential as it goes through the principles of selecting a model: these concepts will apply no matter what method you use.
From there you could sample what you were interested in: chapter 11 on neural networks might be interesting, although you might find a more modern treatment preferable. I like Deep Learning by Goodfellow, Bengio and Courville, which you could browse through and choose relevant chapters.
Support vector machines (chapter 12) are also pretty popular and worth being familiar with, and in the second edition you might read chapter 15 on random forests, too. Chapter 10 on boosting might be fun, too: XGBoost and similar often win machine learning competitions, so knowing the background would be helpful.

Understanding Machine Learning. Part I has a rather theoretical slant; I suspect many practitioners don't know much about PAC learning and VC dimension, although they are interesting if you have the time. In Part II you could review any methods you don't see in ESL, and take a look at the unsupervised learning techniques in Part III. Part IV is again theoretical and probably not as relevant.


Classification vs Regression Algorithms - Should exists algorithms only for Classification and/or Regression

Well you can think of a regression as being a classification with a very large number of ordinal classes, therefore algorithms can be used for both.
Indeed you can build a regression with a random forest, and a neural network can be used both as a regressor or a classifier depending on the activation function of its output layer.


I really like this article because it gives an overview of common approaches to churn prediction as well as a new state-of-the-art method.  There is also code available. However, the state-of-the-art method uses a custom loss function built on top of recurrent neural nets, which is a complex place to dive in.


Hope somebody will safe lots of hours:

def backward_propagation(self, x, y, cache):
    # T - the length of the sequence
    T = len(y)
    # perform forward propagation
    cache = self.forward_propagation(x)

    #...

    # delta for output layer
    dy = cache['y'].copy()
    dy[np.arange(len(y)), y] -= 1.0 # softmax loss gradient
    # print("dy: ", dy)
    dhtmp = np.zeros((1, self.hidden_dim))
    dh_prev = np.zeros((1, self.hidden_dim))
    dctmp = np.zeros((1, self.hidden_dim))

    for t in np.arange(T)[::-1]:
        dV += np.outer(dy[t], h[t].T)
        dhtmp = self.V.T.dot(dy[t]) + dh_prev

        # add to gradients at each previous step
        do[t] = dhtmp * ct[t]
        dct[t] = dhtmp * o[t]

        dctmp += dct[t] * (1.0 - ct[t]**2)

        di[t] = dctmp * g[t]
        df[t] = dctmp * c[t-1]
        dg[t] = dctmp * i[t]

        # backprop activation functions
        diga[t] = di[t] * i[t] * (1.0 - i[t])
        dfga[t] = df[t] * f[t] * (1.0 - f[t])
        doga[t] = do[t] * o[t] * (1.0 - o[t])
        dgga[t] = dg[t] * (1.0 - g[t] ** 2)

        # backprop matrix multiply
        dWi += np.outer(diga[t], h[t-1])
        dWf += np.outer(dfga[t], h[t-1])
        dWo += np.outer(doga[t], h[t-1])
        dWg += np.outer(dgga[t], h[t-1])

        dUi[:, x[t]] += diga[t]
        dUf[:, x[t]] += dfga[t]
        dUo[:, x[t]] += doga[t]
        dUg[:, x[t]] += dgga[t]

        # update deltas for next step
        # here dh is accumulated as shared variable
        dh_prev = np.dot(self.Wi.T, diga[t])
        dh_prev += np.dot(self.Wf.T, dfga[t])
        dh_prev += np.dot(self.Wo.T, doga[t])
        dh_prev += np.dot(self.Wg.T, dgga[t])
        dctmp = dctmp * f[t]

    return [dV, dWi, dWf, dWo, dWg, dUi, dUf, dUo, dUg]


LSTM's for timeseries with additional regressors

You might find this link helpful
https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/


Multi-label classification model in python?

First of all you would need to encode your target columns.We can use sklearn.preprocessing.MultiLabelBinarizer here:

from sklearn.preprocessing import MultiLabelBinarizer
mlb = MultiLabelBinarizer()

X=df[list('ABCD')]
Y=pd.DataFrame(mlb.fit_transform(df[['sex','weight']].values), columns=mlb.classes_, index=df.index)

yields:

In [75]: Y
Out[75]:
    female  male  normal  obese  overweight  underweight
0        1     0       0      1           0            0
1        1     0       0      0           0            1
2        1     0       1      0           0            0
3        1     0       1      0           0            0
4        1     0       0      0           0            1
5        1     0       1      0           0            0
6        0     1       0      0           0            1
7        1     0       1      0           0            0
8        0     1       0      0           0            1
9        0     1       0      0           0            1
..     ...   ...     ...    ...         ...          ...
90       1     0       0      1           0            0
91       0     1       0      0           0            1
92       1     0       0      0           0            1
93       1     0       0      0           1            0
94       1     0       1      0           0            0
95       1     0       0      0           0            1
96       0     1       0      0           0            1
97       0     1       0      0           1            0
98       1     0       0      0           1            0
99       1     0       1      0           0            0

[100 rows x 6 columns]

now we can use one of the classifiers that support multi-label classification (see Support multilabel:)

Example:

from sklearn.neighbors import KNeighborsClassifier

knc = KNeighborsClassifier()

X_train, X_test, Y_train, Y_test = train_test_split(X, Y)

knc.fit(X_train, Y_train)
Y_pred = knc.predict(X_test)


How deep should ones linear algebra knowledge be before starting data science?

I think it truly depends on what you decide to specialize in. Data science is a very broad field, and you can actually work with data without knowing what eigenvalues and eigenvectors are. However, if you want to acquire a intermediate/advanced understanding of statistics or machine learning, you need at least an intermediate/advanced knowledge of linear algebra.

I suggest to take an introductory class on linear algebra on MOOC - just to have a more precise idea of what linear algebra is - and then study some other topics that you are interested in. Linear algebra is a useful tool, but it can be very boring, especially if you are an "applied" kind of guy. Moreover, I think that some concepts like eigenvalues or eigenvectors are easier to understand when seen in an applied context, e.g. principal component analysis.


Hierarchical Clustering and Variable Selection

PCA is only suitable for continuous variables, and sensitive to scaling. So don't use it here.

Instead, you meat want to look at classic information measures whether some attribute is "correlated" with the clusters. For example Gini, mutual information etc.

Similarly, the Elbow method is a bad idea. It never was a good idea in the first place because of scaling... But it's really only suitable for choosing the k in k-means if you do many random restarts and look at the best partitionings found. But even then: there is no mathematical definition of the "elbow". Most of the time, there is no elbow, and people just pick some random k this way because they don't consider this option...nit's probably widely misused. But it certainly must not be used with hierarchical clustering!


Training and cross validation error curves

I think what you're seeing is normal behaviour:

With only few samples (like 2000) it's easy for a model to (over)fit the data - but it doesn't generalize well. So you get high training accuracy, but the model might not work well with new data (i.e. low validation/test accuracy).

As you add more samples (like 9000) it becomes harder for the model to fit the data - so you get a lower training accuracy, but the model will work better with new data (i.e. validation/test accuracy starts to rise).

So:

As the training dataset increases, the training accuracy is supposed to decrease because more data is harder to fit well.
As the training dataset increases, the validation/test accuracy is supposed to increase as well since less overfitting means better generalization.

Andrew Ng has a video about learning curves. Note that he plots the error on the y axis, you have the accuracy on the y axis.. so the y axis is flipped.

Also take a look at the second half of the video. It explains high bias and high variance problems.

Your model seems to have high variance (due to the big "gap" between the two curves) - it's still too complex for the small amount of data you've got. Either getting more data or using a simpler model (or more regularization on the same model) might improve the results.


Anomaly Detection In Univariate Time Series Data Using ARIMA In Python With Updating

Have you considered using a quantile forecast as your ARIMA?

Quantile forecast is when, instead of predicting the average, you predict quantiles. In your case, you could predict for a given day the probability that an observation is below the 0.5%-percentile and the 99.5%-percentile. That range would define the 1% of "abnormal" values for the next days.

Recapitulating, usually in regression you find the average value by minimizing $\min_\beta \sum_i||X_i\cdot\beta -y_i||^2$.

You could find the median value by minimizing $\min_\beta \sum_i|X_i\cdot\beta -y_i|$.

You can also find any given quantile $\tau$ by minimizing $\min_\beta \sum_{i|y_i\geq X_i\cdot\beta}\tau|X_i\cdot\beta-y_i| + \sum_{i|y_i< X_i\cdot\beta}(1-\tau)|X_i\cdot\beta-y_i|$.

Of course, these minimizations problems become more and more complicated. You can only estimate your $\beta$ for this last one using ADMM or gradient descent.

I don't think you'll find python tools for this. You'll find plenty of packages in R though. For instance, see this page:

http://robjhyndman.com/hyndsight/quantile-forecasts-in-r/

One last piece of advise: if you need this forecasting to interoperate with other python code, my recommendation is to do it via CSV files. Just have your python code do the pre-processing and generate a CSV file, then R doing its thing and generating another CSV file, then use this file for the post-processing in python, etc. Some people try to use things like rpy2, but I find it to be overengineering.


Suppose bias as a threshold. Using threshold, your activation function moves across the $x$ axis which may get complicated. Consequently, people usually use the bias term and always centre the activation function which is the step function at zero. There is nothing wrong in both cases.


Random Forest application with 40+ Predictor Variables

The randomForest package supports various tasks using an existing randomForest object.

For instance, it offers the predict method which will perform prediction using a given trained forest and a given dataset. As an example, if you have a trained forest called mod and a dataset called my_data, predictions can be obtained by running predict(object=mod,newdata=my_data).

It's worth reading through the package documentation which gives good examples of using the functionality.

I would note that any new data you supply to a trained forest object should have the same encodings and transformations as the data used to train the forest with.


trade offs between number of features with its score

First of all, the Silhouette score comes always with visual inspection so be careful using it. The idea behind this score is having either Normally distributed clusters or having pretty compact clusters. If the intrinsic clusters within your data are not one of those, then this score is somehow meaningless.

Secondly, the question also needs a re-thinking. "Which features to include in the clustering?" ... well, all!

The characteristic of a dataset is more than just the combination of characteristics of each individual features. In these cases, usually Dimensionality Reduction algorithms are used, which reduce the dimensionality of data and extract/select informative features according to the entire feature set. Note that a feature that you might exclude, makes a meaningful partitioning in combination with some other features.

at the end I recommend having a look at K-Modes which is designed for clustering datasets with nominal attributes. Most probably helps you more than k-mans.

If you like to get some insight about Graph Clustering (which is also related to your problem and is one of typical solutions) you can comment here.

Good Luck!


Preprocessing and dropout in Autoencoders?

I'll go through your questions one by one:

  Should i normalize my numerical data values before feeding to any type of autoencoder? if they are int and float values still i have to normalize?

This is strongly suggested, for two reasons. First, if different variables are on different scales, weights distributions will be unequal. Larger scales will dominate smaller scales during application of gradient descent, and this would lead to many parameters to be undertrained, leading to sub-optimal results. Second, your layers have activation functions that are meant to "learn" non-linear patterns in your data. All the commonly used activation functions (Sigmoid, Tanh, all the ReLU family, you name it) tend to be non-linear only around zero. Normalizing your data helps Neural Networks to learn most from them.

  Which activation function should i use in autoencoder? Some article and research paper says , "sigmoid" and some says "relu" ?

This is more an art than a science, however all the activations form the ReLU family have been proved superior to other counterparts. I'd suggest you to go for some kind of ReLU basically always. Some are more fancier but computationally expensive, usually the rank is:   ELU > Leaky ReLU > ReLU

  Should i use dropout in each layer ? like if my artichare for autoencoder looks like

Use some amount of dropout, but not too much. Dropout is a regularization technique that helps you preventing overfitting. This technique "turns off" some neurons during training sampled randomly, in order to make all neurons specialize during training (it turns your Neural Network in an ensemble of Neural Networks). However, keep in mind that dropout also represents an information loss! For example, if you set a Dropout() layer with dropout probability of 0.5, you'd loose half of the information at that layer at each iteration! I suggest you to use it, but not at every layer, and be parsimonious with it.

  For text dataset , If i am using word2vec or any embedding to convert text into vector then i would have float values for each word , should i normalize that data too ?

No, those are learned automatically by the model, you don't have to worry about their internal values.


Data representation (NoSQL database?) for a medical study

I think you need to define the process and desired outcome a little more. It sounds like you need to:

Define what features you want.
Figure out how to extract those features.
Figure out how to store those features.
Pass that dataset to a ML model for training.

I would figure out what exactly you want for #3 in terms of the number and type of data elements and then choose a data storage method only then.

Unless you are going to be passing the unstructured documents directly to your model, you don't need MongoDB's capabilities. 10,000 records is small, but since you mentioned you want to calculate some aggregated statistics on your patient-level data, you could likely get by with something as simple as MySQL or SQLite.

Spark and Map/Reduce are actually competitors, with Spark stealing MR's spotlight lately. You might need one or the other for feature extraction but they're probably overkill for the rest of the stuff you described.


Function of functions and theirs arguments

I assume you know which distributions are possible once you are inside your function? I would probably just pass a parameter for the distribution name and then a set of parameters in a Python dictionary.

This method also allows for pretty easy testing, because you know that distributions expects only certain parameters, so you can also add small checks to confirm everything is available.

def my_distribution(dist_name, dist_params):
    if dist_name == "normal":
        assert ("mean" in params.keys()) and ("variance" in params.keys()), "Missing expected parameters for {} distribution".format(dist_name)
    # Perform some checks for other distributions as necessary...

    # perform your own steps...

Now you use it like this:

# assume these are provided by your earlier code
distr = "normal"
params = {"mean": 5.0, "variance": 2.0}

result = my_distribution(dist_name=distr, dist_params=params)

Edit:

Here is an example of a single function that can handle multiple distributions in a single call:

def my_distributions(dist_collection):
    # Perform some checks for other distributions as necessary...
    allowed_dists = ["normal", "uniform", "dirichlet", "rayleigh"]
    assert all(
        dist_name in allowed_dists for dist_name in dist_collection
    ), "Input contains disallowed distribution"

    # Do something for each distribution name with its parameters:
    for dist_name, dist_params in params.items():
        print(dist_name, dist_params)

Now the input needs to be specified a little differently:

all_distributions = {
    "normal": {"loc": 1, "scale": 2, "size": 100},
    "uniform": {"low": 0, "high": 1, "size": 100},
}

A single call can now work over many distributions:

my_distributions(all_distributions)

You could of course make classes that hold all configurations and checks, but I would argue you are starting to make things unnecessarily complicated, as Numpy does so much for you already with it's built-in distribution capabilities.


Alternatives to regression to decide weights in an expression

First, consider normalising your data, inputs and outputs. If it does not do the desired behaviour, you can have different options.

A simple solution is to add some higher order polynomials to the current features for each input pattern and use the simple linear regression. The fact about that is this that you can not be sure what features you would need due to the fact that you can not see your data. Consequently, if you don't find the acceptable answers you want you can use the current features the higher order polynomials that you are going to add to train an MLP.


What are the best activation functions for Binary text classification in neural networks?

Relu often works well, but ultimately you need to try. If you really have only one hidden layer, you might have a model which cannot properly learn.

I guess you use a sequential, dense model architecture? Try to add more hidden layers. Don‘t add too much capacity for a start. You could try something like 256, 8, 8, 1 with relu.

Dropout is good to prevent overfitting. It is not important for learning per se.

Just a remark for future posts: always include your basic model code if possible. So people can understand what you are really up to. Also a little data description is useful. Especially with text, preprocessing is really important. So the way you treated your data before it goes into the model is relevant.


machine learning algorithms for 2d data?

You can simply convert a MxN matrix (your image) into a MN dimensional vector. For instance if you have 10 images each of which 10x10 pixels, you can convert them to 10 vectors each of them containing 100 element (you can do it row-wise or column-wise, does not matter). Then putting all these vectors on each other constructs a nxd matrix where n is the number of images and d is the number of pixels in each image.

So far we just prepared data for analysis. Now let's go to the question:

I'm looking for a supervised learning algorithm that can take 2d data
for input and output.

What does that mean?

First- The dimension of input-output is not a property of a machine learning algorithm.

Second- By 2d, you mean a matrix? I'm afraid your data is not really 2d! If you mean a matrix, a MxN matrix is a data of M samples in N dimensional space. also those vectors you may produce from your images are in MN dimensional space.

Third- A supervised algorithm takes some teaching samples (called training) to learn a task. What are those teachers here? If you have a training data which contains some blurred and corresponding original images then you can try to learn the place of white pixels by exploring blurred area. My suggestion:

If a smoothing filter has been applied to your image, the blurred pixels most probably look like a gaussian. So first extract the gaussians.
As you already have the place of sharp whites correspond to those gaussians you can use the mean, location of the mean and also variance as features (inputs) and the place of sharp white pixels as the target (output).

The classifier could be a simple neural network (a MLP should work here) or a simple SVM.

The answer above can not be evaluated without more information about your data and a more specific description of the question (e.g. What are your images exactly? if you can post one of them here I can probably help more)


Adding to what @Super_John said, if adding continents as a Feature, then you can also probably have at-least 2 more features as well,

The Latitude
The Longitude

Also add another temporary column to indicate the Source (like $1$ to $1st$ df, $2$ to $2nd$ df etc), So that we can add Colors to the k-means

So now we can have a k-means Cluster to see whether values are overlapping or not...
(Trying to see it in an Unsupervised Way)

(The analogy is equivalent to the  fact that that you can cluster time(24  hours in a day) in a cyclic fashion , like plotting  $sin(x)$, $cos(X)$ and then trying to cluster them)

Have a look at this answer,
Features Selection, Extraction


Changing the batch size will not change the overall training time too much. Since with any batch size you are passing almost 80K images.
One(and the best) approach will be to use transfer learning.
If you have a compelling reason to do full training, you will need a GPU powered bigger hardware. Google Colab can be an option. There are many other options available
Before that you may try to gauge the model with a sample of ~5k images


Shapley values without intercept (or without `expected_value`)

This is very similar to fitting a linear regression and not including an intercept, and I think they will face similar issues.
To be very concrete, consider an example with $f(x)=1,\ E=1, \ \phi_1=1, \ \phi_2=-1$.
Then your scaling factor is undefined, trying to divide by zero.  Well OK, but you won't often get such exact numbers.  Let's tweak them to
$$f(x)=1,\ E=1, \ \phi_1=1.01, \ \phi_2=-1.$$
Now the scaling factor is $1+1/.01=101,$ so that $\hat{\phi}_1=102.01$ and $\hat{\phi}_2=-101$.  This isn't locally too bad, because it still indicates the relative importance of the two variables in making this prediction.  But if you want to compare this to other points $x$ (where say $f(x)=2,\ \phi_1=1,\ \phi_2=0$, so that $\hat{\phi}_1=2,\ \hat{\phi}_2=0$) things will look quite strange, and aggregating importances over your entire sample may be very misleading.
For the connection to linear regression and more, see this excellent description of shapley values.


What is the difference between Kafka Channel and Kafka Broker?

What is a Kafka Broker?
Kafka is a distributed streaming platform. It is run as a cluster on one or more nodes, and each node in the cluster is called a Kafka Broker.
What is a Kafka Channel?
A Kafka Channel is a type of Flume Channel. Flume is:

Apache Flume is a distributed, reliable, and available system for efficiently collecting, aggregating and moving large amounts of log data from many different sources to a centralized data store.
The use of Apache Flume is not only restricted to log data aggregation. Since data sources are customizable, Flume can be used to transport massive quantities of event data including but not limited to network traffic data, social-media-generated data, email messages and pretty much any data source possible.

A Channel is:

Channels are the repositories where the events are staged on a agent.

So hopefully, it is now clear that a Kafka Channel, is a Flume Channel that uses Kafka.


This is called (perfect) collinearity and happens e.g. when the reference category is included in the set of dummy variables. Use pd.get_dummies(..., drop_first=True) to avoid this.


converting column from a factor to date format YYYY-MM-DDThh:mm:ss.SSSZ

One way to do this is to use the PIPE operator to pass the column of the dataframe into your function, then assign it back to the dataframe.

I created a dataframe with one column holding your example date string five times:

> x <- "2014-04-06T18:42:05.823Z"
> DF <- as.data.frame(c(x, x, x, x, x), colnames="original")
> colnames(DF) <- "original"
> str(DF)
'data.frame':   5 obs. of  1 variable:
 $ original: chr  "2014-04-06T18:42:05.823Z" "2014-04-06T18:42:05.823Z"

I use this PIPE symbol, which allows you to chain methods together, passing the output from one into the input of the following. Using a . symbolises the output from the previous method.

To make the PIPE operator available, load in the magrittr package (I believe it is also included in the tidyverse, so perhaps via the dplyr package:

library(maggritr)

Now I apply the as.POSIXct function to the original column, and save it back to the dataframe in a columnn called new:

DF$new <- DF$original %>% as.POSIXct(., tz = "UTC", "%Y-%m-%dT%H:%M:%OS")
> DF
                  original                     new
1 2014-04-06T18:42:05.823Z 2014-04-06 18:42:05.822
2 2014-04-06T18:42:05.823Z 2014-04-06 18:42:05.822
3 2014-04-06T18:42:05.823Z 2014-04-06 18:42:05.822
4 2014-04-06T18:42:05.823Z 2014-04-06 18:42:05.822
5 2014-04-06T18:42:05.823Z 2014-04-06 18:42:05.822

> str(DF)
'data.frame':   5 obs. of  2 variables:
 $ original: chr  "2014-04-06T18:42:05.823Z" "2014-04-06T18:42:05.823Z" ...
 $ new     : POSIXct, format: "2014-04-06 18:42:05.822" "2014-04-06 18:42:05.822" ...

You can remove the original column if you like, using:

df$original <- NULL


Modelling on one Population and Evaluating on another Population

If the users who you are getting client-side data from are from the same population of users who you would get server-side data from. If that is true, then you aren't really training on one population and applying to another. The main difference is that the client side data happened in the past (by necessity unless you are constantly refitting your model) and the server side data will come in the future.

Let's reformulate the question in terms of models rather than web clients and servers.

You are fitting a model on one dataset and applying it to another. That is the classic use of predictive modeling/machine learning. Models use features from the data to make estimates of some parameter or parameters. Once you have a fitted (and tested) model, all that you need is the same set of features to feed into the model to get your estimates.

Just make sure to model on a set of features (aka variables) that are available on the client-side and server-side. If that isn't possible, ask that question separately.


Do recommendation systems necessarily use machine learning algorithms?

There's nothing about a recommendation system that absolutely necessitates some kind of machine learning. Indeed, I've seen decision systems in use that were essentially just someone's idea about what the customer's preferences ought to be.

A recommender can be based on anything from a few ad-hoc 'common sense' rules, to a logistic regression someone did on some data a few years ago and whose parameters are hardcoded into the system, to a complicated ensemble of machine-learning algorithms that are regularly and constantly trained on new data.

The use of machine learning for recommender systems is partly driven by necessity, partly by fad (at least from what I have seen). If a simple recommender works well, and accurately predicts what the user wants, there's no need for a machine to learn anything. If there's a huge amount of data, hiding some very deep relationships that humans are unable to pick out, that's where machine learning becomes useful.


Should we use only one-hot-vector for LSTM input/outputs?

This depends on what your data is representing and what you want to predict.  My understanding of One-Hot-Encoding is that this should only be used for encoding of categorical features.  For example, if you have a feature representing a category of K classes, you should one hot encode this as well as the Y variable (if you are trying to predict this categorical variable).  Of course, have the final layer be a softmax to output a distribution of size K.
This highly depends on what your data is representing.  If categorical, see above.  If just a simple numeric, you should not one hot encode this.  You could, I guess, if this set of integers is finite and small, but there is no need to learn the extra weights. You should only be using a softmax when you want to output a vector of K dimension where the entries all sum to one (perfect for representing a probability distribution over K classes).  The final layer should output whatever it is you want to predict.  If you want to predict something as simple as a numeric variable at the next time-step, just have a dense layer of size 1, with some activation function (relu probably).  More information about what exactly you are trying to predict is needed to know how to recommend anything concrete here.
I'm not sure about this.  You could have inputs and outputs represented this way, but you wouldn't use a softmax activation at the end.  You would use a dense layer outputting  a vector the size of your X variable, that is, if that's the variable you are trying to predict.  I'm not super well trained in various types of use cases for LSTMs, but from my experience, I can't think of a reason to do this.


If Deep learning or Machine learning is blackbox then why companies are still investing?

You make a huge assumption that because something is a black box, that it is not valid. Almost every decision made by a human is essentially a black box decision. Most could tell you the important factors in a decision, but they can rarely quantify accurately, how each component factored into the decision.

Another problem with your question is that you assume all Machine Learning techniques are black box.  For example, a Decision Tree is indisputably a Machine Learning technique, but it is definitely not a black box algorithm.

That said, in some applications, a black box technique is not desirable. For example, in the pharmaceutical field and credit approvals, you want to be able to explain the mechanics of the model.  But in other applications, black box algorithms trained on data generated by experts may perform nearly as well as experts, and probably better than the average human in a given task. For example, in the field of medical diagnosis from imaging, black box algorithms are enjoying success and they can be built right into the imaging software.  This means that the imaging machine can 'refer' the most challenging instances to a doctor for further diagnosis.  This means a doctor can spend more time with the most challenging patients and this is the real reason companies are investing in all manner of Machine Learning applications; they gain some economic benefit that outweigh the costs.

By all means, there are many failures of Machine Learning, but there are so many examples of Machine Learning that have been deployed and nothing is ever really mentioned about it.  For example, my iPhone employs facial recognition to help me search for people in my photos.  With minimal input from me, my phone has tagged, pretty accurately, all the people I typically take photographs of (mostly family).  I think the key is that we are seeing Machine Learning going mainstream to the point that it is not even really acknowledged anymore.


How to account for reduced student capacity when calculating program retention?

Rationale
Some of the terms are a little vague, particularly what you refer to as eligible students and returned students. I'll set some variables for clarity, but tell me if I defined them incorrectly.
I assume them to mean:

eligible students $ = A $ being the set of all students in the after-school program 2019-2020

returning students $ = A\cap S $ where $S$ is the set of all of summer camp 2020 students
Now we can define retention rate to be $\frac{|A\cap S|}{|A|}$. I base this on the US government definition of university student retention rate, which I think is pretty similar, but please correct me if it's not.

The reasons as to why you might have $|A\cap S|<|A|$ are irrelevant.

Example
Let's say you have $|A|=100$ so that, in your situation, $|S|=90$. For simplicity, let's also say that there are no newcomers to the mix.
The retention is, quite literally, how many students you retained. If you retained $90$ students, then the retention rate is $90/100$. Even though $|S|=|A|-10$, the number of students you retained $|A\cap S|$ as a percentage of the number of students in the original program $|A|$ is still $90\%$.
It wouldn't make sense to normalize this to $100\%$.


Can I apply Feature Selection even though the number of features is smaller than the number of observations?

No matter if the number of features is bigger or smaller of the number of samples, applying feature selection to identify the features with the biggest influence before proceeding with the modeling improves the model performance and helps prevent overfitting and eliminate noise bias (of course, in cases where the sample size is smaller than the number of features the need for dimensionality reduction is even bigger!).
Keep in mind that feature selection is not the only way to achieve dimensionality reduction, it can also be achieved with the extraction of new, more compact features from the initial ones (the only negative thing with that is that you cannot have clear physical interpretation of the selected features).
It is not clear from you question if you have in mind some regression or classification task (or just generally asking), but in both cases the general logic is the same. Nevertheless, I have provided examples for both:

Here is an example applying most common techniques for dimensionality reduction for classification (including feature selection ones) on the Iris dataset.

Here is an example applying feature selection for linear regression on a sample of size 1000 and 100 features.

Also, here is some additional reading in case you want to undertand better the whole dimensionality reduction subject:

An article discussing dimensionality reduction and suggesting alternative methods.

In this link, dimensionality reduction (pluses and minuses) is again discussed and the most popular linear (feature extraction, no feature selection!) method - PCA - is explained.

A review paper discussing state of the art methods for Feature Selection for Classification can be found here.

I hope this helped! :)


It doesn't handle categorical variables very well
It doesn't handle 'soft' boundaries - i.e. areas where some cases appear on either side of a boundary.
See also mathbabe here: https://mathbabe.org/?s=nearest+neighbor - for extended criticism.


Are parametric method and supervised learning exactly the same?

Parametric methods in simple terms follow a particular distribution. The most common example would be that of Normal Distribution, where 64 percent of the data is situated around +-1 standard deviation from the mean. The essence of this distribution is the arrangement of values with respect to their mean. Similarly, other methods such Poisson Distribution etc have their own unique modeling technique.

Parametric Estimation might have laid the foundation to some of the most vital parts of Machine Learning but it is an absolute mistake to think supervised learning is the same thing.

Supervised Learning may include approaches to fit the aforementioned parametric models but this might not always be the case. More often, the data scatter is quite spread out. It might not just be fitting one parametric model but a hybrid of more than one. Supervised Learning also takes into account the error which most parametric models don't consider unless incorporated manually. You could say that supervised learning is an evolved and more robust version of parametric methods which is highly flexible.


If you wish to understand and use Bayesian networks, you can try OpenMarkov, an open-source tool. I recommend you having a look at its tutorial.


How to pass input to deep learning models for Multiple choice question answering problem?

The basic idea of most of the current question answering architectures is:

get a common representation of the question and the input text (e.g., using BERT)
get a representation of the answers
do sort of attention over the answers: compute a scalar score for each of the answers (using a dot-product, linear layer, multilayer-perceptron) and normalize the scores using softmax

The architectures are typically based on the Bidirectional Attention Flow Model, although it was designed for a slightly different task and although the pre-trained word embeddings and RNNs in the model are today usually replaced with BERT-like models. In 2018, there was a competition in question answering at SemEval where many interesting ideas on this problem were presented.


Can Boosting and Bagging be applied to heterogeneous algorithms?

The short answer is yes. Both boosting and bagging meta-algorithms do not assume specific weak learners, thus any learner can do, no matter if uses same algorithm or different one.
The way the meta-algorithms are defined, they use the weak learners as black-box models, without reference to their implementation or algorithmic principle, nor similarity.
For Boosting:

In machine learning, boosting is an ensemble meta-algorithm for
primarily reducing bias, and also variance[1] in supervised learning,
and a family of machine learning algorithms that convert weak learners
to strong ones.[2] Boosting is based on the question posed by Kearns
and Valiant (1988, 1989):[3][4] "Can a set of weak learners create a
single strong learner?"
A weak learner is defined to be a classifier that is only slightly
correlated with the true classification (it can label examples better
than random guessing). In contrast, a strong learner is a classifier
that is arbitrarily well-correlated with the true classification.

For Bagging:

Although it is usually applied to decision tree methods, it can be
used with any type of method. Bagging is a special case of the model
averaging approach.

For Bagging the required condition to improve performance is that the weak learners should be instable (thus perturbed versions of the learners affect outcomes), but other than that, learners are black boxes, as mentioned above.
For further reference:

Boosting, wikipedia
Bagging, wikipedia
The Strength of Weak Learnability
Bagging Predictors


Both of these are feature transformation techniques but they do different things due to their different mathematical properties. In general it is good to normalize the data we are working with, this is usually done with a linear transformation. However, the logarithm is useful when the range of the data is too large.

For example [1, 10, 100, 1000, 10000, 100000], these data points have too large of a range. If we normalize this data with linear transformation using the max, we get: [1.e-05, 1.e-04, 1.e-03, 1.e-02, 1.e-01, 1.e+00]. You can see that the lower values in this data are all essentially 0. However, with the logarithm of this vector we would get [0, 2.303, 4.605, 6.908, 9.210, 11.513]. We can see that these values now have a much better spread.

x = [1, 10, 100, 1000, 10000, 100000]
x_trans_lin = [t/max(x) for t in x]
x_trans_log = [np.log(t) for t in x]

plt.figure(figsize=(10,5))
plt.subplot(1,2,1)
plt.scatter(range(len(x)), x)

plt.subplot(1,2,2)
plt.scatter(range(len(x)), x_trans_lin, label='linear')
plt.scatter(range(len(x)), x_trans_log, label='log')
plt.legend()
plt.show()


Can pinball loss be used to construct a prediction interval?

Yes, your interpretation regarding the pinball loss function seems right. For a given quantile value t between 0 and 1, it gives you the threshold value v.

  Then, can I simply train two models, say one for q=0.05 and the other for q=0.95 in order to get the estimates of the intervals that contain the actual values with the probability 0.95−0.05=0.9?
  `

And of course if you had trained two models to give threshold values for two quantiles, you can say that  $p(\hat{y}_{t, lower} \le y_t < \hat{y}_{t, upper}) = |\delta(q_1-q_2)|$


Information Extraction from Free-form text to create Transactions

My answer is based on couple of assumptions:

user input is more or less standard, so there won't be "Ex 20000"
you have at least majority of forms of input covered

In every representative example of transaction description you would need to mark words of interest, be it name of holder and account number. You can start small with 10-20 examples for start, and then, when you have all required fields marked, you can train a sequence labelling model, or, say it, custom Named Entity Recognition model, which will parse new text and extract required data for you.

How to train actual model is answered in that question: Help regarding NER in NLTK , and how to mark up the data - is more of a question you should answer, since only you do know, what should be marked as account number, account holder etc.

If you would be digging into NER training, I would advise you not to rely only on current word features, but add some regexp-alike features and dictionaries as features, since task seems very limited in terms of context variation.


how to do regression and classification in a same time?

I believe you are just describing standard classification.
Take the image below, the model of the classification is the decision boundary found between the two classes. After we have found this model, using a classification algorithm (clustering or otherwise), for new data we can use it to predict which class the new data belongs to.


What is the difference between AI, ML, NN and DL?

AI: a super general term, means a bit of everything... and nothing at the same time. It's all about building intelligent machines, even though its meaning is not fully developed. It's not used in a rigorous way at all. In fact, scientists prefer to use more technical terms like the others you listed.

ML: it's an approach to data, and it's all based on building models that improve automatically from experience.

DL: Deep Learning, it's the subset of ML that makes use of Deep Neural Networks. So technically it's still Machine Learning, currently its most advanced and promising branch.

NN: The general class of models that is employed in DL. DL is the field of application of NN's.

Personally, I don't really like that Venn diagram you posted. DL being a subset of NN... that doesn't make sense to me.


is it necessary for Artificial NN to be fully connected or only fully connected NN is called ANN?

No, the word artificial in this context is only used to separate the neural networks we build from the natural/biological neural networks that they are loosely based on. But in the realm of data science you can basically use NN and ANN interchangeably and fully connected layers are irrelevant to the terms.

Here is a hierarchy that might make things clearer:

Neural Network
 ├── Natural Neural Network
 |    ├── Brain                         (Not fully connected)
 |    └── ...
 └── Artificial Neural Network
      ├── Dense Neural Network          (Fully connected)
      ├── Convolutional Neural Network  (Not fully connected)
      ├── Recurrent Neural Network      (Not fully connected)
      └── ...


Name of this algorithm for supervised cluster assignment

K means clustering kind of works like this.
It uses the Lloyd algorithm, which basically has two steps.
Per each new sample:
 1) Choose the best cluster, by calculating the minimum distance to available clusters mean.
 2) Recalculate chosen cluster mean.


Can we use HDFS and big data Analytics for processing huge log files being processed through some application on some central server?

If the size of your logs are still growing then a distributed data system is definitely the right way to go. I have been using Mesos in production for almost a year now and it solved the problem of if this processing happens on some server it under or over utilizes resources.

I probably would look into some stacks like this:

Mesos as your fault-tolerant and elastic distributed systems
Spark or some Hadoop-based solutions for log processing and store the output in a DFS like HDFS
Have your applications consume data stored in HDFS as final steps


How to treat input that inherently has a tree structure?

As @Emre mentioned, RNN is a good option. It's worth noting that if the number of possible nodes in each tree is the same or at least has the same upper bound, you could use literally any architecture you want and just pass in the adjacency matrix. Alternatively, you could build an intermediate model to convert your graph into a graph embedding and then once again, you can do basically whatever you want with that.

A pretty big piece of potentially important information here is what you are trying to accomplish, which could have significant consequences for how you want to represent your inputs.


Is pattern recognition the same as unsupervised learning? Is machine learning the same as supervised learning?

Nice topic. I think it depends on how the "learning" process by a machine is defined, so if we assume that a machine can "autonomously" learn from experience (i.e. new data as it becomes available every X time), why not to include also semi-supervised and unsupervised learning styles?
The more data the machine has about the system being modeled, the more info to find more precise patterns which correctly describe the system, for all kind of learning styles.

As an example I made to explain how the one-class support vector machine works, we could see that, although an unsupervised learning process does not have a ground truth (i.e. labels) or a semi-supervised learning has only the "normality" label, the machine is able to better describe the system as it receives more data and as some hyperparameters are adjusted, if you test your model with known points wich should be outside the learnt cluster (where an anomaly score could be also a metric in this case).


Random Forest in R: Error in eval(expr, envir, enclos): object not found

In general column names in R need to be formatted without spaces for R to recognize them properly. Importing files with spaces, R will drop in a period and remove the space

This limit is is true for variables as well, but it will not auto-format them for you.

The most accepted methods for naming are:

stackOverflow camel case

stack_overflow with the underscore and a combination of lower and upper as you see fit

stack.overflow with period (which many dislike because it is often easy to not see the period)

If you use these conventions you will avoid the name issue and be able to focus on the underlying models.

A caveat, which may or may not matter to you. If you are going to pick a naming style and you work with other languages and pass data around, consider picking a style which works in both or all of them. It is easier for you to have one way, and it makes moving files easier if the names are compatible.


Output of machine learning classification prediction on out-of-sample data has too few observations

The actual predictions can be found in newdata.pred$data.
Note that mlr is no longer maintained:

{mlr} is considered retired from the mlr-org team. We won’t add new
features anymore and will only fix severe bugs. We suggest to use the
new mlr3 framework from now on and for future projects.

See the docs: https://mlr.mlr-org.com/
library(mlr)
library(ISLR)

df = ISLR::Auto
df$cylinders = as.factor(df$cylinders)
df$name <- NULL

task = makeClassifTask(data = df, target = "cylinders")
mod = train("classif.randomForest", task)

newdata.pred = predict(mod, newdata = df)
result = data.frame(newdata.pred$data)

    truth response
1       8        8
2       8        8
3       8        8
4       8        8
5       8        8
6       8        8
7       8        8
8       8        8
9       8        8
10      8        8
11      8        8
12      8        8
...


Feature engineering while using neural networks

If you are talking about tabular data (not images, video, sounds etc) then yes. All of preparations you mentioned are applicable to Artificial neural networks.
For instance, data normalization is very important since neurons activations are calculated applying some activation function on a weighted sum(linear combination) of it's inputs. A value of 0.01 and a value of 10000000 in a weighted sum will always have a large value if the weights are at the same scale. Theoretically the network could learn how to deal with such discrepancies on the input values but in a practical sense normalization can make learning faster.
There is a classical paper on efficient backprop, with many tips and tricks, by Yann LeCun.


Find intersection of data between rows and columns

Suppose that you want to know the name of countries with more than ten Ginger:

# read your data using pd.read_csv and specify appropriate inputs based on your excel file in a typical variable named csvFile and transpose it
csvFile = csvFile[csvFile['Ginger'] > 10]

Also if you want to see the countries with zero Ginger:

csvFile = csvFile[csvFile['Ginger'] == 0]


Unable to Use The K-Fold Validation Sklearn Python

Based on the answer here, Since you are doing a classification task, you should be using the metric R-squared (co-effecient of determination) instead of accuracy score (accuracy score is used for classification purposes). You should use something like score for evaluation because your task is regression.


Nested cross-validation generalization error for multiple models

In general you are right and in this answer it has been done as far as I see. The models are compared to each other while the best tuning of them is found, both inside the loop. It looks fine.

About your point, yes. But the point in Machine learning is that at some point we need to stop/limit our attempts as the number of algorithms which can do the task are very large. We usually try to evaluate different families of algorithms and then narrow the search from there but at the end we can never claim that the best answer we found is necessarily the best possible answer. In another POV, this is the main idea behind many research papers in ML. They just creatively find/modify an algorithm and show that it works better than previously applied algorithm through a benchmark dataset.


random forest classifier - impact of small n_estimator and repeated training

there is a parameter in estimators that is called random_state that fixes a seed for your algorithm run and reruns exactly the same so you can expect the same results at every rerun of the exact same code. It makes your code deterministic. But yes, in random forests in particular, due to their nature of averaging across all trees created when the forest is grown, variance in your result becomes less evident. So your reasoning is correct.

If you want to make your low_estimators RF produce the same results at every run, just add random_state and give it a random number


Altough your data is between 0 and 1, the predictions can be outside of this range, as with any satistical model. This shows that you are not able to represent your data properly with your model. However, this is not the important point. If you want to ensure the outputs to be between 0 and 1, you have to change the relu activation for another one, such as sigmoid or softmax. These activations ensure that your outputs will be between 0 and 1, although they might lead to other problems, such as vanishing gradients. What is usually advised is to use relu layers until the last one, which has to be a sigmoid or a softmax.


How can I "export" a model from Orange once I have prototyped a solution?

You can export any trained machine learning model in Orange using the Save Model widget.
This is what the Save Model widget does under the Model tab. Create your model, then click save model and your model will be saved to a pickle file. Then just load your pickle  file in Python.
Here is a like to a similar question on Stack Overflow:
https://stackoverflow.com/questions/45598627/orange-save-model-in-python-script


Multiclass Classification on live sensor data

I would suggest you use a simpler method first, just to have a baseline to compare from.

In your case you can adapt this jupyter notebook to fit your purpose. They train a k-NN model to compare time-series using similarity measures.

After you're comfortable with that, you can move on to using LSTMs for classification

LSTMs for Human Activity Recognition
Multilabel time series classification with LSTM

The last one also works if you are trying to predict multiple labels per sample.


RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed

Okay after 4 days I got my solution eventually. I figured it out that my classes are started from 1 to 12, so, in total I had 12 classes.

The problem was appearing when I was about to calculate the predicted class vs actual class loss with the loss function. Instead of having start from 1 that function always from class 0 instead of 1 and continuously look for 0 to 12 which indicated it do have 13 classes in total which generates an exception.

So how do I solved that? I just docked 1 form each labels and shifted it from 0 to 11 and it went fine. That's how I solved the issue. Mention, I got couple of solution at PyTorch forum regarding the same issue and those wasn't helpful at all, literally!


Time-series classification using convolutional neural networks

For instance, using 1-D convolutions:
https://blog.goodaudience.com/introduction-to-1d-convolutional-neural-networks-in-keras-for-time-sequences-3a7ff801a2cf


What is max_trials and executions_per_trial in keras-tuner

max_trials represents the number of hyperparameter combinations that will be tested by the tuner, while execution_per_trial is the number of models that should be built and fit for each trial for robustness purposes.

For example, let's imagine you have a shallow network (one hidden layer) with the following parameter search space:

Number of Hidden units: 16, 32, 48, 64
Choice of activation functions: ReLu, Sigmoid, TanH

You have a total of 12 combinations (4 times 3).

Then, at this point, when you set up max_trials = 40, you configure the tuner to find 40 random tuples of the # Hidden units and activation functions, e.g. (16, ReLu), (32, Sigmoid), (16, TanH)... up until 40 tuples. For each tuple you will run as many executions as you also set up in execution_per_trial variable, given that depending on how the model runs the optimization process, final results could be very different. For each trial and execution, the tuner will fit the model with as many epochs as you configure in the script.


Binary classifier using Keras with backend Tensorflow with a Binary output

I don't believe that's possible, in order for the model to return 0 or 1, your activation function on the output layer would have to return 0 or 1, which would mean that the activation function is non-differentiable, and you cannot do that.

Also you can simplify your transformer function to:

In [20]: predictions = np.array([[0.1], [0.9], [0.3], [0.6]])

In [21]: (predictions[:, 0] > 0.5).astype(np.int8)
Out[21]: array([0, 1, 0, 1], dtype=int8)

So it is very little extra work, and it gives your more flexibility if you want to see how confident the model is about the prediction, and you can change the threshold if you like.


How do Bayesian methods do automatic feature selection?

Your interpretation of L1 regularization sounds right. Is it used to perform feature selection? Yes, in the broad sense that this 'encourages' coefficients in the linear model to be 0, and those features with 0 coefficients are not used and can be removed.

Of course, this assumption about the prior distribution of coefficients is just an assumption you're adding. You are asserting that you need a lot of evidence to believe the coefficients are nonzero. I'm not sure it's the L1 regularization that's doing the feature selection, but your assumption, implemented by L1 regularization, that's leading you to conclude that some features do not contribute to the model.


NLP : variations of a text without modifying it's meaning

Text summarization can be divided into two categories 1. Extractive Summarization and  2. Abstractive Summarization

Extractive Summarization: These methods rely on extracting several parts, such as phrases and sentences, from a piece of text and stack them together to create a summary. Therefore, identifying the right sentences for summarization is of utmost importance in an extractive method.
Abstractive Summarization: Abstractive methods select words based on semantic understanding, even those words did not appear in the source documents. It aims at producing important material in a new way. They interpret and examine the text using advanced natural language techniques to generate a new shorter text that conveys the most critical information from the original text.

What you are looking for is abstractive summarisation. Since you are working in R there is a nice library called lexRank taking an example from here would look something like
#load needed packages
library(xml2)
library(rvest)
library(lexRankr)

#url to scrape
monsanto_url = "https://www.theguardian.com/environment/2017/sep/28/monsanto-banned-from-european-parliament"

#read page html
page = xml2::read_html(monsanto_url)
#extract text from page html using selector
page_text = rvest::html_text(rvest::html_nodes(page, ".js-article__body p"))

#perform lexrank for top 3 sentences
top_3 = lexRankr::lexRank(page_text,
                          #only 1 article; repeat same docid for all of input vector
                          docId = rep(1, length(page_text)),
                          #return 3 sentences to mimick /u/autotldr's output
                          n = 3,
                          continuous = TRUE)

#reorder the top 3 sentences to be in order of appearance in article
order_of_appearance = order(as.integer(gsub("_","",top_3$sentenceId)))
#extract sentences in order of appearance
ordered_top_3 = top_3[order_of_appearance, "sentence"]

> ordered_top_3
[1] "Monsanto lobbyists have been banned from entering the European parliament after the multinational refused to attend a parliamentary hearing into allegations of regulatory interference."
[2] "Monsanto officials will now be unable to meet MEPs, attend committee meetings or use digital resources on parliament premises in Brussels or Strasbourg."
[3] "A Monsanto letter to MEPs seen by the Guardian said that the European parliament was not “an appropriate forum” for discussion on the issues involved."

EDIT: How I like to think about abstractive summarisation: Y
Using encoder-decoder architecture (extendended with transformers) for seq2seq problems you can essentially get an embeding of your text, where same sentences can be embedded differently in different context, giving same/similiar output.


I have n dimensional data and I want to check integrity, can I downgrade to 2 dimensional feature space via PCA and do so?

By good I assume you're trying to see if the features are different enough? It depends on your data set actually. As smallchess has pointed out, PCA is commonly used and may work with your dataset. Try it out and see if the different classes are distinct? If not, you can try using higher dimensional visualisation tools.

Besides, with PCA you're reducing the dimensions to new features that may or may not be intuitive in meaning. That makes interpretation more complicated. If you're looking to visualise data, why not go with higher dimension visualisations? In case you're not aware of the different available tools..
Here is a question with great answers on Quora about higher dimension visualisation

Of the tools recommended, Andrews plot and parallel coordinates plots are very popular. I would suggest using these tools instead of using PCA and subsequently plotting for the purpose you're describing. If you have no need to visualise the data, sometimes you can just get a statistical summary/description of the datasets (mean, variance, min, max, etc). Below is an example of an Andrews Plot. You can plot the various classes into different colours to see if there's any discernible differences between the classes.

If that's not what you're looking for, to visualise the data to gauge the representation, just proceed to try out and build a model with the features and see if you get a good score?

Edit: Here are some shortcomings of PCA..if there's information that's misrepresented, do let me know!
One thing to note is that with PCA, its transformation is orthogonal. Sometimes your data requires non-orthogonal representation. This is very dependent on how your data looks like. To give an example, take a look at the diagram below.

In the above example, PCA fails to find PCs that maximises the variance due to orthogonality. Independent Factor Analysis may work better. Also, if you're planning to use the transformed data to build your model upon, there may be limitations with the linearly transformed data. Sometimes we need non-linear PCs. See the below example from the scikit webpage.

In such a case, a kernel PCA (utilizing a kernel trick similar in SVMs) may work better by getting a projection to get better linear distinction.


Why does removal of some features improve the performance of random forests on some occasions?

A basic decision tree is pruned to reduce the likelihood of over-fitting to the data and so help to generalise. Random forests don't usually require pruning because each individual tree is trained on a random subset of the features and when trees are combined, there is little correlation between them, reducing the risk of over-fitting and building dependencies between trees.

There could be a few reasons why you get this unexpected improved performance, mostly depending on how you trained the random forest. If you did any of the following, you potentially allowed overfitting to creep in:

a small number of random trees was used
trees with high strength were used; meaning very deep, learning idiosyncrasies of the training set
correlation between your features

and so removing features, you have allowed your model to generalise slightly more and so improve its performance.

It might be a good idea to remove any features that are highly correlated e.g. if two features have a pairwise correlation of >0.5, simply remove one of them. This would essentially be what you did (removing 3 features), but in a more selective manner.

Overfitting in Random Forests

You can read a bit more about the reasons above on Wikipedia or in some papers about random forests that discuss issues:

Random forest paper by Leo Breiman - states in the conclusion section:

  Because of the Law of Large Numbers, they do not overfit.

but also mentions the requirement of appropriate levels of randomness.
Elements of Statistical Learning by Hastie et. al (specifically section 15.3.4 Random Forests and Overfitting) gives more insight, referring to the increase of the number of data samples taken from your training set:

  at the limit, the average of fully grown trees can result in too rich a model, and incur unnecessary variance

So there is a trade-off perhaps, between the number of features you have, the number of trees used, and their depths. Work has been done to control the depth of trees, with some success - I refer you to Hastie et. al for more details and references.

Here is an image from the book, which shows results of a regression experiment controlling the depth of trees via minimum Node Size. So requiring larger nodes effectively restricts your decision trees from being grown too far, and therefore reducing overfitting.

As a side note, section 15.3.2 addresses variable importance, which might interest you.

I assume that you trained ("grew") your random forest on some training data and tested the performance on some hold-out data, so the performance you speak of is valid.


Having trouble scaling scores of logistic regression

Turns out I was using a wrong formula for scaling the scores. The formula
Score_i= (βi × WoE_i + α/n) × Factor + Offset/n
is applicable only if the logistic regression is done after doing WoE Transform, meaning for all features (not for all bins!) a WoE value is assigned and then the classifier is fit.
The way I was doing it was binning first, and feeding columns with dummy values into the classifier.
Reference to correct approach


How to detect if a person is interacting (as in touching) some object in an image?

There are papers that show how to estimate different poses. For example this one gives an overview of different methods: PoseTrack: A Benchmark for Human Pose Estimation and Tracking. You can take this benchmark and group all the poses into needed categories. For example, by manually assigning your groups. And then you use the categorization of poses like in paper and convert to the poses you need.

One of the solutions for pose tracking is Mask R-CNN. It uses a neural network with convolutional layers and predicts in parallel class, box and binary mask. Here is the original Mask R-CNN paper. And here is the ProTracker algorithm that achieved one of the best results on PoseTrack.


When does boosting overfit more than bagging?

I read your question as: 'Is boosting more vulnerable to overfitting than bagging?'

Firstly, you need to understand that bagging decreases variance, while boosting decreases bias.

Also, to be noted that under-fitting means that the model has low variance and high bias and vice versa for overfitting.

So, boosting is more vulnerable to overfitting than bagging.


Try exploring the rich field of "Anomaly Detection in Time Series". Control charts and CUSUMs (or cumulative sum control charts) might help you.

Simple Bullet Graphs might be all you need. Based on historical data and domain knowledge, define normal variance. Then make it clear to stakeholders when the current value is outside of predefined ranges.

Stephen Few is an expert in business dashboards. Any of his books will help you.

If you are open to R, try Shiny to create simple interactive web applications (It is very straightforward). There is also an open source package for anomaly detection, both local and global.

Create quick prototypes and get feedback!


Both the answers are wrong.
An embedding layer is a trainable layer that contains 1 embedding matrix, which is two dimensional, in one axis the number of unique values the categorical input can take (for example 26 in the case of lower case alphabet) and on the other axis the dimensionality of your embedding space. The role of the embedding layer is to map a category into a dense space in a way that is useful for the task at hand, at least in a supervised task. This usually means there is some semantic value in the embedding vectors and categories that are close in this space will be close in meaning for the task.
This is related to one-hot encoding in a sense that it maps a discrete category into a vector feature representation. You could still do this for a neural network but if you use this in a dense layer you would create an enormous amount of weights of which most of them are not used regularly. Putting an embedding layer in between reduces the amount of learnable weights before feeding them to interact with other parts of your input. Another advantage is that the embedding matrix basically works as a lookup table, so you can really use the sparsity of the index of your category to look up what the current value of the embedding is and when applying backpropagating only adapting that entry of the weight matrix.
This blog post explains clearly about How the embedding layer is trained in Keras Embedding layer. Hope this helps.


Depending on your choice of accuracy metric, you'll find that different balancing ratios give the optimum value of the metric. To see why this is true, consider optimizing precision alone vs. optimizing recall alone. Precision is optimized (=1.0) when there are no false positives. Upweighting negative data reduces the positive rate, and therefore the false positive weight. So if you just want to optimize precision, give the positive data zero weight! You'll always predict negative labels and the precision will be ideal. Likewise, for only optimizing recall, give the negative data zero weight - you'll always get the ideal value of recall. These extreme cases are silly for real-world applications, but they do show that your "best" balancing ratio depends on your metric.

As you're probably aware, metrics like AUC and F1 try to compromise between precision and recall. In the absence of prior information, people often try to choose "equal balance" between precision and recall, as implemented in AUC. Since AUC is relatively insensitive to data balance, 1:1 data balancing is generally appropriate. However, in real life you may care more about precision than recall, or vice versa. So, you do need to select your metric in advance, depending on the problem you're solving. Then keep your metric fixed, vary your data balance, and look at your trained model performance on realistic test datasets. Then you can see whether your model is making the optimum predictions, from the point of view of your chosen metric and your real-world dataset.


Can one use non converged results from Logistic Regression?

An iterative algorithm is said to converge when, as the iterations proceed, the output gets closer and closer to a specific value. More precisely, no matter how small an error value you choose, if you continue long enough the function will eventually stay closer than that error value from some final value.

In some circumstances, an algorithm will not converge; it could even diverge, where its output will undergo larger and larger oscillations, never approaching a useful result. More precisely, no matter how long you continue, the function value will never settle down within a range of any "final" value.

You could use the results which did not converge, but it's not recommended.


Computationally Inexpensive Imputation Techniques in R

Take a look at the h20 package https://cran.r-project.org/web/packages/h2o/h2o.pdf.

Everything is designed with parallelization in mind. I've had great success with many of their implementations, in R and Scala.

If you have to do it in R and are going for pure speed I doubt you'll find something faster.


Data Selection according to Feature Values

Ignoring for a moment the variance of the first feature, a straightforward approach is to perform a linear combination of the features $x_1$, $x_2$ and $x_3$, with each of the coefficients being a hyper-parameter you set to indicate the relative importance of that feature (perhaps normalizing the features before hand). This will be your molecule's utility measure.
$$u^{(i)} = \beta_1 x_1^{(i)} + \beta_2 x_2^{(i)} + \beta_3 x_3^{(i)} $$
You can then sort according to this utility and select the top 1000. A natural set of coefficients if each feature was of comparable importance would be $\beta_1=1$, $\beta_2=1$ and $\beta_3=-1$ (negative because we want to select for low values of heuristic value 2).
Knowing that we have the variance of the first feature, we can assume that this feature is distributed normally. The utility is now a random variable distributed according to:
$$U^{(i)} \sim \mathcal{N}(\beta_1 \mu_{x_1^{(i)}} + \beta_2 x_2^{(i)} + \beta_3 x_3^{(i)}, \beta_1^2\sigma_{x_1^{(i)}}^{2})$$
From this point, the method for selecting molecules will depend on certain considerations such as your tolerance to variance and if you are more interested in the collective utility of your selection or if each individually selected molecule must meet a certain standard. If you didn't care about variance at all and simply wanted to maximize the collective expected utility you could bypass modelling utility as a random variable all together and stick with the first approach. An alternative if the variance was a factor would be to select the top 1000 molecules according to their mean utility, excluding molecules which didn't have a high probability of having a utility higher than some threshold $P(U > C) > 0.95$. These are just some examples, the exact approach will depend on your criteria which you need to clarify.
As to your question "How would my protocol be adjusted if there was a constraint of diversity in the selected molecules". Diverse in what sense? Have you been provided with any other features to compare the molecules with or do you mean diverse in the three features already given? Again, another simple approach would be to append another feature calculated as the average distance to those molecules already selected.
One of the difficulties with your question is that it is very broad and open ended. There is not a lot of clarity surrounding what constitutes a good selection of molecules from a bad selection. For example, is it better to have a higher heuristic value 1 or a lower heuristic value 2? These sorts of questions need to be handled by a domain expert prior to applying the data science algorithm. If you were given a set A of 3 molecules and a set B of another 3 molecules do you have a clear way of determining which is the better selection? You need to be able to answer this question before you can begin applying any data science or else you are shooting in the dark.


How to measure a almost (?) ordinal classification?

You could just calculate the mean absolute error.
However, this quality measure (and any other measure taking into account the distance) would not be consistent with the loss function you used while training your classifier I guess. So it would not be fair to your algorithm. It is like if your professor would change the grading system after you submitted your test.
Instead of changing the quality measure you might want to reconsider your approach. The suggestion to use the ordinal regression sounds good. In case if the cost of misestimation between neighbors is similar you could even use plain regression and then round.


Vertical concatenation in a df based on column value_python

Try:
a = ["data1"
,"data1"
,"data1"
,"data2"
,"data3"
,"data2"
,"data3"
,"data4"
,"data4"]

b = ["x;y;z"
    ,"d;f;h"
    ,"p;c;j"
    ,"s;k;a"
    ,"a;w;q"
    ,"o;i;s"
    ,"e;q;j"
    ,"d;f;n"
    ,"q;f;k"]

e = pd.DataFrame({"col_value":a,"concat1":b})

e["vertical_concat"] = e.groupby("col_value").transform(lambda x: ";".join(x.unique()))

Output:


Why are precision and recall used in the F1 score, rather than precision and NPV?

F1 score weights precision and recall equally but there are easy generalizations to any case where you consider recall $\beta$ times more important than precision. See https://en.wikipedia.org/wiki/F1_score:

$F_\beta = (1 + \beta^2) \frac{precision \cdot recall}{\beta^2 precision + recall}$

F1 is just a harmonic mean. The simple mean doesn't quite make sense because precision and recall have the same numerators (true positive) but different denominators (test positive, condition positive). So only a harmonic mean makes sense. I don't know if there's more theory to it than that -- the simplest weighted mean that make sense.

I think I get the gist of your point, which I paraphrase to refer to the fact that precision has "test positive" in the denominator so is quite sensitive to how much the classifier marks positive. For this reason you don't so often see, for example, precision-recall curves. You see ROC curves which are recall-specificity curves (true positive rate vs false positive rate).

That's closer to what you suggest but you're suggesting PPV vs NPV. Sure that could be valid depending on your use case, but I think the argument tends to cut the other way, to recall-specificity instead, not precision-NPV.


Which data mining or machine learning algorithm would be appropriate for learning ordered frequent patterns?

Assuming you only have these two features (var1,  var2), you might want to:
 * Create one-hot encoded features for each variable under each position.
 * Add a column on which variable is first (e.g. two columns - likely to work with trees but not with anything else).
 * Take each possible combination of variables and use that as your only input (e.g. you then take the average of out for that combination, perhaps adding some prior or smoothing).

As the comments mentioned, if out is some discrete event, maybe you'd want to instead look at Markov models.


Prioritized Experience Replay - why to approximate the Density Function?

That's because the probablity density function might not follow an exponential curve, especially if formula is not $p=1/rank$ but is instead proportional to the amount of error. This will be a byproduct of us not keeping the array sorted, because we want to simplify our life. The curve could then look like this:

Here, area under the curve is one, but the curve is not exponential or linear or linear-horizontal.

Roll a dice from 0.0 to 1.0, which by the way has the same probability to roll 0.71 as 0.92 etc. Select an element from the array (doesn't have to be sorted!), scanning the array from left to right.

Consider array with 4 numbers:  {5, 15, 25, 10}. Each represents a portion from  the total sum:

|0.1|     0.3     |           0.5           |0.2|
you got to be pretty lucky to land on 0.1 (within first 10% segment)

The likelyhood of selecting an element is dependant on its "height under the curve". The taller the element, the more likely for our 0-to-1 counter to land on it.

You can't represent such "landing probability mechanism" by the "exponential dice approach" mentioned in my original question.

However, selecting by the "exponential dice" - albeit not precise might nonetheless work. The only caveat is the array's would have to be kept ordered, which is a pain in the rear. So the benefit isn't really there, just keep the array unsorted and go with the "segmented tree" aka "sum tree" approach O(logn) cost on all operations. This would allow you to avoid scanning the array left-to-right, helping you get from O(N) to O(logn) sampling performance

Now please help me with this question instead :)


I can only offer comments base on your statements. First of all,set your goal clear and straight. I think it is the maximization of BigChange and Flat Sensitivities .

I think Number 2 is problematic because of your concern of upcoming images. Classification is not Regression. There is a chance that model will never have a clue about the middle ranges. But you can set the classification threshold for flat and big change prediction for maximum sensitivities. The remaining unclassified images will be middle range changes. The problem here is that there could be a lot of mixed-ups from two different thresholds and unusual metric.

So I think 3 wins because 1 doesn't care if it is small or middle range changes. It is the most straightforward thing towards your goal. Note that even if you are classifying with 3 things the metric for optimization can just involve two things. But you should verify it. I think it is all right to compare the two remaining models. Cross validate and choose the winner base on combined sensitivity metric (including only flat and big changes). Just don't be fooled by randomness in selection.

You might just cross validate three models in the end just to be sure and always keep in mind the unusual metric for optimization. In addition, you should always remember classification thresholding (not target thresholding) for all models and there can always be a choice of no classification for some examples after thresholding.

I'll be happy if you share the result of your experimentation and verify my hunch if it is wrong or not.


Is it better to use F1 score or AUC metric for imbalanced data classification?

F1-score and AUC are two evaluation measures for binary classification but they are not comparable:

F1-score measures the performance of a hard classifier, i.e. a system which predicts a class for every instance. Through precision and recall it compares for every instance the predicted class vs. the gold-standard class.
AUC measures the performance of a soft classifier, i.e. a system which predicts a probability (or a score) for every instance. The difference is that the system doesn't decide which class the instance belongs to, so informally it can be seen as an "unfinished classifier". If one decides a threshold on the probability to separate the classes then it becomes a hard classifier.

Both are fine to be used with imbalanced data, that's not a reason to pick one or the other.
AUC is useful to study the general behaviour of a method without deciding a particular threshold. Sometimes the choice of a particular threshold can have a strong impact on performance, so using AUC avoids the issue completely. However strictly speaking AUC doesn't give the performance of a classifier.


Best framework for recognizing a specific cartoon character's face?

You will most likely need to train a neural network to detect your particular cartoon characters. Although some parts of this are tedious, this type of task is well-documented and there are many user-friendly frameworks available. I would recommend reading the tensorflow object detection api tutorial.

The most difficult part will simply be collecting your data. You will need to sample video frames to train the neural network. Depending on how many characters are in your cartoon, the complexity and variability of the inputs, and network model that you choose, I suspect you will need to collect anywhere from 200-500 distinct samples for training.

After you collect your frames, you will need to annotate your data. "Annotation" is the process by which you manually draw bounding boxes around your characters so that the neural network knows what to look for. This process is described in further detail in the above link. Fortunately, you do not need to program the annotation tool yourself; the Tensorflow tutorial instructs you to install LabelImg, which provides a graphical interface for you to label your selected frames.

After creating your dataset, you can proceed to train your network using the Tensorflow instructions.

If you have a lot of video to inference, I would recommend sampling your frames in relatively large intervals such as 5 or even 10 (depending on the fps of the video). You can then inference 5 to 10 times faster with reasonable accuracy.

For example, suppose your network inferences frames $ x_t $ through $ x_{t+10} $ in 5 frame intervals. If both $ x_t $ and $ x_{t+5} $ have at least one bounding box of reasonable size, then frames between $ x_t $ and $ x_{t+5} $ most likely have characters as well. If frame $ x_{t+10} $ does not have a bounding box of reasonable size, then we might assume that the character(s) leave the frames at about $ x_{t+7} $ or $ x_{t+8} $. This method will let you control the inherent speed/accuracy trade-off.


Tensorflow speech-to-text training on single file with tf.nn.ctc_loss not converging

I have found the solution.

Earlier I have a Dropout layer after single layer of RNN. I have changed the rnn_layer function as below.

Removed the DropoutWrapper from the RNN layer - This helped me to overfit
Added 2 layers of LSTM cell. Single layer LSTM cell taking time to
converge but rather than making LSTM layer wide changed it to deep.
This helped me achieve faster convergence than making network wider.

Below is the code of rnn_layer.

def rnn_layer(input_tensor, n_cell_units, dropout, seq_length, batch_size):
    lstm_fw_cell = tf.contrib.rnn.BasicLSTMCell(2*n_cell_units, forget_bias=1.0, state_is_tuple=True)
# Dropout layer dropped
#    lstm_fw_cell = tf.contrib.rnn.DropoutWrapper(lstm_fw_cell, input_keep_prob=dropout, output_keep_prob=dropout)
# Multi layer RNN
    lstm_fw_cell = tf.contrib.rnn.MultiRNNCell([lstm_fw_cell] * 2, state_is_tuple=True)

    outputs, output_states = tf.nn.dynamic_rnn(cell=lstm_fw_cell,
                                               inputs=input_tensor,
                                               dtype=tf.float32,
                                               time_major=True,
                                               sequence_length=seq_length)

    return outputs


How to handle correlated variables in a dataset?

If your model has dependent variables, then it's hard to interpret the coefficients down the road.

For example, say that your model has 3 variables: profits, costs and revenue.
You trained your linear regression model to predict revenue (silly, I know).

You could get the following coefficients: $c_{cost}=-1,c_{profits}=1,c_{revenue}=0$

Someone looking at these results, could conclude that revenue is not predictive of revenue.

This is kind of a trivial example because the linear dependency is very clear (revenue=profits-costs).

But consider it a motivation for not adding dependent variables to your model.

Correlation is a measurement of pairwise linear dependency.

It's more of a question of what type are your variables.

If your variables are continuous, Dimensionality reduction algorithms like PCA should account for the correlation.


What's the difference between data classification and clustering (from a Data point of view)

Classification is a problem where your input data consists of elements with 2 parts:

Some data features that reflect the traits of an entity
A label that assigns the entity to a group or class.

With that kind of data, you can train a model that receives the data features (first part) and generates the label (second part). This kind of training, where you train a system to generate some output when it receives a specific input is called "supervised learning".
On the other hand, in Clustering, your dataset only has the data features, that is, your dataset does not have the labels. Clustering methods allow you to group the entities in classes without having any labels, normally by defining a priori how many groups you want, and then grouping the entities by their similarity. This kind of training, where there are no labels and you have to learn just from the entity data features is called "unsupervised learning"


Normalise using your training data statistics. Save the values used (e.g. mean and sd per feature), treating them as part of your model. Once you have used these values to transform input, they become fixed translate/scale factors in the model.

Use the same values to normalise test data or new inputs as required. You do not need to calculate new normalisation constants for new data. In fact doing so will most likely reduce the effectiveness of your model.

The same principle applies to interpreting output values if you need to scale those into range that your model produces. Scale according to your training data.