Has there been put any thought into it?

Probably not at a formal research level, it is basically a dead-end, as you have pointed out there will essentially be one correct solution amongst a large number of incorrect ones. No useful heuristic exists to assess partial solutions, so logically RL will not work, as an agent needs to experience (or simulate accurately) success and failure in order to learn correctly. This is a hard limit to all learning algorithms, they can only learn from available data - any combinatorial problem with one special permutation, and zero clues that disambiguate it from a large number of similar permutations is inherently unlearnable.

Only approaches that would work against a generic hash algorithm are brute-force searches that try every possible combination of operations suspected of being in the algorithm, up to a certain size of program. For any real-world secure hashing algorithm, this would be a huge search space, and intractable. For smaller insecure hashing algorithms, it is more feasible, but there would likely be no value to even demonstrating it was possible.

Related to this (when trying to reverse a hash or find collisions, not figure out the algorithm from examples), cryptographic hashes, such as MD5, SHA-1, SHA-256 etc, are designed such that brute-forcing them is the only way to solve them. In the case of MD5 and SHA-1, that has since been broken mathematically, and they are weaker then brute-forcing, but this was done starting with knowledge of the algorithm. Current RL agents and neural networks cannot match a human in learning and understanding abstract mathematics, so that approach is ruled out at least for now.


Is the PR AUC invariant under label flip?

I propose to look at this a bit differently. The different classification status of the two classes are related as follows:

$TP_1=TN_0$
$FP_1=FN_0$
$FN_1=FP_0$
$TN_1=TP_0$

From there we have:
$$FPR_0=\frac{FP_0}{FP_0+TN_0}=\frac{FN_1}{FN_1+TP_1}=1-\frac{TP_1}{FN_1+TP_1}=1-TPR_1$$
and naturally $FPR_1=1-TPR_0$ for the same reasons. Therefore when one switches the labels on a ROC curve, one obtains a mirror image of the curve using the top-left to bottom-right diagonal as the symmetry axis. It follows that the AUC is identical.
However this does not hold with a PR curve, because there is no such symmetry between precision and recall of the two classes. A counter-example should be enough to prove this:
Suppose a test set with 100 instances, 90/10 of which have true class 0/1. Let's assume the classifier predicts all the instances as class 0 except 2 instances as class 1, one true 0 and one true 1.
$$
\begin{array}{|c|c|c|}
\hline
&predicted~0 & predicted~1 \\
\hline
\hline
true~0 & 89 & 1 \\
\hline
true~1 & 9 & 1 \\
\hline
\end{array}
$$
Thus we have:
$P_0=0.92, R_0=0.99, P_1=0.5, R_1=0.1$
[edited] Assuming there is only this point on the PR curve and the two following extreme cases:

Predict every instance as class 0: $P_0=0.9, R_0=1, P_1=NaN, R_1=0$
Predict every instance as class 1: $P_0=NaN, R_0=0, P_1=0.1, R_1=1$

Note: I'm not totally sure about the proper way to deal with $P=NaN$ but I think it's ok to assume P=1 in order to put these points on the curve (this is not essential to the point anyway).
It's easy to see that the AUC is very different whether the focus class is 0 or 1:

In the former case the curve is entirely in the top part of the graph and the area is almost the maximum.
In the latter the curve is mostly in the bottom part of the graph and the area is less than 0.5.


Annotation tool for classification experiments

Try prodigy. An annotation tool powered by active learning.


Functions to do MLE and MSE calculations in R

Have you looked at the Metrics package for R?


Do anomalous input features to autoencoder result in high errors on the corresponding output features?

The output error will be propagated to the subsequent layers, so if your auto-encoder learns a function over all the features, the anomaly will be weaved across all the features you used as input. One trick is to map specific features (or set of features) to different auto-encoders, and train them as an ensemble. Whether you desire to integrate the specific features anomaly in a global outlier score, or maintain outlier scores for subset of features is then a design choice. For a good use case, see this recent paper.


Question about sklearn's StratifiedShuffleSplit

split

Is the object that allows us to do stratified split, and split.split is the split’s object method/function called split that can be used to perform stratified split.

Depending on how many splits you want using the n_split parameter you can split, hence the for loop.


Suggestion to improve the results is to use NER. You can at very least extract names and types. Search a bit more maybe someone has also trained on different types/genres of game also, so you could discriminate on that also.


Loss function in GradientBoostingRegressor

Note that the algorithm is called Gradient Boostign Regressor.
The idea is that you boost decision trees minimizing the gradient. This gradient is a loss function that can take more forms.
The algorithm aggregates each decision tree in the error of the previously fitted and predicted decision tree. There you have your desired loss function.
This parameter is regarding to that.


Correlating activity between entities using Python

You may compute the $id$ co-occurrences frequencies in a given time-window.
Suppose (without loss of generality) your criteria for co-occurrence is that both $id$s must occur on the second t, than using maximum likelihood estimate $P(id_{i}|id_{j})$ is:

$P(id_{i}|id_{j}) = \frac{count(id_{i}, id_{j})}{count(id_{j})}$

and the maximum likelihood estimate for the joint probability is:

$P(id_{i}, id_{j}) = \frac{count(id_{i}, id_{j})}{\sum{k \in IDs}{}
 \ count(id_{k})}$

Where $id_j, id_i \in IDs\ $ and $IDs$ is the set containing all $id$s (AAAA,BBB,CCCC,...),

You can then calculate the pointwise mutual information between each $id$ pair, that is, how ofter two $id$s co-occur, compared with what we would expect if they were independent:

$I(id_i, id_j) = \log_{2}{\frac{P(id_i, id_j)}{P(id_i)P(id_j)}}$

Given you a estimation of how strong is the association between $id_i$ and $id_j$.

The same strategy may be used to find similarities, you may think about each $id_i$ being a $|IDs|$-dimensional vector with the co-occurrences frequencies being the values. You can then apply cosine-similarity or pearson-correlation to find the most similar vectors ($id$s).

EDIT

Complementing my answer follows python demonstrating the ideas above for the sample dataset given in the question.

First we create our dataframe from the data in the question

import pandas as pd
import numpy as np
import collections

d = {'ID': ['AAAA', 'AAAA', 'AAAA', 'AAAA', 'AAAA', 'BBBB', 'BBBB', 'BBBB', 'BBBB', 'CCCC', 'CCCC', 'CCCC', 'CCCC','DDDD', 'DDDD', 'DDDD', 'DDDD'],
     'Time': [1, 6, 5, 2, 4, 2, 4, 6, 3, 3, 4, 1, 6, 7, 4, 5, 3]}

df = pd.DataFrame(d)

Computes the co-occurrences

dfm = df.merge(df, on='Time')
dfm = dfm[dfm.ID_x != dfm.ID_y]  # ID_x and ID_y are created by the merge
df_M = pd.get_dummies(dfm.ID_x).groupby(dfm.ID_y).apply(sum)
print(df_M)

The dataframe df_M represents the co-occurrence matrix

      AAAA  BBBB  CCCC  DDDD
ID_y
AAAA     0     3     3     2
BBBB     3     0     3     2
CCCC     3     3     0     2
DDDD     2     2     2     0

Pairwise Mutual Information

Let N be the total amount of co-occurrences, than I can compute every joint probability simply dividing the co-occurrences in the df_M matrix  by N.

N = df_M.sum().sum()
df_joint_P = df_M/N  # Computes every joint probability P(id_i, id_j)

The probability of each $id$ is the sum of all its joint probabilities

df_ID_P = df_joint_P.sum(axis=0) # Marginalizes to produces P(id_i)

Now that we have all we need to compute a PMI dataframe

idx = [(r, c) for r in list(df_M) for c in list(df_M)]
pmi_dict = collections.defaultdict(dict)
for r,c in idx:
    pmi_dict[r][c] = np.log2(2*df_joint_P[r][c]/(df_ID_P[r] * df_ID_P[c])) if df_joint_P[r][c] > 0 else 0
pmi_df = pd.DataFrame(pmi_dict)
print(pmi_df)

The pairwise mutual information between every pair of different $id$s is different than zero and like bellow

          AAAA      BBBB      CCCC      DDDD
AAAA  0.000000  1.491853  1.491853  1.321928
BBBB  1.491853  0.000000  1.491853  1.321928
CCCC  1.491853  1.491853  0.000000  1.321928
DDDD  1.321928  1.321928  1.321928  0.000000

We can see that every $id$ seems to be almost equally associated with each other.

Cosine Similarity

Now we can compute the cosine similarity, remembering that it is the $l_2$-normalized dot-product

df_NM = df_M.div(df_M.pow(2).sum(axis=1).pow(0.5), axis=0)
df_cos = df_NM.dot(df_NM.T)
print(df_cos)

The cosines are

          AAAA      BBBB      CCCC      DDDD
AAAA  1.000000  0.590909  0.590909  0.738549
BBBB  0.590909  1.000000  0.590909  0.738549
CCCC  0.590909  0.590909  1.000000  0.738549
DDDD  0.738549  0.738549  0.738549  1.000000

Obviously every $id$ is (trivially) most similar to itself but from the data above we see that every $id$ is similar to all the others (confirming the result we found using PMI) but we may notice that $id$ DDDD is more similar to all the other $id$s (at least for this tiny example).


Is ML a good solution for identifying what the user wants to do from a sentence?

Yes this problem is extremely well suited for machine learning. However, I think you should be careful as to which algorithms you tend to use.

A machine learning algorithm should be structured as follows: feature extraction and then your model. These are two things that should be done separately.

Feature Extraction

This is the bag of words, n_grams and word2vec. These are all good choices for text examples. I think bag of words is a good choice in your case. However, if this generates a sparse matrix then maybe n_grams can be better. You can test all 3 methods.

The Model

Theoretically, the more parameters in your model the more data you need to train it sufficiently otherwise you will retain a large amount of bias. This means a high error rate. Neural networks tend to have a very high number of parameters. Thus they require a lot of data to be trained.

But, you have 1000 instances!!! Yes. But, you also have 500 classes. So imagine you have a very young child and you want him to be able to correctly classify 500 different types of images. Then you can't just show the kid 2 different examples of each class for him to truly understand what each class really means.

As a very general rule of thumb, the number of instances you need to train a model increases exponentially with number of classes. So you will need a MASSIVE amount of data to properly train a neural network model.

I would suggest a less intensive model. Moreover, looking at your example, it seems that the classes should be linearly separable. So you can use something really simple, linear regression, logistic regression, naive bayes or knn. These methods would do MUCH MUCH better than a neural network.

My Suggestion

I would start with bag of words and then use knn. This should be a good starting point.

A neural network is 0% recommended for the amount of data you have.


How to choose the features for an algorithm from the given attached screenshot?

Why start with feature selection/engineering if you have so few features to begin with?

Best practice should be to include all features/variables in your first draft model and look at the accuracy, quality of resulting segments, etc.

If there is need to improve and to leave out a feature you could then use the metrics of variable importance of your chosen model (e.g. variable importance from random forest) to decide which feature was least relevant and could be discarded for a slimmer model.

In general model performance shouldn't suffer to much from including irrelevant features and therefore selection isn't the same issue as it would be for less robust models.


Are word embeddings further updated during training for document classification?

Word embeddings are generally used as input features, which like you noticed for image based models, do not get modified during training.

It is in fact quite difficult to update embeddings during training (or at all after they have been computed!), because they are often in some latent space that holds information regarding their relationships to one another. It is for this reason that adding new vocabulary to existing embeddings is extremely challenging, and updating computed embeddings is also difficult without using all the data that was originally used to create them.

I am afraid I don't have an example I can point to where embedding are used during training and simultaneously updated/improved upon. I can imagine that there are ways to store them, however, such that it is technically feasible (although perhaps memory intensive!)


Keras one_hot encoding - what's the point when unicity not guaranteed?

It seems to be bad choice of function names since the one_hot function actually is using hashing and is equivalent to hashing_trick (with standard settings).

See this issue: https://github.com/keras-team/keras/pull/6887

When I do:

from keras.preprocessing.text import one_hot
from keras.preprocessing.text import hashing_trick

text = 'The quick brown fox jumped over the lazy dog.'
print(one_hot(text, n=9))
print(hashing_trick(text, n=9))

I get identical results:

[5, 6, 2, 6, 5, 8, 5, 4, 1]
[5, 6, 2, 6, 5, 8, 5, 4, 1]

So in case you want to have "real" (unique) one-hot encoded results, you need to resort to some other solution (there is no Keras built-in solution for this in the moment as I believe).


What is difference between Fully Connected layer and Bilinear layer in CNN?

I quote the answers from What is a bilinear tensor layer (in contrast to a standard linear neural network layer) or how can I imagine it?.
A bilinear function is a function of two inputs $x$ and $y$ that is linear in each input separately. Simple bilinear functions on vectors are the dot product or the element-wise product.
Let $M$ be a matrix. The function $f(x,y)=x^TMy=\sum_iM_{ij}x_iy_j$ is bilinear in $x$ and $y$. In fact, any scalar bilinear function on two vectors takes this form. Note that a bilinear function is a linear combination of $x_iy_j$ whereas a linear function such as $g(x,y)=Ax+By$ can only have $x_i$ or $y_i$. For neural nets, that means a bilinear function allows for richer interactions between inputs.
Now what if you want a bilinear function that outputs a vector? Well, you simply define a matrix $M_k$ for each coordinate of the output and you end up with a stack of matrices. That stack of matrices is called a tensor (3-mode tensor to be exact). You can imagine the bilinear tensor product with two vectors as $x^⊤M_ky$ computed on each “slice” of the tensor.
Bilinear Models consists of two feature extractors whose outputs
are multiplied using an outer product at each location of the image
and pooled to obtain an image descriptor. 1

Its advantage is that it can model pairwise feature interactions in a translationally invariant manner, which is particularly useful for fine-grained categorization. It also allows end-to-end training using image labels only, and achieves state-of-the-art performance on fine-grained classification.


Image segmentation training set labeling

If you have the labeled training data with you then, you will have to label each pixel in the training set, compute the loss and backpropagate it. The link given below should help.

Suprevised Image Segmentation


How to represent a dataset as a linked list-like graph?

The graph resulting from this kind of dataset is also known as a Network Graph and the kind of analysis you are trying to do is known as Social Network Analysis.

There are many prominent Python libraries for visualization and subsequent analysis of network graphs. The most widely used is NetworkX. It is easy to add nodes and directed edges in a NetworkX graph and visualize them with Matplotlib.

Installing NetworkX is a prerequisite.

pip install networkx

The following code creates a NetworkX DiGraph object, adds edges(and thereby nodes) to it and plots the graph.

import networkx as nx
from matplotlib import pyplot as plt
G = nx.DiGraph()
G.add_edges_from([(1, 3), (2, 3), (2, 4), (4, 1)])
nx.draw(G)

Hopefully, this should get you started :)


I think it is better to use early stopping  method.Too many epochs will lead to overfitting the model.Early stopping allows you to stop the training once the model performance stops improving on a hold out validation dataset.

refer this link to the concept

https://en.wikipedia.org/wiki/Early_stopping


Incrementally Train BERT with minimum QnA records - to get improved results

Remember that BERT was first pre-trained using the concatenation of BooksCorpus (800M words) and English Wikipedia (2,500M words). Then the fine-tuning in your case uses the SQuAD dataset consisting of 100,000+ questions (based on Wikipedia articles) with a learning rate in the order of e-5.

So when you add let's say 100 new domain-specific question/document/answer to your input during the fine-tuning, this will not have a major effect on the learned parameters especially if your new input has out-of-Bert-vocabulary words or uses different English style, as opposed to Wikipedia/Books style.

  Can someone suggest, if they have same or similar experience, and
  suggest please.

Following are some suggestions that might help:

Increase the amount of new input data to help the model converge to your domain-specific use case.
Before fine-tuning,  map recurrent out-of-Bert-vocabulary words of your new input to existing words in Bert vocabulary which have almost the same meaning.
Increase the learning rate for your new input. But this is not very safe as the model may "forget" what he learned and not converge. But as the fine-tuning doesn't take much time you can try it out and see the results.


I found a solution to this problem with some good help from this article.
What I found really convenient is to make two generators. One for features, or X, and one for labels, or y.

def generate_X():
    while True:
        with open("/mypath/myData.csv") as f:
            for line in f:
                x= line.rstrip('\n').split(",")
                x= np.asarray(x)
                x=x[1:]
                x= x.reshape(1,1,12)
                yield x

def generate_y():
    while True:
        for i in range(len(y_train)):
            y= y_train[i]
            yield y

And then I go ahead and use these two generators as input to my third generator which will make batches of a desired size to the fit_generator()

def batch_generator(batch_size, gen_x,gen_y):

    batch_features = np.zeros((batch_size,1, 12))
    batch_labels = np.zeros((batch_size,9))

    while True:
        for i in range(batch_size):
            batch_features[i] = next(gen_x)
            batch_labels[i] = next(gen_y)
        yield batch_features, batch_labels

And last, but not least I am now able to use the batch_generator when training my model

model.fit_generator(batch_generator(128, generate_X(), generate_y()),
                    steps_per_epoch=(len(y_train)/128), epochs=2, verbose=1)


Is GNU Octave a perfect place to code neural networks

No.

The key to implementing neural networks easily and fast is having automatic differentiation. This is the key feature that allows programs using Python libraries like Pytorch or Tensorflow (and Keras), to implement only the forward pass of a neural network and have the backward pass computed automatically.

Octave does not have automatic differentiation. While Matlab's deep learning toolbox seems to include automatic differentiation, I think it is not included with Octave. And even if it were, the deep learning community is mainly focused on the Python ecosystem, and the rest of the environments like R or Matlab are paid almost no attention in comparison.


Random forest with zero precision for unbalanced test data

You mentioned that you're dealing with time-dependent observations, and I'm guessing that's why you chose a train-test split for evaluation.  If you continue along this path, then I think you just need a larger sample of class2 in your test set.

How did you come up with 25 examples of class2 in your testing set?  You should be using stratified sampling to produce your train-test sets.
It looks like you started with a 60-40 train-test split with class1.  If so, then your datasets should have the following distributions:

Train: 900,000 of class 1, 60,000 of class2

Test: 600,000 of class1, 40,000 of class2

Since you have a decent amount of data, you might want to train with more than 60%.  An 80-20 split would probably provide a better estimate of your model's final accuracy.

In any case, I think you can do better than train-test splitting!  There's a modified version of cross-validation designed for time-dependent data.  And scikit-learn has an implementation: TimeSeriesSplit

If you go this route, then you will probably want to use different weights for your classes.  Since class1 outnumbers class2 15:1, then you could assign a weight of 1 to class1 and 15 to class2 (assuming you care equally about the two classes).

To answer a few more of your questions:

  Should I undersample class1 from my test data?

I don't think this will help since you're already adjusting for the class imbalance with sample weighting.

  Are there more parameters besides class weights that I should try to tune?

You should always tune model hyperparameters :) For Random Forests, the most salient parameters are probably the number of trees, the max depth, and minimum impurity decrease.


Eliminate input in gradient by clever choosing of cost function in neural networks

The main reason is that the chain rule is hidden in your justification. The fact that the inner function is linear w.r.t weights then derivative w.r.t weight will always result into input $x$ will be somewhere in $\dfrac{\partial C}{\partial w}.$


problem of entry format for a simple model in Keras

model.evaluate requires both input and output, for example

evaluation = model.evaluate(np.random.random((1, 100)), np.random.random((1, 1)))

I think a step-by-step example would be more beneficial. Here is a working example:

from keras.models import Sequential
from keras.layers import Dense
import numpy as np

N = 1000
dimension = 100

# create some random input features (x) and output (y)
np.random.seed(0)
x = np.random.random((N, dimension))
y = np.random.random((N,))

# split the data into train and test sets
split = int(0.8 * N)
x_train = x[:split]
y_train = y[:split]
x_test = x[split:]
y_test = y[split:]

# build the model architecture
model = Sequential()
model.add(Dense(32, activation='relu', input_dim=dimension))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# train the model
model.fit(x_train, y_train, epochs=100)

# evaluate the model on train and test sets
train_loss = model.evaluate(x_train, y_train)[0]
test_loss = model.evaluate(x_test, y_test)[0]

print('train loss:', train_loss, ', test loss:', test_loss)

# predict (y) for a random input (x)
y_predict = model.predict(np.random.random((1, dimension)))
print('prediction:', y_predict)

which outputs binary_crossentropy loss:

train loss: 0.5500347983837127 , test loss: 0.7403841614723206
prediction: [[0.38731796]]

If you skip the training, i.e. commenting out

# train the model
model.fit(x_train, y_train, epochs=100)

the output will be

train loss: 0.7098221921920777 , test loss: 0.7191445398330688
prediction: [[0.32682237]]


DONUT- Anomaly detection Algorithm ignores the relationship between sliding windows?

The code is available on github and the sliding windows are actually not overlap. Below code from file donut/utils.py, method  minibatch_slices_iterator:

start = 0
stop1 = (length // batch_size) * batch_size
while start < stop1:
    yield slice(start, start + batch_size, 1)
    start += batch_size

I haven't inspect deeper but maybe the windows are not related, as you said.


Siamese Neural Network for massive class variation

I have some discussion about this problem with my friend and found a good solution. The answer is to using CNN not as a classifier but as feature generator (embedded) and store it as vector. The next is to using vector similarities to check whether the person is same or not or who is in the picture.


Different activation function in same layer of a Neural network

At a higher level, using multiple activation functions in a NN may not work as these activation functions operate differently for the same inputs. For an input of -1.56, ReLU will a 0, sigmoid will give a 0.174 and Tanh will give -0.91. These differences will not allow the gradients to flow uniformly during backpropagation.

You may try using two different activation functions in the same layers. I can think of some issues which may arise,

Talking about the combination of Tanh and ReLU functions, we can face the problem of exploding gradients. ReLU functions provide the same inputs as outputs if they're zero or positive. On the other hand, Tanh function provides outputs in the range [ -1, 1 ]. Large positive values will pass through the ReLU function unchanged but while passing through the Tanh function, you'll always get a fully saturated firing i.e an output of 1 always.
Therefore, while using ReLU in a classification problem, we use softmax instead of the sigmoid function. If the sigmoid function is used, you'll probably get an output of ones only.

Instead of the ReLU-Tanh combination, you may want to use the Sigmoid-Tanh combination.

Here, the sigmoid may give an output of 0 ( or a number close to zero ), if given a negative value. These negative values may be supplied by the Tanh function in the preceding layers. You may face overfitting here.


Why don't we transpose $\delta^{l+1}$ in back propagation?


Can I retrain my keras model on small dataset with some empty folders

What you are suggesting is to fine tune the network giving it a portion of the starting data set?

If yes, well you can try, but I am not sure what the outcome would be. But remember to keep a train and test set separated!


Find similar observations in two datasets

Look at "unsupervised nearest neighbor" algorithm. This algorithm needs records to be first expressed as vectors so that "distance" between two point so that it makes sense to talk about distance between two points. For each point in data A, you can look for K nearest neighbors from data B, after expressing all observations in a common vector space.

You will have to handle categorical columns correctly (say using one-hot encoding), as there's no concept of (direct) distance between categorical data.

Python scikit-learn library has a good implementation of this algorithm. Reading the API documentation is a good place to start.


How to extract a column that has the highest value within row in Hive?

I can't test in hive, but a possible SQL query is as follows (greatest returns the maximum value from the list):

select
  case
   when col1 = greatest(col1,col2,col3) then 'col1'
   when col2 = greatest(col1,col2,col3) then 'col2'
   when col3 = greatest(col1,col2,col3) then 'col3'
  end as c1
from test;

Additional note:
you should check, how ties are to be handled, in my solution I simple take the first column.


using feature selection to improve model performance

370 rows, with approx. 900 features is not optimal. I would suggest some dimension reduction. PCA, factor analysis, PLS regression are some alternatives.

You could try a lasso- / elasticnet -regression as well.

Here is a good guide. https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html


What kind of research can be done with an email data set?

The following are some research that can done on e-mail dataset:

linguistic analysis to abbreviate an email message
Categorize e-mail as spam/ham using machine learning techniques.
identifying concepts expressed in a collection of email messages, and
organizing them into an ontology or taxonomy for browsing


Let's start with normal dropout, i.e. dropout only at training time. Here dropout serves as a regularization to avoid overfitting. During test time, dropout is not applied; instead, all nodes/connections are present, but the weights are adjusted accordingly (e.g. multiplied by the keep ratio, which is 1 - dropout_ratio). Such a model during test time can be understood as a average of an ensemble of neural networks.
Notice that for normal dropout, at test time the prediction is deterministic. Without other source of randomness, given one test data point, the model will always predict the same label or value.
For Monte Carlo dropout, the dropout is applied at both training and test time. At test time, the prediction is no longer deterministic, but depending on which nodes/links you randomly choose to keep. Therefore, given a same datapoint, your model could predict different values each time.
So the primary goal of Monte Carlo dropout is to generate random predictions and interpret them as samples from a probabilistic distribution. In the authors' words, they call it Bayesian interpretation.
Example: suppose you trained an dog/cat image classifier with Monte Carlo dropout. If you feed a same image to the classifier again and again, the classifier may be predicting dog 70% of the times while predicting cat 30% of the time. Therefore you can interpret the result in a probabilistic way: with 70% probability, this image shows a dog.


How does the equation "dW = - (2 * (X^T ).dot(Y - Y_hat)) / m" comes in Linear Regression (using Matrix + Gradient Descent)?

dW and db are simply the derivative of the loss function with regards to the weights and biases. Given the loss function
$J = \frac{1}{m} \Sigma_{i=1}^{m}(y_i - h(x_i))^2$
the derivatives of the loss to the weights (dW) and bias are equal to
$\frac{\partial}{\partial W} J = -\frac{2}{m} \Sigma_{i=1}^{m}(y_i - h(x_i)) * x_i$
$\frac{\partial}{\partial b} J = -\frac{2}{m} \Sigma_{i=1}^{m}(y_i - h(x_i))$
As you can see, these equations align with the code you provided. Equation 1.4 from the link you provided is for $\theta_0$, i.e. the bias, which is the same as the code for db (with the only difference being the minus sign, causing by the fact that $y_i$ and $h(x_i)$ are swapped in the loss function between the first and second article).


I will answer, assuming that since you wanted to do text classification, your features were created using some TFIDF feature extraction technique. Next time, please specify the dimensions of your features and not only their storage.

According to the documentation (http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html), the object has a coef_ attributes that stores all of the weights of your model. Since for text, feature vectors are about the size of your vocabulary, it can be fairly large, especially if you do not preprocess your text using stemming or if you do not remove stopwords or irrelevant words.

The documentation also says that the size of the coef_ is number of classes * number of features. So depending on how many features and classes you have, this can get large quickly.

This could explain partially why the object is so large. Other factors could be that Python often stores the predictions in the object after you train it, so that you can access them quickly, so you would need to add another 600,000 numbers to your object. And for this object, it seems to also store the confidence scores, which is another 600,000.


Example of Logistic Regression using a proportion as the dependent variable?

As the link by @Spacedman shows, binomial regression works well. If you don't have the attempts and successes but, instead, have just the proportion, then you'll want to use beta-regression. After a little digging, it doesn't seem like this is available in Python. Here's a blog post demoing it in R.


Appraise the statement: “For the model  = 0 + 1 + , 1 reflects the causal effect of  on .” Ask

Hi and welcome to the forum. Homework is not so well received here in the forum. But still a fair question in my view.

You have two aspects here. In principle you are right that $\beta_1$ is the slope of $x_1$ (you can say the marginal effect of $x_1$ on $y$) and $\beta_0$ is the intercept. This is simply a linear function of form $f(x)=\beta_0 + \beta_1 x$.

However, to claim "causality", a few more things are required. First, you need to make the assumption that there is a causal relation between $x$ and $y$ and $x$ must be exogenous.

Another important aspect is, that if there are additional variables with a causal influence on $y$, say $x_2$, you cannot really claim that $\beta_1$ is the causal effect on $...$, because you omitted $x_2$, so that your model suffers from the omitted variable bias. To claim for causality you need to make sure that your model reflects the data generating process in a proper way.


What is the difference between step size and learning rate in machine learning?

Both learning rate $\eta$ and step size $\Delta w$ are linked to gradient descent. In the most simple case they are linked by :

$$ \Delta w = w(t+1)-w(t) = -\eta \frac{\partial E(w)}{\partial w}$$

Where $t$ is the epoch and $E$ the error function.

In that simple case, they only differ by $ - \frac{\partial E(w)}{\partial w} $, which sometimes lead to use one term instead of the other. However in a more general case (learning rate depending on weights, learning rate depending on epoch, added momentum, or minibatch learning) the distinction may have a greater importance.

Edit : I answered the title of your question, in the body you seems to refer to the number of step, which is the number of iteration or number of epoch of the aforementionned formula.


Missing at random vs missing not at random: What if it is both? (Does one imply the other?)

Missing at random (MAR) means the NA frequency of the variable is never depended on the value of the variable itself.
Therefore in your example the data would be Missing not at random (MNAR)!
Why is this distinction important?
Because when the data is MNAR we have to identify the relationship between missingnes and the value however if the data was truly MAR or MCAR we could ignore the NAs or impute them with simple methods like mean-imputation.
Correctly identifying wether data is MCAR,MAR or MNAR is the only way to correctly identify how to deal with it!
But how can something be MAR and not MNAR?
Imagine you are asking senior citizens their birth month but also measure whether they have Alzheimers or similar memory impairments.
Birth month would likely be MAR as the fact whether it is missing correlates with the Alzheimer variable yet I have no information about the actual birth month from this fact.
I can predict from other variables whether the data is not missing but not what the actual value of it is!


From the Amazing Blog - FloydHub Blog- Attention Mechanisms

Attention Mechanisms

Attention takes two sentences, turns them into a matrix where the words of one sentence form the columns, and the words of another sentence form the rows, and then it makes matches, identifying relevant context. This is very useful in machine translation.

When we think about the English word “Attention”, we know that it means directing your focus at something and taking greater notice. The Attention mechanism in Deep Learning is based off this concept of directing your focus, and it pays greater attention to certain factors when processing the data.

In broad terms, Attention is one component of a network’s architecture, and is in charge of managing and quantifying the interdependence:

Between the input and output elements (General Attention)
Within the input elements (Self-Attention)

Let me give you an example of how Attention works in a translation task. Say we have the sentence “How was your day”, which we would like to translate to the French version - “Comment se passe ta journée”. What the Attention component of the network will do for each word in the output sentence is map the important and relevant words from the input sentence and assign higher weights to these words, enhancing the accuracy of the output prediction.

Weights are assigned to input words at each step of the translation

I recommend having a read through this article - Attention Mechanism

More at - Attention Mechanisms and Memory Networks


'DataFrame' object has no attribute 'to_dataframe'

The function pd.read_csv() is already a DataFrame and thus that kind of object does not support calling .to_dataframe().

You can check the type of your variable ds using print(type(ds)), you will see that it is a pandas DataFrame type.


You should look at literature on unsupervised sentiment analysis. The paper by Peter Turney could be a good starting point.
Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews Turner 2002

You can also check this if you use R https://datascienceplus.com/unsupervised-learning-and-text-mining-of-emotion-terms-using-r/


Keras ImageDataGenerator.flow_from_directory doesn't find images

SOLVED!

Always be sure you have '.png' or whatever other file extension you are using at the end of your filename


Torch and conceptual question about neural nets backpropagation

I have been using torch for a few months now but I will give it a go (apologies if incorrect).

Yes a weight $w$ is updated as follows;

$$ w_{new} = w_{old} - \gamma \partial E/ \partial w_{old}  $$

where $ \gamma $ is your learning rate and $E$ is the error calculated using something like criterion:forward(output,target). The criterion could be, for example, nn.MSECriterion().

To calculate $\partial E/\partial W$  you need $\partial E/\partial y$ gradOutput = criterion:backward(output,target)(gradient respect to output) as well as the input input to the net i.e. your $X$ (e.g. image data) to generate the recursive set of equations which multiply with gradOutput.

model:backward(input, gradOutput) therefore serves to update the weights so that they are ready for the next model:forward(input) as it generates a big derivative tensor $dE/dW_{old}$.

This is then combined with a optimiser such as optim.sgd using optimMethod and the old weights $W_{old}$ to generate the new weights in the first equation. Of course you can just update the weights without an optimiser with model:updateParameters(learningRate) but you miss useful stuff like momentum, weight decay etc.

Got a bit side tracked there but hope this helps.


Word Embedding for Item Names(integer, one-hot encoding)

I'm not sure, if it's possible with this data set. Word2Vec is used to generate word embedding, which works on the principle of "words association" in a sentence.

So I dont think you can apply Word2Vec on this dataset which looks like doesn't have any association, except on some places where you can match(perform clustering) some parameters like:

Units
Size/dimension of the item-name

Interested to know some solution for such types of problems.


How to select best feature set and not ranking for tree based models?

I believe that what you are looking for is Best Subset, Forward Stepwise, or Lasso?

Here is the R implementation Best Subset

If anyone knows a python implementation, please let me know!


Where does the "deep learning needs big data" rule come from

The original slide in question "Scale drives deep learning progress" is possibly what you currently can find at https://cs230.stanford.edu/files/C1M1.pdf (page 13). It may be roughly interpreted with "low bias learners [in that plot, larger neural networks] tend to benefit from more training examples".


What is sensitivity in confusion matrix?

Sensitivity or true recall rate:

Sensitivity is calculated as the number of correct positive predictions (TP) divided by the total number of positives (P)


Classification with heavily overlapping data points of different labels

There are many approaches you could take here. Assuming data overlap in such a way that the classes are still distinguishable from a human expert—human performance is generally regarded as the performance ceiling in supervised classification tasks—you can approach this from the classifier side of the feature side.

First, I would consider the feature representation you're using for your task. A classifier is only as good as the information you give it, and there are many ways you can adjust how you're representing your input data. For example, if you're working in text classification, using a unigram representation will typically lead to different performance as compared with a bigram representation. Similarly, there may be non-linear transformations you can apply to continuous-valued data that a classifier may not pick up on its own. You should consider the domain in which your problem exists and create a set of feature generation and feature selection cross-validation experiments using a hold-out training data set.

Next I would consider what alterations you can make on the classifier end. If your data are highly skewed (i.e., one class is much more prevalent that another), it may not make sense to use an algorithm like Naïve Bayes, at least not with out some sampling approach. If your problem isn't high-dimensional, I recommend generating a graphic that will let you explore the spatial breakdown by class. There are certain overlap patterns that linear classifiers like Support Vector Machines will not be able to classify without using a kernel function, such as a Radial Basis Function.


One tool that may be helpful is Aiden Gomez's blog post. Its strength lies primarily in the fact that he runs through a toy/numerical example, which when paired with the original paper/thesis , serves as a great foundational tool.

I did take a look at the site you mentioned in your other question, it's actually an excellent resource. I'll hop on over and try and clarify what I can for you when I get a chance. It looks like you've misunderstood/overlooked notation which can happen since there are so many components involved.

It may also be worth taking a look at some code. Siraj Raval has a great video on LSTMs and includes the code in the link I've included. No libraries. I wouldn't dive too deep, but it's a great way to see the inner workings of the network.

As far as the CEC goes, there is a reddit post. If you're looking for a more rigorous handling of this topic you can either reference the original paper or often cited paper: On the Difficulty of Training Recurrent Neural Networks.


Evaluating the performance of a machine learned recommendation system

To the extent possible you should try to evaluate based on your data rather than some ad-hoc measure. As you rightly noticed, there is a real risk that the ad-hoc measure would just confirm the predictions of the model, since it uses a somewhat similar method.

I would suggest that you split your data between a training set and test set (or even better use cross-validation), and indeed use top-K accuracy (or something similar) to evaluate on the test set. That would be the safe option for a proper evaluation, and then you could try to see if your ad-hoc measure correlates with it: if it does, then you have evidence that in the future it can be be used instead of a test set.

Side note: your instances don't contain any negative evidence such as resumes rejected by an employer. In case you could obtain this kind of data, it could probably improve the predictions.


Should I repeat the testing/training split while repeating K-fold cross validation?

The purpose of k-fold cross validation is to fit the model multiple times and average out to results to estimate the model's predictive performance.
If having done 10-fold CV once and got one result and then repeating it again and getting a significantly different result, probably means the number of folds is too low, and the k-fold process isn't capturing the inherent variability in the data sufficiently well.
A better test is to change the number of folds and see what happens to the average and variance of the combined output.


Handling features with multiple values per instance in Python for Machine Learning model

A possible resource is featuretools, they do feature engineering on data that has many records. Their examples are not from medical cases but I think it should work for you too.
You can also manually build several features. For instance, given a list of abdomcirc, you can compute its:

mean
maximum
minimum
variance
difference from minimum to maximum
last value (if they are sorted by date)
number of unique values

These features would get most of the information of the abdomcirc list, and this should help your modelling.
I wouldn't go for the first approach of giving lists to the algorithm, although it is possible, I think it is a relatively advanced thing and I wouldn't go for it unless the simpler approaches don't work.


Multivariate Time-Series with Differing Selling Windows

Welcome to the site! This is a fairly broad question and impossible to answer in a single post. So, I would have you start by researching recommender systems and algorithms. On top of that, if you fully want to take the stores into account, I would start collecting a dataset that contains attributes of each store - where are they? How many hours are they open? Do they have a lot of traffic? etc.

If you proceed by using the products, then you would research a "content-based recommender system". If you want to proceed by looking at the stores, then you would research a "community based recommender system". From there you should be able to pick a model and/or choose a hybrid of the two approaches and develop a proper solution.

Good luck!


Suitable sample data set to test machine learning algorithms

The toy examples or the common datasets you are talking about are so because they are simple to visualise it and work with. Their simplicity helps the beginner to train simple models which don't require much compute. The simplicity in the structure of the dataset allows visualisation of the data on lower dimensions.

Reason for using them as test datasets is that they provide us a quick sanity check to see if the algorithm performs or not. The link you provided is specifically for clustering problems. So, datasets which can be easily visualised on 2D plane would be a simple dataset to check the performance of the algorithm via inspection. Had it been a complex datasets like a dataset of human faces, it would be difficult to evaluate the performance of the model through visualisation and inspection.

Some examples for such datasets:

MNIST dataset - collection of handwritten digits used to train classification network to identify the class of digit during test time.

Cifar-10 : collection of RGB images of 10 classes of objects in real world (e.g cars and birds).

Cifar-100: upgrade of Cifar-10. Contains images from 100 classes


Word2Vec Alternative for Smaller Datasets

You can side step the paucity of training data, and indeed training altogether, by using pre-trained embeddings in numerous languages. After that you can calculate your document embeddings using one of these simple algorithms, which basically amount to running dimensionality reduction on the matrix of stacked word embeddings for each sentence using PCA/SVD:

A Simple but Tough-to-Beat Baseline for Sentence Embeddings (code)
Representing Sentences as Low-Rank Subspaces

Note that word embeddings themselves emerge from similar calculations:

Neural Word Embedding as Implicit Matrix Factorization


Standard method to integrate tools coded in multiple languages in an analysis workflow

Luigi is an open source python package by Spotify that does exactly what you described:

  Luigi is a Python (2.7, 3.3, 3.4, 3.5) package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.

Its philosophy is similar to GNU Make, letting you define tasks and their dependencies.

There is also Apache Airflow (originally from Airbnb), another python solution:

  Airflow is a platform to programmatically author, schedule and monitor workflows.
  Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies.

You may found a complete comparative table here.


In a directed graph, how to measure whether a node is more "upstream" or "downstream"?

Please clarify what you are looking for in the presence of cycles.  I'd assume the cycles aren't standalone- any node in a cycle could also have inputs from further upstream, and could also send output further downstream. Even in the unweighted case, it seems to me that some nodes in a cycle can be more "upstream" than others.  For instance, most of the nodes in the cycle might be downstream of a set of source nodes.  One node in that cycle might be the only path out of the cycle to a sink.  Surely that node is more "downstream" than the others in the cycle?

Sorry the above isn't just a comment - I'm too new to have high enough reputation in this group for that.

So, to give an answer as well.  Take the "flow network" analogy to an extreme.  Imagine each source upstream of the target node puts out a water flow, with dye added at the first instant only. Treat the weights as time delays.  For each node, calculate the first time the dye reaches a given node.  Among all the sinks downstream of your target node, pick either the longest time or the mean time (or itself, if the node is a sink).  The target node's position is then just its early arrival time divided by the sinks' mean or longest arrival time.  Computationally, this earliest dye arrival time is done by some modification of breadth first traversal of the directed graph.  Start at the sources with arrival times of 0, and then each node arrival time is just the minimum over all its predecessors.  (As in any graph traversal, visited and completed nodes have to be remembered so cycles aren't followed more than once. This would also be called shortest distance from a set of nodes to a given set of nodes.

But note that cycles aren't given any particular special treatment.   If you really want cycles to matter, you could assume that output flows are split equally, and do mixing calculations, so that the dye concentration would decay gradually over time.  But that would seem ridiculously complex, and take time, for probably no benefit.  But in either case, the nodes in cycles won't all have the same value - some will be more "upstream" than others.

If you really want all the nodes in a cycle to have the same value, you might need a separate pre-processing step to find the directed cycles, and aggregate each cycle into a single node (one for each cycle).  That adds complexity, although it simplifies the shortest distance calculations since then the graph will be acyclic and a trivial breadth first traversal will work.

You probably don't really want to require a global analysis of a graph to get your feature value, although that's what you had to do in your initial analysis too.  If this is something you would run on a regular basis, you could probably just do local updating when changes occur, with limited propagation of earliest arrival times, assuming you could tolerate some "error" in the somewhat arbitrarily-chosen metric anyway.


As a first step, You can take the table of measurements as-is, run k-means on all the columns representing intensities.
In the image, I see pixels in 5 colors (red, orange, green, azure, magenta), so the author of the top image has decided to run k-means with k=5 clusters. (k in in k-means is often chosen to be an odd number)
To get a clearer signal, people often run preprocessing steps such as Principal Components analysis (PCA) on their datasets before running k-means, and then choose the first, say, 10 components (should be larger than 5). Then people run k-means on those 10 columns.
Choosing the number of components to calculate k from, and then choosing a value for k, is basically guesswork, at least initially.
It is also guesswork to assign colors to cluster numbers found by the k-means algorithm.
So you probably have to set a lookup table manually, in another preprocessing step, such as
{
1 => "red".
2 => "azure",
3 => "limegreen"
4 => "yellow",
5 => "orange" }. And then make an x-y plot coloring the pixels accordingly. The sequence is arbitrary.
Sorry, I don't know how to do this in Orange.


How to deal with the difference in the range of Dependent variable in Train and Test data

The assumption in supervised ML is that the test set follows the same distribution as the training set. It's clear from your description that this assumption is not satisfied in your data, so it's not very surprising that things don't work well. Beyond the difference in the range of the dependent variable, if the training and test set have been collected independently it's likely that the distributions of the features also differ.
So how to fix this depends on the goal of the task and in particular why the test set and training set don't follow the same distribution:

If the difference is not meaningful, i.e. there is no reason to assume that real production data would be different than the training set, then the experiment should be redone after properly randomizing the split between training and test set.
If the difference is meaningful, i.e. the test set is purposefully different from the training set because the production data will also be different, then the setup should be redesigned to account for that. For example there could be a step of semi-supervised learning on the test set, which would adapt the model to the new data distribution while leveraging the information from the training data.


In a list, find the numbers where they are bigger than the next numbers

I think the following is the easiest and fastest way:

Create a new vector with the "next neighbors"
Subtract the "neighbors" vector from the original vector
Determine which entries in the new vector are positive

I've actually never used R, but here's how I think it would work:

original_list <- c(7, 7, 9, 9, 9, 10, 10, 1, 51, 51)
# extract the "right neighbors" of the original list elements
neighbors <- original_list[-1]
# fill the missing position after the last element
neighbors <- append(neighbors, 0)
# compute the difference between the original vector and the right-neighbors
diff <- original_list - neighbors
# find the indices of positive elements
idx <- which(diff > 0)

# your output: elements which have a smaller right-neighbor
output <- original_list[idx]


Semantic segmentation in high-resolution images with high variance - cannot avoid underfitting

Given your mask is very small, you should look at reducing your convolutions to 2x2 since that will help aggregate more information from these smaller masks. EfficientNet has 3x3 and 5x5 convolutions which may not be suited for your purpose.
It is a better idea for you to train from scratch using smaller convolutions (2x2). Also, since you will lose the edge of transfer learning, look at self-training, if it is possible for your network.


How to sort list by parameter in python?

You can use lambda function to pass argument by which you need to sort the records.

For example,

sorted_list=sorted(emp_records, key=lambda emp : emp[1])   //Here 1 means ID
print(sorted_list)

You can also use the operator module. The operator module has the itemgetter function, which takes item as argument and sort based on the argument

For example,

from operator import itemgetter
sorted_list=sorted(emp_records, key=itemgetter(1)) // Here 1 means ID
print(sorted_list)


Character-based word representation using bi-lstm

This would generally only be a part of a network, used for some task like classification, entity recognition or machine translation. The loss used for this task will guide the learning of the embedding space. This is different from approaches like word2vec or GLoVe where you use context windows to learn an embedding, which is an unsupervised (or maybe self-supervised) approach.

The approach that you mention here could still be used in these scenarios, by structuring it with a similar loss as word2vec or GLoVe. One advantage would be that you can leverage word similarity (words that end with -ly have a similar function). However these models are usually trained on incredibly large corpora where these character based similarities are not that relevant, and the computational cost increases quite a bit by incorporating bigger models like LSTMs.


Why averaging the gradient works in Gradient Descent?

Each training sample ends up in a distant, completely separate location on the error-surface

That is not a correct visualisation of what is going on. The error surface plot is tied to the value of the network parameters, not to the values of the data inputs. During back-propagation of an individual item in a mini-batch or full batch, each example gives an estimate of the gradient in the same location in parameter space. The more examples you use, the better the estimate will be (more on that below).

A more accurate representation of what is going on would be this:

Your question here is still valid though:

  But why does averaging the gathered gradient work?

In other words, why do you expect that taking all these individual gradients from separate examples should combine into a better approximation of the average gradient over the error surface?

This is entirely to do with how the error surface is itself constructed as the average of individual loss functions. If we note cost function for the error surface as $C$, then

$$C(X, \theta) = \frac{1}{|X|}\sum_{x \in X} L(x, \theta)$$

Where $X$ represents the whole dataset, $\theta$ are your model's trainable parameters, $L$ is an individual loss function for $x$. Note I have rolled labels into $X$ here, it doesn't matter for this argument whether loss is due to comparison of model output with some part of the training data - all we care about is finding a gradient to the error surface.

The error gradient that you want to calculate for gradient descent is $\nabla_{\theta} C(X, \theta)$, which you can therefore write as:

$$\nabla_{\theta} C(X, \theta) = \nabla_{\theta}(\frac{1}{|X|}\sum_{x \in X} L(x, \theta))$$

The derivative of the sum of any two functions is the sum of the derivatives, i.e.

$$\frac{d}{dx}(y+z) = \frac{dy}{dx} + \frac{dz}{dx}$$

In addition, any fixed multiplier that doesn't depend on the parameters you are taking the gradient with (in this case, the size of the dataset) can just be treated as an external factor:

$$\nabla_{\theta} C(X, \theta) = \frac{1}{|X|}\sum_{x \in X} \nabla_{\theta} L(x, \theta)$$

So . . . the gradient of an average of many functions, is equal to the average of the gradients of those functions taken separately. Taking any completely random subset of $X$ will result in an unbiased estimate of the mean gradient, same as taking a random subset of any variable and taking its mean will give you an unbiased estimate of the population's mean.

This will not work if your samples are somehow correlated, hence why you will often see recommendations to shuffle the data prior to training, in order to make it i.i.d.

This will also not work if your cost function combines examples in any way other than addition. However, it would be an unusual cost function that combined separate training examples by multiplying their loss functions, or some other non-linear combination.


Creating dataset - imbalanced or balanced?

The easiest option to close your approach is too just run the model twice in both variations. But in my experience this specific data is bound to be imbalanced.
Your idea to fix the class imbalance is technically known as under-sampling. This is one technique which is used. You can try out a few other techniques which are mentioned here: https://www.kdnuggets.com/2017/06/7-techniques-handle-imbalanced-data.html


Interpreting the evaluation result of multiple linear regression

I am going to answer your questions one after another.

First, what do the 44 degrees of freedom mean?

It simply means that the model you built is constructed by using 44 independent variables. For example a model that looks like y = ax + b has 1 independent variable (i.e. a) and thus 1 degree of freedom. A model that looks like y= ax1 + b*x2 + c would have 2 independent variables (i.e. a and b) and thus 2 degrees of freedom.

Second, what is the Multiple R-squared?

Here for the purpose of interpretating it Multiple R-squared is equivalent to the (simple) R-squared you would have for a linear regression model with 1 degree of freedom. Multiple R-squared tells us the share of the observed variance that is explained by the model. For example if you have a Multiple R-squared of 0.79 it means that your model explains 79% of the observed variance in your data.

Third, what is the Adjusted R-squared and why do we need it?

There are several problems with Multiple R-squared.

Problem 1: Every time you add a predictor to a model, the R-squared increases, even if due to chance alone. It never decreases. Consequently, a model with more independent variables (more degrees of freedom) may appear to have a better fit simply because it has more independent variables.

Problem 2: If a model has too many predictors and higher order polynomials, it begins to model the random noise in the data. This condition is known as overfitting the model and misleadingly high R-squared values and a lessened ability to make predictions.

Problem 1 is caused by Problem 2. And this is where Adjusted R-squared is coming in handy. Adjusted R-squared is an attempt at fixing these problems by factoring in the number of independent variables. The adjusted R-squared tells you the percentage of variation explained by only the independent variables that actually affect the dependent variable.

where:

n is the number of data points you have,
and k is the number of independent variables used to explain their distribution, excluding the constant

If you add more and more useless variables to a model, adjusted r-squared will decrease. If you add more useful variables, adjusted r-squared will increase.
Adjusted R-squared  will always be less than or equal to R-squared . You only need R-squared when working with samples. In other words, R-squared  isn’t necessary when you have data from an entire population.

Here is an interesting series of articles which will help you understand how to use R-squared to interpret the results of your model even better.

Regression Analysis: How Do I Interpret R-squared and Assess the Goodness-of-Fit?
Multiple Regression Analysis: Use Adjusted R-Squared and Predicted R-Squared to Include the Correct Number of Variables


Is it appropriate to use random forest not for prediction but to only gain insights on variable importance?

A Random Forest model can definitely be used to help you determine feature importances. Actually, it is used as a very common strategy for feature selection. If your data is too small, my recommendation would be to treat it as if you were to make predictions with this model, meaning that you should watch out for overfitting and do a proper hyperparameter optimization (you want a simpler and smaller model here), so that the features you find are really the most relevant, and don't just appear important for being biased by the small training set.


Working with inaccurate (incorrect) dataset

I have read @MaximHaytovich's answer and it is a good one. I would just like to suggest some further options, which were generalized in that answer as feature engineering. I would suggest trying to do the obvious first and analyze the data before transforming it to become ready for any machine learning algorithm. First look at the data and learn yourself and hypothesize about what patterns could be indicators of fraud. What is the nature of the data that you are 100% sure is reliable (it is mentioned in the question you posted that there is available data that is reliable) and how informative are they for the goal? The more familiar you are with the data, the better interpretation you can provide to any output you get from the methods you use. Try doing some outlier analysis on the incomes or any features that may be fraudulent that may make sense to do outlier detection on. Are there any suspicious missing values? Try doing clustering (one of the unsupervised algorithms you were talking about) to see if there are any in-betweens or irregularities.


What does sklearn's pairwise_distances with metric='correlation' do?

Y = pdist(X, 'correlation')

Computes the correlation distance between vectors u and v. This is

$$ 1-\frac{(u-\overline{u})\dot{}(v-\overline{v})}{||u-\overline{u}||_2*||v-\overline{v}||_2} $$

where $\overline{u}$ is the mean of the elements of vector $u$, and $x\dot{}y$ is the dot product of $x$ and $y$.

The correlation between any vector which have ONLY TWO entries is always 0 (or nearly zero: $2*10^-16$), why? Because correlation distance measures the distance as the linearity between the data.

When I have [1,2] and [1,2] the equation $y=x$ fits perfectly, when I have [1,2] and [1,3] the equation $y=x+1$ also fits perfectly. The correlation distance says wheter a equation can be drawn for the data, in both cases the equation is perfect.

If you want to try getting a different result, try putting 2 vectors of three elements, and you will see changes.

Try with: [[1,2],[2,3],[3,4]] and [[1,4],[3,8],[-5,6]]. But first, plot them and you will comprehen what 'correlation' measures.


Which Algorithm or combination of Algorithms to use to develop supervised Video Event detection?

I recommend using a 3D convoluted neural network. Traditional 2D convoluted neural networks excel at identifying objects in images, and 3D convolution is used for identifying objects or movements also across multiple frames (i.e., time). Tensorflow offers this capability.


What is the difference between concept learning and classification?

The objective is rather different.
For classification problems, we want to know: GIVEN X, WHAT WILL BE Y?
For concept-learning problems, we are asking: WHAT KIND OF X WILL GIVE US Y?

It may seem similar, but for classification problems, we may use a model that is unable to generalize and answer "WHAT KIND OF X WILL GIVE US Y" until you plug X into the model (e.g. K-nearest neighbor).

On the other hand, concept-learning problems, the main issue is to generalize and answer what can be X specifically.


Assigning a new document to a cluster based on keywords extracted and tf-idf

It seems that you are already half-way there.
If you have divided the documents to clusters, it implies that you already have some feature extraction method that was used to quantize each document to some feature vector.

Now, when you get a new document, you need to extract its features the same way and then use its feature vector to find the best fitting cluster for it.

The best fitting cluster can be determined by using 1NN (Nearest Neighbor) algorithm on either the cluster centers or one of their affiliated document vectors. If you use the vectors and not the cluster centers, you can also you try KNN algorithm (K-Nearest Neighbors).


Choose-many categorical features: alternatives to one-hot encoding?

If your algorithm is based on gradient descent optimization, you can use embeddings, which are dense representation spaces for discrete elements.
Embeddings are supported by most deep learning frameworks such as pytorch or tensorflow.
Update: the fact that you want to have multiple discrete values does not prevent the possibility of using embeddings: you can just add all vectors together in a single value. The most straightforward approach for this would be to have a constant length for the list (equal to the maximum number of elements in all lists, or a sensible maximum value), filling with "padding" items the positions that are not needed. If you want to take the sequential appearance of the elements into account, instead of adding the vectors together you could apply convolutional layers or an LSTM over the embedded vectors.


Error with Jupyter Notebook and Google Cloud

Same issue here. Oddly enough I had another instance regirested before in the same way and it all worked. Unsure what is going on, but suspecting the VM is rejecting to connect for some reason.


What is the purpose of a confusion matrix in a classification problem?

Ok, so let me answer some of these questions for you:

What is the purpose of a confusion matrix?

A confusion matrix is merely a visual help for you to better interpret the performance of your model. It's a way to graphically visualize the true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). Confusion matrices become more and more useful when you deal with a large number of different classes. It can give you some great insights about how your model is doing. Let's say you are training an image detection classifier. It might be good to know that your model is confused between dogs and wolves but isn't confused between cats and snakes.

Another purpose of the confusion matrix is to define a related cost matrix. In my example, being confused between dogs and wolves might be understandable and it shouldn't mean that your model is bad at what it does. However, if it is confused between classes it shouldn't be confused about, this should be properly represented in your performance metric.

Here is a good blog detailing these concepts: https://medium.com/@inivikrant/confusion-cost-matrix-helps-in-calculating-the-accuracy-cost-and-various-other-measurable-a725fb6b54e1

Shouldn't we already know what metric we need to optimize given the type of problem we are trying to solve?

Here you are confusing two things. On the one hand, yes, you should know in advance which metric you want to optimize (i.e. accuracy, precision, recall, etc.) but it doesn't mean that you know the value of that metric in advance. If you dumb down hyperparameter tuning, it's roughly this:

train model $M$ with hyperparameters $H$
evaluate the performance $P$ of model $M$
choose new hyperparameters $H$ and repeat step 1 and 2
pick model $M$ with hyperparameters $H$ such that $P$ is optimized

Computing the accuracy, precision, recall, or F1-score can be done if you know the TP, FP, TN, FP (see this link for more info). So technically, you don't have to create the confusion matrix per se, but you definitely need to compute the TP, FP, TN, FP to evaluate the performance of your model.

if we have enough computational power, we could basically group steps 2 and 5

You can only optimize your hyperparameters if you compute the performance of your model for each set of hyperparameters. You could skip step 3 as it doesn't technically affect your training process. It only helps you understand better what's going on. But you definitely cannot skip step 4.


What is the purpose of the 'train model' step in data mining?

There is a very important difference in Machine Learning (ML) between supervised methods and unsupervised methods:

Supervised learning consists in training a model with some labelled data in order to make the final model able to predict the label on some new (unlabelled) data. This means that the task is designed by choosing exactly what what one wants to predict. For example the task of predicting the author of a text is completely different from predicting the topic of a text, even if the text might be the same.

Note: A "label" is for categorical data, and the task is called classification in this case. The same principle can be applied to numerical values, in which case this is a regression task.

Unsupervised learning consists in detecting patterns in the data with no additional information than the data itself. This means that there's no specific "label" to predict. Very often unsupervised learning is some form of clustering, i.e. grouping instances by similarity. Data mining methods usually belong to this category. In general there is no need for separate training and testing steps with unsupervised methods. However one might need to tune some parameters or sometimes to evaluate the model with some annotated data, so this would require separating training and testing again.


Training a CNN with limited weight sharing

It's important to note that before doing this, they split the frames into individual feature maps.  Note Fig. 1:

The "45 rows" in the upper left of your diagram (Fig 4.) corresponds to 15 frames times 3 feature maps.

The convolution is a 1D convolution, so they are not actually convolving across multiple frames, only across frequencies.  The authors then sum across feature maps in the normal way, meaning summing across frames and features.

Another way of looking at it is in Eq. 9

The variable $O_i$ is a vector across frequency bands only.  The sum is over feature maps which includes frames.


Will historical data lead to target leakage?

I'm going to kind of ignore your question. To me it sounds like you have a dataset containing essentially employee starting and 'end dates' or if the employee is still working for you no end date. This sounds like the standard case for a class of regression techniques called 'survival regression' or survival analysis. I suggest you look into this.

Now for your question, Im not too sure about target leakage as I cannot quite see how, if you have a structural change in employees year by year then theres definitely some leakage yeah.


In general, if the agent is simply not able to take non-valid actions in a given environment (e.g. due to strict rules of a game, like chess), then it is standard practice to have the environment support that by providing some kind of function or filter for $\mathcal{A}(s)$, the set of actions available in state $s$.

However, it does seem like the basic Gym interface does not support it, and has no plans to support it.

It is still possible for you to write an environment that does provide this information within the Gym API using the env.step method, by returning it as part of the info dictionary:

next_state, reward, done, info = env.step(action)

The info return value can contain custom environment-specific data, so if you are writing an environment where the valid action set changes depending upon state, you can use this to communicate with your agent. The caveat is that both your environment and agent would be working to a convention you had invented in order to do this, and the same approach would not extend to other environments or be used by developers of other agents.

Alternatively, you can support calculating what the valid actions are in any other custom way that is not covered by the Gym API - a new method on the environment, a separate "rules engine" for a game etc. Both the agent and the environment could call that in order to perform their roles correctly.

The situation without an official API to do this may be subject to change, but from the linked Github issue it seems that Open AI developers consider a generic interface for this, that accounted for all the different kinds of action space, was more effort than it was worth.


Reverse of Yeo Johnson variable transformation done by preprocess function in caret package to get original Target and predicted values

I am not sure there is an elegant caret solution, but VGAM offers what you are looking for, an inverse of the yeo-jhonson transformation : https://www.rdocumentation.org/packages/VGAM/versions/1.0-3/topics/yeo.johnson.


Proof subtracting baseline doesn't influence gradient can be used to show no gradient exist at all?

After thinking about this, I've realized that $Q(s,a)$ relies on the action and thus cannot be pulled out of the sum in the same way $B(s)$ can. I'm leaving this up for anyone interested in the same thing.


Two different cost functions for neural networks, how they can give the same result?

I don't remember exactly what the book has mentioned, but I guess the difference between the two is due to having one or multiple features. I guess it is already mentioned in the book. They are the same. One is for multi-dimensional input, and the other is for one-dimensional. One sigma is iterating over the features and the other is iterating over the examples. You can put $k$ to one to achieve the simpler formula.


Which kind of model is better for keyword-set classification?

You can leverage word-vector similarity in embedding models.

TL;DR similiar vectors of words (for example fruits) will be clustered together in this high (vector) dimensional space. For every possible class-set you will have a class-set representative (centroid) that is actually a key (so in your case fruit, vegetable etc) all you need to do is train/find a representative word embedding model of your corpus.


Mirror column plot with different y-axis in ggplot

You can either reshape the data into long format (after joining the two datasets) and use geom_col once, or have two geoms as you have in your question. However, the latter needs some variation to have the aes in geom_col and not in ggplot.

Below, I have an example with mpg dataset, illustrating the two options (except the joining). I also included an example with your datasets but since you don't have a reproducible example, I cannot be sure of the outcome.

library(dplyr)
library(ggplot2)
library(tidyr)

## Calling geom_col twice
mpg %>%
  group_by(manufacturer) %>%
  summarise_if(is.numeric, mean) %>%
  select(manufacturer, cty, hwy) %>%
 ggplot() +
  geom_col(aes(x=manufacturer, y=hwy, fill="hwy")) +
  geom_col(aes(x=manufacturer, y=-cty, fill="cty")) +
  theme(axis.text.x = element_text(angle = 90, vjust = 0.4))

## Converting data to long format
mpg %>%
  group_by(manufacturer) %>%
  summarise_if(is.numeric, mean) %>%
  mutate(cty = -cty) %>%
  select(manufacturer, cty, hwy) %>%
  pivot_longer(., -manufacturer) %>%
 ggplot() +
  geom_col(aes(x=manufacturer, y=value, fill=name)) +
  theme(axis.text.x = element_text(angle = 90, vjust = 0.4))

For your datasets, first we join them on date column and then we can use geom_col on each of the desired columns. Note that I have aes in geom_col and not in the ggplot. You need to make sure that your date column is not class of factor (but joining may resolve that issue). If you shared output of dput(head(dataset)) for each of your two datasets, I could further investigate your problem.

recent_school_shootings %>%
  full_join(., school_purchases_month, by = "date", suffix = c("s", "p")) %>%
 ggplot() +
  geom_col(aes(x=date, y=total, fill="Purchase")) +
  geom_col(aes(x=date, y=-killed, fill="Shooting"))

Created on 2020-05-20 by the reprex package (v0.3.0)


When does it make sense to add numbers with different units?

Speaking as an ex-physicist, I would say it never makes sense to add quantities with different units. When problems like this do arise it makes sense to honestly define some scale constants, normalize your quantities with respect to those constants and then add them.
This 'define and normalize' will likely not change much of your procedure, but being explicit about your constants can help to avoid problems later on.
If you want to consider meters and kilogarms to be part of the same metric space, it means that somewhere in your problem there must be a constant with units kilogram/meter, i.e. linear density. Such constants frequently arise in natural sciences, and can be quite interesting in their own right. Charge of electron, speed of light, are just few. I would suggest trying to understand how this linear density constant arises in your problem, might be an interesting insight.

It will become even more interesting if you have two constants with the same units in your problem. Then their relative values can indicate qualitative switch in behaviours. For example, https://en.wikipedia.org/wiki/Reynolds_number are unit-less numbers that indicate whether or not you can have turbulence.

I understand this is not a physics question :-)


Find effective feature on machine learning classification task with scikit-learn

Example code of the nice suggestion from stmax above, with modification to use RandomForest and match the questions sample size and number of features, I hope that helps:

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier

X, y = make_classification(n_samples=10000,
                           n_features=34,
                           n_informative=10,
                           n_redundant=0,
                           n_repeated=0,
                           n_classes=2,
                           random_state=0,
                           shuffle=False)

forest = RandomForestClassifier(n_estimators=250,
                              random_state=0)

forest.fit(X, y)
importances = forest.feature_importances_
std = np.std([tree.feature_importances_ for tree in forest.estimators_],
             axis=0)
indices = np.argsort(importances)[::-1]

# Print the feature ranking
print("Feature ranking:")

for f in range(X.shape[1]):
    print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))

# Plot the feature importances of the forest
plt.figure(figsize=(20,10))
plt.title("Feature importances")
plt.bar(range(X.shape[1]), importances[indices],
       color="g", yerr=std[indices], align="center")
plt.xticks(range(X.shape[1]), indices,rotation=60)
plt.xlim([-1, X.shape[1]])
plt.show()


Comparison of machine learning approaches for a topic in a scientific paper

If I understand, you could evaluate your approach based on your efforts in using the right preprocessing, picking the right features, building the appropriate architecture (choosing the model that best suits the problem).

Each step will make the loss decrease until you reach a point, you will have to search over the hyperparameters to squeeze every percent of accuracy, which is heavily a computational problem.


A feature highly correlated with target variable

You should keep it, the higher the correlation with the target variable - the better the feature.
BUT - you should also make sure this correlation is "real", i.e. not due to data leakage.
(the answer was written using @GeoMatt22 and @Ubikuity comments.)


Business intelligence is perfect for you; you already have the business background. If you want to become a bona fide data scientist brush up on your computer science, linear algebra, and statistics. I consider these the bare essentials.

I don't know about Scandinavia, but in the U.S., data science covers a broad spectrum of tasks ranging from full-time software development to full-time data analysis, often with domain expertise required in various niches, such as experimental design. You have to decide where your strengths and interests lie to pick a position on this spectrum, and prepare accordingly. Useful activities include participating in Kaggle competitions, and contributing to open source data science libraries.


Judging from your question you are probably using this formulae for normalisation:

(x - x_mean)/(x_max - x_min). This is just an approximation of the real normalization formulae. The real one would be:

where mu is the mean and sigma is standard deviation. If your data trends is same throughout, then you can expect the mean and standard deviation to be approximately same, and thus give you a more uniform representation.

Check this Wikipedia article, Feature scaling which says about the schemes used in different ML techniques. Hope this helps!


Can we use prior information to improve the results of a neural network? If so, how do we incorporate it into the learning process?

What you may be referring to is transfer learning.
The process of transfer learning mainly boils down to using layers of big  pre-trained models (super set) in your smaller model (subset) during training.
Additional configuration may include freezing some of the pretrained layers (no updates during training)  and unfreezing others (update weights/biases during backprop).
It is very common in recurrent networks to use pre-trained word embedding layers (word2vec, GloVe).


Use of Random Forest algorithm in PySpark for imputation

You can do the following: use all the other features as input and the missing data as the label.

Train using all the rows that have the column filled with data and classify the others that don't. Use the values predicted by the Random Forest as the value of that field on the subsequent models and transformations.


After carefully reviewing my code I found a bug in my train and test set.

Bug was in this line:

bottom_right[1] *= -1

should be:

bottom_right[:, 1] *= -1

After that my tensorflow accuracy went up to 99% and my Keras accuracy to 89%.

Anyway if someone has better solution or see something I am doing wrong please let me know :)


Optimization of pandas row iteration and summation

This could actually be done quickly and intuitively using linear algebra.

So consider your player as label binarized array (can be done with MultiLabelBinarizer) so you would expect an array of size (2060590, 39) containing 0 an 1, rearrange the columns similar as how you order the your player table (or the other-way around which ever is easier), basically such that first column of your new matrix correspond to the same player on the player table. Finally just apply matrix multiplication, and done.

This is an example using generated sample, but hopefully you get the idea of doing this.

import numpy as np
import pandas as pd
from sklearn.preprocessing import MultiLabelBinarizer

mlb = MultiLabelBinarizer()

sample_active = pd.Series([[100,50,150,200],
                           [100,50,150],
                           [100,50],
                           [100]])
sample_df = pd.DataFrame()

sample_df['id'] = ['fadfsadsa', 'dsafsadf', 'dfsafsda', 'dasfasdfsaf']
sample_df['active'] = sample_active
## sample_df should look close to your original df

classes = [50,100,150,200]

player_df = pd.DataFrame({cl : np.random.uniform(0,1,size=5) for cl in classes}).T
player_df.columns = ['A','B','C','D','E']

sample_transformed = mlb.fit_transform(sample_active.values) ##apply multilabel binarizer

output = sample_transformed.dot(player_df.loc[mlb.classes_]) ##matrix multiply and get your required answer, use loc so the order will be similar as your binarized matrix.

new_df = pd.concat([sample_df['id'], pd.DataFrame(output)], axis = 1)
new_df.columns = ['id'] + list(player_df.columns)

For your case I think this should work :

mlb = MultiLabelBinarizer()
active_transformed = mlb.fit_transform(uuid_df['Active'])
output = active_transformed.dot(points_table_df.loc[mlb.classes_])
df = pd.concat([uuid_df[['UUID']], output], axis = 1)
df.columns = ['UUID'] + list(points_table_df.columns)

Try it!


How to correctly interpret the table function to determine logistic regression accuracy?

The outcome of a multi-nomial or binomial is confusion matrix (2*2 for binomial, n*n for multinomial),

Interpretation of the confusion matrix is for example:

Accuracy of the model : ((TP + TN) / (TP + FN + FP + TN) ) * 100
It means you model could exactly pick this much percent of data was classified correctly.

Precision: (TP / (TP + FP))*100
That measure of correctness achieved in positive prediction i.e. of observations labeled as positive, how many are actually labeled positive

Recall: (TP / (TP + FN)) * 100
It is a measure of actual observations which are labeled (predicted) correctly i.e. how many observations of positive class are labeled correctly. It is also known as ‘Sensitivity’.

F-Measure: ((1 + β)² × Recall × Precision) / ( β² × Recall + Precision )
It combines precision and recall as a measure of effectiveness of classification in terms of ratio of weighted importance on either recall or precision as determined by β coefficient.
generally β is 1.

If you want read them in just normal words in you example:

        Predicted
          M   F
       M 100 150
actual F 100 300

It means your model could predict 100+300 correctly out of 100+150+100+300

Actual number of males where 250 but you could classify 100 correctly and 50 wrongly.

Similarly, Actual number of females where 400 but you could classify 300 correctly and 100 wrongly.

If you any need more detail let me know.

Go through this Link, you will get better idea.
This Link is an interactive chart, which will give you better understanding.


How exp(-z) is working in a sigmoid function in neural networks while z is a matrix?

In many languages and libraries, operations that apply to a scalar can be applied to vectors, matrices and tensors. They're just applied element-wise, and the result is another vector, matrix, etc with each value transformed by that function.


Visualize frequency of 5 Boolean variables together

If you have richer data (ie more than 10 rows), you will want an upset plot.  Upset plots are a way to view information in an intuitive way like a Venn diagram, but is more useful for 4+ categories.
Some references which may give you some ideas and implementation in R:

https://cran.r-project.org/web/packages/UpSetR/vignettes/basic.usage.html (attached image from r-project.org).
https://www.littlemissdata.com/blog/set-analysis


Visualization for comparing the results of two different models

If you have a time series I guess plotting actual vs. predicted values in a graph would work fine. Maybe throw in the MAPE and adjusted R^2 as single values in one of the corners.


I have data with customer personal information and customer transaction. I cannot figure out how to use the data for training my model?

Yes you can.
On the files you have mentioned above i guess, you have merge the data. But keep in mind to remove redundant columns.
After that you have to :
* check if there are any missing values or null values
* find the relation between one independent variable to other independent variable
* find the relation between independent variable and the dependent variable

Then split the data into train set and test data and apply your model


Fast k-means like algorithm for $10^{10}$ points?

k-means is based on averages.

It models clusters using means, and thus the improvement by adding more data is marginal. The error of the average estimation reduces with 1/sqrt(n); so adding more data pays off less and less...

Strategies for such large data always revolve around sampling:

If you want sublinear runtime, you have to do sampling!

In fact, Mini-Batch-Kmeans etc. do exactly this: repeatedly sample from the data set.

However, sampling (in particular unbiased sampling) isn't exactly free either... usually, you will have to read your data linearly to sample, because you don't get random access to individual records.

I'd go with MacQueen's algorithm. It's online; by default it does a single pass over your data (although it is popular to iterate this). It's not easy to distribute, but I guess you can afford to linearly read your data say 10 times from a SSD?


Multiclassification with large number of labels

I think your understanding of the output vector is not correct (Not just for Neural Network but any Model)
We don't encode the output to reduce the dimension. Output has not much contribution on the RAM and computation, it's the input size and dimension. Simple OHE Or just the label encoding will work(Loss function will change as per the case)
What you should decide first-
 Is this a Multi-class problem or Multi-label
 This will decide your Encoding i.e. [1, 1, 0, 0, 1] Or [1, 0, 0, 0, 0]
 This will decide the last layer's Activation Function and the Model's Loss function
It is obvious that for such a high output dimension, you will need a very large input dataset with a good variance over all the 10K class.
To reduce output dimension -
You may use any clustering technique Or your domain knowledge to club different classes.
And then, you should plan about Torch and TensorFlow


Ingredients, Recipes and recipe ratings. I would like to predict the rating based on combination of ingredients

Here are a few ideas to start with:

A simple model would be to use multiple linear regression (MLR), or a Random Forest

If you want to evaluate only based on if an ingredient is used, your input dataset could look like this:

Butter   Flour    Eggs  ...  Test   Smell   Look   Texture
1        0        1          1      5       2      4

If you want to predict on how much of each ingredient (some is better, too much is not good) your dataset could look like this:

Butter   Flour    Eggs  ...  Test   Smell   Look   Texture
1.5       3.0        2          1      5       2      4

You can get some other ideas from this website:

https://machinelearningmastery.com/multi-output-regression-models-with-python/


Additional to the content of the document, the form could contain even more valuable information. I can't comment on the formatting of pdf files, but it should be possible to exploit the pdf metadata. Author, Title, subject or keywords could provide some clue pointing to a common origin in the same organization.


ZeroR as performance baseline for binary classfication model?

Sure, ZeroR is a perfectly fine baseline. In this case I think it's better to call it a random baseline rather than a majority baseline, since it's just what it is.
To my knowledge this is the only basic baseline usable with any classification task. Other baselines would involve something more sophisticated based on the specific task. The standard way to have a more competitive baseline is to used a state of the art method for the task.
Btw the baseline or evaluation method doesn't depend on the learning algorithm, it doesn't matter if it's NB or any other classification method.


How is the evaluation setup for YouTube faces of FaceNet?

The objective function they use to train the CNN minimizes the squared L2 distance (i.e. the squared Euclidean distance) between two similar (positive) images and simultaneously maximizes the distance between two different (negative) images.
That means, the (squared) Euclidean distance between two representations is a measure of their similarity.
Then, recognizing a face in a new image is as simple as 1) running it through the CNN and 2) finding its nearest neighbors with a KNN algorithm.

The last paragraph was only about images - in the Youtube Faces DB, we are handling videos of different persons.
In section 5.7 of the paper, they describe how they evaluate performance:

  We use the average similarity of all pairs of the first one hundred frames that our face detector detects in each video.

So, you were partially right: they just average the independent results over video frames. Probably for performance reasons, they chose to average the first 100 frames. They do describe that increasing this to the first 1000 frames increases performance from 95.12% to 95.18%, which is not significantly more.


RNNs for time series prediction - what configurations would make sense

should there number of units in the hidden layer roughly equal the number of steps you aim to predict ahead? (also, similar thing with the relationship between window size and steps to predict ahead)

Don't forget that a single neuron can saturate, over time - it can assume values closer and closer to -1 or to 1. That means you get nice flexibility, across several consecutive timesteps. Therefore, you don't need number of neurons to be as many as the number of timesteps.

You can verify it by plotting a single neuron of a simple RNN on the graph-paper, with pen and paper. For every square (timestep) draw neuron's activation and previous activation. At some particular timestep assume the input is no longer 0 (for example -1) and observe how the function fluctuates into the wave from that timestep onwards. Pretty flexible for a single neuron to be honest. Even more entangling were we to graph 2 neurons. That wave of only a few neurons can hold and encode a lot of information (exponentially more)

  when using a window, does it make sense to use a set of overlapping-windows, eg. to slide a window of 10 points one point at a time, or non-overlapping windows?
  This is similar to another question:
  sliding window leads to overfitting in LSTM?

Overall, sliding window is a bad approach for LSTM that leads to overfitting. A better approach would be to scan sequence backwards and forwards before outputting an answer. This is Bidirectional LSTM, or even looking into Attention. If you need to output an answer every frame relying on a neuron as is would be reasonable - after all LSTMs are quite good at remembering things due to the "accumulation" and the entangling which allows for oscillations (I've mentioned them above).

  you need some intuitions so you can start form a configuration that "roughly works"...)

Quoting a pretty much legendary post:

  in sum, for most problems, one could probably get decent performance (even without a second optimization step) by setting the hidden layer configuration using just two rules: (i) number of hidden layers equals one; and (ii) the number of neurons in that layer is the mean of the neurons in the input and output layers.

Also a bonus, how to query LSTM might be useful for future thought - LSTM querying approach


I would not concern yourself too much about any structuring of knowledge that declares that one subject is categorised as one thing or another. These structures are often wrong and knowledge in general is more fluid and difficult to define than can be viewed as some kind of Venn diagram.

In addition, both Data Science and AI are poorly defined, and have more in common with marketing terms used to recruit staff and sell products than stricter academic subject areas such as Maths or Modern Languages. You will find Data Scientists working on "AI" projects and "AI Engineers" using basic statistics to analyse data.

Having said that, I would prefer to say that Data Science and AI share a common set of tools, especially when it comes to machine learning. Neither contains the other in my opinion.