It is usually more practical to compute the distances on the fly rather than storing $N^2$ values. If possible, you will want to use a heuristic to select only the items could be sufficiently similar so you don't waste time computing distance to irrelevant items. PostgreSQL can create an index on character trigrams in strings that enables efficient querying for other strings that are sufficiently similar (measured by the Jaccard similarity coefficient).

Storing an $N$ column wide table is not possible because the maximum number of columns allowed in MySQL is 4096. Even if that were not the case, using an RDBMS in such a way is considered bad practice and such a table would be difficult to work with.

If for some reason you must store a dense distance matrix, a format such as HDF5 would be more efficient than an RDBMS.


Bias and variance in the model o in the predictions?

In some cases, you may have a model that's a black box - you feed input features and get output predictions, without knowing or caring what happens in the middle. In those situations, the model is, in a way, defined by its output - two models that produce the same predictions are indistinguishable from one another, even though they may be entirely distinct models. In these situations, saying a model is biased or has high variance is equivalent to saying the predictions of the model are biased or have high variance. It's acceptable to describe both the model and its output in terms of bias/variance. Cross validation provides an unbiased estimate of bias/variance for your model/predictions.


A neural network is the wrong approach for a problem with a small training set. Even if you only have 2 features that are very representative of your function then 16 feature are not sufficient.

As a very general rule of thumb I use 100 examples for each feature in my dataset. This then increases exponentially with every single different class you expect. 16 instances is not enough to train a neural network. You will always have huge error margins when applying your model on a testing set. Even more problematic is the fact that you are using a very deep neural network. This will require even more training instances to properly learn the function.

I suggest you use a general machine learning technique such as SVM. This will likely result in better result. Try these techniques instead and see what results you get: k-NN, kernel SVM, k-means clustering.

But, be warned 16 training instances is still very little.


There are no unseen item types in the given data, by definition. 3 is the count of items seen once, and they are already included in the denominator 18. If the next item were previously unseen, it would become seen once when it appears. Since 3-of-18 examples were seen-once items, this is an estimate of the probability that the next item will be seen-once too on its first appearance.

It is certainly a heuristic. There is no way to know whether there are 0 or 1000 other types out there.


There are many variations of reinforcement learning methods. Some are tweaks to existing algorithms, some are radically different. The problems that you are concerned about with your analysis and suggestion appear to be:

Credit assignment. Whether or not a particular state or state action pair earlier in a trajectory is important to receiving a reward later on, and thus should have any value update due to new experience. This is a core problem to reinforcement learning (RL). All RL algorithms solve it, the differences are in how they deal with levels of difficulty.
Computational efficiency. Whether the algorithm makes best use of CPU cycles and doesn't waste its time calculating many updates of zero in an environment with sparse rewards.
Sample efficiency. Whether the algorithm makes best use of available data to make the most robust predictions of values it can.

Single-step Q learning does address all of these issues to at least some degree:

For credit assignment, the single step bootstrap process in Q learning will backup estimates through connected time steps. It takes repetition so that the chains of events leading to rewards are updated only after multiple passes through similar trajectories.
For computational efficiency, it is not always clear what the best approach would be. Single step Q learning updates are efficient to calculate, so even though each one only updates a small segment of a trajectory, this can be made up, in some environments, by running more updates in the same CPU time.
For sample efficiency, single-step updates on their own are not very good - they can leave many values untouched even though logically they could be updated based on new information. However, there are other approaches to address this - for instance using an experience replay table or Dyna-Q planning can take advantage of previous states and actions to re-calculate Q values without needing to see more experience.

As usual with ML problems, it is not always clear what adjustments and variations of an algorithm may work best. So you need to experiment with your environment.

You also mention this:

  Until the agent finds the cell leading to a positive reward, the agent gets small or no rewards.

This is somewhere that single-step updates can work well. In your grid world example, if you are training to minimise the number of time steps to complete an episode, then typically you will have a negative reward associated with each time step that does not complete an episode. In that case, updating your Q estimates immediately on each action can be a large benefit - it encourages exploration by progressively scoring repetition worse (as the initial agent blunders around randomly) - meaning that the agent tries to find new states and actions that are still scored at original default levels. You won't get that searching behaviour if you wait until the end of the first episode before updating the value tables.

Whether or not this is an issue will depend on the environment, but such time related goals are relatively common, so it is worth bearing in mind.

  Alternative approach
  My question is: why can't we update all the Q-values related to the agent at the end of the episode?

This is a well known common approach in RL. The simplest variant is called Monte Carlo Control, and it uses trajectories from whole episodes to update values of all state/action pairs seen in those trajectories.

More sophisticated approaches blend between single-step Temporal Difference learning (of which Q learning is one example) and whole episode Monte Carlo methods. These are n-step methods (n-step Q-learning is one example) and TD($\lambda$) methods (Q($\lambda$) is one example). These are relatively popular as they combine best features of, and avoid compromises of, the two extremes of whole episode vs single step.

  Is this approach legit?

Yes, in fact as described above, variations of it are used routinely.

  Which problem do we face following this?

When using whole episodes, you face the following problems:

High variance. If either your policy or your environment has randomness, then the final trajectory and reward in each episode will vary a lot. Which means to get reliable estimates of value you may need more samples, not less as you hoped.
Off-policy learning is more complex. Q-learning in particular is an off-policy method, meaning it learns values of its best guess at an optimal policy (called the target policy) whilst still exploring using a non-optimal policy (called the behaviour policy). With single-step updates this difference can be ignored as you always calculate the updated value estimate for an action you've taken (you don't care what policy generated this) using the maximising action for the next step (so you don't use the action actually taken). When you extend to calculating value updates over multiple steps, you need to adjust for differences between behaviour and target policies. If you look at Q($\lambda$) algorithm you will see it copes with this by resetting something called the eligibility trace (which you can think of as a vector that explicitly tracks credit assignment) whenever the agent takes a non-optimal action according to the current value estimates.

There is also Off-policy Monte Carlo Control, which uses importance sampling to decide whether and how much to use particular updates - for learning optimal control typically this will only update from the ends of episodes after the last exploratory action that the agent took.

  The only drawback I can think of is lack of exploration, that can be solved choosing random actions according to a decreasing epsilon value.

Usually you will want some initial high exploration that settles down closer to the predicted optimal policy later on. This or something similar should be standard practice in many environments. However, it would not solve the search problem you mention - for that you need an exploration system that is aware of when states and values are repeated themselves over multiple steps within a single episode. There's more than one way to solve that, too, but the single-step Q learning approach is quite nice when it works because it is a natural consequence of the algorithm.


How to identify text similarity based on training data?

Yes this is a supervised problem. I'd suggest following the example in this article.


Time Series data: How to convert it in a streaming data?

It depends on the answer to these questions:

How is the real-time data provided?
How often and in which format are the predictions required?

It's common to use a database in such a setting. So the ML prediction process would be called regularly, query this database to get the latest instances and generate predictions in the required format (possibly storing the predictions in the database as well).


Can I have a neural network output values >100?

Is sounds like you want to use Neural Networks to do a regression problem instead of classification.

This post gives an example of using a neural network to do regression.


Using pandas, check a column for matching text and update new column if TRUE

You simply need to do:

df['NEWcolumn'] = df['COLUMN_to_Check'].str.contains(pattern)
df['NEWcolumn'] = df['NEWcolumn'].map({True: 'Yes', False: 'No'})


Dataset with disproportionately more of a single label than any other

There are a few ways:

Oversample the under-represented class.
issue: Leads to very constrained boundaries (not smooth) around the under-represented class
Weigh the loss coming from under-represented class more than over-represented class. If you think about it, this is mathematically doing the same thing as 1
Use SMOTE (here) (i recommend this). Basically samples additional points randomly from lines joining nearest under-represented classes and increases the dataset size accordingly. scikit contrib implementation of smote

There are other ways but they offer very marginal improvement over SMOTE


prepare email text for nlp (sentiment analysis)

I have done something similar in the past.  I'll sketch an outline for you.

First you break the text into paragraphs and tokenize.  Then write some regexp rules to capture the data you want to remove.  For instance, if an email signature commonly contains a phone number, paragraph, and a website, you can count those features and flag it based on some threshold you decide.

Next, do likewise with the other features you mentioned.  My experience is it's highly domain dependent so you really need to look at the data and use your best judgement.

The result of this process should be a data consisting of tokenized paragraphs where the paragraph has been labeled 'noise' or 'clean' based on the feature count.

From there, convert your token representation using tf-idf or another type of embedding.  You should be able to use this as input to your favorite classifier, and I have had success using SVMs to that end.

The result is going to be biased towards your rules but you are also leveraging features that are in the labeled examples that are not explicitly in the rules.  Particularly so for longer paragraphs.

It might seem a bit a bit janky but believe it or not it works.


Computing SVD through Eigendecomposition of correlation matrix

The problem is occurring for a few reasons:

You must ensure the correct ordering of eigenvalues/eigenvectors.
You should be taking the square root of the eigenvalues (Lambda_1) to get the singular values.
You need to take special care to ensure you are using the right signs for the eigenvectors as these won't necessarily be correct by default. This is because we assume that the singular values are the positive roots of the eigenvalues, but the two sets of eigenvectors we retrieve don't necessarily respect this assumption when joined in the singular value decomposition.

Here is the working example:
r = 10
XT = X.transpose()
C_1 = XT @ X
C_2 = X @ XT
[Lambda_1, V_hat] = np.linalg.eig(C_1)
[Lambda_2, U_hat] = np.linalg.eig(C_2)

# The eigenvalues/eigenvectors returned from np.linalg.eig are not sorted so may differ between Lambda_1 and Lambda_2
# When performing economy SVD we also want to ensure the largest eigenvalues come first
i_1 = np.argsort(Lambda_1)[::-1]
i_2 = np.argsort(Lambda_2)[::-1]
V_hat = V_hat[:, i_1]
U_hat = U_hat[:, i_2]

# We must take the square root of eigenvalues to get the singular values
Lambda = np.sqrt(np.sort(Lambda_1)[::-1])

# X @ V_hat and U_hat @ np.diag(Lambda) should be equal. But the eigenvectors retrieved may not have the correct signs
# This code checks if  X @ V_hat and U_hat @ np.diag(Lambda) have different signs and then updates V_hat wherever the sign is incorrect
same_sign = np.sign((X @ V_hat)[0] * (U_hat @ np.diag(Lambda))[0])
V_hat = V_hat * same_sign.reshape(1, -1)

V_hat_T = V_hat.transpose()
X_tilde = U_hat[:,:r] @ np.diag(Lambda)[0:r,:r] @ V_hat_T[:r,:]


Different Time series Modeling techniques?

ARIMA is a parameterized model that consists of a few parts. First the I is for integrated, and it has to do with a technique to remove non-stationarity from the data, by looking at differences between sequential values instead of the values itself, the parameter indicates how often we want to do this. AR stands for auto-regressive, it's a regression model that regresses on previous values of itself (lag features), the order determines how far back we look. The MA means Moving Average, which is referring to a linear combination of error terms, this is also parameterized by an order.

Box-Jenkins is a method that uses ARIMA, but it first involves a number of checks before fitting the model, and afterwards does criticism to see how well the model works. This allows for iteration into a better fitting model.

Holt-Winters methods are used for smoothing out time series. Sometimes there is a lot of noise involved in time series, while you are only interested in trends and how values relate to each other. By smoothing it out via for example moving averages or exponential smoothing hopefully you gain more information by trying to filter out noise than you lose by throwing away relevant information.


How much can the AUC improve comparing the raw dataset and the feature engineered dataset?

Yes, the performance can vary a lot using feature engineering.

Example: suppose a dataset where the response variable $y$  is true if $x$ is odd.

x    y

346  F
13   T
178  F
64   F
987  T
...

Most learning models will fail to identify the pattern and will perform poorly, usually falling back to always predicting the majority class. However simply adding a feature $x \% 2$ to the data will allow any model to perform perfectly.

Of course this a toy example, but the point is that a single well chosen feature can drastically change the performance. Naturally the increase in performance totally depends on the data and the nature of the features added.


On what principle did Google's DeepMind learn to walk?

The full method is explained in the paper Emergence of Locomotion Behaviours in Rich Environments by the DeepMind team.
Quoting from that paper:

Using a novel scalable variant of policy gradient reinforcement learning, our agents learn to run, jump, crouch and turn as required
by the environment . . .

So to answer your question, the researchers used a policy gradient method.


I think you got most of it from the way you wrote your question.

How does R-CNN and AlexNet compare?
Are they used for the same purpose or R-CNN does more?

They are different things. AlexNet is a CNN architecture, i.e a neural network with a specific set of layers. R-CNN is multistep method that does object localization and classification using a search algorithm with a CNN (of any architecture) and a SVM.

Does R-CNN use Alexnet as a sub-module?

R-CNN can use AlexNet, or other CNN architectures, as its' feature extractor.


AUC computation on multilabel classification

AUC (Area under the ROC Curve) can not be calculated for multilabel classification directly.
Multilabel classification metrics have to be converted to a binary problem. The two most common conversions are:

OvO - One vs One
OvR - One vs Rest

sklearn.metrics.roc_auc_score discusses this in detail.


What does it mean to have 1 degree of freedom in ANOVA test?

Well having datasets handy would have made it easier to do it myself and explain better.

But let me try to help you get started assuming you are asking to interpret the results rather than trying to solve model/platform/library specific implementation, if not don't bother reading ahead.

As I understood its the linear fitting model here which you are calling with 2-way ANOVA test.So in Linear models each independent variable uses 1 degree of freedom, if there are 6 then 6 DF. Rest are just actually number of observations available which it shows in residual/error. That's why the sum_sq and mean_sq values are same in your output for variables but different for residual.

I found it much more easier to try 2-Way ANOVA  in excel first and trying to interpret the results. please try this link with excellent explanation I found long back-

https://statisticsbyjim.com/anova/two-way-anova-excel/

https://statisticsbyjim.com/hypothesis-testing/degrees-freedom-statistics/

Few more links which clearly explains-

https://support.minitab.com/en-us/minitab-express/1/help-and-how-to/modeling-statistics/anova/how-to/two-way-anova/methods-and-formulas/methods-and-formulas/

https://www.graphpad.com/support/faq/the-anova-table-ss-and-df-in-two-way-anova/

https://people.richland.edu/james/ictcm/2004/twoway.html

And If you are statistically challenged like me and have to brush up to use it effectively in ML, please follow these-

https://www.youtube.com/watch?v=NF5_btOaCig


Multi-class classification with discrete output: Which loss function and activation to choose?

You could try a Quadratic Kappa Metric loss function. This is a summary of what it actually does.

  A weighted Kappa is a metric which is used to calculate the amount of
  similarity between predictions and actuals. A perfect score of 1.0 is
  granted when both the predictions and actuals are the same. Whereas,
  the least possible score is -1 which is given when the predictions are
  furthest away from actuals. In our case, consider all actuals were 0's
  and all predictions were 4's. This would lead to a QWKP score of -1.
  The aim is to get as close to 1 as possible. Generally a score of 0.6+
  is considered to be a really good score.

The detailed explanation can be found here: https://www.kaggle.com/aroraaman/quadratic-kappa-metric-explained-in-5-simple-steps


If you have multi dimensional data, it's very hard to visualize. PCA
  helps us in reducing the dimension of the data set by keeping maximum
  co-variance in top K Features. So PCA will help us in presenting the
  data in most of the cases.

So if you perform PCA, you will definitely lose some data and creating a model with less data, will definitely affect the model outcome.


What loss function should I use for auto encoders for text

The cause of your huge loss is probably a lack of normalization. As an example, in the Keras-blog post you linked, they conduct such normalization by dividing the input by 255, which is the maximum value an element in the input can be of.

As for which loss function to use, if you have a look at this keras-blog post, they construct a LSTM-based auto encoder for text which uses categorical crossentropy as a loss function.


How to handle negative words in word2vec?

When you look at the vectors that word2vec generates - negative words may have unique features but can be treated just like positive words. That is to say, as far as the NN is concerned - these are just similar words. You may have to construct "concept vectors" on top of the word vectors to do what you would like to do.

Your parts of speech tagging should automatically mark negating words as ADV. You can then train on these adverbs in conjunction to your verbs as a positive or negative output. Here's an example using spacy:-

import spacy

nlp = spacy.load('en')        # this can take a while
sample_text = u'Do not go.'
parsed_text = nlp(sample_text)
token_text = [token.orth_ for token in parsed_text]
token_pos = [token.pos_ for token in parsed_text]

At this point token_text will be a list of your words and token_pos will be the POS tagging:-

Do - VERB
not - ADV
go - VERB
. - PUNCT

As you can see, "not" is tagged as ADV here. You can now feed this tagged output (or a better parse tree) into a second network to train for a negative or positive output.

Hope this helps.


When the shallow network has a small number of neurons, how it will need more sum-product?

Shallowness is when the Model has very few Or just one hidden layer.
When it has a lot of hidden layers it will be a Deep Neural Network.
Sum-product is basically a Neuron of a Hidden Layer.

From the paper-
Artificial neural networks with several hidden layers, called deep neural networks, have become popular due to their unprecedented success in a variety of machine learning tasks (see, e.g., Krizhevsky et al. 2012, Ciresan et al. 2012, Goodfellow et al. 2013, Hinton et al. 2012). In view of this empirical evidence, deep neural networks are becoming increasingly favoured over shallow networks (i.e., with a single layer of hidden units), and are often implemented with more than five layers

A single sum-product hidden layer summarizes a layer of product units followed by a layer of sum units.

The exact definition of Shallow and Deep is a moving number depending on the advancement of AI research.
While it is intuitive that if a 3-layered model needs 5-Neurons in each layer then a single Layered model should have 15 Neurons.
But it will actually require more than that
 It's very difficult to get a conclusive answer to this but the two famous reasons are -

Our data has often an underlying Hierarchical nature i.e. Image data
Analogy with Circuit theory (Check Ref#2 and #4)

Practical reasons seem that it worked with GD/Backprop and research moved in that direction. As you may read the listed references and find that None of the experts has a conclusive view on this.
Reference -
1. Deep Learing Book - 6.4 Architecture Design
2. Neural Networks and Deep Learning,  Michael Nielsen
 3. Stanford CS class CS231n
4. Andrew Ng Deep Learning course - C1W4L04


TypeError: 'tuple' object cannot be interpreted as an integer

The solution to the problem is to resize using openCV instead of using PIL and specifying the appropriate pixel width and height.

img = cv.resize(image5 , (150,150))


Difference between cluster centers and means

They are the same.

When you run K-Means, the cluster center changes every iteration. In each iteration, cluster center or mean is given for that specific cluster. In the next iteration, a new cluster may be formed. The cluster centers need to be calculated again.

When the algorithm converges, you get the absolute means of the clusters formed. For most intents and purposes, they are the same.


Why are Machine Learning models called black boxes?

The black box thing has nothing to do with the level of expertise of the audience (as long as the audience is human), but with the explainability of the function modelled by the machine learning algorithm.

In logistic regression, there is a very simple relationship between inputs and outputs. You can sometimes understand why a certain sample was incorrectly catalogued (e.g. because the value of certain component of the input vector was too low).

The same applies to decision trees: you can follow the logic applied by the tree and understand why a certain element was assigned to one class or the other.

However, deep neural networks are the paradigmatic example of black box algorithms. No one, not even the most expert person in the world grasp the function that is actually modeled by training a neural network. An insight about this can be provided by adversarial examples: some slight (and unnoticeable by a human) change in a training sample can lead the network to think that it belongs to a totally different label. There are some techniques to create adversarial examples, and some techniques to improve robustness against them. But given that no one actually knows all the relevant properties of the function being modeled by the network, it is always possible to find a novel way to create them.

Humans are also black boxes and we are also sensible to adversarial examples.


What Base Should Be Used For Negative Log Likelihood?

The change in base is equivalent to multiplying the function by a constant. It does not affect the computation.

$
log_b(x) = \dfrac{1}{log_e(b)}.log_e(x)
$


What are the techniques for anomaly detection of Unsupervised learning problem

You can try these techniques and many more.- All anomaly detection techniques
As discussed in article, these are outlier detection techniques.Are you looking for outliers? better to get some known abnormalities and build a classification.
If supervised not possible, try to fit one of these approaches-
ABOD for identifying abnormalities in high dimensional data
Should clustering be based on distance or density to find outliers(abnormalities)
Connectivity based outlier detection technique
There are other techniques like PCA based, regression based, auto-encoder, knn, weighted Knn and even self organizing map (SOM) . let me know if you need some more information.
Imp-  Know your abnormalities better before jumping to machine learning, I have experienced that even qq plot or just data points 3sd away might give better anomaly detection.


Tensorflow - logistic regression -oneHot Encoder - Transformed array of different size for both train and test

Your shape is (30, 14) and not (30, 15) because there are only 14 unique values in your test (one is missing). In any case you shouldn't fit the encoder on the test set, just on the training set. Then just transform on the test set and you'll get the correct dimensions.

Also as far as I can see W and b are declared correctly. I'd ask however you take a minute to format your question a bit better next time.


How do I factor in features which are IDs?

These IDs should not be represented as numerical values to your model. If you would, your model thinks 2 and 3 are closer together than 2 and 2000, while it's just an ID, the number is just a name.

Some models can deal with them but then they need to be factors or categories (like decision trees). However most models cannot deal with categories at all, there are however numerous ways to solve this problem. The most used one is probably one-hot encoding, which means for every category in your feature you add a column, and you put a 1 if it's that category and a 0 otherwise. Example:

ID | target
1  | 0
1  | 1
2  | 3
3  | 2

To:

ID_1 | ID_2 | ID_3 | target
1    | 0    | 0    | 0
1    | 0    | 0    | 1
0    | 1    | 0    | 3
0    | 0    | 1    | 2

This work very well if you have few categories, however in the case of thousands of IDs this will increase your dimensionality too much. What you can do is collect statistics about the target and other features per group and join these onto your set and then remove your categories. This is what is usually done with a high number of categories. You have to be careful not to leak any information about your target into your features though (problem called label leaking).


What should be the requirement for training data in order to obtain a good regression model using neural network?

If this is your test accuracy (and you make sure not to use any test samples as part of the training), then I don't see any reason not to trust it. Maybe the distribution of the training data is a very good approximation of the real distribution. You can perform a quick sanity check by trying a different separation between the training and test dataset.
Yes, that is correct.
The short answer: you have no way of knowing how it will behave outside the range of trained samples. However, a case where a considerable amount of the test set is taken out of the range of all the training set with a 80-20 division is not likely.
Don't have any personal experience with this question, but from a quick read of several similar questions, it seems that the optimization process would be driven by the target variable which has the largest scale. So you either need to add some weights to the loss function or normalize the outputs (same-same).
Regarding the multi-output vs. multi-model question, I think that the first answer in the following thread is on point:
What is the difference between using single multi-output NN and multiple single-output NNs


The documentation of numpy says you can add the parameter comments='', but warns that np.loadtxt() might not work on the resulting file.


How do I change the marker edge color of a seaborn relplot?

The issue was with the parameter I was using. The parameter "edgecolors" is incorrect. The correct parameter is the non-plural "edgecolor". So the correct call to relplot is:

sns.relplot(data=df, x="values_x", y="values_y", edgecolor=None)

This would remove the marker edges completely, which is what I was trying to do originally.


Derivation of backpropagation for Softmax

We let $$a=\operatorname{Softmax}(z)$$

that is $$a_i = \frac{e^{z_i}}{\sum_{j=1}^N e^{z_j}}.$$

$a$ is indeed a function of $z$ and we want to differentiate $a$ with respect to $z$. The interesting thing is we are able to express this final outcome as an expression of $a$ in an elegant fashion.

If you look at the section of "Derivative of Softmax Function" in your link, using quotient rule:

We can see that

if $i=m$,
\begin{align}\frac{\partial a_i}{\partial z_m} &=\left(\frac{e^{z_i}}{\sum_{j=1}^N e^{z_j}}\right)\left(\frac{\sum_{j=1}^Ne^{z^j}-e^{z_m}}{\sum_{j=1}^N e^{z_j}} \right)=a_i(1-a_m)\end{align}

if $i\ne m$,
\begin{align}\frac{\partial a_i}{\partial z_m} &=-\left(\frac{e^{z_m}}{\sum_{j=1}^N e^{z_j}}\right)\left(\frac{e^{z_i}}{\sum_{j=1}^N e^{z_j}} \right)=-a_m(a_i)\end{align}

If you want to evaluate things in terms of $z$, you can still use the middle term though using the formula

$$\frac{da}{dz}=ae^T\circ (I-ea^T)$$

is more elegant.


You are correct in the sense that when tuning your model via. grid search you are technically not leaking any data. But, recall that tuning your model (via. a specific procedure such as grid search) is one of only many steps you probably took in fitting your model pipeline. In particular, areas such as pre-processing, feature engineering, imputation, model tuning, data aggregations, etc. The point of the test set is to capture the entire model building process and not just the process of model tuning.

Furthermore, it is highly known that validation scores reported during model tuning tend to be optimistically biased (and this bias tends to be worse with smaller datasets). This is because the probability of finding a set of hyper parameters that coincidentally minimizes the error for the validation set but not to the overall population (i.e., overfitting to the validation set) becomes higher the finer your grid is. Imagine theoretically tuning your model to one million different hyper parameter combinations. The probability of selecting a bogus set of hyper parameters (i.e. that are only optimal for the validation set) is now quite large due to the sheer number of possible candidates you have elected to try.

With a test set, this still won't prevent overfitting to a validation set. However, it will allow you to detect the problem and give you the true unbiased measure of model performance.

  At the same time, it feels kind of strange, since with using both CV and Hold-out, I am guaranteed to get some outliers in the hold-out set (TEST in my text above), which will fail.

Indeed, which is a major drawback of cross validation and data splitting especially for smaller datasets (with a lot of outliers/noise). Basically, the performance measure you observe tends to be highly variable with how you split the data in the first place (that is, the seed you choose when splitting your data can lead to large changes in estimated model performance depending on where your outliers fall). The solution is unfortunately, not very glamorous and time consuming. In order to gain more certainty in our estimate, we need more than just a single estimate of model performance. Thus, simply repeat the entire model building process again with a different data partition (a different test set). Repeat this however many times, and average over all repeats. Possibly, form a confidence interval that allows you to see for yourself how variable your model's performance is.

I will also note that there are other ways to get around this problem, such as "optimism adjusted bootstrap", but recent issues have arisen with this method which have potentially shown that for high dimensional data this method does not do well, despite being more efficient than cross validation. Since high dimensional data is the norm these days, I have my doubts but perhaps it may be of use to you.


The only difference is in your example is that you divide by an additional two, because you take the mean per vector instead of the sum. Correctness does not play here because for comparison between different models the only difference is a constant factor and for interpretability it depends on the problem you are solving.

The mean absolute error punishes mistakes linearly while the mean squared error punishes larger mistakes more heavily. This means this depends a bit on what you want to measure, based on the problem you are solving. Next to proper evaluation you could use this same measure to change the KPI you are optimizing directly with a different loss function.


Hyperparameter tuning of neural networks using Bayesian Optimization

Neural networks are optimized by gradient descent which assumes the loss function for parameters is a differentiable function, in other words smooth. Given the nature of the differentiable loss function, Bayesian Optimization could be used for neural networks hyperparameter optimization. In fact, gradient descent can be used to learn the hyperparameters themselves as evidenced in the paper - "Gradient Descent: The Ultimate Optimizer".


How can I find synonyms and antonyms for a word?

The general approach is:

Split the text in tokens.
For each token, find synonyms and antonyms and store them with the token

from collections import defaultdict

from nltk.corpus import wordnet
from nltk.tokenize import word_tokenize

text = """
feels weird; may be a fake!
package came end missing box. since itâ€™s gift i update actual fit.
birkenstock amazing shoe!!!! i wish i ten pairs!
delivered advertised.... shoe looks fake.
second pair i had. nothing beats them.
they totally fake ðŸ˜¡. they felt weird i finally noticed â€œmade germanyâ€ logo above. they also smell like glue leather. infuriating happen!
i've birkenstock wearer 35 years. i wear 10 women's size ones i ordered wear big. i need size down.
great brand, packaging good, dad likes them. the size clearly needs fixed others don't order bigger sizes wanted. some people don't want bigger size actually wear.
false advertising.
a bit loose compared birkenstocks size. still like though. very comfy"""

tokens = word_tokenize(text)

# Store results in a dict where key is token and value is a set of synonyms and antonyms
tokens_syns, tokens_ant = defaultdict(set), defaultdict(set)

for token in tokens:
    for synset in wordnet.synsets(token):
        for lemma in synset.lemmas():
            tokens_syns[token].add(lemma.name()) # Add the synonyms
            if lemma.antonyms():    # Check if antonyms are available
                tokens_ant[token].add(lemma.antonyms()[0].name()) # Add antonyms

# Check for 'and'
token = 'and'
print(f'For the word "{token}":')
print('The synonyms are:', tokens_syns[token])
print('The antonyms are:', tokens_ant[token])

# Conclusion 'and' has no synonyms or antonyms according to WordNet


How to check quality of latent space like in β-VAE article?

Not that I know of. Notice that those heat maps rely on the human knowing what they want, and then devising this heat map approach to visualizing it.

For many papers interested in latent generative disentanglement, the most popular way to "check" the quality is via interpolating in the latent space and visualizing the output, seeing it across each disentangled dimension. One can then "see" whether they are indeed disentangled or not.

Ideally there is some other way to quantify this disentanglement, but it usually requires additional information. For instance, for images, one might have labels that one can use (e.g., brightness, person identity). Or, as in the $\beta$-VAE paper, one can use a toy problem where the underlying latent factors are known. Then one can devise a way to check whether one has actually separated those latent factors. Ultimately evaluation depends on the data and task.

If one just wants to visualize the latent space, usually these interpolations plots are popular. But it is qualitative. Often a quantitative evaluation of the latent space requires an additional downstream task.


For your first question, yes, it is optimized for that size since the original paper for Xception used 299x299 size. But, you can use other sizes. You should resize your images to 299x299 that would be the best.
For your second question, the reason height = width because in the network, the convolutional filters which are used are square (3x3 filters). The reason for using square filter in computer vision is we assume that the features in image are symmetric most of the times (exception being text where the information is more on the vertical dimension than horizontal, 1x2 filters are used there).
For your third question, go for the smaller size, because if you rescale the smaller images to bigger ones, you add useless information into image since it is derived from the smaller image itself. Also, you create a model with more parameters.


In datasets, why don't we represent nominal values that are part of a scale with numbers?

Nice and necessary question to be clear when doing machine learning preprocessing, there are two points here to take into account:

depending on the learning algorithm, you may need to convert your categorical data into a numeric format; e.g. decission trees do not need it and can handle categorical data, whereas another regression algos do need numbers as input
in case you have to convert data into numbers, you have two possibilities:

integer encoding: this is the case you are correctly describing; with this, you have no problem with just changing the labels by integers keeping the ordinal order (in fact it is better so the algo can learn the natural distance between cold and hot as bigger than cold and mild. Nevertheless, for a decision tree algo, it should not matter as these keep being as labels
one-hot-encoding: when the algorithm type requires numbers and there is NOT an ordinal nature among the data, this is the option required to prevent not ordinal data to be understood as being ordinal.
More info here


Train Accuracy vs Test Accuracy vs Confusion matrix

Definitions

Accuracy: The amount of correct classifications / the total amount
of classifications.
The train accuracy: The accuracy of a model on examples it was constructed on.
The test accuracy is the accuracy of a model on examples it hasn't seen.
Confusion matrix: A tabulation of the predicted class (usually
vertically) against the actual class (thus horizontally).

Overfitting

What I would make up of your results is that your model is overfitting. You can tell that from  the large difference in accuracy between the test and train accuracy. Overfitting means that it learned rules  specifically for the train set, those rules do not generalize well beyond the train set.

Your confusion matrix tells us how much it is overfitting, because your largest class makes up over 90% of the population. Assuming that you test and train set have a similar distribution, any useful model would have to score more than 90% accuracy:  A simple 0R-model would. Your model scores just under 80% on the test set.

In depth look at the confusion matrix

If you would look at the confusion matrix relatively (in percentages) it would look like this:

               Actual    TOT
               1    2
Predicted 1 | 77% | 4% | 81%
Predicted 2 | 17% | 2% | 19%
TOT         | 94% | 6% |

You can infer from the total in the first row that your model predicts Class 1 81% of the time, while the actual occurrence of Class 1 is 94%. Hence your model is underestimating this class. It could be the case that it learned specific (complex) rules on the train set, that work against you in the test set.

It could also be worth noting that even though the false negatives of Class 1 (17%-point, row 2, column 1)) are hurting your overall performance most, the false negatives of Class 2 (4%-point, row 1 column 2) are actually more common with respect to the total population of the respective classes (94%, 6%). This means that your model is bad at predicting Class 1, but even worse at predicting Class 2. The accuracy just for Class 1 is 77/99 while the accuracy for Class 2 is 2/6.


K-means program in python for large excel database

First, you can read your Excel File with python to a pandas dataframe as described here: how-can-i-open-an-excel-file-in-python

Second, you can use scikit-learn for the k-means clustering on your imported dataframe as described here:KMeans


Survey papers on Natural Language Understanding

“Grammar as a foreign language” (2015) by Vinyals et al. is not a review paper but is a significant breakthrough by applying sequence-to-sequence deep learning.

"Advances in natural language processing" (2015) by Hirschberg and Manning provides a general overview, including parsing.

"Recent Trends in Deep Learning Based Natural Language Processing" (2018) by Young et al. covers popular deep learning approaches and lists current benchmarks for parsing.


Well, let's start with what label smoothing is. We replace the 0s and 1s in one hot encoding for example for the MNIST dataset with 10 classes by smoothed values e.g.: 0.1 for 1, 0.9 for 9 etc.

If we replace positive classification targets with $\alpha$ and negative targets with $\beta$ the optimal discriminator $D$ becomes:

$$D(x)=\frac{\alpha p_{data}(x) + \beta p_{model}(x)}{p_{data}(x) + p_{model}(x)}$$.

Where the $p_{data}(x)$ corresponds to the original distribution; at least we think. And the $p_{model}(x)$ corresponds to the distribution that we want to target and $x$ are the input features to the discriminator $D$.

Now, when $\beta \neq 0$, erroneous samples from $p_{model}$ have no incentive to move nearer to the data i.e. it reinforces current generator behaviour. Hence, we smooth only the positive labels to $\alpha$, leaving negative labels set to 0. This behaviour prevents discriminator from giving very large gradient signal to the generator.


Welcome to the forum (and machine learning). Choosing an estimator/model can sometimes be the hardest problems, as such there are many different opinions and ways to do so.

The first thing I notice with your problem statement is that you're wanting to predict the Sales (either in $ or quantity, I would assume). Because you're not trying to predict a category or binary label, we can rule out classification models and look at instead regression style models.

A logical starting point for this would be to explore using a Linear Regression. This assumes that you have some existing data with the price, quantity, or target you're trying to predict.

Linear Regression is a logical starting point as they're simple to implement, yet can get some pretty good results. Scikit-learn has a nice package in Python on linear regressions.

If you don't get satisfactory results using Linear Regressions, you may wish to explore Decision Trees, either scikit-learns implementation or xgboost. This rules out classification models.

Briefly, I think it's important to note that the features to your model are nearly all categorical, eg. make, model, etc. To handle these features, you'll need to encode these features, a guide on that is here.

Take your time, explore your data a lot and start with a simple model. The more you understand your data, the more you'll understand the model. Enjoy and have fun! :)


Policy-based RL method - how do continuous actions look like?

The main requirement of on-policy policy gradient methods is that they use a parametric policy $\pi(a|s, \theta)$ that is differentiable with respect to the parameters $\theta$.

This is not restricted to only describing discrete probability distribution functions (e.g. softmax output layer of neural network). A description of any probability distribution function that is differentiable and possible to sample from is all that is required. This is true for the Normal distribution for instance, so one relatively common solution in continuous spaces is for a neural network to output the mean and standard deviation for the distribution of each component of the action vector that accepts continuous values.

Typically the neural network does not perform the sampling to choose an action. This is also true for a softmax output - it is only additional code, outside of the NN, that interprets the values and selects the action. In addition, and unlike softmax, the NN does not need to directly represent the probability distribution function, just enough data to drive the sampling process. However, the nature of the distribution function does need to be taken into account when calculating the gradient in policy gradient methods.

  Thus, it's not obvious how can my action be something continuous like: "turn +19.2345 angles clockwise". Such an action must have already been pre-defined to the "19.2345" value, right?

What the policy might output here, is the two parameters of the distribution $\mathcal{N}(\mu, \sigma)$, which you then must sample to get an action like "turn x degrees clockwise". So for example, the neural network could output $(25, 7)$ and then additional code will interpret those values as describing the distribution and take a sample. If you got a mean 25, standard deviation 7, then at the point you select the action you could get "turn +19.2345 degrees clockwise" amongst a range of other values. The value 19.2345 does not need to be pre-defined or represented in the neural network in order to do that.


How to make machine learning specifically for an individual in a group when we have the data on the group?

There are a number of other options.  The way to select between them is to ask yourself: How are you going to deploy this classifier?  What data will be available to the classifier?

It's possible to build a single model that still has the ability to personalize for a particular person.  You do that by adding another attribute that identifies the person; typically, this would be encoded as a one-hot vector, so you have 3000 data points, and each data point has some features realted to their emotion measurement(s), plus 100 more features for the one-hot-encoded identity vector.

The exact details depend on what data will be available at test time.  Suppose that you will try to predict the number of calories in the evening meal based on the emotion measurements over the past week.  Then you have $100 \times (30-6) = 2400$ data points (the first 6 days from each person aren't usable because you don't have a week of emotion measurements yet).  Each data point has 135 features: 35 features for the emotion measurements over the past 7 days, and 100 features for the one-hot-encoded vector.  Then you train a model on this.

If you use this kind of approach, you need to be careful about how you split the data into training, validation, and test sets.  I suggest you split by person not by data point; so if you want a 60/20/20 split, you pick 60 people to be in the training set (leading to $60 \times 24$ data points in the training set, 20 people to be in the validation set ($20 \times 24$ data points), and 20 people for the test set ($20 \times 24$ data points).  This is based on a deployment scenario where you train a single model once based on some people's data, and then to predict calories for a new person, you don't retrain it based on historical data from the new person.

Hopefully you can see how to adjust this based on the deployment scenario.  The basic principle is to figure out how the classifier will be used in practice, and then choose an evaluation strategy that matches how it will be used.

Another option is to first try to cluster the users into a few clusters, and build a classifier to predict which cluster each person is in.  Then, build one model per cluster.

A third option is to try to identify some additional attribute of the person that will help you predict their calory consumption.  Add this as an additional feature (W).    For instance, this might be gender or demographics or anything else that you think has predictive power.


Confidence intervals for binary classification probabilities

I don't think there is a good way to do this for all models, however for a lot of models it's possible to get a sense of uncertainty (this is the keyword you are looking for) in your predictions. I'll list a few:

Bayesian logistic regression gives a probability distribution over probabilities. MCMC can sample weights from your logistic regression (or more sophisticated) model which in turn predict different probabilities. If the variance in these probabilities are high you are less certain about the predictions, you could empirically take the 5% quantile or something.

With neural networks you could train them with dropout (not a bad idea in general) and then instead of testing without the dropout, you do multiple forward passes per prediction and this way you sample from different models. If the variance is high, again you are more uncertain. Variational Inference is another way to sample networks and sample from these different networks to get a measure of uncertainty.

I don't know from the top of my head but I'm sure you could do something with random forests with the variance between the different end nodes where your features end up, assuming they are not deep, but this is just something I thought of.


Training Encoder-Decoder using Decoder Outputs

I would like to know if it's possible to train a decoder by feeding
  its predictions at time, t-1, into the input at time-step t.

Yes, it is possible to do it.

But I don't see why you would do it.

Υou will have accumulated error propagated and amplified in every new prediction, making your prediction to diverge from the ground truth sooner or later.


How to properly split train and test to measure performances?

welcome to the forum. I dont know what you exactly mean with 1) „common samples“.

But train and test should have no common rows/observations. They should be two distinct sets.

Having duplicates in test is not a good idea since you want to test how well your model works on data not seen so far by the model. Duplicates in test shrink the number of unique test cases. So there is little use of duplicates in the test set.


Predicted and true values distributions comparison

I don't know the method you're using but I suspect that what you observe here is a common problem with supervised learning: models tend to favour predictions close to the mean, that is avoid extreme predictions because these are usually more risky (higher loss if it's a mistake). As a consequence the std deviation of the predictions is often significantly smaller than the s.d. of the ground truth.

Afaik there's no perfect solution. Typically you could try to encourage risky predictions a bit more in the loss function, if that's an option with your method. But in most applications it's safer to learn to live with this issue.


Detecting if a sentence contains a numeric series

So there are many ways to denote a series.  How are you going to parse the series down to determine the values if you don't know the format?

Determining if the label has a series does not get you to the specific numbers in the series.

2,3,5,7  parses out to 4 numbers

Is 6 in 1996?  I assume that is one number and 1996 != 6

"55,56,57" is series with a 6 but but not the number 6

Does 7-9 parse out to 2 numbers or 3 number
Is 6 in 7-9?
If 6 is in 7-9 identifying that as a series does not answer that question.

How many ways can there be represent a series that regex got out of hand?  For each format of series you also need to parse the values.  You need to know the format of the series to parse out the numbers.  I would have a set of regex mapped to set of parsers.

Maybe use machine learning to identity new series formats but you are still going to need to parse out the series.


For visualizing paired 1D data, i.e. true vs predicted counts, you may use something like ggpaired.

You may also visualize the distribution of differences, where each sample is the difference between the true count and its predicted value; excluding the zero differences could better accentuate any deviation.

For a statistical test, you may use Wilcoxon signed rank test (python), or alternatively Sign test if the distribution of differences (true count - predicted count) is not symmetric around the mean.

It is worth noting that a statistical test may lead to "not the same" (rejection of "the same" null hypothesis) for an arbitrarily small difference when a large enough sample is provided, so be careful about the interpretation of "significant". For example, the statement "predicted counts under-estimate the true counts" could be statistically significant but with mean difference 0.05. Although this difference is statistically significant, however, depending on the task, it might be negligible. Therefore, for example, reporting the "mean ± std" of difference along side the statistical test would help interpreting the "significant" better. Also, take a look at this post on effect size.


Shall I use ordinal encoding or One-Hot-Encoding when using DBSCAN for content clustering on websites?

With the information you provide, I suggest you can use both and test based upon a metric like Siluhette Score or Davis Bouldin index to measure the quality of your clusters when using either of both preprocessing techniques. In this way, you will have an objective metric to compare between the two preprocessing techniques and keep the one that maximizes the chosen metric since it means you are taking the preprocessing that generates better-formed clusters


How to normalize data from multiple sources?

Make sure that when you merge your datasets, the column information is as generic as possible, which means: When you merge A with B, make sure you name "Purchases at (any of the) online channel" not "Purchases at A's online channel" and "Purchases at B's online channel.

Merging datasets from completely different datasources is very difficult because they were not thought with the same basic ideas.


Does bias have multiple meanings in Data Science?

Bias can mean different things in statistics:

If your model is biased, it's likely your model is under-fitting.
Some data set is biased in sample collection. For instance, if you assume your sample responses are independent, but somehow it's not, this is a bias in your data set. If you want to sample everybody in the country, but you skip some cities for no reason, this causes bias in your data set.
Your estimators could be biased - the expectation of your estimator is not equal to the true value in the population.
"Bias" is also used to describe an learnable offset parameter when using transfer functions, e.g. in neural networks when calculating activation of an artificial neuron.


How to determine the inputshape of a ANN in Keras

There are basically two parts.

Why there is a comma in the [3,]? In this case you can skip it and just use [3]. You can encounter it in the tutorials, because you can pass the tuple as shape as well - (3,) and if you skip the comma in the tuple, then it will be just number, not tuple. So, it's just more of a python, not a keras itself. Try this in terminal

>>> 3 == (3)
True
>>> 3 == (3,)
False
>>> [3] == [3,]
True

The bottom line is that keras in this case expects 1-dimensional objects of size 3.

About the recurrent networks. In contrast with fully-connected neural nets, the input to the RNN is two-dimensional. I.e. you have a time series with 10 steps, each defined by 3 numbers, then the shape would be (10, 3). The problem is that you don't know beforehand how many steps you will have - it could be 20, it could be 5. So, you use None as a placeholder to say the keras to expect some dimension there.

You can ask, why we use None here and not something else. The reason is numpy has been using similar notations for years. See the example below
>>> import numpy as np
>>> x = np.ones(5)
>>> x.shape
(5,)
>>> x_2 = x[None, :5]
>>> x_2.shape
(1, 5)

In this case None in index tells  numpy to add a dimension. It's not very transparent thing and could be confusing at first, especially with all the other notation, but you will get used to it.
EDIT

Worth mentioning, the number of dimensions of input tensors always will be bigger on one, i.e. 2 dimensions for MLP and 3 for the RNN, because we process data in batches. Thus, the first dimension always will be the number of samples.
Let's look on the examples.

For MLP input tensor shape could be [128, 3], where the 128 is the number of samples and 3 is number of features. input_shape=(3,)
For RNN input tensor shape could be [128, 30, 3], where the 128 is the number of sample, 30 is sequence length and 3 is number of features. input_shape=(None, 3)
For convolutional NN the inputs will be images and shape like [128, 220, 220, 3], where the 128 is the number of images, 220x220 - size of the image and 3 is number of channels (colors). input_shape=(220, 220, 3)

The interesting fact - we asked to specify the input shape not because keras authors are pedants, but because the specific size of the network is depend on it and we need this info during initialization. As batch size doesn't influence the model size, it is omitted by convention. We would probably omitted the sequence length as well, if the RNN was the only architecture, but for consistency with other type of models it is as it is.


Named-Entity Recognition (NER) is one of the techniques you could look at. Different techniques exist, you could first look at pre-trained models of Spacy in Python here. Otherwise, you could train a model on your own training data if you have.

Also, more systematic approaches can be efficient. For exemple, looking at POS tags values, do regex identification...

More generally, this is the field of Information Extraction (IE) in text mining. Some good resources can be found here and here. Sorry, everything is in Python as I work mainly with it.

Good luck!


If I do not do any data normailization, is datagen.fit required in Keras?

According to the comment of the documentation in the code:

# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(x_train)

It does the normalization, reducing mean and dividing by standard deviation, and more things like PCA. So it seems that you don't need to do normalization. That method does that, and normalizing features is required for accelerating training process and caring about all features with different scales the same.


Extracting specific information from multiple unstructured website

Is there any better approach apart from writing custom web scraper/parser?

Probably not. Although this problem might be tractable using machine learning and natural language processing techniques in future, it would be far more work, and a much larger challenge, than the typical 80/20 rule you get from a web scraper plus regular expressions.

E.g. you write a web scraper with your best guess at the rules for extracting names. It finds 800 names correctly out of 1000. You analyse the problem sites, and add custom CSS selectors and tweak some of the regular expressions. That finds 150 of the remaining 200. Repeat the analysis and fix, you are down to maybe 10 problem sites, each requiring an individual fix. It is dull, repetitive work.

The alternative of attempting this problem using machine learning and NLP: First get the true labels for 10,000+ sites . . .


Predicting descrete value problem in regression or classification

IDs are categorical, not numeric. You should be treating this as a multi-class classification problem. Your IDs are locations, a location is a class. The ID is just a identifier for the class.

Since you have a classification problem you should be using precision, recall and f1. However, if it was regression you would have been using mean squared error, mean absolute error and possibly something else.


semi supervised learning doubt only classify points with confidence above threshold

Most sklearn categorizers have probability outputs.

CLF.predict_proba

from that you can decide the threshold.


Why is there a difference in performance across the feature descriptors for the same imaging modality?

Feature-extraction mechanisms like GIST, HOG, etc are built and optimized to improve performance on given datasets.  Because of this, they don't perform as well across datasets.  It's kind of like putting specialized fuel in a vehicle that isn't built to utilize it - it might even do harm.

Hand-engineered features are, as a rule, brittle.  I once heard it said that the dirty secret of machine learning is just knowing how to transform your domain-specific information into meaningful features - after that, you can use an extremely simple classifier and it may do surprisingly well.  The drawback is that the rules you built are very specific to your domain.

Deep neural networks, and convolutional neural networks in particular, are an advancement in that they learn what features are useful about raw data - for CNNs, these are the raw pixel or time-series values.  Instead of hand-building feature extraction mechanisms, these architectures automatically build them.

One benefit of this is that if you use a CNN to identify images in general, you can re-use the top few feature extraction layers of the CNN on a different image recognition dataset, and re-train the bottom few layers to make the network specific to recognizing e.g. dog breeds.  You can transfer what you learned about the statistical structure of natural images in general to other, more specific questions (general -> specific).  In your case, the 'top few layers' are analogous to your GIST/HOG methods - and they wouldn't be expected to perform well when the task changes, because they were constructed for a specific task (specific -> other specific).


How to calculate prediction error in a LSTM keras

You may use the technique explained in the article "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning"

Very briefely the technique consists of applying dropout for training and predictions.

In Keras this can be easily applied passing the training argument in the call of the Dropout layer.

import keras

input  = ...
x      = keras.layer.Dense(100)(input)
dp     = keras.layer.Dropout(0.5)(x, training=True)
output = keras.layer.Activation('relu')

model = keral.Model(input, output)

During prediction you get the mean and standard deviations:

T = 1000   # Do 1000 predictions to estimate uncertainty
predictions = np.array([model.predict(X_test)] for _ in range(T)])

pred_mean = results.mean(axis=0)
pre_std   = results.std(axis=0)

You can increase (or descrease) T if you need more (or less) precision.


Stochastic gradient descent in matrix factorization, sensitive to label's scale?

The larger your target scores, the larger latent variables should be (well, it's not only a magnitude that matters, but also a variance, but it still applies to your case). There's no problem with larger coefficients of latent vectors unless you use regularization (and, likely, you do). In case of regularization your optimal solution will tend towards smaller values, and'd sometimes prefer to sacrifice some accuracy for lower regularization penalty.

Gradient Descent doesn't suffer from problem of large coefficients (unless you run into some sort of numerical issues): if the learning rate is tuned properly (there are lots of stuff on it, google), it should arrive to equivalent parameters. Otherwise nobody guarantees you convergence :-)

The common rule of thumb when doing regression (and your instance of matrix factorization is a kind of regression) is to standardize your data: make it having zero mean and unit variance.


Dueling Network gradient with respect to Advantage stream

I verified the following formulas with Gradient Checking, and it's confirming they are correct. (Getting discrepancy less than 0.00002)

Let's draw the tree diagram of what's affecting what:

If we try to find gradient affecting $a_0$, we have to sum-up the pink chains. They make it evident that $a_0$ has affected all $Q$ variables, thus we sum these chains.

So the formula for entire gradient that has to enter $A$ layer is:

$$
gradForA =  \left[  \begin{array}{c}
 \frac{\partial E}{\partial a_0}
\\\frac{\partial E}{\partial a_1}
\\\frac{\partial E}{\partial a_2}
\\\frac{\partial E}{\partial a_3}
\end{array}
\right]  = \left[  \begin{array}{c}
 \\\frac{\partial E}{\partial Q_0}(1 - u) + \frac{\partial E}{\partial Q_1}(-u)+ \frac{\partial E}{\partial Q_2}(-u) + \frac{\partial E}{\partial Q_3}(-u)
\\\frac{\partial E}{\partial Q_0}(- u) + \frac{\partial E}{\partial Q_1}(1-u)+ \frac{\partial E}{\partial Q_2}(-u) + \frac{\partial E}{\partial Q_3}(-u)
\\\frac{\partial E}{\partial Q_0}(- u) + \frac{\partial E}{\partial Q_1}(-u)+ \frac{\partial E}{\partial Q_2}(1-u) + \frac{\partial E}{\partial Q_3}(-u)
\\\frac{\partial E}{\partial Q_0}(- u) + \frac{\partial E}{\partial Q_1}(-u)+ \frac{\partial E}{\partial Q_2}(-u) + \frac{\partial E}{\partial Q_3}(1-u)
\end{array}
\right]
$$

where $u$ is $\frac{1}{N}$

Bonus:

By looking at the above image, we see that the $v_0$ has similarly affected all $Q$ variables, so its gradient is:
$$
gradForV = \frac{\partial E}{\partial Q_0} + \frac{\partial E}{\partial Q_1}+ \frac{\partial E}{\partial Q_2} + \frac{\partial E}{\partial Q_3}
$$

$gradForV$ is a scalar (not a vector) because value stream only has 1 neuron, always.

Bonus #2

to compute $gradForA$ it's obvious to do via 2 for-loops, nested in one another. However, you can get rid of nesting and improve code performance by re-writing the formula as follows:

$$
gradForA =  R + scalarB
$$

where

$scalarB = (-u)(\frac{\partial E}{\partial Q_0} + \frac{\partial E}{\partial Q_1}+ \frac{\partial E}{\partial Q_2} + \frac{\partial E}{\partial Q_3})$

and

$$ R =  \left[  \begin{array}{c}
 \frac{\partial E}{\partial Q_0}
\\\frac{\partial E}{\partial Q_1}
\\\frac{\partial E}{\partial Q_2}
\\\frac{\partial E}{\partial Q_3}
\end{array}
\right]
$$

The cool thing is that $scalarB$ can be computed once (it's a scalar), and re-used when computing the $gradForA$ vector.

Notice that $scalarB$ and $gradForV$ are almost the same thing, just with extra $(-u)$  So you can re-use this value in your code as well.


The argmax is to get the class label prediction y_pred given a sample x.

# symbolic description of how to compute prediction as class whose
 # probability is maximal
 self.y_pred = T.argmax(self.p_y_given_x, axis=1)

The Loss function is used as optimization objective function to solve the coefficients W and b based on training data. On the tutorial, the loss function is the reverse the likelihood.


What kind of neural network structure is suitable for image to image learning?

I would like to know what kind of neural network is suitable for this task?

A convolutional autoencoder is a good thing to try, but there is a minor issue with it for this type of problem: the latent representation has often lost spatial awareness (especially if FC layers were used at the end of the encoder). For a task like this, a pixel $p_{ij}$ in the input is likely to be far more related (and useful) to the corresponding (same position) pixel $\widehat{p}_{ij}$ in the output, compared to some other, far away pixel $p_{\ell k}$ in the input.

For this reason, I'd suggest using a U-net architecture, as this preserves spatial localities throughout the network.

References:

Long et al, Fully Convolutional Networks for Semantic Segmentation
Ronneberger et al, U-Net: Convolutional Networks for Biomedical Image Segmentation

  And what kind of loss function is suitable?

A good starting point is the standard mean $L_1$ or $L_2$ loss over the output image. Since your images are not natural images, I hesitate to recommend perceptual losses, but they could work as well.

Without more details about your architecture and data, I'm afraid I cannot say much more than these though.


Coming to your first question:

Yes, you can but it's not advisable to use LS as a cost function for classification task since the optimization problem becomes non-convex! More on it, since your model in logistic regression would be sigmoid(a non-linear function) which means your cost function will have a lot of local optima!(think of a surface of the cost function as a Himalayan valley!), So, when you will use Gradient descent(an iterative approach) for minimizing your cost function, GD will be stuck somewhere in the local minima instead of global minimal, so you will not learn the best model parameters for your model on your data! Therefore, we change the cost function from LS to Log loss for classification task that ensures a nice bell-shaped curve surface of the cost function where the gradient descent can reach to global minima and give you the corresponding best model parameters which you can happily take it later to do predictions(on test-set)!
More on it here

Much more theoretical understanding of it you could find in ISLR at page 129 4.2 Why Not Linear Regression?


Are there any algorithms for a classification problem involving unlimited classes, and only a few instances per class

Yes there are couple of methods. I recommend reading the Siamese Networks paper. From there you can check more modern approaches like Prototypical Networks and so on. In general the keyword you are looking for is few shot learning.
All this approaches basically depend on finding similiarities between two or more inputs. One input is your unknown input and the others are your known samples. Your network will predict a class corresponding to maximum similarity.


Gradient Descent Python Implementation isnt converging

I was doubting my implementation all the way but it was the learning rate.
After a lot of experimentation, I found the right one, but I'm very much surprised to see how small the learning rate had to be in order for it to work, i.e alpha = 0.000001


Which are the appropriate prameters for lda modeling?

The main and by far most impactful parameter that you have to search for is k - the number of topics you assume. Conceptually, this is similar to K Nearest Neighbors, in which you have to pre-specify the number of clusters you are searching for and then determine the best fit clusters by properties of the clustering results.

The other parameters, including the method of solving the mathematical problem may produce slightly different results, but none will be nearly as impactful as k.

On a more general note, you can almost always find the answer to this question for an implemented algorithm you're trying to use from the API documentation of the package in question- the topicmodels documentation containing the LDA interface can be found here.

Often, if you don't well understand the properties of the algorithm that you're trying to use, the documentation will also point you to resources which thoroughly describe the algorithm in question. It is this understanding of the algorithm and what it is doing that will enable you to answer the questions like "What parameters should I tune, over what values, and what is likely to change about the outputs when I do so?"


If your total dataset size is too much, take a random sample of n%. I suggest taking a sample of n% of users, then using all rows for each of those sampled users. If you have timestamps, I further suggest making your train/test split based on time. I.e., for a user with 10 purchased items, train on her first 8 items and try to predict her last 2 items.


You could have a look at the AutoML work, which offers a few different ways to optimise a model of parameter space and goes well beyond Sklearn. They even have a tool that wraps around Sklearn!

Here is a summary from their homepage:

Here is a longer description, and here is a full example based on Keras.

These libraries can essentially combine the best known practical methods of optimisation, such as:

standard grid-search over parameters
Bayesian optimisation of parameters
a combination of the two methods: HpBandSter (HyperBand on STERoids)


You can download the corpus from this site. To the best of my knowledge this is the most complete email corpus available. A project to label a subset of this email corpus can be found on this UC Berkley site. I am not sure though whether these emails have the right training labels for you.


About the maximum likelihood, when we convert the maximization problem into minimization, why we take the negative?

It is common to define optimization problems as minimization problems instead of maximization. And by multiplying your target functions with $-1$ you can transform one into the other:

$$\max_{w} \log{L(w)} \Leftrightarrow \min_{w} -log{L(w)}$$

So to maximize the log-likelihood you minimize the negative log-likelihood. Basically it just comes down to conventions in optimization theory.

Moreover, since $L(w) \in [0,1]$ its logarithm $\log{L(w)}$ will be less than or equal to $0$ (note that $log{0}$ is not defined). Accordingly $\max_{w} \log{L(w)}$ means to maximize a negative number which is, at least to me, less intuitive than minimizing a positive number.

The more interesting part is actually the log-transformation which increases numerical stability of your calculations (since it "transforms" the multiplication to a sum and thereby reduces the risk of underflowing).


How to automatically disable register_hook when training model is in eval() phase in PyTorch?

Removing score.retain_grad() and guarding register_hook with if condition (if score.requires_grad) does the trick.
if score.requires_grad:
    h = score.register_hook(lambda grad: grad * torch.FloatTensor(...))

Originally answered by Alban D here.


What is the difference between Pytorch's DataParallel and DistributedDataParallel?

As the Distributed GPUs functionality is only a couple of days old [in the v2.0 release version of Pytorch], there is still no documentation regarding that. So, I had to go through the source code's docstrings for figuring out the difference. So, the docstring of the DistributedDataParallel module is as follows:

  Implements distributed data parallelism at the module level.
      This container parallelizes the application of the given module by
      splitting the input across the specified devices by chunking in the batch
      dimension. The module is replicated on each machine and each device, and
      each such replica handles a portion of the input. During the backwards
      pass, gradients from each node are averaged.
      The batch size should be larger than the number of GPUs used locally. It
      should also be an integer multiple of the number of GPUs so that each chunk
      is the same size (so that each GPU processes the same number of samples).

And the docstring for the dataparallel is as follows:

  Implements data parallelism at the module level.
      This container parallelizes the application of the given module by
      splitting the input across the specified devices by chunking in the batch
      dimension. In the forward pass, the module is replicated on each device,
      and each replica handles a portion of the input. During the backwards
      pass, gradients from each replica are summed into the original module.
      The batch size should be larger than the number of GPUs used. It should
      also be an integer multiple of the number of GPUs so that each chunk is the
      same size (so that each GPU processes the same number of samples).

This reply in the Pytorch forums was also helpful in understanding the difference between the both,


According to the two papers (Limited Evaluation Evolutionary Algorithm (LEEA) and Limited Evaluation Cooperative Co-evolutionary Differential Evolution Algorithm (LECCDE)) you can approximate the result of the evolutionary algorithms by applying the algorithm not on the whole dataset but on mini-batches like in SGD (Stochastic Gradient Descent). This significantly reduces computation time (about 25 times in LECCDE paper for an ANN on MNIST dataset).

According to Prellberg 2018 Limited Evaluation Evolutionary Optimization the training times of LE algorithms is still 10 times longer than the Adam variation of stochastic gradient descent. In your case it is not possible to calculate gradients.

The following changes to the evolutionary algorithm are proposed in the first paper (LEEA):

Use mini-batching while ensuring the diversity of the expected outputs.

  Ideal mini-batch size in LEEA was determined experimentally to be 2. ...To apply the idea of diversity in minibatches, the output range is divided into two equally sized sections, and examples selected so that each mini-batch contains
  one example for both sections of the range. A new mini-batch is randomly generated at the beginning of each generation.

Use ancestor fitness for evaluation.

  f'=(f_p1 + f_p2)/2*(1-d) + f.

  Where f' is the individual's modified fitness. f is the fitness of the
  individual against the current mini-batch. f_pn is the fitness of
  parent n. d is a constant decay value.

The second paper (LECCDE) points to the following speed-up using Limited Evaluation algorithms:

LECCDE vs CCDE for different datasets:

Dataset    Parameters   Speedup             Accuracy
WBC        1652          1.10 times faster  1.2% worse
ESR        9052          2.05 times faster  0.3% worse
HAR        28406         4.00 times faster  0.2% worse
MNIST      47710        25.3  times faster  unknown


For the sake of completeness, fastText uses sub-word information (character level n-grams) when creating embeddings and then for classification with some interesting results:

can give prediction for the unknown words that have some similarity with known ones
with interesting results for modelling syntactic relations like 'went' - 'go' + 'give' = 'gave'


Transform single-label data set into multi-label data set

You should consider using a neural network for this. By using binary crossentropy across multiple categories, you can get a probability "rating" for each category and how it applies to the text. From there, you can develop a script that establishes a threshold (say 0.8) and then creates a new entry of labeled data for that one particular piece of text across multiple categories. There are many examples out there of people taking IMDB data and the movie descriptions and assigning multiple genres to a single movie (like a "horror" movie can also be a "suspense" movie or a "comedy" movie can also be an "animated" movie). Those types of examples should fit what you need here.


Guidance to create a machine learning alg to find multiple overlapping date over multiple period

This is quite easy and you don't need machine learning at all.

Order by Person and Start date. Then simply iterate over each data line for a person and see if the start dates are earlier than the end dates for the previous entry. Add rules as necessary.


You can accomplish this using a Feature Constructor widget with some conditional logic. For example, if looking at the Titanic survival data set, some fares are marked $0. This does not make sense since tickets have a non-zero value. You can create a new variable called fare_new where valid fares are copied over but 0 values are replaced with missing values. See screenshot below:


Where can I find a dataset that contains criminal case sentencing data?

Probably you checked this already, kaggle.com crime tag:
kaggle crime tag
Where yo have some interesting datasets, like:
Executions in the United States, 1976-2016


how to get the classe names in an image classifer after predection?

I get your problem,  the point is that its correct what the model does, but you have to build a look-up table for its answer. Your ground-truth, looks somethink like that [0,0,0,0,1], a one-hot vector for example. You, the human know what this code stands for, for example cats.  just like that you have to build an numpy array, listing the word-embeddings in the correct order and afterwards calling it like: class_names[prediction],prediction being your CNN-result ->[7].

To sum it up, the Dense-layer in the end is giving you with softmax-activation a propability desnity function P. You are using this in comparing it with your ground-truth desnity function q, and calculate the difference. You just use numbers, not words, so you have to write an interpretation of the models answeres, for example in form of a lookup-table.

An example could be like:

pred = model.predict_classes(new_image)
labels=np.array(["cats","dogs","cars","humans"])
print(labels[pred[0]])

-> >>> cats


Alternative methods for improved clustering separation?

Your data doesn't appear to be easily separable. In general, one could apply some kind of transformation that pulls apart the distributions for each class. Having labels available makes it possible, in principle, to learn such a transformation (as @Emre metnioned in the comments). But, there are a couple issues with your particular data set. 1) You don't appear to have many data points (unless you've only plotted a small subset). This would limit you to very simple transformations (otherwise you'd probably get severe overfitting). 2) The points are simply overlapping. A transformation can only work based on its inputs and, if the coordinates are indistinguishable, there's nothing that can be done. In the best case, you might be able to pull the lower left turquoise cluster and the yellow points further from the main mass, but the rest of the points are pretty much intermingled. Any transformation that could manage to separate them in the training data would be very complicated, and probably just reflect sample noise (i.e. it would probably be completely overfit, and not generalize to new data).

The ideal thing would be to find/measure additional (relevant) variables. In this case, the classes may become separable in the higher dimensional space. For example, imagine additing a third axis, where the red points become 'lifted' above the blue points.


Models are trained on training data, and evaluated on test data based on assumption that unseen test data also comes from the same distribution with training data. So when you calculated statistics for training data, based on assumption that test data also comes from the same distribution you should apply same transformations to test data. You should fit MinMaxScaler to your training data, and then use this scaler to transform both tranining data and test data. There are also issues about data leakage, take a look at that: StandardScaler before and after splitting data
For transformations, fit method extract relevant statistics(min, max value for min-max scaling, mean, std for standardization) from the provided data, and transform method transforms each feature individually based on extracted statistics.


How does backpropogation behave when there is a dropout in FC layer?

Correction made (see edit log):

No, there is no need for that.

For all dropped neuron, the corresponding activations are simply zeroed out so the corresponding gradient will also be zero.

Now because you do not want to update the parameters, you simply multiply the gradients with the same dropout mask as was used in your current iteration.

If you're using a library, you just have to do everything as it is usually done because this is already being taken care of.


Your WEBS_4 is an array with two elements, so there's no ip element in it, which explains why this fails:

dbGetQuery(conexion,"SELECT
              data->'results'->'WEBS'->'WEBS_4' ->'ip' as test
                          FROM results;")

You need to get the zeroth element of WEBS_4, and then you can get the ip element of that. This works in a simple PostgreSQL test I set up using psql:

SELECT  data->'results'->'WEBS'->'WEBS_4'->0->'ip' as test
                          FROM results;
  test
---------
 "XXXXX"
(1 row)

So I suspect the relevant R syntax will be (untested):

dbGetQuery(conexion,"SELECT
              data->'results'->'WEBS'->'WEBS_4'->0->'ip' as test
                          FROM results;")


Dataset management: What are some strategies/solutions for efficiently storing datasets with their versions?

I used my existing directory structure with Git LFS and it works well.

Here are the advantages I got by using Git LFS other than the alternative solutions.

It saves space: Since Git LFS manages my dataset files (100MB to 1GB) with pointers and keeping only the minimum in the local memory.
It works seamlessly with a git enabled repository: I was planning to use some storage solution (like Amazon S3) specific for my datasets. But I would have to make extra efforts to manage multiple version and keep my dataset files in sync.


GANs (generative adversarial networks) possible for text as well?

Yes, GANs can be used for text. However, there is a problem in the combination of how GANs work and how text is normally generated by neural networks:

GANs work by propagating gradients through the composition of Generator and Discriminator.
Text is normally generated by having a final softmax layer over the token space, that is, the output of the network is normally the probabilities of generating each token (i.e. a discrete stochastic unit).

These 2 things do not work well together on their own, because you cannot propagate gradients through discrete stochastic units. There are 2 main approaches to deal with this: the REINFORCE algorithm and the Gumbel-Softmax reparameterization (also known as the Concrete distribution). Take into account that REINFORCE is known to have high variance so you need large amounts of data to get good gradient estimations.

As an example of REINFORCE for textual GANs you can check the SeqGAN article. An example of Gumbel-Softmax you can check this article.

Another completely different option is not having a discrete stochastic unit as output of the generator (e.g. generating tokens deterministically in embedded space), hence eliminating the original problem of backpropagating through them.


Feature Scaling and normalization in cross-validation set

Answering your last question first: your mean and std should be computed over the entire image. So for an RGB image you will get a mean/std vector of 3 values for the entire dataset (calculated per channel). Normalizing per pixel is wrong and would result in loss of information (you need to think of an image as a single discreet function, you can't treat each part of it separately without changing its overall meaning).

Regarding normalizing the validation set, you can manually create the validation set and use the "validation_data" argument in the fit method:

# Creating the validation set:
X_valid = X_train[int(0.9*len(X_train)):]
X_train = X_train[:int(0.9*len(X_train))]
y_valid = y_train[int(0.9*len(y_train)):]
y_train = y_train[:int(0.9*len(y_train))]

# pre-processing: divide by max and subtract mean

scale = np.max(X_train)
X_train /= scale
X_valid /= scale
X_test /= scale

mean = np.std(X_train)
X_train -= mean
X_valid -= mean
X_test -= mean

model.fit(X_train, y_train, batch_size=16, epochs=10, validation_data=(X_valid, y_valid), verbose=2)


how to pass parameters over sklearn pipeline's stages?

I figured out how to do that by monkey patching ParameterGrid.__iter__ and GridSearchCV._run_search methods.

ParameterGrid.__iter__ iterates over all possible combinations of hyerparameters (dict of param_name: value). so i modified what it yields (one configuration of hyperparameters params) by adding "km__nbr_features" equal to 'tfidf__max_features':

params["km__nbr_features"] = params['tfidf__max_features']

Important: "km__nbr_features" must be missing from grid_params so the trick works.

Here is some code:

from sklearn.model_selection import GridSearchCV, ParameterGrid
import numpy as np
from itertools import product

def patch_params(params):
    # Updates a configuration of possible parameters
    params["km__nbr_features"] = params['tfidf__max_features']
    return out

def monkey_iter__(self):
    """Iterate over the points in the grid.

    Returns
    -------
    params : iterator over dict of string to any
        Yields dictionaries mapping each estimator parameter to one of its
        allowed values.
    """
    for p in self.param_grid:
        # Always sort the keys of a dictionary, for reproducibility
        items = sorted(p.items())
        if not items:
            yield {}
        else:
            keys, values = zip(*items)
            for v in product(*values):
                params = dict(zip(keys, v))
                yield patch_params(params)

# replacing address of "__getitem__" with "monkey_getitem__"
ParameterGrid.__iter__  = monkey_iter__

def monkey_run_search(self, evaluate_candidates):
    """Search all candidates in param_grid"""
    evaluate_candidates(ParameterGrid(self.param_grid))

# replacing address of "_run_search " with "monkey_run_search"
GridSearchCV._run_search = monkey_run_search

Then i preformed Grid Search normaly:

def create_model(optimizer="adam", nbr_features=100):
    model = Sequential()
    model.add(Dense(512, activation='relu', input_shape=(nbr_features,)))
    ...
    model.compile(loss='binary_crossentropy', optimizer=optimizer,metrics=["accuracy"])
    return model

estimator = Pipeline([("tfidf", TfidfVectorizer()),
                      ('norm', StandardScaler(with_mean=False)),
                      ("km", KerasClassifier(build_fn=create_model, verbose=1))])
grid_params = {
     'tfidf__max_df': (0.1, 0.25, 0.5, 0.75, 1.0),
     'tfidf__max_features': (100, 500, 1000, 5000,),
      ... }

# Performing Grid Search
gs = GridSearchCV(estimator,
                   param_grid,
                   ...)

Update:
In case you used RandomizedGridSearchCV you must monkey patch ParameterGrid.__getitem__ insted.

def monkey_getitem__(self, ind):
    """Get the parameters that would be ``ind``th in iteration
    Parameters
    ----------
    ind : int
        The iteration index
    Returns
    -------
    params : dict of string to any
        Equal to list(self)[ind]
    """
    # This is used to make discrete sampling without replacement memory
    # efficient.
    for sub_grid in self.param_grid:
        # XXX: could memoize information used here
        if not sub_grid:
            if ind == 0:
                return {}
            else:
                ind -= 1
                continue

        # Reverse so most frequent cycling parameter comes first
        keys, values_lists = zip(*sorted(sub_grid.items())[::-1])
        sizes = [len(v_list) for v_list in values_lists]
        total = np.product(sizes)

        if ind >= total:
            # Try the next grid
            ind -= total
        else:
            out = {}
            for key, v_list, n in zip(keys, values_lists, sizes):
                ind, offset = divmod(ind, n)
                out[key] = v_list[offset]
            return patch_params(out)

    raise IndexError('ParameterGrid index out of range')

ParameterGrid.__getitem__ = monkey_getitem__


How to deal with evaluations in a multi step ML pipeline

A pyspark.ml.PipelineModel is the result of calling the .fit() method of a pyspark.ml.Pipeline. This estimator is a sequence of Transformers and/or Estimators, all of which have a .transform() method. When you call the .transform() method of the pyspark.ml.PipelineModel-- say, when you want to make predictions using the trained model-- the .transform() method of each of the objects in the PipelineModel are called in sequence.

In other words, the transformations for each step are applied by calling the .transform() method of the fitted pipeline- the API does this for you, or more accurately, gives you concise tools to do this as you need to.

What you're talking about is slightly outside the scope of this type of object, however. Train/test splitting occurs outside the Pipeline, and importantly so, for methodological reasons.

A PipelineModel couples data preparation steps with an estimator, but not training operations. Splitting the data, for example, is something you'd have to do outside the Pipeline.

The workflow might go:

Split the data into training and test datasets.

Define your data preparation steps and regression estimator, and combine these into a Pipeline.

Train the entire pipeline on your training data:

fit_pipeline = Pipeline.fit(train)

Make predictions on the training set:

train_preds = fit_pipeline.transform(train)

Use something like a pyspark.ml.evaluation.RegressionEvaluator to evaluate the performance on the training dataset:

evaluator = RegressionEvaluator()
evaluator.evaluate(train_preds)

Then, you can make predictions on the test dataset:

test_preds = fit_pipeline.transform(test)

And evaluate them:

evaluator.evaluate(train_preds)

But you have this completed model and data preparation object that can be used to make predictions on similarly structured data now, called fit_pipeline, which is an instance of pyspark.ml.PipelineModel.


Have you looked into a tokenizer? By your question, I can't tell if either (1) you don't know about tokenizing or (2) you are looking for an alternative to tokenizing.

I will assume the former and suggest that you read about tokenizing and go down that road. Keras provides a very simple approach to tokenizing and sequences that you can implement quickly and should get what you're looking for.


Does image resizing lower the prediction accuracy of MLP?

Generally speaking, it highly depends on your data. If you have images of numbers for each image, it may not be that much bad but for images of cats or dogs, you completely put your information away by resizing to that size.

To answer your question, yes. The reason is that it leads to high Bayes error. It simply means that you as an expert can not say what they are. Consequently, it is not possible for the network to learn them. You can easily see the images and figure out that there is not anything to be learned. For instance, in that case, what is the difference between the sky and sea? Can a $64\times64$ image represent them? Can you as an expert find it out without any previous knowledge?


Choosing the right data mining method to find the effect of each parameter over the target

You can try Bayesian belief networks (BBNs). BBNs can easily handle categorical variables and give you the picture of the multivariable interactions. Furthermore, you may use sensitivity analysis to observe how each variable influences your class variable.

Once you learn the structure of the BBN, you can identify the Markov blanket of the class variable. The variables in the Markov blanket of the class variable is a subset of all the variables, and you may use optimization techniques to see which combination of values in this Markov blanket maximizes your class prediction.


Prediction: plugin a corelation table (neuron) into a Time-Series Neuron in Keras/ TF

If you just have a normal correlation table (Which I am not sure about) you can do it like this: For every row you can add a correlation of the organ in a row to other organs. So you will have

Week Baby   Age Organ   Date    IssueFound Problem Type Organ1 Organ2 Organ3 ...
2   Baby2   A0  Organ2  06/24/16    1      F            0.5    0      0.3    ...

If you do not have a row for a new baby you should then format the data this way together with other parameters. The Corr_Organ1 an aggregation of all correlations that lead to Organ1 failure.

Baby  X-Type_cells Organ1 Organ1_time Organ2 Organ2_time Corr_Organ1 Corr_Organ2 ...
Baby1 1024         Yes    06/24/16    No     nan         0.2         0           ...

Then you make iterations: First forecast everything with zero correlations. Then when you calculate the first forecast and Organ 1 failed you update all other correlations.

Starting data:

Baby  X-Type_cells Organ1 Organ1_time Organ2 Organ2_time Corr_Organ1 Corr_Organ2 ...
Baby1 1024         nan    nan         nan    nan         0           0           ...

After first iteration:

Baby  X-Type_cells Organ1 Organ1_time Organ2 Organ2_time Corr_Organ1 Corr_Organ2 ...
Baby1 1024         yes    xxx         no     nan         0           0.8         ...

Then you predict again based on correlations and get the following result:

Baby  X-Type_cells Organ1 Organ1_time Organ2 Organ2_time Corr_Organ1 Corr_Organ2 ...
Baby1 1024         yes    xxx         yes     nan         0.2         0.8         ...

You repeat those iterations until you do not change anything.

The aggregation of correlations could be just a sum or something more complicated.


Python is reading my data with NANS and Infs, but they don't have any

You should use pandas.DataFrame.describe() to find the values that are present in the feature and see which specific values might be an issue.


Different accuracy of random forest and neural net

Different models will give different accuracies. Same model too could give you different accuracies. You can try setting the seeds to predefined values to bring in more consistency in the same model results