Why do we convert skewed data into a normal distribution

You might want to interpret your coefficients. That is, to be able to say things like "if I increase my variable $X_1$ by 1, then, on average and all else being equal, $Y$ should increase by $\beta_1$".

For your coefficients to be interpretable, linear regression assumes a bunch of things.

One of these things is no multicollinearity. That is, your $X$ variables should not be correlated against each other.

Another is Homoscedasticity. The errors your model commits should have the same variance, i.e. you should ensure the linear regression does not make small errors for low values of $X$ and big errors for higher values of $X$. In other words, the difference between what you predict $\hat Y$ and the true values $Y$ should be constant. You can ensure that by making sure that $Y$ follows a Gaussian distribution. (The proof is highly mathematical.)

Depending on your data, you may be able to make it Gaussian. Typical transformations are taking the inverse, the logarithm or square roots. Many others exist of course, it all depends on your data. You have to look at your data, and then do a histogram or run a normality test, such as the Shapiro-Wilk test.

These are all techniques to build an unbiased estimator. I don't think it has anything to do with convergence as others have said (sometimes you may also want to normalize your data, but that is a different topic).

Following the linear regression assumptions is important if you want to either interpret the coefficients or if you want to use statistical tests in your model. Otherwise, forget about it.

Applying the logarithm or normalizing your data, is also important because linear regression optimization algorithms typically minimize $\|\hat y - y\|^2$, so if you have some big $y$ outliers, your estimator is going to be VERY concerned about minimizing those, since it is concerned about the squared error, not absolute error. Normalizing your data is important in those case and this is why scikit-learn has a normalize option in the LinearRegression constructor.


How about using neural networks instead of linear models? The neural net will have an easier time learning arbitrary non-linear relationships.


Is there Orange support for MySQL and SQLite?

In order to be more precise, I have just finished the example of a Python (ver. 3.7) Script to include into the workflow to extract data from a MySQL server via an ODBC connector as following :

from Orange.data import Table, Domain, ContinuousVariable
from pyodbc import connect

connecteur = connect(DSN='basepython') # "table1" from a MySQL server via ODBC connector.
curseur = connecteur.cursor() # Creation of the cursor for data swept.

# Execution of the SQL Query.
curseur.execute("SELECT id, champ1, champ2, champ3 FROM table1")

# All data of "table1" are saved in "donnees".
donnees = curseur.fetchall()

liste_globale = [] # Final list : It will contain the final data.
liste_intermediaire = [] # Temporary list : Only used for conversion.

# Whereas having tuples as list elements, you must convert them into lists included into the main list.
for partie in donnees:
    for i in range(len(partie)):
        liste_intermediaire.append(partie[i])
    liste_globale.append(liste_intermediaire)
    liste_intermediaire = []

# Creation of the data domain : Names of the variables seen in Orange WorkFlow.
identite = ContinuousVariable("Identification")
colonne1 = ContinuousVariable("Colonne1")
colonne2 = ContinuousVariable("Colonne2")
colonne3 = ContinuousVariable("Colonne3")

domaine = Domain([identite, colonne1, colonne2, colonne3])

# Final data
out_data = Table(domaine, liste_globale) # Data seen at the output of the Python_Script widget.
print(out_data) # This line is only used for debugging, you can comment it.

# Don't forget to close the connection at the end.
connecteur.close()

This example is used with the following workflow :

Of course, if you want to connect both MySQL server and SQLite in the same time, you can adapt this script with two connectors, for example :

connecteur1 = connect(DSN="MySQL_ODBC_DSN")
connecteur2 = connect(DSN="SQLite_ODBC_DSN")

And you can work with these two independent connectors as following :

curseur1 = connecteur1.cursor()
curseur2 = connecteur2.cursor()

curseur1.execute("SELECT * FROM mysql_table")
curseur2.execute("SELECT * FROM sqlite3_table WHERE id='2'")

donnees1 = curseur1.fetchall() # If there are several rows to save.
donnees2 = curseur2.fetchone() # If there is only one row to save.

And at the end, don't forget to close each connection :

# Closing of the two opened database connectors.
connecteur1.close()
connecteur2.close()


If categorical variable has some hierarchy, should I just label them or split into dummy variables (One-Hot encode)?

One hot encoding should be performed between independent values like flowers type etc.
Values that you have mentioned are relative rank where bad is better than worst and so on.
So it should not be one hot encoding except converting it into numerical values.


Difference between Sum of Squares and Maximum Likelihood Linear Regression

Suppose you construct a probabilistic model, where $y_i$ are believed to be related to your $x_i$ under the formula;

$$ y_i = w_1 x_i + w_0 + \epsilon_i $$

So $w_1$ and $w_0$ are your target parameters from before and $\epsilon_i$ is an error term which you expect follows some probability distibution, e.g. $\mathcal{N}(0,\sigma^2)$. It is important that its expectation is zero.

Given your data you want to maximise the probability of returning every $y_i$ given your data $x_i$ subject to your model. The probability of getting $y_i$ from $x_i$ is equal to $f_{\epsilon}(y_i-w_1x_i-w_0)$, where $f_{\epsilon}$ is the probability density function of $\epsilon$, i.e. a normal distribution.

And the likelihood of getting every $y_i$ is the product of them all,

$$ L = \prod_i f_{\epsilon}(y_i-w_1x_i-w_0) $$

You want to maximise this value by tweaking $w_1$ and $w_0$, hence the name maximimum likelihood estimation.

Note that this is equivalent to maximising the log likelihood, so;

$$ \log L = \sum_i \log f_{\epsilon}(y_i-w_1x_i-w_0)$$

And if you look at the normal distribution density function you will see that (after ignoring some constants) this reduces to the problem of maximising..

$$ - \sum_i (y_i-w_1x_i-w_0)^2 $$

or in other words minimising the sum of squares akin to OLS.

But like using a different distance function in OLS you can parametrise a different error distribution in MLE.


MSE vs Cross Entropy for training with facial landmark (pose) heatmaps

I experienced a similar problem while doing pose estimation. Your problem may be due to the huge amount of "background" pixels, where the gaussian heatmap has very small values. So the network simply learns the trivial solution and sets the whole heatmap to zero, which results in a small mse error.

Solution to my problem was:

1) weigh the heatmaps: weigh pixels in the loss function, which are "foreground" more. For example you could count the percentage of "foreground" pixels and the percentage of "background" pixels for each heatmap and weigh foreground pixels with (1-percentage of foreground pixels) and background pixels with (1-percentage of background pixels)

2) Increase the maximum of the gaussians to ~12


It sounds as if you don't need clustering.

But rather you are trying to detect near duplicates.

The difference is that clustering tries to organize everything with a focus on the larger, overall structure. But much of your data probably isn't duplicate. Clustering is difficult and slow. Near duplicates is much easier, and much faster (e.g., with MinHash or similarity search)


How to correctly resize input images in a CNN?

Two solutions:

Crop images to a fixed size
Pad images to a fixed size with 0s.

Keep in mind if you are doing segmentation, gt masks have to be cropped/padded too.


Impact of unlabelled documents for label prediction via SVM

It's a slight variation on your problem as described, but if your goal is to build a good model for predicting documents labeled $L$, I would initially formulate this as a recommender system problem, until you've reached a desirable point in the learning curve of your system. I did exactly this in a publication a few years back. In my approach, all labeled documents are used to train the SVM, while all those unlabeled are used for classifying. I use the signed margin distance of each classified document for ranking, and take the top-$n$ as the next $n$ documents the reviewers should assign a label $\in(L, L_{not})$. A side effect of this process is that you get to iteratively evaluate the performance of your classifier as you add new labeled documents to the model, which should get you what you need. If this approach sounds useful to you, I recommend looking over my paper, as I describe some performance metrics useful for this, in addition to outlining the specific procedure to use that will ensure you're not biasing your model.


Difference between sequence length and batch size in time series forecasting

Let's take a TS data = [ 1, 2, 3, 4, 5, 7, 8, 9, 10 ]
Call the function with these parameters
sequence_length=5, sampling_rate=1, sequence_stride=1, shuffle=False, batch_size=2
 shuffle, batch_size has no role in TS data creation. It will come into effect when you iterate on the returned Dataset.
 In this case, we will have the following data points,
[ 1, 2, 3, 4, 5 ]
[ 2, 3, 4, 5, 6 ]
[ 3, 4, 5, 6, 7 ]
[ 4, 5, 6, 7, 8 ]
[ 5, 6, 7, 8, 9 ]
[ 6, 7, 8, 9, 10 ]
batch_size
When you will iterate on this dataset, you will receive 2 records in each iteration.
If shuffle=True, records will be shuffled before batching.
for batch in dataset:
  inputs, targets = batch

In the above snippet, inputs will be a batch of records, not just one record. You may have the batch_size=1 if required.
targets

Targets corresponding to timesteps in data. It should have same length as data. targets[i] should be the target corresponding to the window that starts at index i (see example 2 below). Pass None if you don't have target data (in this case the dataset will only yield the input data)

This is a general-purpose function.
It is not deciding the Target on some logic i.e. Autoregressive approach. It expects that the targets will be provided, otherwise, it will just return the Predictors.


How to handle Tokenized text content which is given in number?

Well, in general case, machines do not understand the text, but they understand the numbers. Thus, we always tokenize the text followed by converting them to some form of numbers. We build a vocabulary of words from the given document, where each word can be assumed as a number corresponding to its index in the vocabulary. Further, this number is converted to one-hot vector representations.
Since, you already have sequence of numbers instead of words, you can make an assumption that you can convert them directly to it's one-hot vectors. Or in a advanced way, you can use a embedding layer to learn the embeddings (especially if you are using the neural networks.)


Amount of multiplications in a neural network model

Essentially you are correct, there are a lot of calculations necessary to process inputs and train neural networks.
You have some terminology a bit wrong or vague. E.g.

In a feedforward meural network each neuron of the first layer multiplied with all the neurons of the second layer.

The neurons do not multiply together directly. A common way to write the equation for a neural network layer, calling input layer values $x_i$ and first hidden layer values $a_j$, where there are N inputs might be
$$a_j = f( b_j + \sum_{i=1}^{N} W_{ij}x_{i})$$
where $f()$ is the activation function $b_j$ is the bias term, $W_{ij}$ is the weight connecting $a_j$ to $x_i$.
So if you have $M$ neurons in the hidden layer, you have $N\times M$ multiplications and $M$ separate sums/additions over $N+1$ terms, and $M$ applications of the transfer function $f()$

And in addition to a forward pass in a typical Neural network they also have a backward pass that because of my calculations doing earlier they are 1,800 derivatives (gradient) for the entire backward pass.

It doesn't work quite so directly, and there as a small factor of more calculations involved (you do not calculate each derivative with a single multiplication, often there are a few, some results are re-used, and other operations may be involved). However yes you do need to calculate a derivative for each weight and bias term, and there are roughly that number of weights in your network that require the calculations done.
Your suggested numbers are actually quite small compared to typical neural networks used for image problems. These typically perform millions of computations for a forward pass.

That's why a CPU computer takes so long to train a model because it has to do about 3,600 (1,800 + 1,800 ) mathematical operations.

Actually that is a trivial number of calculations for a modern CPU, and would be done in less than a millisecond. But multiply this out by a few factors:

You must do this for each and every example in the training data

Your example network is small, think bigger

This does not include the activation function calculations - typically slower than a multiply

Your rough estimate ignores some of the necessary operations, so as a guesstimate, multiply number of CPU-level operations by 3 or 4 from your analysis.
. . . and the number of operations does start to get to values where CPUs can take hours or days to perform training tasks in practice.


Python's fuzzywuzzy uses Levenshtein Distance which looks at character level differences.

You have to explore other approaches to text similarity. Find algorithms that nonlinearly weight n-grams differences, such as Q-gram.

python-string-similarity repo has implementations of many text similarity algorithms.


Deploying ML/Deep Learning on AWS Lambda for Long-Running Training, not just Inference

I've used ECS (Fargate) to train models, the retraining trigger could be the start of an ECS service. While ECS has a little latency, it handles well long runtimes.
You can then serve the model via a lambda.


I would like to know which metrics can be useful to do such things

Since you are clustering your data, why not use the same metric that you used during clustering for comparing your data inside your clusters? I assume this should technically give you exactly what you need.

  there is no threshold to tell whether individuals in a cluster share similar pattern or not

This hardly depends on your data and what you want to do with your similarity. If you just want to retrieve the most similar point, you can do it pretty straight forward.

If you, on the other hand, want to remove outliers using this measure, you can surely also do so, but you might as well just use an algorithm that deals with outliers in the first place.


Best Data Science & Machine Learning Conferences 2019?

Any of the Strata conferences (East/West/Europe). Should hit all of your requirements.


Actor Network Target Value in A2C Reinforcement Learning

The target will still be a form of a return estimation ($V(s_t)$, $Q(a_t,s_t)$, Advantage, n-step reward, etc). For example in your case the $Q_w$ that Critic estimated.
You will need to review a bit Policy Gradient methods in this order: PG Theorem, REINFORCE (Actor only method) then AC (Actor-Critic) and then A2C. I will give you a conceptual explanation, abstracted from the math behind. The general form of Policy Gradient is:

likelihood of action given state multiplied by a form of return: $log\pi_{\mathbf{\theta}}(a_t|s_t)\cdot R_t$

What this tells us is: maximize the likelihood of a specific action multiplied by the return. In classification we do know the correct class (action) but here this is not the case. The only learning signal comes from the reward. Therefore multiplying the likelihood of an action with a form of return for selecting that action, that will decrease/increase the probability of selecting that action again given at the current state. The PG theorem states that this is the direction to update the weights $\theta$ in order to maximize return.
In A2C, you can have various implementations: 2 separate networks or 1 network with 2 separate heads. Actor is responsible for learning the distribution of actions that maximize the return given state. Critic is responsible for estimating the return from current state (and action). Thus the loss function in A2C is usually the Policy Gradient loss plus the Mean Squared Error between expected return and observed return (plus entropy for exploration). Please refer to the original A2C paper for the equations. As you can see we have 2 main loss functions (one for each network (head) ).
The return, in any form, affects training in 2 ways:

Learning signal for the actor: Multiplies the likelihood of actions and by doing so increases/decreases the probability for selecting these actions. This will change the parameters of the policy in order for the policy to favor rewarding actions.
Learning signal for the critic: MSE target.

As you can see in a loose sense you are doing again supervised learning for both Actor and Critic but with classification loss for Actor and regression loss for Critic. The why this occurs comes from the Policy Gradient theorem that shows that in order to maximize the return following a parametrized policy we need to update the parameters towards the direction that maximize the PG loss function (for the Actor).
Please note that for various reasons using the return might not be that of a good idea. That's why you will see the PG loss in various forms ( e.g. policy likelihood multiplied by the advantage $A(t)$ or $V(s_t)-R_t$, $Q(a_t,s_t)$ etc)


Why is the cosine distance used to measure the similatiry between word embeddings?

You're asking two questions here.

Does this mean the magnitude of the vectors is irrelevant?

Yes. Cosine distance is $ D_{cos} =  \frac{A \cdot B}{\|A\|\|B\|} $, which just comes from the definition of inner product, $A \cdot B = \|A\|\|B\|\cos\theta$.

Why is the cosine distance used? Or, to think of it another way, why is the answer to (1) a desirable property to have in a distance metric?

In a word embedding, we choose a dimensionality $d$ for the embedding. This is the number of components in our embedding space. The components (or, linear combinations of the components) are meant to encode some kind of semantic meaning. Classic examples are like that the vector for "queen" plus the vector for "man" should be near the vector for "king". That sort of thing. There's a direction that roughly corresponds to "royalty" and a direction for gender.
Look at your example, where $b = 3a$, $a=[-1,2,-3], b=[-3,6,-9]$. This is a perfect illustration of why we use cosine similarity. They have very different magnitudes but point in the same direction. They have cosine distance 1, and we want that because it means that they have the same relative proportion of each component.
If we use euclidean distance, $a$ and $b$ are $\sim7.48$ units apart.  It would be easy to find another vector $c$ that is around the same distance from $a$ as $b$ is, in a completely different direction. If our space has been learned properly, $c$ should have completely different semantic meaning from $b$, but they're both the same distance from $a$. The euclidean distance doesn't measure the similarity that we want very well.


I understand your pain!

Can you not simply use tools to read the filenames and filter those as required, thereby letting your development language do all the escaping etc. for you?

In Python, for example, that'd mean using something like: os.walk() or os.listdir().


Need some advice on approach to select only the informative emojis from the data set?

One option is to reframe it as a word embedding problem. Emojis can be embedded in a vector space along with comments and hashtags. Then distance measures and clustering can be used to find the emojis that are associated with different sentiments.


Is it a best practice to exclude retweets from the data set?

No. I do not believe so and I can explain a few reasons why.

If an entity wants to create waves in twitter with false tweets retweets are probably apart of the plan.
If you want to detect tweets generated by bots looking at the statistical data on said tweets and retweets like time stamps could be relevant to detecting if the tweet is generated by a bot.
If You have a way of checking retweets by bots then removing all retweets would also remove that data.

You should remove retweets if.

The project is focused on analysis of text to determine if a tweet is bot or not.
There is no labeled human or bot retweet data.


A way to explore your dataset would be to use different techniques and see how they preform. The first approach I would consider is a tree based learner of modest complexity (or at least not boosted), like C4.5. It's possible (or perhaps likely) that won't perform as well as some more powerful approaches, but it has the added benefit of being relatively readable.

If you are familiar with a decision tree visualisation, you'll know that you can identify (leaf)nodes with high(er) impurity, and a large(r) population quite quickly. You can find an example of such a visualisation here. Those nodes can help you understand your domain, and possible how to improve performance (For instance by boosting, something LGBM already does for you).

If you want to optimise the LGBM straight away, another thing that could help here is to cluster your dataset (kMeans could be a good starting point) and then calculate performance metrics per cluster. Possibly you could also consider clustering the scored items, including the classification, or even the evaluation (TP, FP, TN, FN). If used right that could help identify clusters of false negatives for instance, and the commonality they share. Disclaimer: I have never done this myself.

PS: Lot's of times I find it helpful to use KNN (over a subsample of the feature space) and Naive Bayes. They can quickly tell you something about the global correlations. Other golden oldies are feature importance rankers and a correlation plot.


Machine learning or NLP approach to convert string about month ,year into dates

I got my answer , NLTK is good to go for this problem.
You may use sutime with python wrapper :

Python wrapper for Stanford CoreNLP's SUTime

The usual approach in NLP is to collect a dataset required for training. Process that dataset so that the words in the dataset are converted into numbers.

One simple example of converting it into numbers is to make a large dictionary of words from the dataset and use the index of each word in the dictionary as the representing number


Prohibitive size of random forest when saved to disk

The size of each tree depends very much on its depth. Thus, change the maximal depth (max_depth). Try to set it to finite number (as opposed to the default "None") and then try to reduce this number. In addition (or as alternative) try to increase min_samples_split or min_samples_split.

You can also analyze you features and keep only important ones. The simplest way would be to have a look at the clf.feature_importances_ of your forest. (In general, finding important features is an art and science on itself.) Exclude non-relevant features and rebuild the forest.


1x1 convolutions, equivalence with fully connected layer

The interpretation that the 1d convolution given in the OP can be duplicated with four separate fully-connected layers is correct (see diagram).  Also, in at least some implementations, kernel weights used during a 1x1 convolution can be made trainable the same way weights in a fully-connected layer can be made trainable.  These points made, every fully-connected layer can not be mathematically duplicated by an equivalent 1x1 convolution. This is based on the definition that 1x1 convolution performs a "column-wise dot product" such that every pixel column in a multi-layer feature map is reduced to a single number (pixel).  A fully-connected layer intermixes weights differently from the way weights are intermixed when performing a 1x1 convolution.  In summary, fully connected layers and 1x1 convolutions each have their own use cases -- some overlap among these use cases exists; however, the two are not intended to be mathematically equivalent in a general sense.


How to predict probabilities from a new data set from an already built and validated model in Python?

Welcome to Data Science on Stack Exchange,
You should apply the same steps 1-3 to the new dataset as you did to the original train and test datasets. Smote is a special case and couldn't be applied anyway as you don't have the response in the new dataset. In general, your train, test, validate, and any new datasets should go through the same data pre-processing. The new dataset will replace your Test dataset in your prediction.
There may be more steps you can add, but the current arrangement is fine.


Is there any named entity reconginition algorithm trained for the french language?

Yes, there is a french model free and ready to use via the spaCy package!

Here are the small amd medium sized models, that should be ready to go.

Here is the basic summary of the dataset, shown at the spaCy website:


Support Vector Machine analysis - Python, how do I determing whether to use linear, square, or other types of SVM models?

By Default, SVM in Sklearn uses RBF Kernel. You have to try out all the 3 kernels, with different Gamma and C.

SVM treats outliers better and add a penalty on every outlier it detects.

You should understand your data well or vizualize well to know what kernel fits well.
SVM literally adds a new dimension to the data to create a hyperplane.


Clusters: how to improve results for text classification

It can be improved in many ways. However, the best start would be if you elaborate on your pipeline together with choosen parameters, as well as your data.


A simple method is to compare similarity based on words. TF-IDF can be used considering the importance of each word. The similarity of words can be measured in various ways, such as cosine similarity. There are several recent algorithms:

W. E. Zhang, Q. Z. Sheng, J. H. Lau, and E. Abebe, “Detecting duplicate
posts in programming qa communities via latent semantics and association
rules,” in 26th International Conference on World Wide Web (WWW),
Geneva, Switzerland, 2017, pp. 1221–1229.
B. Xu, D. Ye, Z. Xing, X. Xia, G. Chen, and S. Li, “Predicting semantically linkable knowledge in developer online forums via convolutional
neural network,” in 31st International Conference on Automated Software
Engineering (ASE), 2016, pp. 51–62.
Y. Zhang, D. Lo, X. Xia, and J.-L. Sun, “Multi-factor duplicate
question detection in Stack Overflow,” Journal of Computer Science
and Technology, vol. 30, no. 5, pp. 981–997, Sep 2015.
M. Ahasanuzzaman, M. Asaduzzaman, C. K. Roy, and K. A. Schneider,
“Mining duplicate questions in Stack Overflow,” in 13th International
Conference on Mining Software Repositories (MSR), 2016, pp. 402–412.
Y. Mizobuchi and K. Takayama, “Two improvements to detect duplicates
in Stack Overflow,” in 24th International Conference on Software Analysis,
Evolution and Reengineering (SANER), 2017, pp. 563–564.


You can see the result by writing down what you get explicitly in both cases. What you want to get after summation in the update rule is:

$\sum_x \delta^{x,l} (a^{x,l-1})^T = \delta^{1,l} a^{1,l-1}+\delta^{2,l} a^{2,l-1}+...$

which is obtained by the form given in the text.

Keep in mind that Hadamard product can be done only for vectors of same dimension and result in another vector of the same dimension. I suppose you can treat $\delta$ as an $m$ dimensional vector (m being number of training examples) with vector components and take a Hadamard product of it with $a$, but you will end up with another $m$ dimensional vector. And at that point, summing over $x$ becomes meaningless. You can say "Do a Hadamard product and then sum the rows", and this will give the above result.

Edit:

I will try to further clarify the dimensions of the objects given. Let us assume that the layer l has j neurons, and the layer l-1 has k neurons. That makes $w^l$ an $(j x k)$ dimensional matrix. The update equation has to conserve the dimensionality, so the right hand side also has to be matrix of the same size.

Now, $a^{(l-1)}$ is a vector with k components, corresponding to the activations from the neurons of the layer l-1. Likewise, the error of the layer l has j components, one for each neuron.

To get a $(j x k)$ matrix, we take the outer product of these two vectors, that is why there is a transpose in the equation. And finally we sum over all training samples to finalize the calculation.


The formula is defined in section two of Gower's A general coefficient of similarity and some of its properties.

  Abstract
  A general coefficient measuring the similarity between two sampling units is defined. The matrix of similarities between all pairs of sample units is shown to be positive semidefinite (except possibly when there are missing values). This is important for the multidimensional Euclidean representation of the sample and also establishes some inequalities amongst the similarities relating three individuals. The definition is extended to cope with a hierarchy of characters.


Faster grid search with small dataset to derive best params instead of full dataset?

This idea of using a small sample to the data set to search for the hyperparameters is called multi-fidelity methods.  A good starting point is the book by Frank Hutter, Lars Kotthoff, Joaquin Vanschoren Automated machine learning: methods, systems, challenges which is open access.


How many dimensions should matrix be reduced to with PCA?

Being PCA the same as SVD, I'll explain the approach with the last one.

SVD decomposes a matrix $A$ in three others, so that $A = U * \Sigma * V^t$, where $\Sigma$ is diagonal consisting of $A$'s singular values in descending order.

The sum of all the singular values of $A$ is known as energy of matrix $A$. You want to retain most of the matrix's energy when reducing dimensions, so what you have to do is calculate how many top singular values you need to retain to preserve the most of the energy.

There are two ways of achieving this:

'Elbow method': plot the singular values and see from which on the energy isn't adding much to the total energy anymore.
calculate how many of the top singular values you need to preserve most of the energy. Usually you want to preserve $\simeq 90\%$ of the energy. So you have to find $m / \frac{\sum_{i=0}^{m} sv_i}{\sum_{i=0}^{n} sv_i} \simeq0.9$.

I recommend you expand my answer with the links below. I especially recommend the mentioned book.

SVD on Wikipedia: https://en.wikipedia.org/wiki/Singular_value_decomposition
Leskovec's book Mining of Massive Datasets, the chapter on Dimensionality reduction: http://infolab.stanford.edu/~ullman/mmds/ch11.pdf


Is the search space of Hyperparameters Continuous or Discrete?

Continuous means only you have continuous variables. It can be convex or concave. It might not even be differentiable. Gradient descent only applies to differentiable convex problems (or convex approximations of concave problems).

Example:

learning rate of SGD is continuous hyper parameter.
NN optimizer (SGD, Adam, RMSprop) is discrete parameter

From my perspective: right now, rarely we have resources to perform cross validation on a number of deep models. Most of the hyper parameters are tuned using intuition, a bit of math and tuned in isolation.

If you can afford it:

grid search to initialize solutions
if your'e unhappy with the results:

run some hill climber on space, like simulated annealing
run simple GA, there'll be problems because of mixed space (some params are continuous, some are discrete), you'll need custom operators


"Oracle" refers to something that has access to the ground truth. It has the perfect information of which in practice, we rarely have some luxury. In such paper, you can typically see that it reflects a comparison with a model with perfect information.

From Candes and Terry Tao's paper:

"Even though $n$ may be much smaller than $p$, our estimator achieves a loss within a logarithmic factor of the ideal mean squared error one would achieve with an oracle which would supply perfect information about which coordinates are nonzero, and which were above the noise level.
To see why this is true, suppose one had available an oracle letting us know in advance the location of the S nonzero entries of the parameter vector

From Hai Zhou's paper:

We show that the adaptive lasso enjoys the oracle properties; namely, it performs as well as if the true underlying model were given in advance

Notice that oracle procedure/properties is also defined in Hai Zhou's paper reflecting the ability to rediscover the ground truth.


How to apply Content-based image retrieval for scanned images?

Your problem is studied under the rubric of "image (Quality) assessment" and, more specifically, "document image assessment". Here are links to and abstracts from some relevant surveys:

Document Image Quality Assessment: A Brief Survey

  To maintain, control and enhance the quality of document images and minimize the negative impact of degrada- tions on various analysis and processing systems, it is critical to understand the types and sources of degradations and develop reliable methods for estimating the levels of degradations. This paper provides a brief survey of research on the topic of document image quality assessment. We first present a detailed analysis of the types and sources of document degradations. We then review techniques for document image degradation modeling. Finally, we discuss objective measures and subjective experiments that are used to characterize document image quality.

Subjective and Objective Quality Assessment of Image: A Survey

  With the increasing demand for image-based applications, the efficient and reliable evaluation of image quality has increased in importance. Measuring the image quality is of fundamental importance for numerous image processing applications, where the goal of image quality assessment (IQA) methods is to automatically evaluate the quality of images in agreement with human quality judgments. Numerous IQA methods have been proposed over the past years to fulfill this goal. In this paper, a survey of the quality assessment methods for conventional image signals, as well as the newly emerged ones, which includes the high dynamic range (HDR) and 3-D images, is presented. A comprehensive explanation of the subjective and objective IQA and their classification is provided. Six widely used subjective quality datasets, and performance measures are reviewed. Emphasis is given to the full-reference image quality assessment (FR-IQA) methods, and 9 often-used quality measures (including mean squared error (MSE), structural similarity index (SSIM), multi-scale structural similarity index (MS-SSIM), visual information fidelity (VIF), most apparent distortion (MAD), feature similarity measure (FSIM), feature similarity measure for color images (FSIMC), dynamic range independent measure (DRIM), and tone-mapped images quality index (TMQI)) are carefully described, and their performance and computation time on four subjective quality datasets are evaluated. Furthermore, a brief introduction to 3-D IQA is provided and the issues related to this area of research are reviewed.

No-reference image quality assessment algorithms: A survey

  valuation of noise content or distortions present in an image is same as assessing the quality of an image. Measurement of such quality index is challenging in the absence of reference image. In this paper, a survey of existing algorithms for no-reference image quality assessment is presented. This survey includes type of noise and distortions covered, techniques and parameters used by these algorithms, databases on which the algorithms are validated and benchmarking of their performance with each other and also with human visual system.

Objective image quality assessment: a survey

  Image quality assessment (IQA) is critically important for the image-processing field. IQA aims to build a computational model to predict human perceived image quality, accurately and automatically. Until now, great efforts have been employed to design IQA metrics. In this paper, we systematically and comprehensively review the fundamental, brief history, and state-of-the-art developments of IQA, with emphasis on natural image quality assessment (NIQA). First, the definition of image quality is discussed, which contains three aspects and lead to different philosophies of designing IQA metrics. Afterwards, classic NIQA metrics are presented with some further discussions. Widely used databases and the performances of classic NIQA metrics on them are also listed. We highlight the most significant works and some open issues about the developments of IQA, and provide the benchmarks for the researchers and scholars who work on IQA.

Research papers specifically on document image assessment

Document Image Quality Assessment Using Discriminative Sparse Representation

  The goal of document image quality assessment (DIQA) is to build a computational model which can predict the degree of degradation for document images. Based on the estimated quality scores, the immediate feedback can be provided by document processing and analysis systems, which helps to maintain, organize, recognize and retrieve the information from document images. Recently, the bag-of-visual-words (BoV) based approaches have gained increasing attention from researchers to fulfill the task of quality assessment, but how to use BoV to represent images more accurately is still a challenging problem. In this paper, we propose to utilize a sparse representation based method to estimate document image’s quality with respect to the OCR capability. Unlike the conventional sparse representation approaches, we introduce the target quality scores into the training phase of sparse representation. The proposed method improves the discriminability of the system and ensures the obtained codebook is more suitable for our assessment task. The experimental results on a public dataset show that the proposed method outperforms other hand-crafted and BoV based DIQA approaches.

Discrete Orthogonal Moments Based Framework for Assessing Blurriness of Camera Captured Document Images

  One of the most widely used tasks in the area of image processing is automated processing of documents, which is done using Optical Character Readers (OCR) from document images. The most common form of distortion in document images is blur which can be caused by defocus, motion, camera shake etc. In this paper we propose a no reference image sharpness measure framework using discrete orthogonal moments and image gradients for assessing quality of document images and validated the results against state of the art image sharpness measures and accuracy of three well known Optical Character Readers.

Document Image Quality Assessment Based on Texture Similarity Index

  In this paper, a full reference document image quality assessment (FR DIQA) method using texture features is proposed. Local binary patterns (LBP) as texture features are extracted at the local and global levels for each image. For each extracted LBP feature set, a similarity measure called the LBP similarity index (LBPSI) is computed. A weighting strategy is further proposed to improve the LBPSI obtained based on local LBP features. The LBPSIs computed for both local and global features are then combined to get the final LBPSI, which also provides the best performance for DIQA. To evaluate the proposed method, two different datasets were used. The first dataset is composed of document images, whereas the second one includes natural scene images. The mean human opinion scores (MHOS) were considered as ground truth for performance evaluation. The results obtained from the proposed LBPSI method indicate a significant improvement in automatically/accurately predicting image quality, especially on the document image-based dataset.

No-reference document image quality assessment based on high order image statistics

  Document image quality assessment (DIQA) aims to predict the visual quality of degraded document images. Although the definition of “visual quality” can change based on the specific applications, in this paper, we use OCR accuracy as a metric for quality and develop a novel no-reference DIQA method based on high order image statistics for OCR accuracy prediction. The proposed method consists of three steps. First, normalized local image patches are extracted with regular grid and a comprehensive document image codebook is constructed by K-means clustering. Second, local features are softly assigned to several nearest codewords, and the direct differences between high order statistics of local features and codewords are calculated as global quality aware features. Finally, support vector regression (SVR) is utilized to learn the mapping between extracted image features and OCR accuracies. Experimental results on two document image databases show that the proposed method can accurately predict OCR accuracy and outperforms previous algorithms.

A deep learning approach to document image quality assessment

  This paper proposes a deep learning approach for document image quality assessment. Given a noise corrupted document image, we estimate its quality score as a prediction of OCR accuracy. First the document image is divided into patches and non-informative patches are sifted out using Otsu’s bina- rization technique. Second, quality scores are obtained for all selected patches using a Convolutional Neural Network (CNN), and the patch scores are averaged over the image to obtain the document score. The proposed CNN contains two layers of convolution, location blind max-min pooling, and Rectified Linear Units in the fully connected layers. Exper- iments on two document quality datasets show our method achieved the state of the art performance.

Metric-based no-reference quality assessment of heterogeneous document images

  No-reference image quality assessment (NR-IQA) aims at computing an image quality score that best correlates with either human perceived image quality or an objective quality measure, without any prior knowledge of reference images. Although learning-based NR-IQA methods have achieved the best state-of-the-art results so far, those methods perform well only on the datasets on which they were trained. The datasets usually contain homogeneous documents, whereas in reality, document images come from different sources. It is unrealistic to collect training samples of images from every possible capturing device and every document type. Hence, we argue that a metric-based IQA method is more suitable for heterogeneous documents. We propose a NR-IQA method with the objective quality measure of OCR accuracy. The method combines distortion-specific quality metrics. The final quality score is calculated taking into account the proportions of, and the dependency among different distortions. Experimental results show that the method achieves competitive results with learning-based NR-IQA methods on standard datasets, and performs better on heterogeneous documents.


suggest ingredients based on recipe title

If the user inputs a title, then you could construct a system which finds the most similar titles in the corpus, and outputs the ingredients in the retrieved recipes. Some ideas below:

1) Represent the titles in a common vector space using a vocabulary-based vectorization and use Jaccard or Cosine similarity to find the most similar titles.  examples of similarity measures

2) Convert the words in the title to their respective word-embeddings using FastText, GloVe, or Word2Vec (publicly available), and then use cosine distance or  word-movers distance to find the most similar titles. example of publicly available GloVe word embeddings

3) Create sentence embeddings for the title by taking the average of the word embedding vectors or using InferSent embeddings, and then use a distance function to find the most similar titles. InferSent repository link

spaCy is a good Python package for performing these type of NLP tasks link


LSTM - divide gradients by number of timesteps IMMEDIATELY or in the end?

Answer to your question is it doesn't matter. The gradient is just a product of Jacobians (because of the chain rule), so it doesn't matter if you multiply the result or the intermediate multiplicand, these give the same result. Or, better yet, is to multiply the function you're differentiating, since $\tfrac{d(\alpha f(x))}{dx} = \alpha f'(x)$. So instead of manipulating the gradients you should just average your loss over whatever you want it to be averaged over.

Note that I didn't assume anything about the function above. It's because these are fairly general basic rules, and they apply to all kinds of functions, RNNs and LSTMs included.

That said, you might not want to actually average your loss (remembers, it's the same as averaging the gradients) over the timesteps. Average over inputs x – yeah, sure. Over timesteps? Not necessarily a good idea.

If you do average over timesteps, then you essentially decrease importance of longer sequences, and thus your model will care less of making more mistakes in them, whereas if you do not average over timesteps (or if they are all of the same length), then every timestep-loss has the same weight, and the model cares about all of them equally.


Is it updated before weight update or after weight update?

It doesn't usually matter, provided your NN framework has separate gradient calculation stage from parameter update stage.

The gradients should be collected with all weights and parameters at their current values (during back-propagation). Then the parameters are updated, using those gradients. There should be no interaction between the parameters during the update step.

However, if you have implemented the training code yourself from scratch, and have interleaved the gradient calculations and weight updates, then this only works in a pure online situation with weights updated as the gradients are calculated. In that scenario, you should probably update the weights for a layer before updating the PReLU parameter for the same layer, to prevent changes in the activation function altering the gradients during your calculations (this is not a concern with usual weight updates).

Alternatively, separate your gradient calculations from the weight update steps. This is more flexible and allows you to use other more advanced optimisations and layer designs.


Semantic segmentation with greyscale images

Use ImageNet, convert every image to grayscale and train VGG16.


Classification problem using features with unequal sizes

I am not sure if what I am gonna say is totally correct but you could try it. So instead of treating each of the pixels as part of one feature, you could append everything to make one array. That array would be of size N+3+1. So this would be the size of one row in data matrix and further you could provide weightage to the elements. Like if pixels are more important then mulitply them by some factor. One problem that I see in dealing with this might be standardization of data matrix.


Classify text labels in to a similar category

One way to do it could be with fuzzy string search. Levenshtein distance algorithm is what you may use for it.

  ... the Levenshtein distance is a string metric for measuring the difference between two sequences. Informally, the Levenshtein distance between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other.  From wikipedia

More on Levenshtein distance -

https://stackoverflow.com/a/5859823/5406448


Will cross validation performance be an accurate indication for predicting the true performance on an independent data set?

First off, a pragmatic answer: don't discount the possibility that the test set comes from a somewhat different distribution than the data set you're using for training and cross-validation.  You might think that shouldn't happen, but in practice it does seem to occur.

That said, let's go with your hypothetical and assume that the test set comes from exactly the same distribution as the rest of your data.  In that case, it is possible for cross-validation to lead you astray about which model is better, if you're using cross-validation to select hyper-parameters.

You can use cross-validation to either (a) select hyper-parameters, or (b) estimate the accuracy of your model -- but not both at the same time.

It appears you're using cross-validation to select the optimal hyper-parameters: you try many different choices for the hyper-parameters, for each choice estimate accuracy of that choice using cross-validation, and select the best choice.  When you do that, there's no guarantee that the resulting accuracy (with the best parameter) will be predictive of performance on the test set -- it might be an overestimate (due to overfitting).  If it's more of an overestimate for M1 than M2, then you might see what you saw.

If you want to both select hyper-parameters and estimate accuracy, I suggest that you have a separate held-out validation set for estimating accuracy, or use nested cross-validation.  See https://stats.stackexchange.com/q/65128/2921 and http://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html.


For numeric data(observations) in order to drop you can use missing and delete:

data olddata;
set cleandata;
if missing(coalesce(of _numeric_)) then delete;run;

Note: untested code


Data visualization with extreme far away points

If you want to see the distribution of the data that is hidden in the bottom portion, you can add a histogram or probability plot, or even a violin plot. Each will show the distribution of the data more clearly than this boxplot does, and you can still see the true value directly. You can also add some jitter to the boxplot to see more of the overlapping points displayed.
Jitter:

Violin:

Probability Plot with Boxplot:


This is indeed a simple problem if tried to be tackled using semantic segmentation. Semantic segmentation itself is a computer vision problem that could be understood as an extension of object detection and could be understood as follows:

Using semantic segmentation done using a network called as UNET, the model could be trained for the required image and then it can be extended to find the boundary of the required object and finally extract it. UNET architecture could be understood using the following diagram:

UNET's are generally used to create mask that could be XOR with the actual image and the background of the image could be subtracted easily.

Completely explaining image segmentation using UNET or any other technique falls beyond the limit of the answer and thus a better explanation could be found in this article

If you want the practical implementation/codes of the same, they could be found here on kaggle kernels of the following contests:

TGS Salt Identification Challenge
Carvana Image Masking Challenge


If I understand your question correctly, then this is not possible for a "generic" model. For a generic model, you can only use generic methods. One of the most generic would be bootstrap sampling. In this scenario, you take your model and train it on different bootstrap samples and derive confidence intervals from the distribution of these predictions.

However, it sounds like you are looking for a method that does not require training. In this case, you would need to use models from which confidence intervals can be derived directly based on the model assumptions, such as linear regression, Gaussian processes or Bayesian models.


Implement the following loss function without interrupting the gradient chain registered by the gradient tape

Your intuition about counting the 2 is pretty good. For this type of problem, you can use the dice loss function wich use your idea but sligthly differently.

the values of predicted and ground truth components (each pixels) are either 0 or 1, representing whether the pixel bayond to the label (value of 1) or not (value of 0). Therefore, the denominator is the sum of total labeled pixels of both prediction and ground truth, and the numerator is the sum of correctly predicted boundary pixels because the sum increments only when predicted and ground truth match (both of value 1).

as cited here:

For example, if two sets A and B overlap perfectly, DSC gets its maximum value to 1. Otherwise, DSC starts to decrease, getting to its minimum value to 0 if the two sets don ‘t overlap at all. Therefore, the range of DSC is between 0 and 1, the larger the better. Thus we can use 1-DSC as Dice loss to maximize the overlap between two sets.

The dice loss is commonly used in segmentation task because it is resilien to class inbalance (eg too much background)

you can implement it in tensorflow keras like this;

def dice_coef(y_true, y_pred):
    y_true_f = K.flatten(y_true)
    y_pred_f = K.flatten(y_pred)
    intersection = K.sum(y_true_f * y_pred_f)
    return (2.0 * intersection + 1.0) / (K.sum(y_true_f) + K.sum(y_pred_f) + 1.0)

def dice_coef_loss(y_true, y_pred):
    return 1-dice_coef(y_true, y_pred)

(in the implementation, we use a smoothing termes (here it's 1, the lower the value is the better) The smoothing therme is here to avoid dividing by 0 if the input as no label in it)


In general, you can use brute force or a smart feature matcher implemented in openCV.
Another approach is seeing the task as image registration based on extracted features.

The question may be what is the relation of HoG and SIFT if one image has only HoG and other SIFT or both images have detected both features HoG and SIFT.


You could consider finding watermark an object detection problem, not an image classification problem. Faster R-CNN and "You only look once" (YOLO) are both general object detection architectures.


Alterable Similarity Join for Time Series

Dynamic Time Warping might be what you're looking for: it measures similarity between two time series based on the optimal alignment between the two sequences. For example point $i$ in sequence 1 might be better aligned with point $i+3$ in sequence 2 based on the evolution of the sequences (as opposed to Euclidean distance which would always compare $i$ in seq. 1 with $i$ in seq. 2). The alignment method is inspired by the Levenshtein edit distance method, which can be implemented efficiently with dynamic programming.
There are many software implementations available.


Calculating RMSE AND R-squared from the confusion matrix

The confusion matrix suggests that you are performing classification rather than regression. RMSE and R-square are measures associated with continuous variables; For categorical variables, I'd suggest using Accuracy / Recall / Precision / F1 score to measure the performance of the model.

https://www.quora.com/How-is-root-mean-square-error-RMSE-and-classification-related


How to get feature importance from RandomForest using scikit-multilearn library?

First, to directly answer your question, the easiest way to get Feature Importance using scikit learn is this, where model is the variable holding your classifier.

print(model.feature_importances_)

However, this method only exists on some of the Ensemble models, namely:

AdaBoostClassifier
AdaBoostRegressor
ExtraTreesClassifier
ExtraTreesRegressor
GradientBoostingClassifer
GradientBoostingRegressor
RandomForestClassifier
RandomForestRegressor
RandomTreesEmbedding

If you're wondering why, there's a fantastic free book online all about Interpretable Machine Learning. Here's an excerpt:

  The easiest way to achieve interpretability is to use only a subset of
  algorithms that create interpretable models. Linear regression,
  logistic regression and the decision tree are commonly used
  interpretable models.

Once you've chosen the right model, beware of using feature importance! See here, it ranks random data very highly. The article proposes using Permutation Importance instead, as well as Drop-Column Importance.

They created a library called rfpimp for doing this, but here's a tutorial from scikit themselves on how to do both of those with just scikit-learn. I've pasted the permutation importance example from that tutorial below:

from sklearn.inspection import permutation_importance

result = permutation_importance(rf, X_test, y_test, n_repeats=10,
                                random_state=42, n_jobs=2)
sorted_idx = result.importances_mean.argsort()

fig, ax = plt.subplots()
ax.boxplot(result.importances[sorted_idx].T,
           vert=False, labels=X_test.columns[sorted_idx])
ax.set_title("Permutation Importances (test set)")
fig.tight_layout()
plt.show()


Did anybody ever use mean pooling and publish it?

I just found an answer myself:

  Boureau, Y-Lan, Jean Ponce, and Yann LeCun. "A theoretical analysis of feature pooling in visual recognition." In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 111-118. 2010.

See also: shortscience summary


Single machine learning algorithm for multiple classes of data : One hot encoder

I know this is late, but the question got bumped by Community, so...

Your discussion and data suggest that what you want to produce is the model
$$y = (3x_1+1)x_2,$$
but that is not a linear model and so linear regression will not find it.  You can try any number of other nonlinear models, the best type depending on your real use-case.  For instance,

If you really just want a linear model for each value of $x_1$, then it's probably best just to split the data along $x_1$ as you started with.  You might need to examine your motivation behind wanting just "one machine learning model".
You could introduce a new feature, equal to $x_1 x_2$.  This loses some information from option (1) when $x_1$ has more than two values, but it might be suitable.
You could use a tree-based model with linear regression done at each leaf, as in  https://stats.stackexchange.com/questions/78563/regression-tree-algorithm-with-linear-regression-models-in-each-leaf .  If the model fitting procedure decides to split only on $x_1$ and regress only on $x_2$, it mostly recovers option (1), though there's no reason to necessarily expect that.


Take a look at association rule learning (https://en.wikipedia.org/wiki/Association_rule_learning). A really common algorithm is the Apriori agorithm. You could use the package apyori, it works great: https://pypi.python.org/pypi/apyori/1.1.1


Finding correlation between MNIST digits

You can’t. Correlation is a measure of a variable changes as another variable changes. One goes up by a certain amount, the other usually goes up too: positive correlation. And so on.

What you can calculate is how similar or how different images of 1s are, compared to images of 7s. You could average all images of each, by summing the images to get one images with very high pixel values, and then dividing all pixel values by the number of images that you summed.

Then you can represent the average 1 and the average 7 as a long vector of 784 pixels and calculate the distance between these two as a measure of their similarity.


You need to list the set of nominal values for the class in the arff file, comma-separated and enclosed in curly brackets {}. Here's an example:

@RELATION iris

@ATTRIBUTE sepallength  REAL
@ATTRIBUTE sepalwidth   REAL
@ATTRIBUTE petallength  REAL
@ATTRIBUTE petalwidth   REAL
@ATTRIBUTE class    {Iris-setosa,Iris-versicolor,Iris-virginica}

In your case, Iris-setosa,Iris-versicolor,Iris-virginica would be replaced with happy_pleased,...

If this still doesn't work, you can tell WEKA which attribute is the class in the explorer by choosing the correct attribute in the drop-down list above the histogram in the bottom right corner.


This can be thought of as an active learning problem. You want feedback from the learning process to influence what kind of input examples you next train on.  Active learning is still a difficult problem in most settings.

As @cohoz points out, hill climbing is an intuitive option in this case, but only if you always want to increase or decrease your measure of frustration. If instead the goal is to explore frustration as a function of as much of the music parameter feature space as possible, you will want to read about active learning and uncertainty.


This is rather common.  The algorithm for KernelRidge requires a SVD to be performed.  Sadly, the SVD cannot handle a sparse matrix so the _pre_compute_svd function in sklearn just converts the matrix into a dense matrix and moves on.  This tends to blow up memory rather quickly.

You have a couple of choices.  Rewrite the method to handle sparse matrices or just use a different method.  SVR would be the most similiar alternative.


Taking Neural Network's false positives as the recommendation system result?

1) Is regression, not classification.

You should try to think of it from a decision tree perspective instead of NN. Your attributes for a customer are all products of this customer and all additional information about that customar.

Since the task is regression, all leafs of the decision tree will have the label 0 (if the decision path leads to the statement that the customer doesn't have the product) or the label 1 if the decision path leads to the statement that the customer has the product.

When building the decision tree, we start at the root where we have to decide which attribute we will use as the first node from where all the customers will take the corresponding branch depending on their value of the chosen attribute. This attribute is choosen by using entropy to calculate the attribute with the highest information-gain value. This simply means that we choose the attribute, that divides the customers into sets where each set has ideally the same amount of customers with 0 and 1.

This is repeated for each new node that branch from the parent node as long as there is an attribute left with information-gain above a threshold value or when the tree becomes to deep (since it is NP-complete).
Now we count at each leaf the amount of zeros and ones and label the leaf as max(0,1).
The conclusion of this: if a customer landed at a 1-leaf although he was 0 means, the highest information-gain attributes have the same values for this customer as they much more often do for customers who are 1.


logistic regression : highly sensitive model

Actually, what is happening is natural. There is a trade-off between sensitivity and specificity. If you want to increase the specificity, you should increase the threshold of your decision function but note that it comes at a price and the price is reducing the sensitivity.


XGBOOST - different result between train_test_split and manually splitting

All looks correct, but you have to get to know about some details.

train_test_split splits arrays or matrices into a random train and test subsets.
If you not specifying random_state, you will get a different result, it means that
every time when you run your code train and test datasets would have different values each time.

If you fixed specific value like random_state = 42 then yours data in test and train set conserve the same values.

So in the first way splitting,  model is learning on the specific(always the same) chunk of data, which could give a good results, in second way every time model is learning on different chunks of data, so results could be worse(by the way, its looks like overfitting on chunk(in first way of split).

In general no matter what way you choose, but you have to use cross-validation to create a good performing model.


How to detect the begin word and end word in a sentence with machine learning

For this kind of problem, you can study some of the latest research areas of reading comprehension in which people are trying to predict the starting index of the answer & with the help that index they are predicting ending index of the answer.

Following screenshot took from BiDaF research paper
Refer to BiDaF for more details


Feature selection before or after applying filter in Time-series forecasting

I find your question confusing (this might be my fault). Let's see if I understand you correctly.

Feature selection is not applied before or after the filter. You choose to include a feature, or not. There are many ways to make a selection of features. If you use decision trees (or algorithms based on decision trees like XGBoost or random forest), then you can calculate the feature importance and use that to select features.

Your set-up to predict ozone concentration contains feature processing (the filter) and then an algorithm (boosted trees) that makes a prediction. You use a set of features, run them through a filter, you train the trees. Now you can use the filter + the trained trees to get your prediction (ozone concentration). You can also look at the feature importance of all the features that you used, make a selection, and retrain with a reduced set of features.

What makes no sense to me is to use a different set-up without the filter, to train the trees on the raw unfiltered features, and then look at the feature importance, because you have trained your trees on input data that the trees will never see in the set-up you're actually trying to use.

So I think the answer to your question is "after". :-)


Is it possible to extract specific words from a sentence in Hindi/Marathi?

Based on your comment I understand that you want to identify a particular syntactic category of words, for instance nouns. The task which identifies these categories is called Part Of Speech (POS) tagging.
I'm not familiar with any specific resource for Hindi or Marathi but there are probably some software that you can use. From a quick search I found a few links below:

General resources for Indian languages
A POS tagger for Hindi
Another more recent POS tagger for Hindi


Confusion about neural network architecture for the actor critic reinforcement learning algorithm

Both architectures work. It is probably more common to use two separate networks for simpler problems, and a combined network for a more complex problem, such as one involving machine vision.

In general, you need the two functions - policy and value - to be separate. There is no reason to expect them to have too much in common, in terms of the overall mapping from state to their output. However, if the state requires a lot of non-linear interpretation to get meaningful features, such as image, audio or video input, then it could be an overall benefit if the two functions share the lower-level feature representations. Not only will this encourage better generic learning of the low-level features (because they are effectively being trained with twice the data per time step compared to if they were separate), but the calculations should be faster too.

For a similar reason, if the agent works from natural images, you could use either pre-processed features from upper layers of some model trained on ImageNet, or you could start the network initialised with the first layers of such a model. This might apply for other RL agents, such as DQN too - if this works, then clearly sharing that part of the network when you have more than one function to work on could work also.

If your state data is simpler, e.g. a few positions and velocities, or one/both policy and value have a simple relationship to state, then a joint network may be less useful.

When the policy and value functions are in a shared network, they may still both have more than one dedicated layer, as it is expected that there is not any simple linear relationship between the two functions (a NN could learn this by itself, but the assumption is good enough that it may as well be hard-coded by splitting the network into two branches with maybe a couple of hidden layers each before output).


Size of folds in k-fold cross-validation

I think cross-validation is almost always superior to a simple train-test split. The only problem is computation time (for a 5-fold cross validation you have to train your model 5 times).

If your problem allows it, you should always use cross-validation.

The only reason you shouldn't use cross-validation is if your model takes too long to train (e.g. state-of-the-art image recognition networks require weeks of training on GPU clusters).


What to optimize for when asked to find the most important features

Ideally you should select features in the same way you select the best model hyperparameters, with a validation set.

You are interested in how the features work on unseen data, not your training data.

Depending on your feature selection algorithm, feature search will be intractable with larger data.  Naive feature selection is O(2^n) AND some model hyperparameters could be dependent on your choice of features.


Instead of using embeddings and doc2vec, maybe you should start by an easier and more straightforward information retrieval approach. Start by building a bag-of-words representation of your long document. Given its vocabulary $V$ of size $|V|$, you represent it as well as the short texts for the second document in a space $\mathbb{R}^{|V|}$. This will give a vector for the first document and several vectors, one for each of the lines, for the second document.

Proceed by calculating the similarity (euclidean distance) of the vectors of the lines of the second document with the vector of the first document.

There are many ways in this line of though that would enable you to use word embeddings if you want, but I would suggest starting with the simplest approach first.


Can you use zero padding? It's a pre-processing technique for CNNs, it consists in creating a frame of zeros around the image, so that all input image will have the same size.

The CNN will then learn autonomously to ignore the zeros. It's a common technique, Keras layers already have padding built-in arguments.


What would be the best way to impute data?

You can treat the missing feature as the target variable of a sub-problem and create a classifier (e.g., a linear model, SVM, etc) for it.


Input standartization for Deep Learning - Proper Scaling

First, it is important to keep in mind that neural networks (like many other machine learning algorithms) work over the domain of the real numbers, i.e. they assume for granted the properties of this space. Some properties that are relevant to this context:

additive identity is 0 and multiplicative identity is 1
there are inherent (intuitive to us) total order and a standard distance

A side note on the implications of this: if one is planning to use neural networks for categorical or ordinal data, one needs to consider what mapping into the space of real numbers is reasonable and whether the standard mapping (e.g. to natural numbers or via string similarity) makes sense in the context of the problem.

From theoretic viewpoint scaling is a homogeneous transformation, i.e. it preserves the properties of the space up to a scalar, thus it doesn't matter what you choose. Since 1 is the multiplicative identity (nothing changes when you multiply by one) it makes sense to use it as a scale, because it simplifies calculations.

Another side note: This is actually valid also in a more general context. For example the machine representation of floating point numbers is via the so-called scientific notation. From this perspective, 0.01, 0.1, 1, 10, 100,... all have the same significand and only the exponent changes. Thus, the difference between the examples you mention is actually quite small.


Assuming you are talking about Basemap Matplotlib Toolkit, you have to manually download the Windows binary installers.


the feasibility of image processing techniques for physics based images

I think your intuition is correct, that it doesn't make sense to "clean" the matrices, if they are generated perfectly.  I agree with @SyedAliHamza that noise should be eliminated as much as possible in the simulation.

On the other hand, depending on what analysis you are trying to do, you may want your model to generalize better. In that case, it would be good to apply some data augmentation, meaning adding some noise or other types of augmentation to your matrices.


What should be the value of batch_size in fit() method when using sgd (Stochastic Gradient Descent) as the optimizer?

First, using the appropriate terminology you can say batch Stochastic Gradient Descent and batch Gradient descent are in the extreme ends, where Stochastic Gradient Descent is training with $batch size = 1$ and for batch gradient descent with $batch size=n$ where $n$ denotes the number of data points.

In the appropriate terminology, what we are often using (and similar to your example as well) is called mini-batch gradient descent. Note that the term mini here does not mean it is necessarily very small like 4, 32 or 64 but instead can be anything bigger than $1$ but smaller than $n$. In practice, people use the term mini-batch gradient descent and stochastic gradient descent interchangeably. This is because in practice they behave similarly.

I personally do not think that such practice (using SGD and minibatch SGD interchangeably) is bad, because I don't think that it differs a lot such that it requires a specific new term.


How to implement random cropping during training?

For segmentation, random cropping cannot be included inside the model definition, since random crop of same state also has to applied on mask. I suggest before each epoch, apply the 'random_crop' function on the entire dataset and shuffle(very important) the dataset.


PMML XML Model == same as config file or implementation?

Essentially, the PMML is just a set of formulas (and the corresponding coefficient values).

The story goes something like this:

You develop a model capable of taking some inputs and outputting certain coefficients that correspond to a certain formula.
Then, these ouputs are turned into a PMML file (an XML-based encoding of your model's outputs).
We can now say, "please evaluate this formula, using these coefficients. As input for the independent variable(s), please use these attributes from my dataset here."

Take a very simple linear regression, for example.

The PMML consumer application parses your PMML file, and finds

model =
coefficient = <5.46>
X = < attribute:  colour of the leaf >

It now goes ahead and scores the formula for the reggression

Note: there’s got a to a file somewhere in your application which contains a column called “colour of the leaf”

And, there you have it.


I'm planning to record video now and then analyze it in the future once AI programs are more accurate, what resolution and FPS should I record at?

This question is definitely opinion-based and so is my answer, but let me try to clarify a few things:

Don't count blindly on AI/ML to become more accurate in general, and even less for a particular project. Imho for your project to be future-proof, you need to test the current technology with a sample of your data (or some similar data) and make sure that the results are at least "encouraging", i.e. decent enough. If the result are terrible or cannot be obtained at all, then there's probably a problem with the whole design and it's unlikely that future technology will solve this.
It's not always by providing more detailed information to ML that it gives better results, it strongly depends on the task. This is why I recommend testing first (see point 1).
Technically it's always possible to downsize resolution but not the opposite, so from this point of view it's clear that the higher the resolution the better. A simple estimation says that with your characteristics a single daily video would require around 2GB, so 2 years would require around 1.5TB.


Yes, these features should not be scaled together. Normally ALL features are scaled individually, so grouped scaling is the exception not the norm. From the Wikipedia on feature scaling:

  Since the range of values of raw data varies widely, in some machine
  learning algorithms, objective functions will not work properly
  without normalization. For example, the majority of classifiers
  calculate the distance between two points by the Euclidean distance.
  If one of the features has a broad range of values, the distance will
  be governed by this particular feature. Therefore, the range of all
  features should be normalized so that each feature contributes
  approximately proportionately to the final distance.

You can scale some features together if you know that they exist on the same scale, i.e. that "Sales January" is scaled together with "Sales February". But you should not scale things together that exist on different scales, i.e. "Sales January" and "Discount Ratio" should be scaled separately.


Which part should be frozen during transfer learning?

Ideally I follow the below rules for performing transfer learning, I could do successful object detection using Yolo model by traning only last layer and was able to do leaf classification by training good number of layers.


What is the depth of an image in Convolutional Neural Network?

It means that the number of filters (a.k.a. kernels, or  feature detector) in the previous convolutional layer is 96. You may want to watch the video of the lecture, and in particular this slide, which mentions that a filter is applied to the full depth of your previous layer:


Optimum minimum number of instances in weka's j48

If there were an optimum in terms of number of instances or attributes, then that would be set in the program.  No, such a parameter generally has an optimum that depends on the data itself, and needs to be tuned to get optimal performance.


Is there a name for this form of sampling?

It is called resampling and is quite famous for dealing with imbalanced datasets. See, for example, https://www.kaggle.com/rafjaa/resampling-strategies-for-imbalanced-datasets for implementation.

Congrats on good intuition!


Where the game theory is applied when it comes to reinforcement learning?

It is not used directly in this case, and AlphaStar makes no breakthroughs in game theory. The blog's wording here is not super precise.

The point of the quote was to explain the extra challenge, which occurs in many games where opponents can react to each other's choices and there is often a counter-strategy to any given policy. Rock-paper-scissors is the simplest game that has this challenge, but it is common in many strategy games, as the game designers typically don't want a single best strategy to dominate the game, often going to some lengths to balance options in the game so that more of their game content is used and to keep a level of uncertainty and excitement in the game-playing community.

The actual breakthroughs in regards to the quote in your question, are in finding ways to perform the kinds of long-term exploration that allow for different high-level strategies. Many RL algorithms perform relatively local exploration which would be too weak to keep track of entirely different strategies and decide when to use them.

The way that Deep Mind team approached this is explained in their blog:

  To encourage diversity in the league, each agent has its own learning objective: for example, which competitors should this agent aim to beat, and any additional internal motivations that bias how the agent plays. One agent may have an objective to beat one specific competitor, while another agent may have to beat a whole distribution of competitors [ . . . ]

So Deep Mind have not resolved any of that at a theoretical level, and have not used game theory in any direct sense. However, they have identified the kind of game theory scenario that applies, and have used that in the design, making steps in an engineering sense towards practical solutions.

Other solutions in RL might also apply, such as hierarchical RL for capturing high-level actions as strategies to inform lower-level decisions, or using slow changing noise functions to drive exploration (as opposed to something which changes faster such as epsilon-greedy).

In general, game theory is related to reinforcement learning, in that both construct a formal view of optimising utility:

Game theory is useful for analysing multi-agent scenarios, but generally analyses optimal policies for relatively simple single-step or repeated games.
Reinforcement learning is well-described for single agents, and deals well with sequential decision making, but does not have much quite as much material for dealing with competitive and co-operative multi-agent environments - typically treating other agents as "part of the environment".

There is enough cross-over between the two theories that they can be used to inform each other in an intuitive way, as Deep Mind have done here.

In more tractable game environments, game theory is able to determine stable and effective policies - for instance in rock-paper-scissors, the Nash equilibrium policy (one which players will be punished for moving away from) is randomly selecting each action with 1/3 probability. Note this is not necessarily the optimal policy - that depends on the opponent's behaviour - but it is an expected stable outcome for two rational and capable opponents to arrive at.

If you develop a rock-player-scissor learning bot using RL, and it learns this strategy through self play, then you can be relatively happy that your learning algorithm worked. That would be one way of using RL and game theory together.

Deep Mind don't know the Nash equilibrium of Star Craft strategies, and in fact the strategies are only loosely defined in terms of low-level actions, so it is not clear whether it is possible. The analysis of strategies supplied in the blog (e.g. a "rushing" strategy) are based on observations of the game and adding a human narrative to help understand what is going on. In practice, it is the sampling of opponents each preferring a different strategy or set a particular goal in the game, that trains a single neural-network based bot that has experience of countering multiple strategies and can express actions that optimally beat any strategy that matches patterns it has learned in self-play and observes an opponent using.


Visualizing N-way frequency table as a Decision Tree in R

You could try

library(data.tree)
test$pathString <- with(test, paste("lm", Type, Treatment, round(res, 2), sep="/"))
(tree <- as.Node(test))
#             levelName
# 1  lm
# 2   ¦--Quebec
# 3   ¦   ¦--nonchilled
# 4   ¦   ¦   °--36.97
# 5   ¦   °--chilled
# 6   ¦       °--30.11
# 7   °--Mississippi
# 8       ¦--nonchilled
# 9       ¦   °--24.31
# 10      °--chilled
# 11          °--17.45
plot(tree)

The plot uses the DiagrammeR package.


In convolution layers, weights are the filter values.
Weights of a CNN network is determined by the number and size of filters. Weights doesn't depend on shape of input image.
Filters of pre-trained network are capable of detecting different features in an image. Consider sobel filter which detects edges in an image. A 3x3 sobel filter can be
used to detect edges in an image of any size. Similarly, those filters can be reused instead of again learning from scratch.

That's why, weights of CNN network can be transferred to another CNN network with different input shape.
However, number of layers, no. of filters, filter sizes and number of image channels must be same.

For transfer learning, best practices would be to use pre-trained model for similar task and don't change the input shape to very small or large.

On the other hand, weights of Fully Connected(Dense) layer can't be transferred. Because, those weights depend on image size. Dense layers at the end of CNN network should be cut off and recreated to adopt to change in image size.
It would be more clear if you get to know how the trainable parameters are calculated in CNN and Dense layers. It would be very long if I try to explain in this answer. You can google search on that.


How to deal with time series which change in seasonality or other patterns?

After reading your question, I became curious about the topic of time series clustering and dynamic time warping (DTW). So, I have performed a limited search and came up with basic understanding (for me) and the following set of IMHO relevant references (for you). I hope that you'll find this useful, but keep in mind that I have intentionally skipped research papers, as I was more interested in practical aspects of the topic.
Resources:

UCR Time Series Classification/Clustering: main page, software page and corresponding paper
Time Series Classification and Clustering with Python: a blog post and an ipython notebook
Capital Bikeshare: Time Series Clustering: another blog post
Dynamic Time Warping using rpy and Python: another blog post
Mining Time-series with Trillions of Points: Dynamic Time Warping at Scale: another blog post
Time Series Analysis and Mining in R (to add R to the mix): yet another blog post
And, finally, two tools implementing/supporting DTW, to top it off: R package and Python module


LabelEncoder converts strings to integers, but you have integers already. Thus, LabelEncoder will not help you anyway.

Wenn you are using your column with integers as it is, sklearn treats it as numbers. This means, for example, that distance between 1 and 2 is 1, distance between 1 and 4 is 3. Can you say the same about your activities (if you know the meaning of the integers)? What is the pairwise distances between, for example, "exercise", "work", "rest", "leasure"?

If you think, that the pairwise distance between any pair of activities is 1, because those are just different activities, then OneHotEncoder is your choice.


Which of the following linear regression model is better?

Which one of the two models is better?

It depends what you care about.

  Why?

If you just want the highest $R^2$, $R_{adj.}$ or $R_{pred}$, obviously Model 1 is better. In fact, in your example, all performance metrics would tell you to go for Model 1! But there are other aspects to picking a model...

Many people care about the density of explanability (I made that term up); think of it as the explanatory power normalised by number of degrees of freedom (also referred to as the elegance of the model). If that was your preference, then Model 2 look more attractive, as its metrics drop by only 1% and 1% again, compared to Model 1's 10% and 10% drop.

Some relevant background

There is the term Minimum Description Length, which formalises Ockham's Razor, stating:

  the best hypothesis (a model and its parameters) for a given set of data is the one that leads to the best compression of the data.

This can help decide which of two models is best, by assessing how dense the explanability of each model. In simple terms: assume models A and B both produce 80% accuracy, but A uses an extra parameter, then B is the more elegant model as it uses its input more efficiently. In other words, its compression of the information is higher.

Here is a really nice blog post (published today!) that explains how Ockham's Razor, Bayes' and Shannon Entropy all come together to equally define how thie density of information in a model is a really important criterion. Spoiler: all three boil down to the same guiding principal.

Another way people try to decide which regression model is best, is to use another metric such as the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC), both of which try to find a compromise between the performance of a model and the number of degrees of freedom (i.e. number of parameters) that were used. We pick the model with the lowest score. For example, the AIC is defined as:

$$
AIC = 2k-2 ln({\hat {L}}),
$$

where $k$ is the number of parameters and $\hat{L}$ is the performance metric (here the Maximum Likelihood is implied). So in plain English, the AIC gets larger and alrger as we include more parameters and out model becomes more complex. However, if the model fits the data more accuractely with each parameter, that is subtracted and so reduces the AIC value. The BIC essentially works the same way. I think it is intuitive what we are trying to do here.

Summary

Coming back to my first answer: "It depends what you care about" - it hopefully now clear that the selection is subjective. If you can live with high model complexity, you might just go for the best $R_{pred.}$ and so Model 1. On the other hand, if you enjoy an elegant model with high information density, the deltas of 1% between your three metrics will lead you to select Model 2.

EDIT

A term I was searching for came back to me: parsimony. A parsimonious model is one that reaches accuracy expectation and does so with a few parameters as possible. It is all about maximum information density and explanatory efficiency. Check out this thread that is relevant to your questions.


It depends how much noise:

If it's only a little noise, say for instance 2% of the target values are off by a small value, then you can safely ignore it since the regression method will rely on the most frequent patterns anyway.
If it's a lot of noise, like 50% of the target values are totally random, then unless you can detect and remove the noisy instances you can forget it: the dataset is useless.

In general ML algorithms are based on statistical principles, to some extent their job is to avoid the noise and focus on the regular patterns. But there are two things to pay attention to:

Is the noise truly random, or does it introduce some biases in the data? The latter is a much more serious issue.
Noisy data is even more likely to cause overfitting, so extra precaution should be taken against it: depending on the data, it might be necessary to reduce the number of features and/or the complexity of the model.


How to balance class weights correct for a CNN in Keras, given an unbalanced data set?

If the "cost" for experimenting is not really that big I suggest you take the time to experiment and take this as a learning opportunity and just try if it could actually work.

There are many approaches to address class imbalance and setting class weight is one of them and the easiest to implement.

Change loss function (for example to focal loss for binary classification with extreme imbalance)
Oversampling and Undersampling
Setting class weights
Use specific algorithm that are build to address this problem e.g. siamese network which is very useful when you only say have very few training sample of object of interest.
etc.

Specifically for your case, I can tell you the specific case that it could fail base on my experience. So basically this very likely fail when you have extreme class imbalance say like 1% positive and 99% negative. How this could fail is simply because using class weighting in this case will put very high value on the positive sample and if your model fails to detect this, the penalty is very high and hence lead to unstable training. To top it off consider a hypothetical situation your model predict the positive class correctly on epoch 10 and then it fails on epoch 11. For this case you might get a loss for example 1.3 for epoch 10 but then on epoch 11 your loss could go to say like 37.7 simply because it fail to detect said sample. This could also affect any callbacks that utilize this loss.

In summary if the situation could be as I described then don't use this otherwise just play around and find out what's best for you.


Keras decision threshold for Multiple Label prediction

You can look at your multi class classification as 28 separate binary classifiers (1 for each output feature).

The output of a binary classification is the probability of a sample belonging to a class, so the threshold per each feature is 0.5.


Undersampling for credit card fraud detection before or after Train/Test Split

Do the undersampling after the train/test split and only in the train split: you want to somehow "weigth" your learning algorithm in order to prevent it to be biased towards the majority class, indeed, other techniques can be applied, and always on the train split, what eventually is the data for the learning algorithm. But... your test set should represent the real data distribution. In this case, you will asses if whatever technique you have applied to prevent a biased algorithm towards the majority class has not eventually affected its performance on it.
Some other approaches for this problem (fraud vs non-fraud):

Anomally detection
One-class classifies: One Class SVM


Logistic Regression or regression SVM for probability of outcome

Applying regression methods would render useless, as we usually feed binary (or multiple) categories as target variable in classification problems. However, if you have somehow got probability values for each sample, you can perform regression as well - I can't think of such a rare case, though.

If you are looking for prediction probabilities, Logistic Regression is the best call. Having some experience with sklearn, I can tell that SVM, by default, does not compute probabilities. If you wish to obtain those prediction probabilities, SVM uses another set of algorithms, which cause serious slowdown in process and extra computational cost.


Difference between zero-padding and character-padding in Recurrent Neural Networks

If implemented properly, there should be no difference. The very first thing that happens with the indices is corresponding embeddings are loaded. From this perspective, there is no difference between having the pad embedding at the 0th or at the 1000th position.
When you use padding, you should always do masking on the output and other places where it is relevant (i.e., when computing the attention distribution in attention), which ensures no gradient gets propagated from the "non-existing" padding positions.


How do I use TF*IDF scores for my machine learning model?

The numbers on the left are also important, they are basically the indexes in the following format

(document_number, token_number)

TF-IDF is computed for all the unique tokens in all the documents.

Let me know if you have any further doubt.

Vote me if i was able to help ;)


Probabilities of a Poisson distribution not making sense

This seems quite right to me. A poisson(1000) law has a standard deviation of $\sqrt{1000}$ which is something around $31.6$.

At such values, the Poisson law pretty much behaves like a normal distribution, so the probability of getting a value greater than 1100 would be very close to 0 (around 0.1%).

Perhaps what you need is not a Poisson law, but a truncated normal distribution?


Leave - one - out - Cross Validation KNN R

To learn how its done, simplest way is to give a try on sklearn and then try to implement the same by yourself-

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.LeaveOneOut.html

and a very basic level explanation  here-
https://medium.com/@svanillasun/how-to-deal-with-cross-validation-based-on-knn-algorithm-compute-auc-based-on-naive-bayes-ff4b8284cff4


How to validate recommender model in healthcare?

You should still be able to use a validation set to evaluate the model, whether or not you pursue an experimental approach. (Specific features of your model and investigations may tweak these, but this is based on what's already been posted alone).

There is nothing wrong with A/B group assignment and testing in a medical context, with a few caveats (this list is not exhaustive):

The relevant clinical/medical knowledge must be in a state of
equipoise (it's not already clear that one approach is better than
another, or which is better is genuinely not known).
Individuals should be aware that they are participating in a study, and that they are being routed to
group A or B, and have the option to decline their assignment (or,
conversely, they have been made aware of the experimental assignment
and have consented to participate in advance).
An institutional review board should evaluate your proposed
experiment and signed off on it. This, of course, presupposes that
you have access to such a board composed of members able to make
those assessments.

Those can be a tall order, but you don't necessarily have to perform a prospective, double-blind experimental study in order to glean some information. A retrospective study could provide some insight as well, and your process for the validation set would be something like:

Prepare your recommender model
Feed your data through the model, without looking at actual outcomes
Match your model output to actual outcomes to see whether or not
people followed the recommendation (whether they ever saw that
recommendation or not)
Compare the results of people that ended up going with each
recommended approach (A vs. B), as well as those who "followed" the
recommendations or not (Recommended-A-did-A vs. Recommended-A-did-B,
etc.)

Retrospective studies are generally not as good as well-designed, well-executed prospective experimental studies, but they can still provide a lot of information. In situations where prospective experimentation is impossible or undesirable, the information a retrospective study provides may be the very best you can actually get.


Predicting a numeric variable from many numeric variables, how to choose a proper structure?

How much data do you have? this is a regression problem that doesn't require using neural networks, you can solve this by using much simpler algorithms ( SVM, Linear Regression , Tree-based regressors ) ( I'm assuming you don't have millions of rows of data ). As in for the inputs, you can either use the numeric variables x1 to x8 or create some other variables based on those depending on which is more correlated with y. It is the algorithm's mission to go through your data and come up with a function that gives you a y for your X .


Neural network options for simple data classification

You may not be very familiar with deep learning. Each kind of network is used for a special kind of task, you cannot just stack LSTMs, GRUs, dense layers and other stuff without supervision. If you have a task that your patterns are local and they may be in multiple locations in an input pattern, you can employ convolutional layers for feature extraction and you can employ dense layers for classifying those extracted features. If you want to classify data which there is a kind of sequence in it, temporal data, you can employ LSTMs and GRUs and you can stack them for better accuracy and you can use their output and feed them to other networks based on your need. MLPs are good for learning non-linear mappings.

Dropout is used for avoiding overfitting.