normalization of probabilities in predicting a poly-neuron output in neural nets

You are correct, but this is not called normalization. You can simply use the highest probability output for category. This is what softmax does for you. For example 2 output neurons can have 0.1 for dog and 0.9 for cat as the loss. Softmax will it just convert it to [0,1] meaning no dog but a cat is on the image.


Does it make sense to apply clustering on aggregation of data?

Since some features are missing for specific sources, the missing values are not missing-at-random but are systematically missing. In this situation, I'd advise against doing clustering on the combined data set with all available features. If missing values were occurring at random, you could have used some missing value imputation method before performing cluster analysis. However, since the values are systematically missing, imputation would be difficult to tackle. (You could try to predict those missing values, but I am afraid that will add a lot of unnecessary noise in the data.)

I'd recommend choosing from one of these two options:

Perform clustering on the combined data set, but use only those features that are non-missing across all sources.
Perform three different cluster analysis, one for each source. This way, you can ensure that you are using as many features (information) as possible. The determination of "abnormal" behavior can then be determined within each source. This can be an added benefit since it would allow you to be more specific about why a use might be abnormal, as you have more features that can be used to explain this. The results can also be then summarized across all sources to create one consolidated report.


How do I select the "best" unsupervised machine learning algorithm to cluster my specific dataset?

This is a non-trivial question. There is no general law to find the "best" algorithm or the "correct" amount of clusters (assuming you don't know the correct number of clusters).
As you already mentioned, certain algorithms make assumptions on the shape of your clusters or your data in general. One thing I suggest is to look at your data, check the assumptions and rule out algorithms where assumptions are violated. dimensionality reduction methods like PCA, t-SNE, UMAP, etc. are very helpful here if you work with high dimensional data.
Further have a look at the complexity of your clustering algorithm. If your are familiar with the Bias-variance tradeoff and Occam's razor you already know that simpler algorithms are less prone to overfitting and more likely to give you the correct results. BIC and AIC are quite popular measures in this regard.
Generally for unsupervised clustering it is a good idea to look at a number of different algorithms, compare them with different metrics and look where you can find the highes "agreement" among them.


Assuming the implementation is as simple as possible, with no advanced concepts, is it likely for something like this to happen or is it definitely an error in the implementation?

In my experience, using the simplest possible network, and simplest gradient descent algorithm, then yes this happens relatively frequently. It is an accident of the starting weight values, and technically a local minimum of the cost function, which is why it is so stable when it happens. In the basic implementation you have then there are only 6 starting weights. If they are selected randomly, the chances of a "special" pattern (such as the weights to hidden layer being all positive or all negative) are relatively high (1 in 8 for all positive or all negative weights between input and first hidden layer).

This is also why the values sum to 2 - given the the network is stuck on the wrong part of the error surface, it will still minimise the cost function as best it can given that constraint, and this will usually end up with compromise values that still meet statistical means overall in the predictions. If you doubled up some, but not all of the input/output pairs (e.g. a training set of 6 inputs $\{(0,0:0), (0,1:1), (1,0:1), (1,1:0), (0,1:1), (1,0:1)\}$, then the network may converge to different wrong mean value when it failed.

  Is it because of the very symmetrical nature of an XOR, which makes it impossible to handle for a single neuron?

You don't have a single neuron here. Unless you mean in the output layer? In which case, no, this is not to do with having a single neuron in the output layer.

Pretty much any more advanced NN feature, or simply more randomness, will stop this problem happening. E.g. make the middle layer have 4 neurons instead of 2, use momentum terms, a larger dataset with random "mini-batches" sampled from it.

In general this kind problem does not seem to happen on larger, more complex datasets and larger more complex networks. These can have other problems, but getting stuck in a local minimum far away from the global minimum tends not to happen. In addition for those scenarios, you typically don't want to converge fully into a global minimum for your dataset and error function, but are looking for some form of generalised model (that can predict from input values that you have not seen before).

On a practical note, if you want to add an automated test showing your NN implementation can solve XOR, then either use fixed starting weights or a RNG seed that you know works. Then your test will be reliable, even if the NN is not in all cases.


What is used for Machine Translation besides RNN

Recursive Neural Networks assume you have a relation between the current input and the previous input, if you organize data correctly CNNs can be used in place, some CNNs models have score higher than RNNs in BLEU metric. There is an article in Medium about it Why Convolutional Neural Networks are a Great Architecture for Machine Translation

Some implementation examples are Facebook's architecture proposed in the paper A novel approach to neural machine translation and this one from a Stanford's professor.

Also CNN's have been used by Google for language understanding and even in some fast-compact models or their new AI that talks on the phone.


Autoencoder implementation using ImageDataGenerator

As the following images from the paper shows, during training time you create patches from larger images. These patches have no defects and can therefore be seen as 'good' images. To get the accompanying 'bad' images with defects you synthetically generate these defects on the 'good' images.

Your data generator should therefore follow the following steps:

Read in full size 'good' images
Create patches from the full size images
Synthetically add defects to the patches from step 2
Return both the 'good' patches from step 2 and 'bad' patches from step 3

You then train your model using the 'bad' patches as the input and your 'good' patches as the output.


Not sure if it is domain adaptation as it has been described in literature, because the performance of your classifier in the target domain depends more in the alignment between the two domains due to the mistaken spelling etc. instead of measuring the divergence between the distribution of the two domains. If the distribution of the data between source and target is different then after the alignment you should use any appropriate method in order to tackle this difference and use the available information from the source domain.
P.S. Have you used the "levinstein distance" in order to align the two domains? (Match together the most similar ones etc...)


No one answered the question, then I will answer it myself. Thanks to @user12075 for the link. I didn't find it when I was googling it.

https://stackoverflow.com/questions/15388831/what-are-all-possible-pos-tags-of-nltk

From the above link, I know that nltk uses The Penn Treebank's POS tags.

nltk.help.upenn_tagset() will give you the list.


Feature selection for Support Vector Machines

Personally, I like to divide feature selection in two:

unsupervised feature selection
supervised feature selection

Unsupervised feature selection are things like clustering or PCA where you select the least redundant range of features (or create features with little redundancy). Supervised feature selection are things like Lasso where you select the features with most predictive power.

I personally usually prefer what I call supervised feature selection. So, when using a linear regression, I would select features based on Lasso. Similar methods exist to induce sparseness in neural networks.

But indeed, I don't see how I would go about doing that in a method using kernels, so you are probably better off using what I call unsupervised feature selection.

EDIT: you also asked about regularization. I see regularization as helping mostly because we work with finite samples and so the training and testing distribution will always differ somewhat, and you want your model to not overfit. I am not sure it removes the need to avoid selecting features (if you indeed have too many). I think that selecting features (or creating a smaller subset of them) helps by making the features you do have more robust and avoid the model to learn from spurious correlations. So, regularization does help, but not sure that it is a complete alternative. But I haven't thought thoroughly enough about this.


what is the interest of TimeDistributed after an LSTM layer?

Ok let's say you have an LSTM() layer with return_sequences = True set. That means each LSTM cell in it is outputting its value. The output of the layer is therefore a sequence of outputs, not just the final one. That means the output is a processed time series, with sequential information in it.

You want Dense() layers to take this information and use it to process the signal further. In particular, you want some Dense architecture to process each of these sequential outputs one by one. By using the TimeDistributed() wrapper it's like if you're iterating the same Dense() architecture on this sequential output.

I typically use TimeDistributed(Dense()) at the bottom of a seq2seq Decoder, in order produce the final output sequence.


Need input on which features to drop in classification model

Welcome to the community Sai!

Let's assume your problem is a regression problem (i.e. you have continues target).

There are some points:

First of all there is no written rule for this. Feature Engineering is an EDA kind of thing. There is no final solution. Your model selection strategy will choose some.
Just as a reminder, if these are Pearson correlations, be careful that nonlinear correlations might exist which is not captured by linear correlation analysis. Plus the fact that linear correlation analysis always comes with visual inspection.
Negative correlations inhibits information. If whenever a variable increases the other decreases, then knowing it tells you about the other one! So take them into account. You better use Mutual Information for checking dependencies.
Sparse Linear Models seem to be fruitful here. I suggest letting LASSO or Ridge Regression choose the final set of features.
In case you really insist on your current way (e.g. if your supervisor asked for it), use the threshold as the hyperparameter of your model selection and find the optimal. It means, you train and validate your model using different threshold (which results in different sets of features) and according to empirical error (validation error) you choose the best threshold.

Hope it helped. Good Luck!


A single column has many values per row, separated by a comma. How to create an individual column for each of these?

It would be better if you could provide some code which allows us to reproduce at least part of your DataFrame, such as this:

import pandas as pd
df = pd.DataFrame({'code': ['6254', '5854, 5676, 7265, 6051', '5815']})

At the start, your dataframe looks like this:

                     code
0                    6254
1  5854, 5676, 7265, 6051
2                    5815

A possible solution would be to do this:

df['code'].str.get_dummies(sep=', ').add_prefix('code')

which gives you this:

   code5676  code5815  code5854  code6051  code6254  code7265
0          0          0          0          0          1          0
1          1          0          1          1          0          1
2          0          1          0          0          0          0


How to visualize mass spectrometer data with plotly?

There is a Needle Plot example here (dash-bio.plotly.host) or here (dash-gallery.plotly.host) which is suitable to visualize the data in 2D as well as a 3D Stem Plot example to visualize the data in 3D.


ship class Object detection with custom model weights

You should be able to use keras.models.load_model to load the hdf5 model file. See also the tensorflow documentation.
# saving the model
model = ...  # Get model (Sequential, Functional Model, or Model subclass)
model.save('weights.hdf5')

# loading the model
from tensorflow import keras
model = keras.models.load_model('weights.hdf5')


Classifying points exactly on decision boundary

It is up to the user how they want to handle such cases. The loss for the points on the decision boundary will be 0. It's a design decision the user has to take. One example could be to consider such cases in misclassification and update your weights while training.

If $y (w \cdot x + b) > 0$: $\text{no loss}$

If $y (w \cdot x + b) \leq 0$: $\text{loss} = −y (w \cdot x + b)$

During inference, if such cases arise, you can call a random classifier to classify it in one of the classes.


You're looking for this answer: https://www.quora.com/Why-is-Python-a-language-of-choice-for-data-scientists


If your model is overfitting then you can use Dropout, L1 & L2 Regularization. It helps to improve your model's accuracy.


Need help with LDA for selecting features

Try before doing LDA look at the data - like doing TF, IDF and TFIDF analysis to identify such words which happen in all subject. If You have some taxonomy of Your product definition - consider using it. In my case it was really helpful.

I experimented with LDA topic modeling for recommendation systems purposes. I've few runs for offers in our marketplace service. It is in some aspect similar to Your problem with products and a list of keywords. But in our case, we do not have a product definition and an offer description is created by a seller. So You can imagine what tokens soup You will get :-)

The key point for me wast to analyze TF, IDF and TFIDF inside categories where I've tested LDA. E.g. for Toys category (http://allegro.pl/zabawki-11818?ref=simplified-category-tree) ~ .5M items, LDA based only on its names results in:

TOPIC:
lego    0.02282500803373657
puzzle  0.016679246682968853
gra     0.01092812347216676
klocki  0.010650330607355207
trefl   0.006797740719581059
zabawka 0.006365688767908592
lalka   0.0063546887775893105
maskotka        0.006111593588558912
dzieci  0.005888846995546501
24h     0.005736876716620104

TOPIC:
lego    0.02296167186908205
puzzle  0.016613064812376045
gra     0.010928234683331952
klocki  0.010607432835126579
trefl   0.006689061156804526
lalka   0.006317220992078405
zabawka 0.0062838718987015575
maskotka        0.006121026383268814
dzieci  0.0058569443925760144
24h     0.00578544965708585

TOPIC:
lego    0.02285507528449409
puzzle  0.016539723998111246
gra     0.010892673154789407
klocki  0.010624244094063881
trefl   0.006683424205358961
lalka   0.006350949499009139
zabawka 0.006324510823409019
maskotka        0.006148070280704085
dzieci  0.0058401580163702565
24h     0.005782372531889951
...

And the rest looks almost the same. Before running LDA I just did basic text preprocessing - our dictionary based item-name stop words removal and text tokenization (but without steaming). Even giving more terms for topic descriptions results doesn't look promising. Then, after exporting word frequencies with its offer categories by looking at data and plots I've decided to remove terms with TF-IDF above some threshold. Yes, above - I've used the calculation provided by Spark 1.3.1 implementation (HasingTF + IDF from mllib, no ml). After doing this I received:

TOPIC:
interaktywny    0.0026985965530064884
pony    0.0026721637823727343
dmuchany        0.0015777211309733249
baterie 0.0012447186564456534
pojazdy 0.0011171017074143481
thomas  9.476877418459519E-4
dinozaury       8.823212274401862E-4
monsters        7.426822230409613E-4
heroes  7.365256561824247E-4
ninjago 7.344344320326593E-4

TOPIC:
pony    0.0026867880330479327
interaktywny    0.0026803244251279693
dmuchany        0.0015797940843537916
baterie 0.001263424692131187
pojazdy 0.0010685733429671523
thomas  9.87888159025782E-4
heroes  8.26066846602493E-4
ptaszek 7.952454816383102E-4
dinozaury       7.849838129629457E-4
batman  7.408185413186286E-4

Where results started to differ from each other. Taking like last time only 10-terms for every subject description. Still, it's not perfect but better.

So, for me doing the experiments which few categories, the way to overcome problem was to remove it based on TF-IDF value. But, for every category the threshold was calculated separately. Mainly: mean(tf-idf) + 3*sd(tf-idf).

I know that Idf multiplication factor in Tf-Idf should resolve this problem by itself and the term should be punished for occurring in every document. But, using Spark implementation (idf = log((m + 1) / (d(t) + 1))) and our data it was simpler during a quick and dirty experiment to filter it that way.

When I find few sec. I will get back on this and place results with code and online to share this.


decision rules for each feature (binary classification)

You are describing every binary classifier.  However, you are missing a key point. If your classes are separable by the value of just ONE feature, you can do what you're saying and find e.g. F1 > 1.23 as a threshold.  If classification involves a combination of features, you will need to describe some combination of thresholds for each feature, or (equivalently) some relationship between the features that tells you about the class label.  It's the job of every binary classifier to do exactly this - they just do it in different ways.  See for example this post.  Your desire to have a combination of fixed sets of thresholds will only work if you can have a set of thresholds that will encompass/describe/classify every combination of feature values.

If you want a set of easy-to-read thresholds like you mention, you should read about decision tree classifiers.  They'll do something like what you want - but will also ensure that you provide a class label for every possible combination fo features values. The nice about about decision trees is that they'll let you leave out your current feature selection step - they just do it for you by (1) picking the feature that best discriminates the two classes overall, (2) picking the threshold value of that feature that gives the most information about the class label (usually), and (3) repeating (1-2) several times.


What regression model can handle tiny amounts of data?

The description of your problem is vague. In case you want to do causal analysis, you need to consider the bias-variance-tradeoff. For a causal analysis you would stick to low bias (e.g. using a "best linear unbiased estimator" like ordinary least square regression), while most ML models, such as neural nets or boosting, go for low variance.

Still not clear to me: how does your data look like. If you have more features/variables ($X$) than observations ($i$) your problem is high dimensional. For a high dimensional problem, use the Lasso. If $i$>$X$ and you have say at least 40+ degrees of freedom, you could simply apply OLS regression. If you (can) use OLS and you care for significance and confidence bands, use robust standard errors (hc2, hc3).

As mentioned above, you will likely have high variance using OLS or Lasso. What you can do to work on that in a causal model, is to make linear transformations of $x_i$, e.g. add polynomials, or you can add interactions of two $x$, e.g. $x_3=x_1 x_2$.

Finally, you could use generalized additive models (e.g. with regression splines) to model "complexity" if it originates from non-linearity in $X$. However, in this case you go in the direction of decreasing variance (at the cost of probably introducing unwanted bias).

I guess the nature of your problem is of "high dimension". This is a little special (and I'm no expert here). Have a look at "Elements of Statistical Learning", Chapter 18. The book gives a really good overview of possible options to deal with this problem.


How to fight underfitting in a deep neural net

The problem with deep networks is that they have lots of hyperparameters to tune and very small solution space. Thus, finding good ones is more like an art rather than engineering task. I would start with working example from tutorial and play around with its parameters to see how results change - this gives a good intuition (though not formal explanation) about dependencies between parameters and results (both - final and intermediate).

Also I found following papers very useful:

Visually Debugging Restricted Boltzmann Machine Training
with a 3D Example
A Practical Guide to Training Restricted Boltzmann
Machines

They both describe RBMs, but contain some insights on deep networks in general. For example, one of key points is that networks need to be debugged layer-wise - if previous layer doesn't provide good representation of features, further layers have almost no chance to fix it.


Can anyone recommend me a very good pre-trained model for face or head detection?

Using facenet pretrained model to detect faces in image, but i thing ti will not detect the head, so i suggest that you train object detection model to detect faces and heads.


Text classification problem using Python or R

Take a supervised approach:

For each group "Blood test", "Stool test" etc,

Take a subset of your rows as a training set, say 200 rows.
Create a new numeric or logical column, "IsBloodTest" or similar for this new subset of your dataset

For each row in these 200

Classify them manually: if it is a blood test, assign 1, if not assign 0 to "IsBloodTest"
Split the "Description" column into word vectors. This creates a Document-Term matrix with only 1 or 0 values in the cells (Word is present/not present)
Classify the rows of the sparse new matrix (omitting your own newly manually created attribute) with the Multinomial NaiveBayes algorithm. This assigns another column with "0/1" prediction, say "PredictedBloodTest"  to  your training set.
With these 0/1 values in the two new columns you can build a confusion matrix.

Perform Cross validation experiments to check your confusion matrix, or use an extra validation set. Label more rows manually if necessary.

If the classification result is good enough, use it for the remaining 1000s of rows from your data set.

This procedure is simple, but it might get you results faster than learning more sophisticated approaches first.


Square Root Regularization and High Loss

The problem with using $L_p$ norm with $p < 1$ for regularization is the gradient.

Regularization term is used to force the parameters to be closer to zero. For this to work, when a parameter goes closer to zero, the gradient of regularization term, i.e. its contribution in updating the parameter, should decrease as well or at least remain constant.

However, by going closer to zero, the gradient increases unboundedly for $p < 1$, which causes radical changes in parameters, leading to your observations. Here is a visual plot for root $r$, absolute $a$, and squared $s$ regularization terms in one dimension (drawn here):

As you can see, for root regularization, the gradient explodes to infinity as we get closer to zero which is in conflict with our intention to bring the parameters close to zero.

For the sake of completeness, here is what a 1D parameter update with root regularization looks like:
$$\theta_{n+1} = \theta_n + \alpha \frac{d L(\theta)}{d \theta}\Bigr|_{\theta=\theta_n} = \theta_n + \alpha \left(\frac{d l(\theta)}{d \theta}\Bigr|_{\theta=\theta_n} \pm \lambda \frac{1}{\color{red}{\sqrt{|\theta_n|}}}\right)$$
where $L(\theta) = l(\theta) + \lambda \sqrt{|\theta|}$.

You can spot the trouble! The more we get close to zero, the less we get close to zero!


How to define the number of features to select in RFECV?

That's the point of RFECV over RFE: the former selects the best number of features through cross-validation.  If you want 15 features, use RFE instead (or some other feature selection method).

From the API docs,

  cross-validated selection of the best number of features

and from the User Guide

  RFECV performs RFE in a cross-validation loop to find the optimal number of features.

To respond to the comments (my response is a bit too long for comments):

Yes, RFECV is meant to produce the optimal number of features.  RFE is run from the full feature set down to 1 feature on each of the cross-validation splits, then those models are scored on the test folds and averaged; then the best-scoring number of features can be taken and then RFE is run again down to that number of features.  The thought is that (barring some special domain knowledge) RFECV is better than RFE, except that it may take rather longer to run.  By definition, the RFECV's accuracy (on the CV-splitted training set) will be better than RFE with any other fixed number of features.

Now, the usual caveats apply.  The number of features returned by RFECV may not necessarily generalize the best to unseen data, especially if that data isn't perfectly iid with the training data.  Especially with small datasets, the splits made in the CV may affect the results, and perhaps going from the smaller CV train sets to the whole train set should allow more (or fewer?) features.  And the top features on each of the splits may not be the same, so again when going to the full training set the question of increasing (decreasing?) number of features is reasonable.


too few data while too many degrees of freedom in linear regression

Good question. The solution lies in understanding how weights play into the model's prediction. I've glossed over a bunch of issues here, but the big picture is what is most important.

Theory

When you create a model all the weights are psuedo-randomly initialized. This solves your problem of not having enough degrees of freedom to solve a linear regression. We've interpolated our weights and our model will update only the weights it needs as it trains.
Additionally, your model will probably find that most of your weights converge to zero. If you are using sklearn's load_digits() function, I recall that about half of each image was just white pixels. It is likely that there is a significant percentage of pixels that do not matter across all images (and thus might be assigned a weight of zero). Your question of how can you trust the prediction of the model is thus answered and we can modify our statement about degrees of freedom to: There cannot exist a greater number of discovered statistical trends than there are data samples. It is hard to measure "number of discovered statistical trends", but it follows that there is a finite number and that they are less than the number of samples minus 1.

Example

With a fully-connected, shallow neural network consisting of the input layer with weights and an output layer with two neurons, the input to the activation function of each of the output neurons will be something like the following:

$$f(B_1+\sum_{i=1}^{i}W_i*{I_i}_A)=P$$

where

$P$ is the class probability,
$B_1$ is the bias of the output layer,
$W_i$ is the weight of a specific neuronal connection,
${I_i}_A$ is the output of the connected neuron if activated (in this case the pixel value), and
$f()$ is the activation function of the output layer neuron.

If we didn't initialize $W_i$ to something/anything, we would be unable to solve the equation (more variables than samples). However, by randomly initializing them, the equation immediately becomes solvable and we can use stochastic gradient descent or a different algorithm to optimize our weights. I hope this shows how pseudo-randomly initializing $W_i$ solves the mathematical problem of degrees of freedom.


What is the best way to organize the datasets for my task?

When you uniformly take samples from a society, definitely  chance of selecting from cities is directly related to their population. Therefore, more users will be selected from more populous cities and it's one of the most important characters of the problem you trying to solve. I think if you want to balance the data-set, you ignore one this important character of your data and also your problem.

I strongly recommend to continue with the unbalanced data-set and handle it by choosing an appropriate loss function and evaluation method.

Disclaimer:

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)


How to get Real life Social Network Data?

Take a look at these links:

https://opendata.stackexchange.com/questions/1670/any-cdr-call-data-record-dataset
https://stackoverflow.com/questions/26229570/dataset-needed-for-cdr-analysis
http://vlado.fmf.uni-lj.si/pub/networks/doc/seminar/nicta02.pdf


Visualizing Support Vector Machines (SVM) with Multiple Explanatory Variables

Does it matter that the model is created in the form of SVM?

If no, I have seen a clever 6-D visualization. Its varieties are becoming popular in medical presentations.
3 dimensions are shown as usual, in orthographic projection.
Dimension 4 is color (0..255)
Dimension 5 is thickness of the symbol
Dimension 6 requires animation. It is a frequency of vibration of a dot on the screen.
In static, printed versions, one can replace frequency of vibration by blur around the point, for a comparable visual perception.

If yes, and you specifically need to draw separating hyperplanes, and make them look like lines\planes, the previous trick will not produce good results. Multiple 3-D images are better.


Cleaning data largely varies from data to data. Considering you are talking mostly about unstructured data, it can be of two types, namely Image data and text data. The process of cleaning both these data depends on the type of goal required to be achieve using these. A little brief process could be explained as follows:

Image data: While performing convolutional neural network on the data, the process of feature extraction from the data is automated. Specific pre-processing need to be carried out aiming it to the target. This includes:

Keeping the data set close to the actual data on which the model is to be tested.
Removing any unnecessary images that do not co-ordinate with the corresponding label.
Resizing the images as per the model requirement.

Text data: Text data has a robust procedure for of pre-processing consisting of Removing stop words, tokenizing, lemmetizing the tokens etc. At times when we are using the word to vec approach, we need to convert the text tokens to their corresponding word-vectors.

Generally functions designed for these processes are fed into the pipeline and so individually every entry is not required to be processed in a similar way.

There are some papers and blogs I would suggest you to read: Towards automated patient data cleaning , Turn Messy Data into Tidy Data


Correlation between nominal categorical variables

The Chi-Squared test of independence (and subsequent Cramer's V test) give an indication of the relationship between two categorical variables.


How to optimize XGBoost performance accuracy?

There are three main techniques to tune up hyperparameters of any ML model, included XGBoost:

1) Grid search: you let your model run with different sets of hyperparameter, and select the best one between them. Packages like SKlearn have routines already implemented. But also in this case you have to pre-select the nodes of your grid search, i.e. which values have to be tried by the routine

2) Random search: similar to Grid Search, but you basically only choose the parameters boundaries, and the routine randomly try different sets of hyperparameters.

more informations about method 1 and 2 are here.

3) Bayesian optimization algorithms; this is the way I prefer. Basically this algorithms guesses the next set hyperparameter to try based on the results of the trials it already executed. An easy to use and powerful is SMAC.


Your loss function estimates the conditional expectation and the predictions mainly hover around the mean of the time series. I would argue that predictions are good, but the problem is poorly designed. One approach would be to partition the problem into two subproblems: probability of your series being $\mathbb{P}(Y>0)$ and predicting the conditional expectation by training only on non-zero values $\mathbb{E}[Y|Y>0]$. The final prediction would then be $\mathbb{P}(Y>0)\mathbb{E}[Y|Y>0]$


Complex HTMLs Data Extraction with Python

I ended up using BeautifulSoup to do the job. The final code is not that clean - as I'm kind of a newbie - but it does what I wanted. You can find the source code and the dataset I've extracted so far in this repo. You can also check out an article I did on this in my website: fanaro.com.br.


Update a pandas data frame column using Apply,Lambda and Group by Functions

I used 'Apply' function to every row in the pandas data frame and created a custom function to return the value for the 'Candidate Won' Column using data frame,row-level 'Constituency','% of Votes'

Custom Function Code:

def update_candidateresult(df,a,b):
max_voteshare=df.groupby(df['Constituency']==a)['% of Votes'].max()[True]
if b==max_voteshare:
    return "won"
else:
    return "loss"

Final Code :

 df_andhrapradesh['Candidate Won']=df_andhrapradesh.apply(lambda row:update_candidateresult(df_andhrapradesh,row['Constituency'],row['% of Votes']),axis=1)


Dummy encoding the categorical variables using the changed version of OneHotEncoder

Did you try using

X = pd.get_dummies(X)

?

Otherwise make sure that your X is either a 2D Array or a dataframe for OneHotEncoding to work.


Is it possible to classify documents of corpus using labels?

You could simply encode the documents using BERT and cluster the documents based on their content provided they sufficiently different in terms of the kind of content they contain.

Another approach would be to train a document segmentation model which would segment documents based on their semantic structures and then classify the documents based on their masked skeletons. This however would require a large dataset to train. Fortunately you can find one online called PubLayNet. Augment that with a few representations of your documents for better generalization over the test set.

I've read about the second approach being implemented to classify patents, legal documents, research papers etc. With good results. However it would take a long time to train.
I'd recommend simply clustering the documents based on their text embedding (point 1) and then naming the clusters. If that doesn't work satisfactorily, try the deep learning method for document semantic masking.


Gaussian distribution in python without using libraries

You need to sort arr. For example you sort df.Age then apply the function and after plotting you will get a beautiful chart.

For example, I used your function and a range from 0 to 99 that is already sorted:

import numpy as np
import math
from matplotlib import pyplot as plt

arr = np.arange(100)
y=gaussian_transform(arr)
plt.plot(arr,y)

and got the following plot:

To make the plot smooth you need to add more points to the chart. In the following code I used vector functions of numpy to make the computation faster and write less code. I also used the linspace function to fill in the space between max and min of the data with more points for smooth charts.

import numpy as np
import math
from matplotlib import pyplot as plt

def gaussian_plot(arr , n=1000):
    u = arr.mean()
    sig = arr.std()
    x = np.linspace(arr.min(), arr.max(), n)
    e = np.exp(-a)
    y = 1/(sig*math.sqrt(2*math.pi)) * e
    return x, y

arr = np.arange(100)
x, y=gaussian_plot(arr, 10000)
plt.plot(x,y)

And the result:


In regression, is a higher adjusted R-Squared ALWAYS better?

Generally, a higher adj. R-square is better. In your case, you might be better off working on the representation of temperature in the model. It depends on the data, really, but you could try polynomials for temperature (squared term or so) or you could make „classes“. Plot temperature against y, to see if there is a non-linear trend in the data. If so, you need to try to represent this structure in your model, because OLS regression is parametric by definition.

Adjusted R-square balances the advantage of adding a new variable and the disadvantage of losing degrees of freedom. So in your case, the explanatory power of temperature seems to add little to your model. This may change if you change the representation of the variable, so that it better fits the data generating process. You need to try.


Is there any logic to adding a threshold to see if two variables are related?

I think you may have a sample size effect on your correlation coefficient. You may find the following references useful:

At what sample size do correlations stabilize?
The effects of sample size and variability on the correlation coefficient.
And also this post in stats.stackexchange demonstrating the effect of subsampling on correlation coefficient.


You can think of examples as vectors in $\mathbb{R}^p$, where $p$ is the number of features. Two examples will be very similar if the distance between them is close to $0$ (in the extreme case, if two examples are equal their euclidean distance is $0$). One way to measure the distance is using euclidean distance, but other distances can be used, as cosine distance or $L^p$ metrics. In fact, if $p$ is very high, then Euclidean distance is not a good measure, as it tends to make the distances too uniform (see this paper).

Edit: When $p$ is very high:

See this magnificient answer to the issues that very high $p$ may have.


Softmax function result for already normalized probabilities

Summing to 1 is just one property of the softmax function.

The softmax function takes the exponential of each value and divides it by the sum of the exponentials of all values. This tends to cluster values towards the mean, as you've seen in your example.

While the outputs of a softmax look and smell like probabilities, their values don't actually correspond to the likelihood of sampling from that distribution. They're better off thought of as the confidence of your prediction.


It sounds like you are making this more complicated than it is with trying to predict spend at the cohort level. It is best to use cohorts for understanding "why" something occurred and time series analysis for understanding "what" will occur.

The Forecast package in R seems to be solution to your problem. Just forecast Monthly spend and be done with it!


To answer your question no.

The term "imbalance" usually refers to classification problems. For your case, i.e. a regression problem you can only look at the distribution of your target variable.

If by "balance" you mean them having a uniform distribution, you could argue that they are, if fact imbalanced. However, I'd argue that this is not the problem here. When steering you rarely need an extreme angle, I think this dataset represents this well.

Your goal is also a bit unclear. Are you trying to "predict the steering angle" or  "follow the lane"? If it is the first, I'd first suggest changing your metric. Use one that tells you actually how well you are doing, e.g. MSE, MAE.

A lot of things could be to blame for a poor performance. I don't thing the distribution of the angles is one of them.


images from training set are different from images of test set

The way you use ImageDataGenerator is wrong. The .fit() method is trying to read in the directory path which is a string. To be able to run your code, you should remove the train_datagen.fit(training_set) and your script should work fine and all the images should have been preprocessed the way you want them. Here's a link on how to use the ImageDataGen.

The .flow_from_directory() function is used for raw images. The .fit() method is used on numericals.


Non-prediction Applications of Machine learning

The most obvious applications are indeed the supervised learning approaches (surrogate models, prediction). But there is much more than that! Other usual applications include:

Clustering: This can be seen as a different kind of prediction, but not in the classic supervised learning fashion. For instance, I have been using a clustering algorithm on a 3D geometry (CAD file) to make almost adjacent elements become actually aligned.
Anomaly detection
Study of extreme values (how to learn the behaviour of extreme values without actually observing such values)
Feature importance (determine which sets of features impacts the result the most) or, more generally, data mining
Inference (use a dataset to improve the prior model prediction)

Of course, most of these applications are more or less close to prediction, but not trivial prediction.


No, your model isn't supposed to know about your test data, if you include clues in your training about what's in your test data, you will do something called
Data Leakage.
Data leakage would lead to Over-fitting which will give you good results on that particular test set, but won't generalize to other data.
Lets say, you deploy this model in production and feed into it real-life data that it never encountered before, the predictions will be below the expectations you had in the training/testing phases because of those two phenomenons i mentioned.
I suggest you tweak a bit more your model during the training phase, maybe clean your data more, do something called OverSampling and UnderSampling if the target classes are imbalanced (You have for example 90% / 10% proportions in your training dataset), pick better features etc..
In conclusion: adjusting your model to have good predictions on your test data, in particular, is not good practice and will lead to creating a bad model on unseen data.


Two different pytorch networks, combined loss, back propagation and optimizer step

A single final_loss.backward() would calculate the gradients but there are some strange things in your scheme (these may seem strange due to lack of information in your question):

The first strange this is: why having two separate optimizers? It could only be justified if you purposely want different optimization algorithms for each network.

The second and most strange thing is: why would you want to have a combined loss? The only reason that comes to my mind would be that module1 and module2 share some parameters. Otherwise, I see no point in combining two totally unrelated losses because they could have different scales, leading to one of the partial losses to have little effect. Minimizing each loss separately would lead to a much better result.


Why does GridSearchCV (sklearn) change the value of n_samples?

The CV stands for CrossValidation, meaning it will split up your training set in a number of folds (in this case 3), train on n-1 of those folds and test on the remaining one. This is why your training is now done on 32 instead of 50 samples. Crossvalidation is useful for estimating how well your model (including specific hyperparameters) does on unseen data.


How to find similarity between different factors in a dataset

One way is to normalize your quantitative  values (play, eat, drink, sleep rates) so they all have the same range (say, 0 -> 1), then assign each game to its own "dimension", that takes value 0 or 1. Turn each row into a vector and normalize the length to 1. Now, you can compare the inner product of any two people's normalized vectors as a measure of similarity. Something like this is used in text mining quite often

R Code for Similarity Matrix

Assumes you've saved your dataframe to the variable "D"

#Get normalization factors for quantitative measures
maxvect<-apply(D[,1:4],MARGIN=2,FUN=max)
minvect<-apply(D[,1:4],MARGIN=2,FUN=min)
rangevect<-maxvect-minvect
#Normalize quantative factors
D_matrix <- as.matrix(D[,1:4])
NormDMatrix<-matrix(nrow=10,ncol=4)
colnames(NormDMatrix)<-colnames(D_matrix)
for (i in 1:4) NormDMatrix[,i]<-(D_matrix[,i]-minvect[i]*rep(1,10))/rangevect[i]
gamenames<-unique(D[,"game"])
#Create dimension matrix for games
Ngames<-length(gamenames)
GameMatrix<-matrix(nrow=10,ncol=Ngames)
for (i in 1:Ngames) GameMatrix[,i]<-as.numeric(D[,"game"]==gamenames[i])
colnames(GameMatrix)<-gamenames
#combine game matrix with normalized quantative matrix
People<-D[,"name"]
RowVectors<-cbind(GameMatrix,NormDMatrix)
#normalize each row vector to length of 1 and then store as a data frame with person names
NormRowVectors<-t(apply(RowVectors,MARGIN=1,FUN=function(x) x/sqrt(sum(x*x))))
dfNorm<-data.frame(People,NormRowVectors)

#create person vectors via addition of appropriate row vectors
PersonMatrix<-array(dim=c(length(unique(People)),ncol(RowVectors)))
rownames(PersonMatrix)<-unique(People)
for (p in unique(People)){
  print(p)
  MatchIndex<-(dfNorm[,1]==p)*seq(1,nrow(NormRowVectors))
  MatchIndex<-MatchIndex[MatchIndex>0]
  nclm<-length(MatchIndex)
  SubMatrix<-matrix(NormRowVectors[MatchIndex,],nrow=length(MatchIndex),ncol=dim(NormRowVectors)[2])
  CSUMS<-colSums(SubMatrix)
  NormSum<-sqrt(sum(CSUMS*CSUMS))
  PersonMatrix[p,]<-CSUMS/NormSum
}
colnames(PersonMatrix)<-colnames(NormRowVectors)
#Calculate matrix of dot products
Similarity<-(PersonMatrix)%*%t(PersonMatrix)


after cross-validation, what measure of 'stopping' training should i use when training final model on whole dataset?

If you use the scikit-learn GridSearchCV class (from sklearn.model_selection) together with the scikit-learn wrapper in keras, you can get your final model refit on the whole training set directly via the best_estimator_ attribute (i.e. the model instanced with the best hyperparms found in the CV process) already refit with the whole training dataset.
I have seen some source which make a "manual" retrain on the whole training set, but it is not necessary as scikit-learn already let's you accessing the refit model on the whole set.
Interesting links:

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html
for using the keras wrapper for scikit: https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/


How to model & predict user activity/presence time in a website

I suggest approaches based on neural networks for time to event data. Depending on your data I think you can also use

https://www.slideshare.net/mobile/datasciencelondon/survival-analysis-of-web-users

https://arxiv.org/abs/1807.04098

https://arxiv.org/abs/1801.05512

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/


How to classify movement data (time series) in real time

Supervised learning for classification in machine learning trains a model in order to determine which distribution a certain novel instance belongs to. Model complexity is directly correlated with the performance of a specific algorithm given a set number of instances.

This means you should always start with understanding your data before selecting a model.

Your dataset

Your dataset is comprised of instances $x_i \in \mathbb{R}^{300}, i \in {1, ..., n}$ and $10 < n < 50$.

This is a very small number of instances $n$. Usually I recommend to have ten times more instances than features for a two class classification problem. Furthermore, less than $50,000$ instances is not sufficient for training any deep learning model. You will need to consider using traditional machine learning algorithms (SVM, KNN, Random Forests, etc.). Lastly, the class imbalance in your dataset will introduce some bias, this will lead to poorer results.

All hope is not lost, we can still do something to solve this problem!

Feature selection

The first step which I would suggest is to consider performing some feature reduction. You want to map your $X \in \mathbb{R}^{300}$ to a lower dimensional set $\mathbb{R}^{p}$ where $p<300$. However, in removing features you will evidently be losing information, so you want to be able to learn what are the relevant features to your decision smooth/shaky. For example, when classifying cat/dog, the weight may be a highly informative feature thus its contribution in the decision should be highly considered, whereas the number of feet would not provide any useful information.

Some techniques to do this include PCA, LDA, and theres others as well. You can try them, its one line in scikit-learn.

Choosing an algorithm

Due to the heavy class imbalance, I would suggest that classification based algorithms will perform poorly. In contrast, I would attempt to use anomaly detection algorithms. These techniques will learn the distribution of your nominal set (smooth) and attempt to detect when signals differ significantly from this distribution and will then classify them as anomalous (shaky).

Many of these techniques exist, please refer to the following answers:

Using time series data from a sensor for ML

How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries


is it possible to output more than 2 nodes away from a node in a decision tree? if yes, how to do that with sklearn?

It is possible to make more than a binary split in a decision tree. Chi-square automatic interaction detection (CHAID) is an algorithm for doing more than binary splits.

However, scikit-learn only supports binary splits for many reasons. One primary reason to limit to just binary splits is that the library can support as many splitting criteria as possible with the same API. For example, Gini Impurity only supports binary splits.

In practice, only supporting binary splits is not an issue because a series of binary splits can model any number of simultaneous splits.


How to Obtain Output of Intermediate Model in Keras

The key is to first do .get_layer on the Model object, then do another .get_layer on that specifying the specific layer, THEN do .output:

layer_output = model.get_layer('Your-Model-Object').get_layer('the-layer-contained-in-your-Model-object').output


Data augmentation: rotating images and zero values

Keeping the values as zeros will introduce some bias to your network. Given, you have this corner effect for the majority of your dataset, you do not want the network to identify a high probability of making the corners black. Thus, you should fill them, you can extend the edge, do a reflection, wrapping. You can also do some more complex function, like take average of a few patches in your image then place them in the missing areas.

Keras has a very nice function that can do all this for you.

import numpy as np
from keras.datasets import mnist
from keras.preprocessing.image import ImageDataGenerator
import matplotlib.pyplot as plt
%matplotlib inline

# load data
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.astype('float32')

# set up your data generator
datagen = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rotation_range=15,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    vertical_flip = False)

# Fit the generator using your data
datagen.fit(X_train.reshape((len(X_train), 28, 28, 1)))

# Black images
image = X_train[5]
plt.imshow(image,  cmap='gray')
plt.show()

plt.figure(figsize=(12,12))
plt.subplot(4, 4, 1)
plt.imshow(image.reshape((28,28)),  cmap='gray')

for j in range(15):
    augmented = datagen.random_transform(image.reshape((28,28,1)))
    plt.subplot(4, 4, j+2)
    plt.imshow(augmented.reshape((28,28)),  cmap='gray')
plt.tight_layout()
plt.show()

# White images
image = -1*X_train[5]
plt.imshow(image,  cmap='gray')
plt.show()

plt.figure(figsize=(12,12))
plt.subplot(4, 4, 1)
plt.imshow(image.reshape((28,28)),  cmap='gray')

for j in range(15):
    augmented = datagen.random_transform(image.reshape((28,28,1)))
    plt.subplot(4, 4, j+2)
    plt.imshow(augmented.reshape((28,28)),  cmap='gray')
plt.tight_layout()
plt.show()


Why large weights are prohibited in neural networks?

Many strategies used in machine learning are explicitly designed to reduce the test error, possibly at the expense of increased training error. Generally,
regularization is any modification we make to a
learning algorithm that is intended to reduce its generalization error but not its training error.

The L2 parameter norm penalty commonly known as Weight decay is one of the simplest and most common kinds of regularization technique which forces the weights to become smaller, by adding a parameter norm $Ω(θ) = 1/2 || w||^{2}_2$ penalty to the objective function.
For example, in linear regression, this gives us solutions that have a smaller slope, or put weight on fewer of the features. In other words, even though the model is capable of representing functions with much more complicated shape, weight decay has encouraged it to use a simpler function described by smaller coefficients.

Intuitively, in the feature space, only directions along which the parameters contribute significantly to reducing the objective function are preserved relatively intact. In directions that do not contribute to reducing the objective function, movement in this direction will not significantly increase the gradient.So, Components of the weight vector corresponding to such unimportant directions are decayed away through the use of the regularization throughout training.

Another simple explanation is when your weights are large, they are more sensitive to small noises in the input data. So, when a small noise is propagated through your network with large weights, it produces much different value in the output layer of the NN rather than a network with small weights.

Note that weight decay is not the only regularization technique. In the past few years, some other approaches have been introduced such as Dropout, Bagging, Early Stop, and Parameter Sharing which work very well in NNs.

There are other interesting findings in this very rich chapter.


There are certain pre-trained models available regarding sentiment analysis or emotion analysis like textblob , flair and deepmoji. Here the link to the in the medium ->
https://medium.com/@b.terryjack/nlp-pre-trained-sentiment-analysis-1eb52a9d742c

But if you have your own set of emotions for which you want to classify, then the best way would be to gather data with labels (the emotions) and train a model from scratch.


Why underfitting is called high bias and overfitting is called high variance?

How can one understand it intuitively?

Underfitting is called "Simplifying assumption" (Model is HIGHLY BIASED towards its assumption). your model will think linear hyperplane is good enough to classify your data which may not be true. consider you are shown a picture of cat 1000 times, Now you are blindfolded, No matter Whatever you are shown the 1001th time, probability that you will say cat is very high(You are HIGHLY BIASED that the next picture is also gonna be a cat). Its because you believe its gonna be a cat anyway. Here you are simplifying assumptions

In statistics, Variance informally means how far your data is spread out.
Overfitting is you memorise 10 qns for your exam and on the next day exam, only one question has been asked in the question paper from that 10 you read. Now you will answer that one qn correctly just like in the book, but you have no idea what the remaining questions are(Question are HIGHLY VARIED from what you read). In overfitting, model will memorise the entire train data such that it will give high accuracy on train but will suck in test. Hope its helps


It's hard to put a general rule with this kind of methods, they depend heavily on the data at hand. You should know pros and cons of each, try all methods and see which performs best on the validation set. Remeber that:

over-sampling the less frequent class can bring a lot of repetition in the data, because you just replicate the observations
under-sampling can be bad because you throw away information in the most frequent class, and so you can lose performance

Hybrid methods tend to outperform both, see SMOTE or ROSE.

For example, SMOTE generates new information from the minority class (synthetic observation which are derived from the true samples), and also under-samples the majority class (you choose the final ratio of the two).

Again you need to test on the data the best solution.


How to run a python script on GCP Compute Engine?

First, you will need to install Cloud SDK: https://cloud.google.com/sdk/downloads#apt-get

Then, the simplest way is to run your script through your Terminal (mac and I presume the instructions also work on Linux):

Configure your project: gcloud config set project insert_your_project_name
Set up SSH keys: gcloud compute config-ssh
Connect to the VM: gcloud beta compute ssh vm_name --internal-ip
Run script: python your_script.py

You can also connect PyCharm directly to GCP and run everything on your VM but you will need PyCharm Pro, otherwise the deployment option is not available. Let me know if this works.

Also, if you want to use the interactive version of setting your project then in step 1 do this instead: gcloud init


Normalizing the final weights vector in the upper bound on the Perceptron's convergence

The Perceptron's output $f$ is
$$
f(\overline\theta \cdot \overline{x}) = \begin{cases} 1 &\text{ if } \overline{\theta }\cdot \overline{x} > 0 \\ 0 &\text{ if } \overline{\theta }\cdot \overline{x} \le 0\end{cases}
$$
Here, $\overline{x} = (1, x_1, \dots, x_n)$ where $(x_1, \dots, x_n)$ is the input vector.

You can see that the output only depends on the sign of the product $\overline{\theta }\cdot \overline{x}.$
Therefore the output will not change if we multiply $\overline\theta \cdot \overline{x}$ with a positive constant:
$$
f(\overline\theta \cdot \overline{x}) = f(c\times \overline\theta \cdot \overline{x}) \qquad \text{ for all } c > 0.
$$
In particular, we may choose $c = 1/\lVert \overline\theta \rVert.$ So we get
$
f(\overline{\theta }\cdot \overline{x}) = f(\tilde\theta\cdot \overline{x}),
$
where $\tilde\theta = \overline\theta / \lVert \overline\theta \rVert$ is the normalized vector.
Conclude that there is no loss of generality in assuming $\lVert \overline\theta \rVert = 1.$


First you do not always need to normalize (standardize) the input vectors (feature vectors), sometimes is good, sometimes is bad. In general you scale your feature vector when the magnitude of a feature dominates the others, so the model cannot pick up the contribution of the smaller magnitude features. Read here for a detailed explanation.

Second there are two general classes of machine learning problems: classification and regression. In a classification type problem the output (dependent variable) is discrete, so you do not need to normalize it.

In a regression type problem scaling the output do not affect the shape of your function see here. Moreover it does not affect error functions like Mean Squared Error, i.e. the error is also scaled.


Does the choice of normalization change dramatically the result of a KMeans

Yes. Feature scaling can completely change the clustering result.

People usually scale data to [0:1] or to have a standard deviation of 1.

However, that is nothing but a heuristic.

In many cases, the need for scaling is nothing but a symptom, caused by inappropriate data for the method. You can't just fix this by some naive scaling, but it's just a hack that often works.

For statistically meaningful results, all axes should be scaled to reflect attribute relevance, such that a difference of 1 unit is of the same importance in each attribute.


Clustering with geolocation (lat/long pairs) attributes

Don't treat clustering algorithms as black boxes. If you don't understand the question, don't expect to understand the answer.

So before dumping the data and hoping that magically a desired results comes out, understand what you are doing...

Standardizing latitude/longitude is a horrible idea. These values are angles on a sphere. Linearly scaling these values breaks everything that these values mean. There are many valid transformations - even rotations can be good to get a desirable Mercator protection, for example. But standardizing them, I cannot imagine what this would be good for.
Mixing variables with different meaning rarely works well. It's not just the problem of scale. Scaling often helps as a heuristic to prevent one variable dominating another. It also has the nice property that it doesn't matter if your data were feet or yards. But the need to do so usually means that there is something wrong with your approach at a deeper level: that you apparently are trying really hard to compare apples and oranges... You'll get out some result. It's probably even interesting. But once you try to explain or act on it, you're back to square one: what does it mean if you scale your data this way, and why is that better than the infinitely many alternative ways, infinitely many of which lead to other results?


Outlier detection is part of data preprocessing and used to remove some of the rare events but it could happen that rare events are important to us like fraud detection in that case it becomes important and so we can't do outlier detection beforehand.
In that case we do various approaches like undersampling of majority events or oversampling of minority class. For various approaches refer this
You could also look into KPCA. Having said that there is no particular solution it all depends upon your dataset.


Action value and target value in DQN refer to the same thing in terms of what they measure, but are obtained and used in different ways.

An action value does not approximate the reward of a given action, but represents the related concept of expected return - the expected discounted sum of future rewards when taking action $a$ in state $s$. You see it very often associated with the action value function $q(s,a)$.

Action values are not inherently approximated, they are more conceptual than that. However, in practice in Reinforcement Learning (RL), the action value function must be learned, and some form of approximation used based on observed data. When an action value function it is approximated by a neural network during learning you may see it written $\hat{q}(s,a,\theta)$ which makes explicit that the NN is approximating some "true" function that you don't know, and that it is parametrised by $\theta$, the neural network weights and biases.

During learning, the agent takes actions and observes resulting states and rewards. It must use these observations to improve its estimates of values. To do so in DQN, the agent constructs a temporal difference (TD) target - for single-step Q-learning this is $G_{t:t+1} = r_{t+1} + \gamma\text{max}_{a'}\hat{q}(s_{t+1},a',\theta)$. This is the "target value" and there are multiple ways to construct it, giving rise to variants of RL algorithms.

The value $G_{t:t+1}$ is also an estimate of expected return. Technically it is a sampled estimate, which may vary and may be biased (due to starting conditions of the neural network). However, it includes some real experience, so may be used to improve the neural network through training. You can use it like ground truth associated with the state and action, to train the neural network.

The TD target or "target value" gets its name because by updating a Q table or training a NN with it as a ground truth, the estimator will output values in future closer to the supplied value. The estimator "gets closer to the target".


How can there be more true positive than positive?

Of course it is impossible to have a higher than $100\%$ true positive rate in your results.

The author does explain how he arrives at his results, but I must admit it is not at all obvious. In the subsection Performance Metrics of section 2 Experiment Protocol the author explains that the metrics are calculated as

  The TP10 metric is the number of true positives found in the 10 test
  cases that are predicted most strongly by the classifier to be
  positive.

This mean that the TP10 is only calculated using the testing set. The author also explains in the subsection Experiment Procedure of section 2 Experiment Protocol states

  The condition on the second loop ensures that there are 40 positives
  available for training plus at least 10 others in the test set
  (likewise, 200 negatives for training and at least 50 for testing)

So we see that there is actually a minimum of 10 testing instances with which the TP10 is computed.


What are the criteria for updating bias values in back propagation?

Actually, weight values and bias values are updated simultaneously in each pass of backpropagation. That’s because the orientation of loss gradient vector is determined by the partial derivatives of all weights and biases with respect to the loss function. So if in each pass, you want to move in the correct direction towards the minimun of loss function, you must update both weights and biases at the same time and in the correct orientation.


You can simply use df.loc to achieve what you want. Assuming indice is the list of indices where you want to make the changes, do:

main_df.loc[indice, 'column_name'] = new_value


Interpreting Results of Multivariable Regression / how to transform variables to improve results

Your VIF values are extreme, and your residuals plot is a right-opening megaphone (vice a null plot, which is what you want). It looks like the residual plot is being strongly affected by BTC.

If I were working this I would look into Principal Component Analysis (PCA) for two reasons:

1) It can help mitigate your multicollinearity and drive down your VIFs

2) It can reduce the number of predictor variables you're using (linear regression is better on fewer variables)

I would also consider removing the BTC data point just to see what happens. This can give you some insights that can help you build the model. Good luck!


How to modify the resnet 50 in Pytorch (pre-trained) to give Multiple Outputs for Multi-Label Classification

A softmax layer helps:

class ResNet(nn.Module):
def init(self):
    super(ResNet, self).init()
    resnet = models.resnet50(pretrained=True)
    modules = list(resnet.children())[:-1]
    self.resnet = nn.Sequential(*modules)
    self.fc=nn.Linear(2048,10),

def forward(self, x):

x1 = torch.softmax(self.resnet(x), dim=-1)
#x1=self.resnet(x)
#x1 = x1.view(x1.size(0), -1)
#x1=self.fc(x1)

x2 = torch.softmax(self.resnet(x) dim=-1)
#x2= x2.view(x2.size(0), -1)
#x2=self.fc(x2)

return x1,x2

To know more about the dim attribute, corss reference here: https://stackoverflow.com/questions/52513802/pytorch-softmax-with-dim

I hope this what you were looking for.


If you can, it is preferable to use the normal equation to estimate the coefficients for multivariate linear regression. Since the normal equation is a closed-form expression, it will be faster than gradient descent.

Given you have relatively few examples and features, inverting the matrix is not an issue.


ValueError: Expected 2D array, got scalar array instead using predict method

Try:

Y_pred = regressor.predict(np.array([6.5]).reshape(1, 1))

Scikit does not work with scalars (just one single value). It expects a shape $(m\times n)$ where $m$ is the number of features and $n$ is the number of observations, both are 1 in your case.


Variance in cross validation score / model selection

Since you want your model to be a general solution, you want to include all your data when building the final model.  You are correct in saying that keeping the model with the best validation score in the CV is overfitting.  Including these inner random states help generalize your model, and since you have already tuned your model parameters using CV, you can apply these parameters to the final model.

As for feature selection, you want to separate the data used to perform feature selection and the data used in cross-validation, so feature selection is performed on independent data in the cross-validation fold.  This prevents biasing the model.  If you were to select your features on the same data that you then use to cross-validate, you will likely overestimate your accuracy.

Here are some other great posts that help:
https://stats.stackexchange.com/questions/11602/training-with-the-full-dataset-after-cross-validation

https://stats.stackexchange.com/questions/27750/feature-selection-and-cross-validation

Check out Dikran Marsupial's answers to both, they are really good.


The connection between optimization and generalization

What part the optimization alg. (Grad. Descent) plays in generalization of the learning algorithm?

Actually for generalizing you have to find a model that does not overfits the training data. For doing so, there are numerous approaches, like L1/L2 regularization which adds noise to the cost function and somehow weights to prohibit the network from having large weights which may lead to overfitting. Take a look at here. Other techniques are drop-out which adds noise to the activations to help the network not to depend on special nodes. These techniques add some noise to different parts of the network which lead to a cost function with a high error value. After finding the error back prop techniques try to set the parameters/weights to go downhill of the cost function. These techniques help the algorithm not to overfit which means the constructed model will be able to generalize, although you have to test it using cross-validation data and test data. Consequently, optimization techniques by themselves always try to reduce the amount of error and they somehow always lead to overfitting because they try to reduce the cost and this is regularization techniques and other variants that have to be used to construct a cost function which its optimum points do not lead to overfitting. This link also may help you.

  Or the influence of the data [4] plays a more important role? Although [6] sheds the light on the role of the model parameters from a neuroscience point of view in terms of the effect of neurons in generalization, the question on how to learn such models remains unsolved.

Data is the most important part of the learning process. Your data have to be representative enough. Deep learning problems are considered those problems which may have better results if their training data increases. If the model should be able to generalize, it is vital to use data from the real distribution. This is just for training. You can use some techniques in order not to overfit the data using your training data-set. It is customary to add small noises to the input to let the model generalize well. Transforming data by translation, rotation and even distorting image inputs are examples of adding noise to the input to avoid overfitting. Although changing the data is dangerous because it may change the distribution of the real population. So, you can do something with your input data to avoid overfitting and let your model to generalize well.

  So how can we really better understand the generalization considering different factors as mentioned above?

Generalization techniques are used to make the network generalize well. The customary approaches are:

Drop Out
L1/L2 Regularization
Early Stop
Adding Noise and data-augmentation

It should be considered that adding noise to the input should be without changing the distribution. The ratio of signal to noise also should not be small because the information may be lost compeletely. Depending on your problem each of them make work well but I guess there is no consensus which one is the best but in deep-learning the first one is so helpful.

  What is the connection between optimization and generalization (as defined in [5])?

What I'm saying is based on my experience. Optimization itself always lead to overfitting. You have to use generalization techniques to avoid that. To help you figure out the problem, suppose that deep-learning algorithms are able to learn all functions. If you provide them with relatively small number of data, they will memorize a hypothesis, and they do not learn the problem.


Python to clean miswritten words with repetitive letters such as "wwwwooorrrrddss" to "words"

A simple two part solution from this site
remove any letter sequences longer than two (probably not good for welsh)
def reduce_lengthening(text):
    pattern = re.compile(r"(.)\1{2,}")
    return pattern.sub(r"\1\1", text)

print(reduce_lengthening( "finallllllly" ))

Then using pattern.en to check spelling.
from pattern.en import spelling

word = "amazzziiing"
word_wlf = reduce_lengthening(word) #calling function defined above
print word_wlf #word lengthening isn't being able to fix it completely

correct_word = spelling(word_wlf)
print(correct_word)

NLTK is another common toolkit that can help with this


How to tackle a multilabel classification problem

LSTMs, like any other neural net, implicitly support multi-label classification.  You should ensure that your output layer has $n$ neurons, one for each class, and you should use logistic activation rather than softmax activation (which is typically used for the final layer in multi-class problems).
Each neuron $N_i$ in the final layer will output a value between 0 and 1.  If the output of $N_i$ is greater than 0.5, then the example is a member of label $i$.
Here's a pretty good article that might help.


Increase accuracy of classification problem

I'm not familiar with scikit but I'm assuming that TfidfVectorizer represents bag of words features right? By this I mean that it treats all the instructions in an instance as a set, i.e. doesn't take into account their sequential order.

I'm also not familiar with compilers but I'm guessing that the order of the instructions could be a relevant indication? I.e. a compiler may generate particular sequences of instructions.

Based on these remarks I would try to represent instances with n-grams of instructions rather than individual instructions. Then you can still use some kind of bag-of-ngrams representation, possibly with TFIDF, but I would start with simple binary or frequency features. A simple feature selection step with something like information gain might be useful.

[edit] N-grams take order into account locally. In a bag of words model, words (or instructions in your case) are considered individually of each other: for instance the sequence push, push, mov is the same as push, mov, push. With bigrams this sequence would be represented as (push,push), (push,mov) whereas the second one is (push,mov), (mov,push). This means two things:

Higher level of detail about the instance, which can help the model capture the relevant indications
More features so higher risk of overfitting (the model taking some random details as indication, which lead to errors on the test set).


Clustering with multiple distance measures

This is just a wild guess, but I was wondering whether your custom distance function is indeed a distance function.
In particular, your problem might occur because your distance function $d(x, y)$ does not separate observations: This is the case if there exist two observations $x_1$ and $x_2$ that are distinct, $x_1 \neq x_2$, but have zero distance, $d(x_1, d_2) = 0$. Then $x_1$ and $x_2 $ would necessarily be put into the same cluster by your algorithm.
If this happens for many observations in your test set, then you will observe large clusters.

You might be able to check on paper that your function satisfies the four axioms of a distance function (see the Wikipedia article above); if that is not possible, you could at least check it by looping over your test set.


How to model user's buying behavior on Amazon?

How will you validate your algorithm?

Rather than trying to answer the second question, consider that your answer to the first question might need revision...

What methods have you used for validating learning methods throughout your data science class?  First you want to define a specific set of numerical metrics to assess the success or failure of your model. Second, what methods can you use to create a very realistic testing population (more realistic than modeling the population)?  The first hint I will give is that the Amazon Reviews Dataset is very large, so your data is very amendable to this method.  The second hint I will give is that this method is likely the one that you have used in 95% of the supervised learning problems that you have worked on in class...

Hope this helps... I will edit this if need be based on comments added by the OP, but don't want to provide the solution right away in order to elicit some organic thought e.g. since this is a class problem and the Prof is also trying to help you come up with the right solution on your own.


Convert non-binary value into binary value?

What I understood from your question is, for your OneVsAll classifiers, is it ok to convert feature set in to binary or not?

  If this is correct, you should not convert your features into binary.
  From the shared data set, it seems you have continuous data set and if
  you convert into binary one, it will become categorical data, which
  will have different model representation. So according to me you
  should consume your data as it's. Though you can perform data
  normalization.

  Note: Multinominal Logistic Regression works on the similar principal
  i.e. OneVsAll. So you can try this one as well.


Find outliers in Hive - SemanticException

Seems like Hive doesn't let you use avg in a where clause. You can solve this with a subquery.

SELECT id
FROM
    (SELECT id, AVG(id) as avg_id, STDDEV(id) as stddev_id FROM data)
WHERE id < avg_id + stddev_id AND id > avg_id - stddev_id


ggplot (in R)/matplotlib (in Python) with linear model?

I don't know what you mean by "with linear model" in the title, but here's code that generates a toy dataset and replicates your plot.

library(tidyverse)
x<-crossing(year=paste("Year", 1:3), avg=c("Above 4.0", "Below 4.0"))
x$dat<-replicate(6, tibble(wrkday=runif(100, 1000, 2000)))
x %>%
  unnest(dat) %>%
  ggplot(aes(dat)) +
  geom_histogram(aes(y=..ncount..), bins=20) +
  facet_grid(avg~year)

EDIT:

Given your comment I think I understand what you mean by "with linear model."  You want to see how wrkday varies as a function of year, faceted by avg.  Here's some code:

library(tidyverse)

x<-tibble(year=sample(factor(paste("Year", 1:3)), 600, replace=T),
          avg=sample(c("Above 4.0", "Below 4.0"), 600, replace=T)) %>%
  mutate(wrkday=rnorm(600, mean=1000*(as.integer(year)-1)/2, sd=300))

ggplot(x, aes(x=as.integer(year), y=wrkday)) +
  geom_jitter(width=0.2) +
  geom_smooth(method="lm") +
  facet_wrap(~avg)


How to manipulate this column of less than/greater than?

This is a categorical variable. There are multiple ways to handle categorical variables in the literature -

Dummification (mostly for nominal variables)

The variable is split and converted into multiple binary variables (the number of variables should be 1-(number of classes) ).

Python example -

import pandas as pd
series = pd.Series(['<=50k', '<=50k', '<=50k', '<=50k', '<=50k', '<=50k', '<=50k',
                    '>50k', '>50k', '>50k', '>50k', '>50k', '<=50k'])

print(pd.get_dummies(series).iloc[:, 1:])

Output

>50k
0      0
1      0
2      0
3      0
4      0
5      0
6      0
7      1
8      1
9      1
10     1
11     1
12     0

Notice how the information that was string has been now encoded in numbers. The reason was dropping a column is so that there is no perfect multicollinearity.

Ranking(in case of ordinal variables)

These are the cases where you can clearly rank the variables. In your case it would make more sense to rank them as we know then >50k is actually more than <50k.

Use the ranking method in you salary case and the dummification method for nominal cases (like location variables - where it can't be ordered)

Python example -

import pandas as pd
series = pd.Series(['<=50k', '<=50k', '<=50k', '<=50k', '<=50k', '<=50k', '<=50k',
                    '>50k', '>50k', '>50k', '>50k', '>50k', '<=50k'])
uniq = series.unique()
ranks = { uniq[i]:i+1 for i in range(len(uniq)) }
print(ranks)
series_ranked = series.apply(lambda x: ranks[x])
print(series_ranked)

Output

0     1
1     1
2     1
3     1
4     1
5     1
6     1
7     2
8     2
9     2
10    2
11    2
12    1
dtype: int64

Again, in no way is this an exhaustive answer, but rest assured these are the most popular methods used to deal with categorical variables. Hope that helps. Cheers!


How can we use the cosine similarity formula on document feature vector without a direction?

From this
"Cosine similarity measures the degree to which two vectors point in the same direction, regardless of magnitude.

When vectors point in the same direction, cosine similarity is 1; when vectors are perpendicular, cosine similarity is 0; and when vectors point in opposite directions, cosine similarity is -1. In positive space, cosine similarity is the complement to cosine distance: cosine_similarity = 1 - cosine_distance.

For example, the cosine similarity between [1, 2, 3] and [3, 2, 1] is 0.7143."

Also for angle and "Direction", google results says-

Here is a another nice explanation-

https://www.machinelearningplus.com/nlp/cosine-similarity/

by this article-

"When plotted on a multi-dimensional space, where each dimension corresponds to a word in the document, the cosine similarity captures the orientation (the angle) of the documents and not the magnitude. If you want the magnitude, compute the Euclidean distance instead.

The cosine similarity is advantageous because even if the two similar documents are far apart by the Euclidean distance because of the size (like, the word ‘cricket’ appeared 50 times in one document and 10 times in another) they could still have a smaller angle between them. Smaller the angle, higher the similarity."


I understand the data you have is just a bunch of unlabeled images of tires and you need to predict the tread depth.

Your solution 1 requires a different dataset of labeled images. But if you had that, you could use it to train a model and forget about your unlabeled data. If the images are not very similar, maybe you could label a few and use transfer learning to tune the last layers of your network to the actual data.

Solution 2, treating it as a regression problem, also requires labels. But I think it’s the correct approach. You need to label the data. If labeling is a lot of work, then perhaps you can train a model on a different but similar problem. I can’t think of any similar problems that might be useful, so I would say you’re down to putting in some manual labor and label a part of your dataset.

When that’s done, I would expect that a good solution is a CNN with some FC layers to generate the prediction. You’re not going to have too many labels though, so you want to use a model that is as simple as possible.

There could be some domain-related knowledge that you can use, like the fact that tires have markers on them, in the tire grooves, and if these markers are flush with the outer surface, then you know that the tread depth is at or below minimum and the tires need to be replaced. But I don't think that's the question you're asking, so I would insist you need to label the dataset.


For some reason in cross validation we also need to set the label column of the evaluator (even tho it's already set for the estimator. So all you need to do is change BinaryClassificationEvaluator() into BinaryClassificationEvaluator().setLabelCol("churn") where "churn" is the name of your target variable.


Sales Prediction for Fashion Retail Data

If this is your first project start with something simple - e.g. linear regression, lm(sales ~ data). Once you get it up and running you can try something more sophisticated.

I suggest avoiding neural networks in sales prediction - neural networks are sexy, but require huge amounts of data to train properly. This is often not feasible in sales prediction environment.


In ranking applications of information retrieval, training data consists of queries and documents matching them together with relevance degree of each match.

For example when searching in something in google, the training data may be prepared manually by human assessors (or raters, as Google calls them), who check results for some queries and determine relevance of each result.

It is not feasible to check the relevance of all documents, and so typically only the top few documents, retrieved by some existing ranking models are checked.

Training data is used by a learning algorithm to produce a ranking model that computes the relevance of documents for actual queries.

Rank profiles can have one or two phases:

Phase one should use a computationally inexpensive function to rank candidates
Phase two is run on a small candidate set

In short, the query selection and first phase ranking reduces the size of the computation - then machine-learned models can be used on the second-phase ranking on rerank-count documents per node. This makes the ranking scalable (see sizing):

Control the second phase candidate set size
Add content nodes to rank less documents per node

Let's now answer your questions:

  1) Let's say we have qid's in our training file. Does it mean that the optimization will be performed only on a per-query basis, all other features specified will be considered as document features and cross-query learning won't happen?

It means that in order to rank all candidates it will rank only the first ones in order to be computationally less expensive. So it will only look in the first query for the extensive ranking search.

  2)Let's assume that queries are represented by query features. Should we still have qid's specified in the training file or we should just list query, document and query-document features?

Yes, the rank_feature query is typically used so its relevance scores are added to other scores query.

Note that qid's are only used in training set https://github.com/Microsoft/LightGBM/issues/1398

Sources:
 - Wikipedia - Learning to rank
 - Vespa - Tank
 - Vespa - Phased Ranking
 - Elastic - Rank feature query


Shapley contribution when coalition is 0

In the blog post, the author defines the characteristic function $v$ to be the sum of conversion across all subsets of the coalition.  (If I'm understanding the raw data correctly, this is the number of conversions that would've occurred if you limited the channels to that coalition, which seems a sensible measure.)  You can check that in the code definition for v_function.  So in your example, while no customers converted after hitting all four channels, still $v(C_1, C_2, C_3, C_4)\geq 30$, since it includes the sum of the two other channel combinations you've listed.


Dimensionality reduction convolutional autoencoders

Why don't you use a lower number of filters in the last convolution? Instead of 128 you can just choose whatever number you want, e.g. 10.
Also, normally after the convolutional (and pooling layers), you flatten the output (therefore losing the spatial information) and then project with a dense layer onto the final representation space. You can control the dimensionality of the representation space with the shape of the matrices of the last dense layer.


What is advantage of using Dryad instead of Spark?

Dryad is an academic project, whereas Spark is widely deployed in production, and now has a company behind it for support. Just focus on Spark.


Clustering uncertain data with independent uncertainty per dimension

Model the measurement process

For each dimension you could attempt to model the distribution of the measurement process variance. Perhaps every dimension has the same measurement process or perhaps it is distinct. Combine all measurements from all dimensions if the process is the same.

If you have enough data then you can build an empirical model of the distribution of errors using perhaps a kernel based estimator.

Otherwise you might choose to use a Gaussian distribution formed from the mean value of each measurement and an overall measure of the variance of the measurement process. Then you replace observations you have with a sample drawn from a  distribution with the observation mean and overall measurement process variance.


If RAM memory is a problem during training, and you have run for 1500 episodes, it would suggest to me that you are saving some information in each episodes (e.g. appending to a list). That will slowly add up over time and cause a crash.

After a quick look at your code, it looks like the scores and ep_history grow with every loop. You could consider writing that information to disk e.g. to a json, file or pickle object, with something like this

import pickle

if episode % 1000 == 0:
    with open(f"./results_dir/scores_{episode}.pkl", "wb") as file_handle:
        pickle.dump(scores, file_handle)

You should of course retain the amount of history required for your print() statements and computing average scores over time.

You rarely need to worry about freeing up some RAM yourself, as Python (and then afterwards, the operating system itself) will start to allow memory to be overwritten. The results is the same, they are essentially being deleted.


Ordering scrambled 1D data sets by continuity

I ended up soling this by cutting the datasets into segments and then gluing them together. Here are some of the disentangled datasets:

First I cut all the datasetes into continuous segments by identifying the discontinuities (I looked at peaks of the second derivative). I then sowed the segments together based on which segment is the closest to the 2nd order interpolation from the neighboring segment.

There were some special cases that I had to deal with - for example actual divergences in the signal. I dealt with these by identifying segments that cannot be continuously sowed into any other segments (these are the ones going to +-infinity) and gluing them to other segment returning from +- infinity.

Ofri


Clustering combining numeric features and weekday & hour cyclic features

The essential transformation in @AN6U5's answer ist done in the following lines:

  df['hourfloat']=df.hour+df.minute/60.0
  df['x']=np.sin(2.np.pidf.hourfloat/24.)
  df['y']=np.cos(2.np.pidf.hourfloat/24.)

in the first line he transforms minutes into hours by dividing them by 60
so for example 20 Minutes are converted to 0.3333 hours

After that, in line 2 and 3 he converts this float number from polar coordinates to cartesian coordinates (https://en.wikipedia.org/wiki/Polar_coordinate_system)

So chaning this from hour to weekday you just need to adapt the first line.
Imagining a a clock where 00:00 is Monday, followed by Tuesday (clockwise), and so on ... you need to convert hour into weekday (for simplicity I assume weekday has the values 0-7). So first you divide your hour by 24 which transforms it into days and then further you divide it again by 7 which gives you a float number in weeks. Then add your weekday to the hour and then proceed with line 2 and 3 just as given, except that you correct the 24 to 7.

As a formula: hour/(24*7)+weekday = weekfloat
I haven't tried it out by myself but I think this should do it.

Alternatively, when you have two cyclic features you could transform weekday and hour into spherical coordinates. This would leave you with three coordinates, x,y and z but also preserves the 'closeness' within one feature itself.


Is there a model that can adapt to additional new training data with different columns?

A tree-based algorithm can do that.

The point is that you need to train the model with the union of the possible columns that can exist the different batches.

Moreover you need to account for missing values so that the model can learn to recognize a missing and handle them: you need to recode the missings with a proper value, for example you can create a new level for categorical variables and recode the numerical in the standard way (zero, mean, extreme value, etc.)


There exists multiple ways, each consisting on summarizing the embeddings of each word from a document.

The most common ones when using linear models like Logistic Regression are:

max of word vectors
mean of word vectors
concatenation of word vectors (but implies a fixed length)

However, it is important that using such approaches often results in :

either equal or worse performance than using bag-of-words
less interpretability (if applicable)

The reason is the loss of information (especially for long documents).

When using word embeddings (and so called word2vec), the best choice is often to use models that can handle a second dimension in your input.

For that, recurrent and convolutional neural networks have proven to be effective. The former operates at the word level by recurrently feeding the word embeddings, the latter operates at the whole document level (represented by a matrix of $n_{words}\times d_{embedding}$ as you mentioned) by locally convoluting filters with windows of the document.


What are real world applications of Doc2Vec?

I will answer your second question first, doc2vec and word2vec both are primarily good representations of text data that capture the semantics of words and documents. So whenever you are working with text data, you need a representation for it and that is what word2vec and doc2vec provides. Now think of any real world task on text data, like document similarity, using doc2vec you can find cosine similarity between two documents easily, now think of real world applications for it, finding duplicate question on a site like stack overflow, ranking candidate answers for a question answering model, features for text cladsification like sentiment analysis ( word2vec doesnt work well here, the context of good and bad are quite similar so it struggles to differentiate between positive and negative reviews). So these are just representations, you can apply them to any NLP and a lot of IR task.
To answer your first question, a model is not just task dependent but also data dependent. So you can read Mikolov's paper to find out how each model works for the baseline tasks but a good idea is to try both models on your data and extrinsically evaluate which algorithm performs better.


Feature selection vs Feature extraction. Which to use when?

Adding to The answer given by Toros,

These(see below bullets) three are quite similar but with a subtle differences-:(concise and easy to remember)

feature extraction and feature engineering: transformation of raw data into features suitable for modeling;
feature transformation: transformation of data to improve the accuracy of the algorithm;
feature selection: removing unnecessary features.

Just to add an Example of the same,

  Feature Extraction and Engineering(we can extract something from them)

Texts(ngrams, word2vec, tf-idf etc)
Images(CNN'S, texts, q&a)
Geospatial data(lat, long etc)
Date and time(day, month, week, year, rolling based)
Time series, web, etc
Dimensional Reduction Techniques (PCA, SVD, Eigen-Faces etc)
Maybe we can use Clustering as well (DBSCAN etc)
.....(And Many Others)

  Feature transformations(transforming them to make sense)

Normalization and changing distribution(Scaling)
Interactions
Filling in the missing values(median filling etc)
.....(And Many Others)

  Feature selection(building your model on these selected features)

Statistical approaches
Selection by modeling
Grid search
Cross Validation
.....(And Many Others)

Hope this helps...

Do look at the links shared by others.
They are Quite Nice...


Identify social users using graph embeddings

This question can have many ways to answer but here is what I think:

Lets assume the example of very common, facebook.

Very naive sense

Number of friends is equal to the 'social-ness' of the person. In graph context, a node with more degree will considered to be strongly connected in the graph. But ofcourse we know this is not at all a good way of defining the social-ness.

Every human point of view

Ask yourself a question about when do you like or comment in someone's post or photo ? The answer most probably will be when you and that person have good terms or friendship or relationship. So, with this kind of approach you can say that the person with more number of likes/comments in their posts is more social than the one with less likes/comments.

Now, coming to your question, it depends on how you define 'social' as I can also say a person with similar interest as the community with largest number of members with similar interest is most social. So, lets say we have a facebook graph and you want to measure social-ness of vertices you need to exploit the edge information also which is nothing but likes/comments/messages shared between two people. You can convert this edge data into some kind of number to denote the strength of that edge. Also, in the facebook scenario there is a directed edge between any two nodes (people) as it can happen that one person likes/comment/message the another person but vice-versa is not true. So, there are many more kinds of data/information that you can use in facebook graph.

Then as answered in this post graph embedding will give you the representation of a graph/subgraph as a whole. So, you need vertex embeddings in this case so you can represent  every vertex separately and compare against one another.

One approach is to use deepwalk and get vector embeddings based on topology (you need to modify it a bit as the original method is only for undirected graphs). Then you can form another vector like [1050, 300, 500, 2, ...] where each element will be some kind of number denoting number of total likes, number of total comments, number of friends, frequency of messages, frequency of likes etc. You can then train a neural network which basically outputs a number between 1-5, 1 being least social and 5 being most social. So, in this way you can take advantage of vector embeddings.

This is one approach that I can think of. If you have any other thoughts please share them.