Linear SVM in matlab and python giving different results

In Matlab, you are separating a train-test (HoldOut Validation) type of data separation. You get the accuracy of the test set.

In Python, you are making a 10-fold Cross Validation where you get the resulting accuracy of the 10-fold, not using any seperate test set.

This two methodologies are definitely not the same, you need to have the same train-test structures to compare the two fairly.


Validation loss increases and validation accuracy decreases

What you are experiencing is known as overfitting, and it’s a common problem in machine learning and data science.
Overfitting happens when a model begins to focus on the noise in the training data set and extracts features based on it. This helps the model to improve its performance on the training set but hurts its ability to generalize so the accuracy on the validation set decreases.
To deal with overfitting, you need to use regularization during the training. You can try adding dropout layers or batch-normalization layers, adding weight regularization, or you can artificially increase the size of your training set by performing some data augmentation.
You can read more about it in the following post:
What are the possible approaches to fixing Overfitting on a CNN?


Define difference between feature selection and feature reduction

Feature selection refers to deciding on what features to include in model training.

Feature reduction refers to assigning weights to features regarding how important they are for training. In linear regression, this is known as Ridge Regression (shrinking features based on L2 norm), Elastic Net, and Lasso (shrinking based on L1 norm). Lasso can shrink features to zero, so that they are effectively excluded from training (this would be similar to selection iff features are shrunken to zero).

Ridge and Lasso are very well described in the book „Introduction to Statistical Learning“. If you want to know more about Ridge/Lasso, have a look there.


How to find correlation between categorical data and continuous data

For your specific case I would recommend the "grouped mode" because that would be the value you are interested in imputing (I did the same for this kaggle challenge).
On more general terms we have to understand the scale of each variable. We often talk about categorical data but in more detail we have to differentiate between "nominal data" and "ordinal data".
Income brackets are ordinal, that means there is a clear numerical hierarchy, while other data such as the "Embarkment" here is more nominal, that means there is no order or numerical relation.
Therefore a correlation is impossible and it is more helpful to think in terms of grouped distributions.
What does that mean for imputation
Finally besides simply imputing the grouped mode (that is the most frequent value per group) you could try imputation algorithms like MICE which are in a way predicting the missing value based on other variables. This is possible for nominal as well as ordinal categorical factors.
Please be aware that imputation is highly depended on the question of why a value is missing. You have to make sure that the value is not MNAR before you try imputation at all.


Avoid reloading DataFrame between different python kernels

If it's important for your use cases, you could try switching to Apache Zeppelin.
As all Spark notebooks there share the same Spark context, same Python running environment.
https://zeppelin.apache.org/

So what you're asking happens natively in Zeppelin. Or to be complete, it is an option to share the same Spark context / same Python envrionment between all Spark notebooks (they're called 'notes' in Zeppelin):

So you can choose to share context Globally (default Zeppelin's behavior), Per Note (the only possible Jupyter's behavior), or Per User.

If you can't / don't want to switch to Zeppelin, look at other options of sharing common dataframes between your notebooks using:

Apache Arrow
Feather

ps. You can't import ipynb files to Zeppelin currently as of now (it has its own notebook format stored as a json file), until https://issues.apache.org/jira/browse/ZEPPELIN-1793 is implemented; although it's not that hard to convert them manually in most cases.


Churn models often simply predict a binary output: will the student churn? Yes or No, 1 or 0. In your case there is an added component, namely the time factor of 6 weeks, so the question is more "when is the student likely to churn?".

Does your dataset include how long students stayed on the course before churning (i.e. leaving the course)?

At the highest level, you could model this problem as either a classification problem or a regression problem (with some post processing).

Classification approach

If you choose classification, you should form your target variable (the actual churn information) for each student into several discrete classes. For example, you could create 3 classes:

0 -> the student didn't churn
1 -> the student churned later than 6 weeks
2 -> the student churned within 6 weeks

Then you can select any model that can consume your data (input features) and classify each case as one of those three cases. Something like a decision tree might work well as a base line model. You could then perhaps try an SVM model.

Regression approach

In this case, you would be predicting the exact time each student will churn. For this, your dataset must contain e.g. the number of days or weeks into the course that churn students left the course.

In this case, you target variable is then simply this information, and your model will predict numbers, like 3.7 weeks or 26 weeks; you will then need to simply post-process these results into your 3 categories (as listed above).

In this case, again, you could try decision tree (regression variant) or an SVM model.


Binary Classification on small dataset < 200 samples

This post might be of interest. Basically by selecting the model with the best crossvalidation score, you already account for overfitting.

Also, you should separate your dataset into two parts. For the first one (validation) you run the crossvalidation on to select a model, in your case LinearSVC. For the second one (testing) you run crossvalidation again, but this time only with LinearSVC to get unbiased estimates of the accuracy.


You have to create multiple training sessions, in which you selectively freeze the layers you don't want to train. Each Sequential() model can be seen as a list of layer objects. Each of these layers have the argument trainable = True/False. You Freeze the layers you don't want to be trained classically, and proceed to

Then, using Keras Models, you connect layers that are not adjacent as you depicted in the picture. Once all layers have been trained, you transfer the weights into a news Sequential() shell to compose a final model.

It's going to be time consuming, but it's doable.


How to calculate VIF in python 3.6 with scipy 1.0.0?

First, ss module is in scipy.stats not scipy.stats.stats as of scipy 0.17. Also, it is deprecated, but you seem to have figured that out.

Second, in the current linear_model.py on github, the ss package is no longer imported:

from scipy.linalg import toeplitz
from scipy import stats
from scipy import optimize

What version is your statsmodels? Sounds like you may need to update it.


Decision trees are supervised methods, so they need to be trained on some annotated data. Thus the general idea is the same as for any text classification: given a set of documents (for instance represented as TFIDF vectors) together with their labels, the algorithm will calculate which how much each word correlates with a particular label.

For instance it might find that the word "excellent" often appears in documents labeled as positive, whereas the word "terrible" mostly appears in negative documents. By combining all such observations it builds a model able to assign a label to any document.


Recommendations for Data Visualization solution for large CSV file on a non-profit website

Shiny could be an interesting option: it's an R library which lets the programmer generate interactive web pages from an R program.

All the R libraries for data manipulation/visualization can be used (e.g. the great ggplot2 library for graphs)
The interactive pages are rendered with very little effort required on the programmer side
Very flexible, allows quite sophisticated visualizations (see the gallery) and the result looks very professional imho.
It requires the Shiny server to run on the site, it can be installed quite easily in my experience. There is a commercial version but the free version doesn't have any important limitation, as far as I can tell.

Drawbacks:

I'm not sure I would recommend this to somebody who doesn't have any experience with R, it could be a quite steep learning curve to learn the language just for using Shiny.
It might not scale well with a high amount of traffic, depending on the application and the memory resources.


Which machine learning/AI method is this problem related to?

For example I should delete words like and, the, etc. ...

In natural language processing, this first task is called stop word removal. You can identify them by looking at the words' frequency over the documents; uninformative words appear very often.


Lambda Architecture - How to implement the Merge Layer / Query Layer

What you are asking about is, in my view, the main problem of implementing a lambda architecture.  Here are some suggestions on how to solve it.

The combination of Spark and Spark Streaming largely supersedes the original lambda architecture (which usually involved Hadoop and Storm).  Read here an example of how to use a SparkContext and a separate StreamingContext to produce different RDDs, one for batch processed results and another for real-time results.

Once you have replicated that in your system, you still have to think about how to query both kind of RDDs.  The trivial case would be to just union both of them:

scala> rdd1.union(rdd2).collect

Or maybe you can create a new DStream, similar to stateStream in the linked example, where some keys are kept for real-time results, and others for batch results.


Approaches to Bag-Of-Words Information Retrieval

While reading blogs and papers is helpful to identify the latest and greatest, having a solid foundation helps a lot, too. But I assume you already have gone over Manning's great (and free in e-book form) book on IR, right?

http://nlp.stanford.edu/IR-book/

It contains information on creating your own thesaurus from your document collection to solve synonymy problems, LSA for polysemy, etc..

As for similarity measures, you will see there that Okapi BM25 (Robertson et al.) is considered superior to cosine similarity (but more expensive to implement and run).
Regarding the current state of the art, there was a small emergence of Bayesian Network-based classifiers in the early nineties (starting with Turtle & Croft), but that went quiet for a while.
However, right now, using BNs for IR is again finding some revival, particularly in  biomedical IR.
In that respect, I think most ongoing work is directed towards using Bayesian models incl. topic models and deep learning for word-sense disambiguation (WSD) and semantic similarity.
Here is a pointer to a recent paper with good references on the topic.

http://arxiv.org/abs/1412.6629


how to build a lift chart using scikitplot

The mistake is not to have read the documentation ;)

scikitplot.metrics.plot_lift_curve(y_true, y_probas,...)
Parameters:

y_true (array-like, shape (n_samples)) – Ground truth (correct) target values.
y_probas (array-like, shape (n_samples, n_classes)) – Prediction probabilities for each class returned by a classifier.

The second parameter y_probas should be a double-dimension array of the predicted probabilities for every class, not a single dimension array of the predicted class.
You can also check the example provided in the documentation which contains:
y_probas = lr.predict_proba(X_test)


One idea that could be leveraged - use all the model answers and learning material, find some sort of semantic similarity between all that text and the student answer. The higher the similarity with a model answer with a specific grade tag, the higher the probability of assigning that answer that particular grade. You can couple this with a metric of similarity between the student answer and learning material, more similarity between the two, higher probability of a better grade.

This seems like a very hard problem considering it is still a very active area of research in NLP, Some tools or concepts that might be interesting to you - Latent Semantic Analysis, word embeddings, text/document similarity, semantic coherence.


How do I produce generative neural networks?

Feed-forward neural networks are discriminative models, i.e. they model $P(y|x)$.

Something like restricted Boltzmann machines look a lot like neural networks, and share some of the proprieties, and are generative models, i.e. they model $P(y,x)$ --- which, of course, you can always turn into a $P(y|x)$ using Bayes' rule, but the opposite is not true.

Libraries like Theano have support for RBMs.

EDIT: since the writing of my reply, generative adversarial networks have been invented. (Actually, they were invented in 2014, but only in 2016 were they taken to their full potential.) In these networks, you feed noise to the neural network and it generates data samples. This is achieved by co-training it against a regular discriminative neural network. Both networks are trained at the same time: the discriminative one must classify each sample as being either true or synthetically created by the other neural network. And the generative neural network must fool this one, so they get better and better, and, in the end, you have both a discriminative and a generative model!


Is the answer already there?

"
The function  can capture non-linear interactions between features. You can think of the lattice parameters as the height of poles set in the ground on a regular grid, and the resulting function is like cloth pulled tight against the four poles.

With  features and 2 vertices along each dimension, a regular lattice will have  parameters. To fit a more flexible function, you can specify a finer-grained lattice over the feature space with more vertices along each dimension. Lattice regression functions are continuous and piecewise infinitely differentiable.

"
EDIT-

Specifically for these pictures, they have given the example of dateset ranking restaurants(links are below). These graphs shows number of reviews(Y-Axis)/Average rating (x-axis).It has source code also.
https://www.tensorflow.org/lattice/tutorials/shape_constraints
https://www.tensorflow.org/lattice


How is computed the train data set score and why?

Overfitting

Computing the training score and the test score is usefull to detect overfitting

for exemple here we can see that the training score (in blue) keep improving but at some point the test score (in red) stop improving, that's the sign of overfitting.

But for a final model the training score is not usefull and only the test score should be used.


Adjust predicted probability after smote

No, any adjustment to the probabilities will presumably be monotonic, so the rank-ordering of the predictions will be the same, so the AUC will be the same.
See, e.g., https://datascience.stackexchange.com/a/58899/55122

See also the more complex "probability calibration" techniques.

Also, if you see better results after smote+undersampling, and can share your data and work, I'd be very interested.  I haven't yet seen an example where training on the original dataset doesn't do just as well (with appropriate thresholding).


It looks fine to me :)  the only problem is that your plot (resulting from In [18]) is being displayed on your computer in a separate window somewhere - maybe you have to find it. Once you close that window, your iPython prompt woill return to In [19]. You could alternatively press Ctrl-C in the iPython session, but this will end the session.

If the problem persists (and you cannot even find/close the plot from plt.show()) - maybe have a look at this post, which discusses the various backends used for matplotlib.

I assume you do this to begin with:

import matplotlib.pyplot as plt

plt is a module (as the error message says), so you cannot use it directly like that, but rather the functions that are contained in that module, like scatter() (which worked for you) and then others like plot(), hist() and so on.

Have a look at the relevant documentation with lots of exmaples.

Here is a full example:

import matplotlib.pyplot as plt
import numpy as np

# evenly sampled time at 200ms intervals
t = np.arange(0., 5., 0.2)

# red dashes, blue squares and green triangles
plt.plot(t, t, 'r--', t, t**2, 'bs', t, t**3, 'g^')
plt.show()


What are the elements in a BERT word embedding?

Some points first:

BERT is a word embedding: BERT is both word and sentence embedding. It needs to be taken into account that BERT is taking the sequence of words in a sentence into account which gives you a richer embedding of words in a context but in classic embeddings (yes, after BERT we can call others "classic"!) you mostly deal with neighborhood i.e. the semantic of the word vector is kind of the average of all semantics it had in the training set.
it did not take homonyms into account: Either a typo from you or I did not understand homonym very well. To be honest had to search it! Google says "two or more words having the same spelling or pronunciation but different meanings like "right" and "write". That is not a problem for word embedding. Maybe you meant something else?!
BERT tackles this problem by taking context into consideration: All embeddings take context into consideration. The difference is in capturing context when the sequence of the words are taken into account e.g. modeling sentence.

About questions:

Is there still one vector for each word token?: It's one vector per token per layer. Means that for finding a single vector for a word you take n layers and sum the values up. As you know, more you go towards later layers (i.e. toward the output layer) information (features) encoded in layers gets richer. One could also concatenate all after each other and get a higher dimensional representation. Be aware that it is completely task dependent. BERT authors tried different things and came up with summing up last four layers for NER task.

The interesting thing in comparison with classic embeddings is that you can now encode the sentence-dependent semantic of a token! Means that in word2vec (trained on general corpus text like Wikipedia) you have 1 vector for the word apple and if you inspect it you probably see that it is in a relation with both iphone and fruit (never tried it. Just made up an example to make my point for you. Let me know if you tried and something else came out!) but with BERT, you can encode the sentences containing similar words in different context and check the encoding of those words in the sentence. You surprisingly see how it captures the semantics!

The last but not the least is this blog post which is the base of my answer.

Hope it helped!


Same confusion matrix when changing DecisionTreeClassifier parameters

I ran your script and this is what was returned for all 3 confusion matrices:
[[4 0 0]
 [0 3 1]
 [0 0 2]]

This confusion matrix indicates to me that your model is working wonderfully on the testing data. There is a single exception where your model predicts the wrong class (the non-diagonal "1").
Since you are giving the same random seed every time, your training and testing indices are the same for all 3 models you are investigating. Meaning: they are all being trained on the same data and tested on the same data.
It is quite possible that, in this test data set, 9 out of 10 samples are very easy to classify, while there is 1 case where it is much more non-trivial. The contrast between trivial and non-trivial might be so stark that adjusting hyperparameters in your DecisionTree isn't yielding any difference in performance.
So this could explain why your output is the same for all those fields you mentioned.
As for the cross_val_score - this method randomly mixes your entire dataset and gives you 5 successive CV trials. As this process is actually random from your first model through to your third (it doesn't depend on your np.random.seed(0)), the splits are different and you will yield similar (but not identical) results.
As a general note - be careful how you're currently using cross_val_score. In your current implementation you are feeding a pre-trained model to a method where it expects an un-trained model, and you are asking it to cross validate. As a result it's likely that cross_val_score is re-training your model in a way that you might not be intending.


How can I merge 2+ DataFrame objects without duplicating column names?

Welcome to the site!

Problem 1 seems to be an issue that could be solved by treating your dataframes like tables in a relational database. By this I mean: you have your main data table and secondary tables called 'headers' and 'supporting_data', and you store each header in the 'headers' table once with an ID (a key, maybe just an integer) assigned to it, and in the main table you just store the integer corresponding to the header data. If you ever need to refer to the header data, you look up the integer and go over to the 'headers' table to retrieve it. You can do something similar with the supporting data, but not knowing what that data is it's up to you to figure out if you should parse out parts of that data or store it as whole text blocks or whatever. But if you do this, you'll just have two integers on each row of the main table instead of all that superfluous repeated data.

Problem 2 I'm less sure about, but I'd look into the pandas concat function. I think you should probably just have a case column containing 'A', 'B', etc, and be appending the rows from those files into the main table; concat should be able to do that if all the data types work out, etc, etc.

Good Luck!


Try adding dropout layers to your network, this should work very well to reduce the amount of overfitting of your network.


Structure the dataset for financial machine learning

You are going to have to consider three different factors:

1 - What data are you going to have available when you run your predictions? Are you going to have to pre-process that data? Are you going to have the time to do it? You should be setting your focus on this and work backwards from what runtime predictions look like

2 - When it comes to time series, you have to think of it in terms of (1) lookback windows, or how many periods prior are you considering and (2) time shifts, or how many periods forward are you predicting? This can result in an amazing number of combinations for you to model. You should end up with data sets where you have n features for X time periods and your target variable (your labeled data) is the result Y time periods from X where you are creating a prediction.

3 - The cardinal sin of time series is that you should never model on data that was not available at the specific runtime date. It's a common beginner mistake where you mix up your time periods and end up taking in data that was not actually available when it was released; in other words, it will not be available to you (or at least not correct) when you go to make a prediction. This can be common in financial data where entities can come back and "re-state" numbers at a future date. You have to make sure you are modeling on data as it stood on the target date you are modeling.

With these thoughts in mind, you should be on your way to prepping your data in a way that makes sense.


Which observation to use when doing k-fold validation or boostrap?

I recommend using the second option you presented.  I would use $T$ with 10-fold CV to select my modeling technique and optimal tuning parameters.  Take a look at what performed the best ("best" being the model that gives us the best error, but also doesn't have the error fluctuate too much from fold to fold).  After selecting a model, you can use the model on $V$ to get a realistic error rate.

The reason I don't recommend the first option is: There are varying degrees of over fitting that can occur when going through model selection and model tuning, then using that same data to get an error rate.  CV is a great way to limit this overfitting and it gives us a sense of performance variance which is great, but a classic hold-out validation set is the gold standard for model performance.  In your case the first option might not be wrong (depends a lot on data/techniques), but if a hold-out validation set is available I would go for that.


Should Feature Selection processes be apply on training data or on all data?

Like any preprocessing step, feature selection must be carried out using the training data, i.e. the process of selecting which features to include can only depend on the instances of the training set.

Once the selection has been made, i.e. the set of features is fixed, the test data has to be formatted with the exact same features. This step is sometimes called "applying feature selection" but it's an abuse of language: it's only about preparing the test data with the features which were previously selected during the training stage.

Applying feature selection on the test data is a mistake because the training depends on it, so that would mean that the model "has seen" the instances of the test set and therefore invalidate the results on the test set.


Why Gradient methods work in finding the parameters in Neural Networks?

If you have a convex cost function, gradient descent will find the global minimizer. Typically, the cost functions associated with neural networks are not convex - technically. However, if you read this question on Stats.SE, you find that the cost function of many neural networks is "close enough" to convex so that gradient descent still works. You'll find, technically, a local minimizer, but neural networks are often invariant to node-switching (as mentioned in the answer to the question to which I linked), so that you're still finding a very good minimizer.


Search and Replace in specific column of csv with python

You can do it easily by using pandas, like this:

import pandas as pd

data = [['Austria', 'Germany', 'hob', 'Australia'],
        ['Spain', 'France', 'Italy', 'Mexico']]

df = pd.DataFrame(data, columns = ['A','B','C','D'])

# Values to find and their replacements
findL = ['Australia', 'Mexico', 'United States', 'hob']
replaceL = ['Kangaroo', 'Spider Monkey', 'Eagle', 'Test']

# Select column (can be A,B,C,D)
col = 'C';

# Find and replace values in the selected column
df[col] = df[col].replace(findL, replaceL)


Tokenisation with Spacy - how to get tokens to the left/right of token

the token.lefts and token.rights attributes return a generator of the immediate children of the word, in the syntactic dependency parse. It does not just return the tokens on the left and right of the given token.

see : https://spacy.io/api/token#rights

If you want the adjacent tokens for a doc, you can do :

for i in range(len(doc))[1:-1]:
    print(doc[i-1], doc[i+1])

It will print the adjacent tokens for all tokens of the doc, starting at the 2nd token and finishing at the penultimate one.


Implementation of Tree Kernels in Python

I assume that the ones from here are Moschitti's version, as the author is associated with Moschitti in some papers. I'm unable to confirm, though.


Response variable with very high number of class levels in R

Cluster your n response classes into m clusters, where m < 1,000. With an incoming instance i assign it to one of the m clusters using your favorite classification algorithm. Once the instance has been assigned to a specific cluster, you just need to run it through a classifier trained on the classes that comprise that particular cluster.

I've used this particular architecture to assign texts to one of n possible "emotions", which I illustrate below. The incoming text was first scored with a binary classifier trained on positive & negative texts. Given the text's probability of being positive (as predicted by the classifier), the text was then routed to either a series of all positive or a series of all negative emotion classifiers where it was a assigned a specific emotion.

This same sort of design should work your case. First you assign your instance a cluster. Given that assignment, send your instance to the next stage - where the final classification occurs.


Question-answering (QA) is sometimes used to refer to the task where the input to the system is a question and a list of possible answers (normally only a handful) or a paragraph where the answer is supposed to be found, and the expected answer is the index of the correct answer or the start/end positions where the answer located within the text.

In chatbots, the task is normally defined more generically as "answer the given question". This can be approached in many ways, like with chit chat chatbots, where you simply train your system with pairs of question and answer much like a machine translation system, or with information retrieval chatbots, which make use of documents to identify the relevant pieces of composes the answer based on them.

Therefore, in the usual QA definition, QA is much more constrained (i.e. easier) than chatbots.


Strange behavior in current Keras setting (tensorflow-1.7.0 Keras-2.1.5) when using CPU

I recently had a similar trouble with porting working models to tensorflow 1.7. I found that the initialization was the culprit. Adding
model.add(Dense(2048, activation='relu', kernel_initializer=initializers.glorot_normal())) instead of model.add(Dense(2048, activation='relu')) fixed the problem.


How to construct a Decision tree in R where the training data has a frequency associated with each class

To use the column named Freq as your case weights, you can call rpart with the argument weights=Freq.


Exporting R model to OpenCV's Machine Learning Library

Instead of exporting your models, consider creating an R-based interoperable environment for your modeling needs. Such environment would consists of R environment proper as well as integration layers for your third-party libraries. In particular, for the OpenCV project, consider either using r-opencv open source project (https://code.google.com/p/r-opencv), or integration via OpenCV C++ APIs and R Rcpp package (http://dirk.eddelbuettel.com/code/rcpp.html). Finally, if you want to add PMML support to the mix and create a deployable-to-cloud solution, take a look at the following excellent blog post with relevant examples: http://things-about-r.tumblr.com/post/37861967022/predictive-modeling-using-r-and-the.


Cross validation and evaluation: neural network loss function continuously decreases in cross-validation

In the evaluation type B approach, your neural network weights and biais are not reset before each loop of cross-validation. The neural network is then learning from one loop to another, so you see the MAE continuously decreasing.
A solution is to store your weights and biais before fitting the model and load them at each loop so they have the same init.
You can use those methods to do so:
model.save_weights('model.h5') # right after model instantiation
model.load_weights('model.h5') # in the loop before fitting

In evaluation type A, because you instantiate the model in the loop, weights and biais are reset so you don't see the phenomenon.


This is an interesting question. Fixed effects have been developed in a linear model world (OLS-like). So maybe the twist in a highly non-linear world (like NN) is a little different. As far as I know, NN has not yet had too much impact in econometrics. So it seems that there still is need to bridge some empirical concepts.

Let's think about fixed effects (FE). FE uses the "within variation" of each subject to which a FE is applied to estimate something. So what is the problem with deriving this within variation (the change from time t=1 to 2, e.g. x2i - x1i) and plug it into NN manually. Remember that an NN without any hidden layer is just a linear like model. If you do something like this you may probably not need a LSTM layer since all that is relevant in this setting is the change from t1 to t2. Normal dense layers will do, I guess.

Another alternative would be to use "dummy FE", so just plug in one dummy for each subject to be controlled by fixed effects. This data representation could probably work well in an LSTM setting. However, I think we cannot strictly speak of a fixed effect model in this case, since - by the nature of NN - there is no "separate constant term" (like in OLS) for each subject. But is this really a problem? I don't think so, because what you still do in the end is, to control for each subject separately.

A more fundamental question is what you gonna do with the results. FE in economics are primarily used in causal modelling (FE should make sure that certain model assumtions are met). In NN - generally a predictive modelling setup - there are way too many "coefficients" (weights) to make a meaningful causal statement. So I guess that its not so much about the strict econometric meaning of FE, but more about adequate control for FE in NN. In this case, adding dummies for each subject should be a working solution.


Deep Learning vs gradient boosting: When to use what?

Why restrict yourself to those two approaches? Because they're cool? I would always start with a simple linear classifier \ regressor. So in this case a Linear SVM or Logistic Regression, preferably with an algorithm implementation that can take advantage of sparsity due to the size of the data. It will take a long time to run a DL algorithm on that dataset, and I would only normally try deep learning on specialist problems where there's some hierarchical structure in the data, such as images or text. It's overkill for a lot of simpler learning problems, and takes a lot of time and expertise to learn and also DL algorithms are very slow to train. Additionally, just because you have 50M rows, doesn't mean you need to use the entire dataset to get good results. Depending on the data, you may get good results with a sample of a few 100,000 rows or a few million. I would start simple, with a small sample and a linear classifier, and get more complicated from there if the results are not satisfactory. At least that way you'll get a baseline. We've often found simple linear models to out perform more sophisticated models on most tasks, so you want to always start there.


It is a problem specific and data specific problem. For one problem, 95% accuracy may be good, for some other it might not be very good and there might still be room for improvements. Likewise, it might happen that the problem is well defined, your architecture is good but due to a lot of noise in the data, you can't do better than 95%. In that case, we will say the results are good.

Ensembles: Ensembling is a general term for combining many classifiers by averaging or voting. It is a form of meta-learning in that it focuses on how to merge results of arbitrary underlying classifiers. Generally, ensembling improves the final results but again, not necessarily. If you are ensembling two highly correlated models, your ensembled model will give almost the same result as your stand-alone models.


Activity recognition with binary sensors

First you need to find a way to formulate this as a classification problem, where you have pairs of features (from your sensor data) and labels (the activity).

Bag of features

A simple way is to cut up the observations into a series of time-windows, where inside each window you ignore the temporal parts of your sensor data (a "bag of features" representation). Keep track of the current (or most common) state of each sensor, and count the number of sensor transitions that happened (ex: C08open->close: 2 times).

To select the appropriate length of time-window you should do some Exploratory Data Analysis to find typical activity durations.

Both during training and prediction, the time-windows should be computed with overlap.
Say that each window is 5 minutes long. Then the next window contains the 4 last minutes, plus 1 new minute of data. Overlap is usually in a range of 50-90%.

Note that some activities are quite time-dependent. Like make dinner or make lunch. So you should include also a feature time_of_day which is seconds/minutes since midnight.

Temporal models

Some transitions between activities are probably much more common that others, and this is a source of information that can be used. Example: making dinner -> having dinner probably way more likely than making breakfast -> making dinner.
A Hidden Markov Model (HMM) would be the standard way to model that. It can give you the most probable sequence of activities for a whole day.

There are some ways to use CNNs and RNNs on these kinds of datasets, but unless the dataset is huge I would probably not try that.


I'm not sure what the state of the art is right now, but I had a similar problem for which I designed a neural network. What I did was make a model that predicted the class probabilities from the inputs (in your case the image) and then made another model that took both the normal inputs and my probability predictions. I saw a very large increase in the accuracy for the second part after I had fed the class predictions into the feature prediction section.

You could design this as one model with two outputs where the class is an intermediate layer but still an output. Your loss function would be adding the loss for the class prediction and the feature prediction. Weighting the loss for the class prediction would cause the model to lean more towards correctly predicting the class over the other features. If you must absolutely predict the class first and then the features, you can first train the model on the class loss, then train it again with the feature loss after freezing the weights associated with/preceding the class predictions.


What expresses if/how two variables are dependent on each other?

I think what the clarification has in mind is that correlation does not imply causation, that is, that two variables are mathematically correlated, does not mean they are dependent on each other.

Here a wikipedia article if you'd like to expand on reading.

Here is a page full of examples where two variables, although highly correlated, are very clearly independent one from the other. For example: Per capita consumption of chicken vs Total US crude oil imports, or Per capita consumption of mozzarella cheese vs Civil eng. doctorates awarded.

Having shared my understanding of the clarification, now going back to your question:

  What is there that is related to if/how two variables are dependent on each other?

I think what does show if two variables are dependent on each other is not correlation alone, but a further analysis based on common sense.

Hope this helps.


Contributions of each feature in classification?

This is called feature ranking, which is closely related to feature selection.

feature ranking = determining the importance of any individual feature
feature selection = selecting a subset of relevant features for use in model construction.

So if you are able to ranked features, you can use it to select features, and if you can select a subset of useful features, you've done at least a partial ranking by removing the useless ones.

This Wikipedia page and this Quora post should give some ideas. The distinction filter methods vs. wrapper based methods vs. embedded methods is the most common one.

One straightforward approximate way is to use feature importance with forests of trees:

Other common ways:

recursive feature elimination.
stepwise regression (or LARS Lasso).

If you use scikit-learn, check out module-sklearn.feature_selection. I'd guess Weka has some similar functions.


Tensorflow, how many layers does this model have?

When talking about feed forward neural networks one distinguishes between input layer, hidden layers and output layer. Usually you have a single input and output layer and one or more hidden layers.
In your case, the input layer with 20 input neurons is not explicitly mentioned in the code but its still there. Further, there is one hidden layer with 200 neurons and one output layer with a single neuron.
So I would say you have a three layer network. But maybe there are other conventions on how to count the number of layers in a feed forward neural network. Some might exclude the input layer.


How often do we see normally distributed data

Your confusion is apt. Normally distributed data doesn't show up that often. Most of the real world datasets are more complex than normal. Many of natural occurring phenomenon (think: height of people in certain population) might be normal. But most of the cases where human behaviour plays a strong role (donations as you mentioned, incomes, peoples preferences) will show other distributions like fat tailed distributions or power law distributions.

But the result that you highlighted talks about a result in statistics called central limit theorem, which states that the mean that you infer from averaging will be normally distributed (irrespective of distribution of data). I'll explain below with an example.

Imagine you want to talk about heights of all males in United States. The first question that you might ask about such data is what's the central tendency (mean). But you may not have data about every male in the United States (getting that data will be too expensive). So you take a sample of let us say 100 people from each state. There's no way for you too know the distribution or shape of heights the entire population (you haven't collected that data!). You take the sample that you collected and calculate average of the heights. Can you make some statements about the how close is the average to the real mean of entire population? The computed average after sampling is a random variable since you will get different answer for every different sample. Central limit theorem says that this random variable is normally distributed with its mean same as the population mean and your estimate will get pretty closer as you have bigger samples. (Central limit theorem works in case of i.i.d. samples, that is samples are independent of each other and they are picked from same population.)

So in summary, normal distribution shows up more often with statistical tests because you are talking about distribution of average of sampled data, not the actual distribution of data (which can be non-normal). In a way, central limit theorem is why most of real world applications of statistics from simulations to election studies (psephology) works!

All the theory above answers the initial part of your answer. But for specifically running ANOVA test, you need the data to be (close) to normal distribution. Look at histogram of the data (yes, I know you have done sampling only once) and you can run tests for normality assumptions.


Can we compare a word2vec vector with a doc2vec vector?

In paragraph vector, the vector tries to grasp the semantic meaning of all the words in the context by placing the vector itself in each and every context. Thus finally, the paragraph vector contains the semantic meaning of all the words in the context trained.

When we compare this to word2vec, each word in word2vec preserves its own semantic meaning. Thus summing up all the vectors or averaging them will result in a vector which could have all the semantics preserved. This is sensible, because when we add the vectors (transport+water) the result nearly equals ship or boat, which means summing the vectors sums up the semantics.

Before the paragraph vector paper got published, people used averaged word vectors as sentence vectors. To be honest, in my work these average vectors work better than document vectors. So, with these things in mind, in this way it could be compared.


Since topic modeling is unsupervised, it's not usually evaluated against labeled data. Instead people devise measures which evaluate the clusters, typically based on how much the most probable words for a topic are semantically relevant.

You might find data ideas in this paper: https://www.cs.cornell.edu/~laurejt/papers/authorless-tms-2018.pdf


Regarding "modification" of feature columns in supervised learning

I think what you're talking about is called compound features, and it's extremely important because sometimes it captures interactions between certain features which are not readily apparent in considering each column vector (or column) independently.

The answer to your question is, it may or may not, there's really no way to know without looking at the data, for example if we have a dataset consisting of the following features $x_1, x_2, x_3, x_4$. You could create many datasets consisting of mathematical operations between these variables, like you have mentioned in the question.

Now for the reason as to why someone would want to do this. Consider a supervised learning scenario where you're trying to classify a nominal feature with respect to some other features. This is a purely hypothetical example, but let's say we're predicting skin color which has $3$ or $4$ distinct classes based on facial features (absolute nose position in a grid, position of left eye, etc). While you may attain low error, it may be worthwhile considering compound features like position of right eye - position of left eye which could give us more information than their absolute positions.

To answer the latter question, there isn't any real algorithm which does this because they are feature combinations which are based on the data and anything that does this would have to be a trial and error mechanism. Because underlying data similarities may not be apparently visible. One way you could solve this is in your elementary data processing phase, consider the actual possible relationships between data, or at least, what you perceive them to be.


In the paragraph following equation (1):

$T$ is the number of leaves in the tree.

$\gamma$ is a hyperparameter that affects how much regularization occurs on the size (number of leaves) of the tree.
Now it turns out that you can interpret $\gamma$ (at least roughly, see note at bottom) as ([source]):

Minimum loss reduction required to make a further partition on a leaf node of the tree. The larger gamma is, the more conservative the algorithm will be.

You can see that from equation (2), the regularized objective:
$$\mathcal{L}(\phi) = \sum_i l(\hat{y}_i, y_i) + \sum_k \Omega(f_k),\\
\text{where }\Omega(f)=\gamma T + \frac12 \lambda \|w\|^2.$$
By making the split, you increase $T$ by one, so the penalty increases by $\gamma$, and so your base loss term $l$ needs to decrease by at least $\gamma$ for this to be an overall improvement.  Note: Of course, this ignores what happens to the leaf weights $w$ in splitting one node into two.


What knoweldge framework supports this type of query?

Well, to answer my own question, it turns out the issues I was struggling with are almost exactly the issues that motivated the development of and that are addressed by graph databases.  I would have been out of luck if this was 1999, but it turns out 2016 is a good time to be alive for databases that go beyond the traditional relational model!  I guess all this NoSQL stuff that the kids are always talking about is more than just hype.  :)


How to estimate a user's gender based on what apps the user download?

If you have data on how many downloads have been done for each category, you can use that data to perform logistic regression, where your target variable would be 0/1 classification of male/female. This will form your benchmark result and then you can improve upon the results. Especially look for feature selection, because there might be categories of apps which have nothing to do with target. You should also try bi variate profiling of variables with target to see if there is good correlation with target.
Try using some advanced techniques once you have done some data exploration.
KNN, Random Forest and Neural network for classification might give good results. Also cart might work fantastically here.


Is there a way to classify an alphanumeric string?

Find a minimal implementation of (binary) text classification based on n-grams below. It's written in base R. You first need to extract the ngrams and apply some model (Lasso in my case) to predict each ngram's class. You can easily expand this to multiclass by changing the model accordingly. Note that this is just a minimal example with room to improve.
See also here: https://github.com/Bixi81/R-ml/blob/master/NLP_ngram_short_text.R
Find a Python implementation of GLMnet here: https://web.stanford.edu/~hastie/glmnet_python/
Data:
# Dummy data
# Note the little differences in the strings, however there is a clear pattern
df = data.frame(text=c("ab13ab12ab16","ag16ag16fg16","ab12ab12ab12","fg12fg16fg16","ab16ab12af12","fg16fg16fg16"),target=c(1,0,1,0,1,0))

head(df)

          text target
1 ab13ab12ab16      1
2 ag16ag16fg16      0
3 ab12ab12ab12      1
4 fg12fg16fg16      0
5 ab16ab12af12      1
6 fg16fg16fg16      0

Ngrams:
# Set up lists to post results from loop
ngrams = list()
targets = list()
observation = list()

# Loop over rows of DF
counter = 1
for (row in 1:nrow(df)){
  # Loop over strings in first colum per row
  for (s in 1:nchar(as.character(df$text[[row]]))){
    # Get "ngram" (sequence of two letters/digits)
    substring=substring(as.character(df$text[[row]]), s, s+1)
    # Append if >1, also post row and target
    if (nchar(substring)>1){
      ngrams[[counter]]<-substring
      targets[[counter]]<-df$target[[row]]
      observation[[counter]]<-row
      counter = counter+1
    }
  }
}

# Lists to DFs
ngramdf=data.frame(ngram=matrix(unlist(ngrams), nrow=length(ngrams), byrow=T))
targets=data.frame(ngram=matrix(unlist(targets), nrow=length(targets), byrow=T))
obs=data.frame(obs=matrix(unlist(observation), nrow=length(observation), byrow=T))

# Get dummy encoding ("one hot") from all the ngrams
dummies = model.matrix(~ . -1 , data=ngramdf)

# Bind target and dummies to train DF
train = cbind(targets,dummies)

Estimate some model:
# Now apply Lasso to predict the classes
# https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html
library(glmnet)
cvfit = cv.glmnet(as.matrix(train[,-1]), as.matrix(train[,1]), family = "binomial", type.measure = "class")

# Predict class per ngram
classes = predict(cvfit, newx = as.matrix(train[,-1]), s = "lambda.min", type = "class")

# predicted probability per ngram -> needs (type="response")
probs = predict(cvfit, newx = as.matrix(train[,-1]), s = "lambda.min", type = "response")

# Calculate average prob that a sequence (so ngrams per row) belongs to some class
result = cbind(probs,obs)
# Get mean per row
result = aggregate(. ~ result$obs, result[1], mean)
# Bind original text and target
result = cbind(result,df)
colnames(result)<-c("observation","estimated_prob", "text", "original_target")

Result:
  observation estimated_prob         text original_target
1           1      0.7173567 ab13ab12ab16               1
2           2      0.3020248 ag16ag16fg16               0
3           3      0.7674571 ab12ab12ab12               1
4           4      0.2922798 fg12fg16fg16               0
5           5      0.6815783 ab16ab12af12               1
6           6      0.2393033 fg16fg16fg16               0


Why are mBART50 language codes in an unusual format?

It encodes the language and its regional variant, the same way as locales are encoded. hi_IN then means Hindi as spoken in India, en_US would mean American English, en_GB British English. My guess is that en_XX means English in general.
Anyway, the first part of the locale code is the ISO 639-1 language code which is the same as langid uses.
Btw. langid works fine for documents, but not that well for isolated sentences. For isolated sentences, pre-trained FastText classifier delivers much better results.


when can xgboost or catboost be better then Logistic regression?

XgBoost often does better than Logistic Regression. I would use CatBoost when I have a lot of categorical features or if I do not have the time for tuning hyperparameters.

You should invest time in a boosting model for sure (they will always take more time than Logistic Regression) because it is worth it. If you are impatient, try CatBoost instead of XgBoost to see the improvement in accuracy. That will definitely give you the motivation to spend more time tuning your boosted trees.


Alternative means of clustering streams of incoming facial recognition data

You are describing incremental learning, input data is continuously used to extend the existing model's knowledge.
There is a Python implementation of incremental DBSCAN.
There is no current Python implementation of incremental HDBSCAN.


How to incorporate the uncertainty of the model coefficients in the prediction interval of a multiple linear regression

One possible solution is Bayesian linear regression. Bayesian linear regression estimates a posterior distribution for each coefficient. From that posterior distribution, a credible interval can be calculated.


python xgboost DMatrix - get feature values or convert to np.array

Currently we cannot direct extract data matrix from DMatrix, mainly due to DMatrix is a internal data structure that may or maynot sit in memory.

--by tqchen commented on 3 Aug 2015

(Source)


Is it correct to use non-target values of test set to engineer new features for train set?

Definitely (2). You should not include the testing values when calculating features over the data (or normalizing or scaling the data, etc).

Let's take a step back for a second. Why are we holding out a test set at all? Well, we're training a model on a dataset so that in the future, when we collect more data from the same domain, we can use our model to predict the target for that new data. We don't just want to do as well as possible on the data that we have, we already have the labels for it. So we hold out a test set to measure the performance on, so that we can get a sense of how well the model's predictions generalize to new data.

If we give the model information about the test set (which putting the test samples into a mean calculation would do), we'd improve the model's performance against that particular test set -- but presumably not against any new data that was collected later, when the model was actually deployed and in use! All we'd be doing is making our performance metrics less effective measures of whether or not the model generalizes.

So even while the performance would appear better, it would both possibly not reflect an actual improvement (depending on the actual distributions of train, test, and post-deployment collected data), and would harm our ability to say for sure whether our choices in model design improved our generalization performance at all.

I actually talked about this a little bit with respect to feature scaling in this answer.


regression network that will be optimised on a subset of the data

Your reasoning sounds similar to curriculum learning: the idea is to pass training data to the network not randomly, but based on some scoring function $f$. It was proven that a network learns faster and better, even though the process itself is not well understood. I think it is a better approach than the one you suggested: I am afraid that, if you pass only poorly predicted data after a while, the network will forget what learned before.
What you can also do is to perform data augmentation to compensate your unbalanced data (if I understood well, higher ranges in your dataset are not well represented compared to a certain range).


How to make an effective sampling from a database of text documents?

Off the top of my head, I would consider, at least, two approaches, as follows.

Graph sampling. If you can (and it makes sense to) model your population (database of text documents) as a graph, then consider graph sampling. Check this relevant answer of mine on the topic on Cross Validated site (contains many references - I would start with this nice overview paper).
Topic modeling. Another alternative is topic modeling (also referred to as topic models). See this introductory paper as well as this more detailed paper on the subject. Also, take a look at this interesting paper on topic modeling of streaming documents (or other situations, such as large databases, when unsupervised solutions make more sense). Finally, speaking about software for topic modeling, while many options do exist, I suggest investigating MALLET, which is an interesting Java-based software package for "statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text".


Why do a lot of people use ipython notebook over python file when doing analyzing data? Is it the same in industry?

iPython notebooks are great for some cases. I use them because of:

Easy in-place editing and immediate execution, very friendly for quick and experimental stuff
In-place visualization. Also, ability to have multiple figures on the same page, compare them, re-run figures, move the cells. Much more convenient than multiple and independent OpenCV's imshow windows.
They are more convenient for step-by-step iterative analysis
Since they are collection of code blocks prepended with a description and followed with the results, they are great for tutorials and guides.

You can compare the notebooks to test stands, where you can quickly assemble something working to demonstrate to the others, while Python modules are more like building blocks to be used either for demonstration or for production later. Thus, combination of both is very powerful.


train_test_split() error: Found input variables with inconsistent numbers of samples

You are running into that error because your X and Y don't have the same length (which is what train_test_split requires), i.e., X.shape[0] != Y.shape[0]. Given your current code:

>>> X.shape
(1, 6, 29)
>>> Y.shape
(29,)

To fix this error:

Remove the extra list from inside of np.array() when defining X or remove the extra dimension afterwards with the following command: X = X.reshape(X.shape[1:]). Now, the shape of X will be (6, 29).
Transpose X by running X = X.transpose() to get equal number of samples in X and Y. Now, the shape of X will be (29, 6) and the shape of Y will be (29,).


How does Tensorflow compute gradients of reduce_min operation?

A minimum operation is differentiable, or at least you can easily express the partial derivatives w.r.t. its inputs:

$f = min(x_1, x_2, x_3 ... x_n)$

$ \frac{\partial f}{\partial x_i} =
\begin{cases}
    1,& \text{if } argmin_i(x_i) = i\\
    0,              & \text{otherwise}
\end{cases}$

This does not hold strictly when multiple values share the same minimum value. However, that is not a problem in practice for gradient-based optimisers in TensorFlow, which can simply set all tied indices to have partial derivative of 1 (or a fraction $\frac{1}{n_{min}}$), with little impact to the eventual result, because ties for values will happen rarely. Ties may happen frequently enough in a ReLU-based network that the TensorFlow developers have considered a best response for them - I don't know specifically what TensorFlow does for that situation.


K-Means visualisation problem 8 numerical feutres

It's not clear enough what you try to do. If I understand correctly, you want to train a K-Means clustering and visualize the results. However, you have 8 dimensions in your dataset and obviously, you cannot plot such a space.

What you can do is to reduce the dimensionality in 2 dimensions and then create that plot.

For example,

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans

# read my data with pandas into a dataframe
data = pd.read_csv("data.csv")

# run a KMeans model with 3 clusters. Change that number to what you want
clustering_kmeans = KMeans(n_clusters=3, precompute_distances="auto", n_jobs=-1)
clusters = clustering_kmeans.fit_predict(data)

# run PCA to reduce the dimensionality to 2 dimensions
reduced_data = PCA(n_components=2).fit_transform(data)

# create a new dataframe that contains the 2 dimensions and the cluster label
results = pd.DataFrame(reduced_data,columns=['pca1','pca2'])
results['label'] = clusters

# plot the results with a scatterplot
sns.scatterplot(x="pca1", y="pca2", hue=label, data=reduced_data)
plt.show()


python print values seasonal_decomposition

seasonal_decompose returns an 'object with seasonal, trend, and resid attributes.'  We can access the data by calling the object:

res = seasonal_decompose(series, model='additive', freq=365)
residual = res.resid
seasonal = res.seasonal
trend = res.trend
print trend

etc...


Will one hot encoding / unbalanced columns cause bias to Clustering Analysis?

To answer your question we need to understand what the aim of the clustering analysis that you are doing. Some of goal's of clustering analysis are:

Outlier Detection,
Pattern Detection,
Grouping Data together, etc

Now depending on the type of data, we can choose the algorithm that best fits the data at hand. If you have only numerical features, then you can go for KMeans. If there is a only categorical features we can choose K-Prototypes algorithm and if there is a mix of both categorical and numerical, we can use the K-Medoids algorithm.

To answer your question specifically, we need to choose an appropriate clustering algorithm while using one-hot encoded data to avoid the issues that you are worried about. Since one-hot encoding is intended for categorical features, we have to go with an algorithm that is specifically designed for that. For example K-Prototypes or K-Medoids.

I would recommend you to understand the exact business problem that you are trying to solve. Correlate that to the data at hand and then use an appropriate mentioned algorithm to solve the problem that you are facing.

Regards,


Can colors be detected using Neural Nets?

Normally color spaces are not considered to be one dimensional.

Given three types of human cone cells, the most natural approach is probably not to use a single but three neurons. If color input is encoded as RGB values, then one neuron would be for red, one for green, and one for the blue channel.


How to rank documents using Bag of words approach

If you're just looking to rank documents according to how many appearances your words w1,..,wn contain, then there's no need for clustering or machine learning in general: Clustering your 50 results will give you a partition of these results into clusters containing results that are similar to one another and different from the results in other clusters.

If you just need to rank by word occ, just count the frequencies of your words in each document (including synonyms, which you can get e.g. from Wordnet automatically if you prefer) and sum them up.


anomaly detection - conceptual help on machine learning

There is two options:

You have a label

Then what you probabibly have is a binary classification problem, probably unbalanced. If this happens follow this https://towardsdatascience.com/practical-tips-for-class-imbalance-in-binary-classification-6ee29bcdb8a7

You have no label

Then you have an unsupervised learning problem, in finance Isolation Forest is really used. If this is the case have a look at this https://towardsdatascience.com/outlier-detection-with-isolation-forest-3d190448d45e


How to encode high cardinality categorical data?

You may want to look into Target Encoding as an example:
https://contrib.scikit-learn.org/category_encoders/targetencoder.html
Another post from the forums:
Problem with converting string to dummy variables


PCA and k-means for categorical variables?

No - PCA and k-means can not be used on categorical variables. Both PCA and k-means require numerical variables.


Best approach to find optimal solution to linear equation by group in R

You could use linear regression without intercept in order to accomplish this.

If I understand correctly, your implementation of linear regression in Update 3 had two issues: first, you fit the linear model with intercept, but then drop the intercept when applying the model. Second, there is a typo in the following line

linear_discount = 1/linear_discount-1

In the code snippet below I extended your analyses by two approaches: linear regression without intercept and linear regression with intercept. The former case should correspond exactly to your requirements, i.e. using a single relative discount rate per customer. The latter one corresponds to the case with relaxed assumptions, i.e. one relative discount rate and one absolute discount (independent on the price level).

Note that for the illustrative dataset you provided the optimal (in terms of mean squared error) discount rate would be zero.

df_ext <- df %>%
  # mean hack
  mutate(Discount_indiv = (Old_Price-New_Price)/New_Price) %>% # Identify optimal discount individually
  group_by(Customers) %>%
  mutate(Discount_mean = mean(Discount_indiv),
         Old_Price_mean = New_Price + New_Price*Discount_mean) %>% # Average individual discount to get approximate discount for customer

  # incorrect linear regression (fitted with intercept, then intercept dropped)
  do({ co <- coef(lm(Old_Price ~ New_Price, .))
  mutate(., Discount_wrong = co[2])
  }) %>%
  ungroup %>%
  mutate(Discount_wrong = 1 / Discount_wrong-1) %>%
  mutate(Old_Price_wrong = New_Price+New_Price*Discount_wrong) %>%

  # correct linear regression w/o intercept
  group_by(Customers) %>%
  mutate(Discount_regr = coef(lm(Old_Price ~ New_Price -1))[1] - 1,
         Old_Price_regr1 = New_Price+New_Price*Discount_regr) %>%

  # correct linear regression with intercept (i.e. two discounts per customer)
  mutate(
    Discount_abs = coef(lm(Old_Price ~ New_Price))[1],
    Discount_rel = coef(lm(Old_Price ~ New_Price))[2] - 1,
    Old_Price_regr2 = New_Price + New_Price*Discount_rel + Discount_abs) %>%

  #calculate residuals
  ungroup() %>%
  mutate(Resid_mean = Old_Price_mean - Old_Price,
         Resid_wrong = Old_Price_wrong - Old_Price,
         Resid_regr1 = Old_Price_regr1 - Old_Price,
         Resid_regr2 = Old_Price_regr2 - Old_Price)

#transform data for visualisation
df_gat <- select(df_ext, matches("Customers|Price")) %>%
  gather(key="Approach", value="Old_Price", -Customers, -New_Price)

ggplot(df_gat, aes(x=New_Price, y=Old_Price, group=Approach, colour=Approach, shape=Approach)) +
  geom_line() + geom_point(size=3) + facet_wrap(~Customers, ncol = 1)

select(df_ext, matches("Customers|Disc"))
select(df_ext, matches("Customers|Price"))

#calculate mean square error for all approaches
select(df_ext, matches("Customers|Res")) %>%
  group_by(Customers) %>%
  summarise_all(~mean(.^2))

Results for all approaches: MSE by customer & some visualizations

  Customers Resid_mean Resid_wrong Resid_regr1 Resid_regr2
  <fct>          <dbl>       <dbl>       <dbl>       <dbl>
1 A               1.71        3.62       1.          0.222
2 B               1.71       11.5        1.000       0.222


What are advantages or disadvantages of training deep learning model from scratch?

(Suggestions and edits will be appreciated)

let us discuss advantages of training a deep learning model from scratch:

Building and training NN from scratch is of a great use in the research field.
You will know your model to the most basics and can modify it in case needed as per the requirements.
It will be more efficient in terms of size and training time(because there may be less number of layers as compared to any pretrained model trained for more than one purpose).Best example is VGG16.In case of VGG16 it is trained for 1000 different categories and even if we want only 2 categories we have to train the whole model,because we can't change the layers without deep knowledge about the architecture of VGG16.
For any beginner(like me) it will be beneficial to start building from scratch and the use the pre-trained model.
Don't require mastery in Deep Learning to use pretrained models.

Disadvantages:

Many pre-trained models are trained for less or mode different purposes,so may not be suitable in some cases.
It will take large amount of resources(time and computation power) to train big models from scratch.


Information about LTSM RNN backpropagation algorithm

Here is a good tutorial on LSTM from scratch with forward and backward pass.


Model ensemble with Spark or Scikit Learn

As you said in your question, there is no way to do that with the baseline algorithms provided in MLLib.

Two ways you could do that is by either :

Creating a function to generate a Pipeline
Creating a Meta Estimator that would take your base learners and the forking column.

The first one is what you have stated in your question and the second one has been explained by Brian Spiering.

As you said making a custom Estimator and Transformer will make it works nicely with all the MLLib Tools as Tuning Tools.

If you want a really precise example how to do so, there is my library which implements meta algorithms for Ensemble Learning with Spark.

There is a way to train multiple estimators in parallel quite simply in the Stacking Classifier, and for all of them you have an example on how to ask for an estimator as the parameter (to chose either your logistic regression or decision tree).

You can pick ideas out of that !

Edit : Similar question on StackOverflow


How many neural networks can a GPU run simultaneously?

This is sort of like asking "how long is a rope?". To be able to answer how many neural networks you can run simultaniously on a single GPU one would require a lot of information. For example:

How much memory does your GPU have?
Are we talking training or inference?
If we are talking training, is there a performance requirements and demand for a specific number of samples per batch?
How big is each image that should be processed?
How big is the neural network?
Datatype: float16 or float32 or something else?
Deeplearning-framework used
E.t.c.

If you were to provide this information, one might be able to guess how many you can run in parallell, but it would still be hard.

Best way is if you simply try running your network and monitor the memory usage on your GPU, using for example nvidia-smi, and find the peak memory usage and then simply check of many of that fits on the total GPU-memory. However note that you need to check the specifications for the framework e.t.c. to verify how they allocate GPU-memory (for example many frameworks allocate all the memory that's avaliable)


What to give as predictors to predict future values?

Welcome to the site!

Firstly, when we use any kind of predicting algorithm, then you need to have the future values of the independent variables(features for explaining the target variable), for Prediction.

Secondly, If you don't have such values then you cannot use predicting algorithms, we have forecasting algorithms to do such task. To apply this you need to have data which is time series data. From the sample and graph I think your data is time series data.

FYI, you can also use additional features even in time series. For Feature Engineering on Time Series data you can go through the link.

If you need any additional information let me know.


How to decide optimal threshold for my classification model from FPR, TPR and threshold

I’m willing to bet that you have a 100 value array for both the ROC and the data by coincidence. You aren’t getting a threshold for each data record but for each threshold evaluated.

At this point, you have to define what ‘optimal’ means in your context. By default (in many R packages at least), sensitivity and specificity are equally weighted, in which case the ‘optimal’ threshold corresponds roughly to the threshold where a 45 degree tangent line intersects the ROC curve. You can choose any threshold you want for a given ROC, depending on how you weight sense/spec.  R has a few packages that help compute that; I’d be surprised if Python did not have an equivalent.


Including the dependent variable in your data to perform principal component analysis?

You need to remove the dependent variable before performing PCA.
Otherwise you are essentially using the dependent variable to explain itself.

Also, think about prediction:
You might want to use your model to predict unknown GPA scores based on information about amount of alcohol, amount of study, IQ and SAT. In that case, it is impossible to include information on GPA scores.


Is there a machine learning model suited well for longitudinal data?

Assuming we are not talking about a time series and also assuming unseen data you want to make a prediction on could include individuals not currently present in your data set, your best bet is to restructure your data first.

What you want to do is predict daily outcome Y from X1...Xn predictors which I understand to be measurements taken. A normal approach here would be to fit a RandomForest or boosting model which, yes would be based on a logistical regressor.

However you point out that simply assuming each case is independent is incorrect because outcomes are highly dependent on the individual measured. If this is the case then we need to add the attributes describing the individual as additional predictors.

So this:

id | day | measurement1 | measurement2 | ... | outcome
A  | Mon | 1            | 0            | 1   | 1
B  | Mon | 0            | 1            | 0   | 0

becomes this:

id | age | gender | day | measurement1 | measurement2 | ... | outcome
A  | 34  | male   | Mon | 1            | 0            | 1   | 1
B  | 28  | female | Mon | 0            | 1            | 0   | 0

By including the attributes of each individual we can use each daily measurement as a single case in training the model because we assume that the correlation between the intraindividual outcomes can be explained by the attributes (i.e. individuals with similar age, gender, other attributes that are domain appropriate should have the same outcome bias).

If you do not have any attributes about the individuals besides their measurements then you can also safely ignore those because your model will have to predict an outcome on unseen data without knowing anything about the individual. That the prediction could be improved because we know individuals bias the outcome does not matter because the data simply isn't there.

You have to understand that prediction tasks are different than other statistical work, the only thing we care about is the properly validated performance of the prediction model. If you can get a model that is good enough by ignoring individuals than you are a-okay and if your model sucks you need more data.

If on the other hand you only want to predict outcomes for individuals ALREADY IN YOUR TRAINING SET the problem becomes even easier to solve. Simply add the individual identifier as a predictor variable.

To sum it up, unless you have a time series, you should be okay to use any ML classification model like RandomForest or boosting models even if they are based on normal logistical regressions. However you might have to restructure your data a bit.


selecting variable randomly at each node in a tree in Random Forest

To train a tree in a random forest, at each split m variables are randomly selected from the global variable set, and the variable that maximizes information gain is chosen to be split on. Source: https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf (the original random forest paper).


CNN 3D line angles prediction regression - results of training of phi depend on theta

In this case changing CNN to a simple MLP with 3 layers changed the behaviour of the network completely. It becomes fast to train, accurate and more importantly, phi results do not depend on the range of theta used during the training (although they depend on particular theta value, as expected). This does not explain why in CNN there is dependency. I guess that convolution filters can't separate parts of the phase space, but I don't have intuition why.


How do local minima occur in the equation of loss function?

The equation you used for gradient descent isn't general; it's specific for linear regression.  In linear regression, there is indeed only a single global minimum and no local minima; but for more complex models, the loss function is more complex, and local minima are possible.


This is discussed in stackoverflow, just to recap:

Define your your customized cost function, e.g.:

 def new_cost(y_pred, y_true):
    # perform calculation for new cost
    return 'new_cost', score

Then pass it to the feval argument (see official doc) for training like:

model = xgboost.train(params, dtrain, num_rounds, watchlist, feval=new_cost)

There are many examples out there if you search harder for more concrete use cases.


SVC aims to find a separating hyperplane. That data you have is separable. Thus the hyperplane will go between x3 and x4. This are your support vectors. Now C is a parameter which allows you to make a trade off between errors allowed and the width of separating hyperplane. But your data does not need this trade off since it is already separable. Because of that the needed error to be accepted is zero. This penalty error in svc equations is multiplied with C. Because C is multiplied with 0 it does not matter the value of C, their product will be zero anyway.


How to alter word2vec wikipedia model for n-grams?

To clarify, an n-gram usually refers to a sequence of characters, i.e. the word "clear" is comprised of the trigrams {cle, lea, ear}. I think the term you are looking for is "multi-word phrases".

Embedding collections of words is referred to a couple different ways, including "sent2vec", "doc2vec" or "thought vectors". These terms generally refer to embedding a "complete" set of words,  either a sentence, paragraph, collection of paragraphs, or a document. A common --
if somewhat inelegant -- approach to using a pre-trained word2vec model to embed multiple words is to embed each word separately and then take the average as the embedding for their combination.

I think you'll find this article relevant: Representation learning for very short texts using weighted word embedding aggregation


Cluster documents and identify the prominent document in the cluster?

A very simple approach would be to find some kind of centroid for each cluster (e.g. averaging the distributions of the documents belonging to each cluster respectively) and then calculating the cosine distance of each document within the cluster from the corresponding centroid. The document with the shorter distance will be the closest to the centroid, hence the most "representative".

Continuing from the previous example:

import pandas as pd
import numpy as np
from sklearn.metrics import pairwise_distances
from scipy.spatial.distance import cosine
from sklearn.cluster import DBSCAN
from sklearn.preprocessing import StandardScaler

# Initialize some documents
doc1 = {'Science':0.8, 'History':0.05, 'Politics':0.15, 'Sports':0.1}
doc2 = {'News':0.2, 'Art':0.8, 'Politics':0.1, 'Sports':0.1}
doc3 = {'Science':0.8, 'History':0.1, 'Politics':0.05, 'News':0.1}
doc4 = {'Science':0.1, 'Weather':0.2, 'Art':0.7, 'Sports':0.1}
collection = [doc1, doc2, doc3, doc4]
df = pd.DataFrame(collection)
# Fill missing values with zeros
df.fillna(0, inplace=True)
# Get Feature Vectors
feature_matrix = df.as_matrix()

# Fit DBSCAN
db = DBSCAN(min_samples=1, metric='precomputed').fit(pairwise_distances(feature_matrix, metric='cosine'))
labels = db.labels_
n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)
print('Estimated number of clusters: %d' % n_clusters_)

# Find the representatives
representatives = {}
for label in set(labels):
    # Find indices of documents belonging to the same cluster
    ind = np.argwhere(labels==label).reshape(-1,)
    # Select these specific documetns
    cluster_samples = feature_matrix[ind,:]
    # Calculate their centroid as an average
    centroid = np.average(cluster_samples, axis=0)
    # Find the distance of each document from the centroid
    distances = [cosine(sample_doc, centroid) for sample_doc in cluster_samples]
    # Keep the document closest to the centroid as the representative
    representatives[label] = cluster_samples[np.argsort(distances),:][0]

for label, doc in representatives.iteritems():
    print("Label : %d -- Representative : %s" % (label, str(doc)))


Bayesian network for classification using PyMc or PyMc3

pymc will not provide you pretty sklearn-style .predict method for this case, however you can do it on your own. The idea is simple enough: you should draw coefficients for the classifier using pymc, and after it use them for the classifier itself manually.

You can see a very basic example at this blogpost or more complicated case at pymc3 documentation.


Check this post.

  In the cases where the frequency is related somewhat with the target variable, it helps the model to understand and assign the weight in direct and inverse proportion, depending on the nature of the data.

Check also this thread.

  What's the rationale behind it?

High cardinality may result in dimensionality curse and actually decrease the quality of your model.


MDP - RL, Multiple rewards for the same state possible?

it is possible that performing an action $a$ that takes us to state $s′$, could result in multiple rewards?

Yes, that is true the general case that any $(s,a)$ pair can result in a range of results for both $s'$ and $r$. Also $s'$ and $r$ can vary independently, provided each ones distribution only depends on $(s,a)$. In practice $r$ often depends strongly on one or more of $s$, $a$ or $s'$ (if it depends on the latter, then it still depends in absolute terms on just $s$ and $a$ because $s'$ does - it is just that values of $s'$ and $r$ are allowed to correlate).

This does not affect the statement about $p(s',r|s,a)$ being deterministic. It is the probability of specific $s',r$ results occurring that should be deterministic and depend on $(s,a)$ in a Markov Decision Process.


What kinds of data other than geographical are topologically spherical?

Any data consisting of a three element vector of numbers  $v =\{x,y,z\}$ could be viewed as topologically spherical if, after converting to polar coordinates, there is only one datum for every value of the 3D angle $\theta$. Thus, every point in the data represents an exterior of a topological sphere. This is taking the term "topological" literally and formally.

There are also graph structures (acyclic) that have neighborhood structures that allows the graph to be mapped topologically to a sphere. For example, all graphs that can be mapped to Platonic Solids (i.e. vertexes to vertexes, edges to edges, 1:1) are topological spheres. This can include irregular neighborhood structures, too.

As far as practical examples of these, none come to mind immediately.


How to drop points from a data series for presentation?

Indeed this problem isn't a very simple one to deal with, despite looking very easy to conceptualise. There exist a certain number of techniques for "reducing" the number of points of timeseries, one being called "downsampling".

A little litterature : https://skemman.is/bitstream/1946/15343/3/SS_MSthesis.pdf

hope this helps,

Cheers


You can normalize each criteria independently in values between 0 and 1 without taking into account the other criterias, it will work better for most classification methods k-nearest neighbors, random forest, neural network, etc.
$$x^*_{i,j}=\frac{x_{i,j}-x^{min}_j}{x^{max}_j-x^{min}_j}$$


Concatenating embedding and hand-designed features for logistic regression

I think that I may understand where they are coming from. The key difference is if we're talking about inference or training.

Inference

There is nothing stopping you from concatenating your features. Whether they are hand-designed or a neural network's output, features can be combined however you please.

Training

If you don't intend to update your embeddings through training, then, there is no problem. Your embeddings will be treated as some other features and you can train your downstream model.
It gets tricky if you want to train both your downstream model and your embedding layer. Indeed, you will only be able to train your embedding layer if your full model (NN + downstream model) is differentiable. This is for instance not possible if the downstream model is a KNN model.
I assume this is what they meant when they said you couldn't concate embeddings and hand-designed features.


There are a couple of things you can do.

Sample a representative, but small set of your data, which will allow you to compute PCA in memory. But seeing as you have 600,00 observations this will most likely not result in any meaningful results.
Use incremental PCA, here is a link: http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.IncrementalPCA.html#sklearn.decomposition.IncrementalPCA

But the main problem you have is that a number of samples are less than the amount of observations you have. I would recommend a different approach to dimensionality reduction. Autoencoders would be my recommendation to you. Autoencoders can be trained in an iterative fashion, circumventing your memory issue, and can learn more complicated projections than PCA (which is a linear transform). In case you want a linear projection you can have an autoencoder with one hidden layer, and the solution found by the neural network will be equal to the solution found by PCA.

Here are a couple of links you will find helpful:

http://ai.stanford.edu/~quocle/tutorial2.pdf
https://www.quora.com/How-is-autoencoder-compared-with-other-dimensionality-reduction-algorithms
https://www.cs.toronto.edu/~hinton/science.pdf


High Level Understanding of Neural Machine Translation

In encoder-decoder architecture, we first represent the input sequence by a fixed vector. It is assumed that this fix vector represents the complete meaning of the sentence. Now decoder uses this fix vector to generate the output sequence.

Answers of your questions:

If you use 1 recurrent unit, it outputs the value at each time instant which is not suitable for machine translation as translation of first word may be in the last or somewhere else in target language.(I am not sure how will you map the output and input in that case.)
Regarding synonyms, system can handle synonyms as words which are synonyms have very close word vectors.
Regarding Word-Order, it can learn the word order if you have enough training data. Decoder also uses beam search to find the best possible translation. I will suggest you to read this paper thoroughly.
https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf
Regarding evaluation, there are various metrics which are designed to evaluate the machine translation system. BLEU is one of the most famous one out of all. It uses various reference translations. You can read more about BLEU in the paper by Kishore Papineni et. al.(I can not post more than two link in answer as my reputation is less than 10.)

I will also suggest you to read the below paper regarding neural machine translation.
https://arxiv.org/pdf/1409.0473.pdf

Hope it helps!!


Differentiating between normal and attack clusters

First of all please ignore my previous answer. I did not understand you properly.

The difficulty of the your problem goes back to the distribution of normal points in train and test. The distribution of new normal points are so different than what your model can learn from only green points (training set) so I assume any model will have a relatively poor performance on it. Anyways, having a look at scikit-learn documentation is good if you did not.

About Clustering as you asked I have an idea. An active are in Graph Analysis and Network Science is Community Detection (which is simply the clustering of graph nodes). The idea is that, assume all your points in this figure are nodes of a graph and based on their Euclidean distance in PC space, you put an edge between them (either binary or weighted). Then you can use Modularity-based algorithms to find dense subgraphs. I assume if you apply it to your data it will work. Also a similar idea is Spectral Clustering which internally uses the idea of graph partitioning.

According to the density of clusters, I would also propose to use density-based clustering approaches. They might be able to distinguish between those overlapping clusters "a little bit" (e.g. this paper). This paper has an approach to find the peaks of densities. Sounds compatible with your data.
Please note that those overlapping clusters are anyways difficult to distinguish.

My apologies for the first misunderstanding and hope it helps.

Good Luck!


Can I use fit() function with images in Keras?

Yes you can. fit() takes the images in a numpy matrix while fit_generator() takes a generator object (processing images in batch)

Guide for using fit().

Guide for using fit_generator.


What is the meaning of "probability distribution of p(x)" of something uncountable?

Let me preface this answer by saying I am not an expert on variational autoencoders, but I think your conceptual gaps don't have anything to do with autoencoders.

First, it is possible to specify a probability distribution over categorical outcomes or continuous outcomes.  For example, a sample from a normal distribution is a continuous outcome, and the normal distribution describes the distribution of outcomes (or, equivalently, the relative likelihood of outcomes with various values).  When you sample from a normal distribution, the probability of getting any particular numerical value (like 0, or 3.14) is infinitely small.  Because of this, people typically talk about the probability density of a continuous distribution -- which can be interpreted as the likelihood of a sample being from a given (small) region.  The probability value is called a probability density because it represents probability mass divided by space (or volume).

Second, it is common to represent images as a vector of numbers. For a grayscale image, this might be 1 numeric value per pixel (it's slightly more complicated for color images).  You can think of the vector representation of an image as a point in space.  If you have a bunch of images, you can talk about the density of images in this space. Some regions of space have a lot of 'image points', and some have very few.  You can approximate this density by fitting a high-dimensional probability distribution to the image points in your dataset -- like fitting a smooth curve to a binned histogram of image counts.  Because of this, each image maps to a specific value for the fitted probability density.  That means you can talk about p(image) -- it really means, if we look 'near' the vector representation of this image in our vector space, how many images are there, relatively speaking? Some images are in high-density regions and some are in low-density regions, which will reflected as high and low values for p(image).


arima analysis after clustering time series

Here's a reference for your problem. If you read the first two pages, you'll get a hint at what features to use for time series clustering.

https://www.jstatsoft.org/article/view/v062i01


Understanding LSTM input shape for keras

(9999,1) has 9999*1 elements = 9999. However, (9999,20,1) will have 9999*20*1 elements, which are not available. Break your data into a batch/sequence length of say 99.
Then reshape it into (101,99,1)


Combining heterogeneous data sets for more powerful machine learning

Following your considerations:

  IMDB and RT don't have all the same movies

I would build a larger dataset of movies, aggregate all observations available, and run a model on that.

  There are words local to each dataset. i.e. the word "rotten" might be
  a frequent word on RT, but never appears in the IMDB dataset.

I think this doesn't constitute a problem once you aggregate data in a larger dataset. At that point you can run some NLP method on the whole aggregated corpus. Let me know what you think.

  The outcome measures are different. RT uses a 5 star scale, whereas
  IMDB uses a 10 star one. How do we deal with this heterogeneity?

You just need to uniform the scales on a given interval like [0, 1]. Then all votes could be compared.


Are deep learning models way over the required capacity for their datasets' estimated entropies?

The current thinking is that it is easier to fit an overparameterized neural network, since the local extrema are different ways of expressing the same thing, whereas in a minimal neural network you have to worry about getting to the global extremum:

  The subtle reason behind this is that smaller networks are harder to
  train with local methods such as Gradient Descent: It’s clear that
  their loss functions have relatively few local minima, but it turns
  out that many of these minima are easier to converge to, and that they
  are bad (i.e. with high loss). Conversely, bigger neural networks
  contain significantly more local minima, but these minima turn out to
  be much better in terms of their actual loss. Since Neural Networks
  are non-convex, it is hard to study these properties mathematically,
  but some attempts to understand these objective functions have been
  made, e.g. in a recent paper The Loss Surfaces of Multilayer Networks.
  In practice, what you find is that if you train a small network the
  final loss can display a good amount of variance - in some cases you
  get lucky and converge to a good place but in some cases you get
  trapped in one of the bad minima. On the other hand, if you train a
  large network you’ll start to find many different solutions, but the
  variance in the final achieved loss will be much smaller. In other
  words, all solutions are about equally as good, and rely less on the
  luck of random initialization.

CS231n Convolutional Neural Networks for Visual Recognition


If you want to proceed on your existing path I suggest normalizing each term's frequency by its popularity in the entire corpus, so rare and hence predictive words are promoted. Then use random projections to reduce the dimensionality of these very long vectors down to size so your clustering algorithm will work better (you don't want to cluster in high dimensional spaces).

But there are other ways of topic modeling. Read this tutorial to learn more.


What's the input dimension for transformer decoder during TRAINING?

For feeding word one by one in transformer network we pass the whole sentence along with a mask to the network. And the mask will do the job by unmasking one new word at a time.


Structuring extensive medical histories + demographic information for prediciting future medical outcomes

There are many options and many different problems here, I think you should start simple and then try to improve on it.
Imho the most basic option is to restructure the dataset so that there's one instance for every patient at any given time $t$, for instance every month. The binary label represents whether the patient was admitted to hospital in the period $t$ to $t$+60 days. The features should represent whatever relevant indicators can help the model based on past data (before $t$), for instance mean/standard deviation of the blood pressure in the past N months, possibly with various values of $N$ and/or a sliding window. This way both short-term and long-term information can be integrated. Occasionally you might have missing values, it's important to take this into account.
In this design there will be several instances for every patient, and this could cause a bias in the evaluation. There are two options: either there is one independent model for every patient, but this is probably too complex. Or the datasets is made of all the patients, and in this case it's important to avoid having the same patient in the training set and test set.
The problem of patients who would take an ambulance to the ER on a regular basis is a preprocessing problem: if possible, these instances should either be discarded or re-labelled as if it was a regular consultation, for example.


Neural Network with Connections to all forward layers

What you describe has been explored in Deep Residual Neural Networks.

A residual block will combine two or more blocks from a standard architecture like a CNN with a skip connection that adds the input to the first block to the output of the last block.

The intuition is that deep networks have a harder and harder time learning the identity function between layers, which has been proven to be useful especially in image recognition tasks. Residual connections also mitigate the problem of vanishing gradients.

Residual connections help solve the "degradation" problem, where deeper architectures lead to reduced accuracy. For example GoogLeNet won ILSVRC in 2014 with a 22-layer CNN, but in 2015 Microsoft ResNet won with a 152-layer Res Net.


Given is the result of the model performance. Help me with this MCQ

I think answer are options 3) and 4)

Clearly there seems to be overfiting towards dev set as the dev set error is low and almost equal to training set error.

And  increasing the number of samples helps minimizing overfiting problem..


If you up to a predictive model, you look for a model which performs well on the test set and the metric of interest is the mean squared error which indicates by how much you fail to predict $y$ on average. So don't use $R^2$. Just compare all models based on MSE.


LSTM input and output for sentiment analysis

Here it looks like the model takes in sequences of words, which are turned into embeddings, which are then put into the LSTM layer. The output of the LSTM layer is a 100 (because this is what is specified in the model) dimensional vector. This 100 dimensional vector is then put into a Dense layer outputting a 1-dimensional object, which I am assuming is some sort of prediction. Does this help? I feel like I may be missing the question.


Reducing a page of content to a short paragraph

Look into TextRank algorithm, here is the paper paper. You can find a neat python implementation here textrank implementation If I am not wrong gensim also offers an implementation.