Since you had not provided the tag on which language you are using. I am providing a sample solution in python with pandas for loading the data and transforming.

>> import pandas as pd
>> sample = pd.DataFrame({'a': ['1 inch', '2 meters', '1 metre', '2.5 inches', '2.8 meters']})
            a
0      1 inch
1    2 meters
2     1 metre
3  2.5 inches
4  2.8 meters
>> def get_number(a):
       for _ in a.split(' '):
           try:
               return float(_.strip())
           except ValueError:
               pass
>> m['b'] = m['a'].apply(lambda a: get_number(a) * 0.0254 if 'inch' in a else get_number(a))
>> m
              a       b
  0      1 inch  0.0254
  1    2 meters  2.0000
  2     1 metre  1.0000
  3  2.5 inches  0.0635
  4  2.8 meters  2.8000

Hope this gives you a general idea of how to do it.


How should I construct a binary classifier for small set of positive data and million of unlabeled data?

This is called PU Learning, and it can be used when using a probabilistic classifier and certain assumptions are met about how the data is labeled.
If the assumptions are met, you

Label positive, already labeled instances as positive
Labeled unlabeled instances as negative
Train a probabilistic classifier.

This produces the same ranking of class probabilities as a classifier would if trained on a dataset labeled with true positive/negative labels.
This video covers the assumptions pretty well and the Elkan paper is pretty accessible.


difference between feature interactions and confounding variables

Some remarks (as far as I understand your questions):

In a causal model you need to reflect the "data generating process" (DGP). The DGP is a theoretical construct. You need to come up with an idea what is relevant for your research question, so what $X$ explain $y$ in a causal way. You can also include variables which are not so important. In tendence underspecification (exluding important variables) is a real problem. Overspecification can also be a problem but the consequences are less salient.
I think the wording is unclear here. In some fields "adjusting for confounders" means to "control" for all relevant variables (aka. confounders). First thing to do: get the wording right and understand what you mean when you say "confounders".
Where does this claim come from ("feature interaction"). In general, Logit is a model that is linear in the way you include variables, so if you think about interaction of two or more $x$ in your model, then no: Logit does not consider interaction unless you specify your model so to include interaction terms.
An easy way of thingking about interactions is the interaction of a continuous variable $x_1$ with a dummy variable (1 or 0) $x_2$. In the model $$y=\beta_0+\beta_1 x_1$$ you only have one intercept ($\beta_0$) and one slope ($\beta_1$). In a model with interaction between $x_1$ and $x_2$ you have $$ y=\beta_0+\beta_1 x_1 + \beta_2 x_2$$ where there is the same slope for $x_1$ but a different intercept in case $x_2=1$. Another way to think about interactions is to add "squared" terms because a squared term is just the interaction $x_1 * x_1$.
I don't understand this: needs a reference! I guess you speak about the inclusion of indicator variables (aka dummies) as interactions??!
No you can use any numbers as $x$.
Look here and read this book.


The problem was that I was using a MinMaxScaler which scaled the inputs differently for the different cases. Once I set them to the same scale, this issue was resolved.


One option is Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between phrases. WMD is based on word embeddings (e.g., word2vec) which encode the semantic meaning of words into dense vectors.

  The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

Source: "From Word Embeddings To Document Distances" Paper

For your problem, you would have to generate candidates then use WMD to find the best matches for a given phrase.


Show distribution of users affected by outlier response times

One option would be look at conditional probability based on percentiles.
First, find percentiles based on all the data. For example, 99th percentile is 1432 milliseconds.
Then, find percent of a specific user request above that threshold. For example U1 has 50% of requests above 99th percentile.
This could be made into a cross-tabs table for easier exploration. The rows would be each user and the columns would be binned response time percentiles.


How to learn certain Maths to understand machine Learning papers?

As Elias mentioned, the expectations are related to random variables and you would be good to go if you know about conditional probability, multivariate probability, joint and marginal distributions. I would suggest you take a course that has a syllabus on the lines of https://secure.oregonstate.edu/ap/cps/documents/view/134169.


Try it the R tidyverse way:

library(dplyr)
dfr <- read.table(text = "id;attribute;time
A;AB;1
B;AB;2
C;AB;3
B;AB;4
B;AB;5
A;AB;6", header=TRUE, sep=";")

dfr %>% group_by(id, attribute) %>% summarize(mean=mean(time))

Result

# A tibble: 3 x 3
# Groups:   id [?]
      id attribute  mean
  <fctr>    <fctr> <dbl>
1      A        AB 3.500
2      B        AB 3.667
3      C        AB 3.000


How to implement LSTM using Doc2Vec vectors?

Weights are nothing but the pretrained word vectors . You can use any of word2vec or Glove embedding and create an embedding matrix to get that . Essentially each row of the matrix will be the vector for a word in the word2vec/Glove vocabulary. Please have a look here


Deep learning theory: why are hidden layers necessary?

Every input node can be connected to every output node, but that wouldn't account for non linearity of features and relationships amongst each other that are predictive. A neural network without any hidden layers, is just a regression. I think this is a great intro to deep learning.
In answer to your 2nd question: The number of nodes and connectivity are hyperparameters that should be optimized. Also, people use dropout (ie dropping out nodes) as a form of regularization.


How can I specify I want output of which units in LSTM?

If you use:

LSTM(10 , return_sequences = True)

you will only get as output something of size (batch_size, num_timesteps, 10).  The "10" for LSTM unit-size is a hyper-parameter you should be tuning.  However, what you want is something the size of (batch_size, num_timesteps, num_features).  This is why you want to put a Dense layer after the LSTM layer.  That being said, you will also want to add a TimeDistributed wrapper around your Dense layer, so that a Dense layer is applied to every time-step (1):

TimeDistributed(Dense(num_features))

Now the output of the TD dense layer will be of size: (batch_size, num_timesteps, num_features), which I'm assuming is what you want.  You will have to set up your Y data to be of the appropriate size for this to work.  It will have to be of size (batch_size, num_timesteps, num_features).  You mentioned that you are curious to know how to use just part of it.  At prediction time, post training, you would call your model.predict and it will return something the size of (batch_size, num_timesteps, num_features).  You can just use numpy array slicing to access what it is that you want.


How to represent audio data in a format that can be used for preprocessing and modelling?

Audio .wav codec file has a 44 byte header which will give you critical data like bit depth ( CD quality audio is 16 bits per sample), sample rate ( CD quality uses 44,100 audio samples per second ), number of channels, etc ... the balance of bytes in a .wav file is the payload which is the audio curve stored as a set of integers which define the height of the audio curve across a succession of instances in time. The beauty of .wav is its not compressed so all bytes after the header only store the audio curve.

Many audio libraries will allow you read a wav file and return back a floating point array of samples per channel.  Visualize a dog bark, this sound propagates from the dog and lands on the membrane of a microphone.  Similar wobbling different mediums.  This wobbling recorded over time can be represented as a time series curve ... as its digitized its transformed from analog to digital ( ADC ) the original analog audio wave is ~sampled~ X number of times per second to generate a simplification of that analog curve during this analog to digital conversion.  Each audio sample represents the height of that audio curve at a specific instant in time.

If you only devote 1 bit to record the audio curve, for a given sample, this will transition from 0 to 1 as the audio curve wobbles above and below the stationary silence flat line mark.  Whereas a bit depth of 16 bits which can store 2^16 distinct integers allows far greater granularity and so more accurately models the height of the audio curve.

Unlike image processing where conceptually the entire dataset can be ~consumed~ in an instant of time, audio has an inherent dimension of time.  Audio is a time series.  Another attribute of audio is the simplicity with which any arbitrary audio curve can be represented by a set of pure sin curves, each with an amplitude, frequency and phase shift.  Joseph Fourier outlined the theory whereby time domain data like audio can be represented equally well in either the time domain ( wav codec or analog audio curve ) or in the frequency domain.  It does not go unnoticed that this symmetry implies a conservation of information.

To an arbitrary degree of precision a window of audio samples ( in the time domain ) can be transformed using a Fourier Transform into its frequency domain representation with a controllable degree of information loss or lack thereof.  Be aware that as you increase the number of samples in this window you decrease the temporal specificity of the resultant spectrogram ... meaning the ability to pin point when a frequency occurred is aided by using the lowest number of audio samples in your window send into the FFT call ... although the frequency resolution in the freq domain is increased by feeding in a larger number of samples to your FFT call ... there is no such thing as a free lunch

So now you have options ... perform your analysis of audio while in its native time domain or send it into a FFT call and have the same information as a set of frequency bins where each bin has values for ( frequency , amplitude, phase shift )

This should give you ideas to tease apart and dig into - take care


Coursera ML - Does the choice of optimization algorithm affect the accuracy of multiclass logistic regression?

Limits of numerical accuracy and stability are causing the optimisation routines to struggle.

You can see this most easily by changing the regularisation term to 0.0 - there is no reason why this should not work in principle, and you are not using any feature engineering that particularly needs it. With regularisation set to 0.0, then you will see limits of precision reached and attempts to take log of 0 when calculating the cost function. The two different optimisation routines are affected differently, due to taking different sample points on route to the minimum.

I think that with regularisation term set high, you remove the numerical instability, but at the expense of not seeing what is really going on with the calculations - in effect the regularisation terms become dominant for the difficult training examples.

You can offset some of the accuracy problems by modifying the cost function:

def compute_cost_regularized(theta, X, y, lda):
    reg =lda/(2*len(y)) * np.sum(theta[1:]**2)
    return reg - 1/len(y) * np.sum(
      y @ np.log( np.maximum(sigmoid(X@theta), 1e-10) )
      + (1-y) @ np.log( np.maximum(1-sigmoid(X@theta), 1e-10) ) )

Also to get some feedback during the training, you can add

                       options = {
                           'disp': True
                       }

To the call to minimize.

With this change, you can try with regularisation term set to zero. When I do this, I get:

predict_one_vs_all(X_bias, theta_all_optimized_cg)
Out[156]:
94.760000000000005
In [157]:

predict_one_vs_all(X_bias, theta_all_optimized_bfgs)
/usr/local/lib/python3.6/site-packages/ipykernel/__main__.py:2: RuntimeWarning: overflow encountered in exp
  from ipykernel import kernelapp as app
Out[157]:
98.839999999999989

The CG value of 94.76 seems to match the expected result nicely - so I wonder if this was done without regularisation. The BFGS value is still "better" although I am not sure how much I trust it given the warning messages during training and evaluation. To tell if this apparently better training result really translates into better digit detection, you would need to measure results on a hold-out test set.


Predict if user will come the next day or not

I think that you should stop thinking about modeling and instead start thinking about the meta datasets that you can draw from what you already have. I have also downvoted the previous answer you received because it's making the same mistake.

One of the most consistent mistakes I see in data science (by all levels of scientists) is that they skip the data transform/wrangling stage and just go straight into modeling. That's a MASSIVE mistake and the answer you seek starts well within this stage.

So, I want you to take a step back and think about your data. What do you need to know about a user? What are some of the features of each user? What are the features of when they visited? What day? What time? Was it the morning or the afternoon? How many days in a row was it? What is the gender of the user? And so on and so on.

Your goal is to create a dataset that has one line per user and the same features for all users. From there, you'll be in a better position to make better decisions about modeling and anything else you want to know about your data.


How to recommend items after finding similar users in recommendation system

You can use some kind of distance metrics.

For example, you identify three users that have the lowest distance from your target user and propose one item from each, or three items from the closest user, whatever works best for your usecase

I guess you've used clustering to cluster the users together. Maybe that using product clustering in conjunction with user clustering can give you good results?


What is the difference between an RMSE and RMSLE (logarithmic error)?

Let's start with the definitions:
$$ \text{RMSE} = \sqrt{\frac{1}{N} \sum_{i = 1}^{N} (y_i - \hat y_i)^2}, $$
$$ \text{RMSLE} = \sqrt{\frac{1}{N} \sum_{i = 1}^{N} (\log y_i - \log \hat y_i)^2}, $$
where $y_i$ is the target value for example $i$ and $\hat y_i$ is the model's prediction. Both metrics quantify the prediction error, so in general a high RMSE implies a high RMSLE as well.

RMSLE has the meaning of a relative error, while RMSE is an absolute error. Choosing one depends on the nature of your problem. Imagine that the target spans values from around 1 to around 100. Is predicting $\hat y = 1.01$ for true $y = 1$ as bad as $\hat y = 101$ for $y = 100$? Then RMSLE might be a good choice. With RMSE, on the other hand, predicting $\hat y = 2$ for $y = 1$ is as bad as $\hat y = 101$ for $y = 100$.

In general, you would often want to use the logarithm (or maybe the square root or some other power $0 < \alpha < 1$) for strictly positive targets that span a large range.


What are the rules when extracting SVO triples from preprocessed text?

The goal of Subject-Verb-Object (SVO) triples is to extract a single triple for a sentence.
The sentence:

A rare black squirrel has become a regular visitor to a suburban garden.

results in the following SVO:

(squirrel, become, visitor)

Triplet Extraction From Sentences by Rusu et al. outlines how to do that. First, you'll need a parse tree ( Stanford Parser and OpenNLP are the most common). The three items then can be extracted:

The subject will be found by performing breadth first search and selecting the first descendent of NP that is a noun.

… the predicate of the sentence, a search will be performed in the VP subtree. The deepest verb descendent of the verb phrase will give the second element of the triplet.

… we look for objects. These can be found in three different subtrees, all siblings of the VP subtree containing the predicate. The subtrees are: PP (prepositional phrase), NP and ADJP (adjective phrase). In NP and PP we search for the first noun, while in ADJP we find the first adjective.


which is better : F1-score of 'N' in imbalanced data or 'N+3' in balanced data?

Second.
why? you said you objective is to minimize misclassifications which equates to maximizing
F1 score, and you achieve that in inbalaced situations when you down-sample the majority class (one approach to maximize)
Potential Pitfall:
It could be that you just got rid of of some important infomration in the majority class and you are over-fitting on what is left.
In some cases 1500 datasamples would suffice, but I it could be that it doesn not have enough information. I would rather advise to upsample the smaller class, for example using ADASYN, SMOTe , knnSMOTE etc...
Only you can answer this question, by knowing some meta-information about your dataset, quality rules, are you expecting huge covariate-shift, is the dropped data really that different in distribution, etc...
Then do the analysis. If there is one thing you should always ask for is diverse and informative dataset, and more data will (almost) always beat better algorithm.


Grid Search Pool Classifiers for Dynamic Classifier Selection Ensembles

Here is my solution to this problem :
Let's say I have the classifiers pools I want to cross validate :
grid = dict()

grid[ 'pool_classifiers' ] = [ ( 'pool_01' , [ LogisticRegression() , DecisionTreeClassifier() , GaussianNB() ] ) ,
                               ( 'pool_02' , [ LogisticRegression() , DecisionTreeClassifier() ] ) ]

Then I can use the following function :
def ola_cv( X , y , cv ) :

  scores = dict()

  for pool_classifiers in grid[ 'pool_classifiers' ] :

      scores[ pool_classifiers[ 0 ] ] = list()

  for train_ix , test_ix in cv.split( X , y ) :

    X_train = X[ train_ix , : ]
    y_train = y[ train_ix ]

    X_test = X[ test_ix , : ]
    y_test = y[ test_ix ]

    for pool_classifiers in grid[ 'pool_classifiers' ] :

      for model in pool_classifiers[ 1 ] :

        model.fit( X_train, y_train )

      ola = OLA( pool_classifiers = pool_classifiers[ 1 ] )

      ola.fit( X_train , y_train )

      y_pred = ola.predict( X_test )

      score = accuracy_score( y_test , y_pred )

      scores[ pool_classifiers[ 0 ] ].append( score )

  for k in scores.keys():

    print( f'pool_classifiers : {k} | accuracy : {np.mean(scores[ k ])} ({np.std(scores[ k ])})')

  return scores

to obtain for each classifiers pool mean accuracy ( and standard deviation ) on 10 * 3 = 30 folds.
NB : Mayne grid should be an argument of the ola_cv function.


Training the Discriminative Model in Generative Adversarial Neural Network

In normal GANs, there are no labels, the training is completely unsupervised.

The role of the discriminator is to tell apart samples generated by the generator from those taken from the training dataset. The training dataset is just a bunch of images. The discriminator is trained to output 0 for data generated by the generator (i.e. fake data) and 1 for real data (so the discriminator has a single output). This should answer points 1 and 2.

The training of discriminator and generator takes place alternatively in a loop: first we train the discriminator, then the generator, then the discriminator again, etc. It is possible (and common) to train the discriminator a few times per each time we train the generator. This should answer point 3.

It is also possible to use labels, but not in the way you were suggesting. When labels are used, we have Conditional GANs (https://arxiv.org/abs/1411.1784). In this case, the label is supplied as input to both the generator and the discriminator. The generator has to generate data that is associated to the supplied label. The discriminator has to tell apart fake data from real data, given the label.


Xgboost performs significantly worse than Random Forest

Different algorithms need to be tuned in different ways. For example, one important parameter in boosting is ETA or learning rate. This determines the change in weights after each boosting step. This parameter is important to reduce overfitting in boosting.

To understand the different parameters you can refer to this.

Another important thing to look at is the objective function. This will determine what your algorithm is trying to optimize.

Lastly, please make sure you are selecting the model that actually maximises the metric that you care about. You can do this by selecting an appropriate eval_metric. In this case, you probably want to use something like F1-Score or precision at a fixed recall.


What algorithm could be used to fuzzy merge multiple datasets?

The first thing is to define a distance metric to say how close potential keys are. If the data is text, Levenshtein distance  is a common distance metric. If the data is numeric, then Euclidean or Manhattan distance could work. If the data is geographic, then Haversine is a good choice.

After defining a distance, a threshold needs to be set to for merging. The threshold could be picked by a person or learned by a machine learning algorithm. Typically, there is not enough data to train a machine learning algorithm. It is often good enough to just pick a threshold based on domain knowledge.

If you can use R, there is the fuzzyjoin package.

Given that you have to search the cartesian product of the space, evolutionary algorithms will take too much time.


Should Principal components be normalized before applying K means on them?

No - There is no need to normalize after Principal Component Analysis (PCA) because each dimension is on the same scale.


Is there a rule of thumb for a sufficient number of trials for hyperparameter search

In a pure random search, 60 points is often given as a rule of thumb, because provably with probability 95% such a search finds a hyperparameter combination in the top 5%.
However, that 5% is as a percent of the volume of the search space, so giving much-too-broad a search space, the best 5% might not be a fantastic score for the model.  So it does seem to depend, not exactly on the number of hyperparameters, but on the volume of good combinations, which in turn indirectly depends on the number of hyperparameters.
Bayesian optimization should take fewer iterations (but not necessarily less time!) to reach an optimum than a pure random search, and in particular should rule out swaths of bad volume relatively quickly.
So, to conclude...no, I don't actually have a rule of thumb.  60 is probably a decent bet, given the tradeoff of the last two paragraphs; I'd go up to 100 if the training isn't too expensive.  Also consider whether your package allows you to continue a Bayesian search: you could analyze the results so far (mostly, see if the last few iterations have clustered around some point, and how widely the objective function varies) to see if you want to proceed.  Finally, note that scikit-optimize sets a default of 50, but there doesn't seem to be much behind that.


Is a 100% model accuracy on out-of-sample data overfitting?

High validation scores like accuracy generally mean that you are not overfitting, however it should lead to caution and may indicate something went wrong. It could also mean that the problem is not too difficult and that your model truly performs well. Two things that could go wrong:

You didn't split the data properly and the validation data also occured in your training data, meaning it does indicate overfitting because you are not measuring generalization anymore
You use some feature engineering to create additional features and you might have introduced some target leakage, where your rows are using information from it's current target, not just from others in your training set


Data Annotation: "labeling" target vs features

Those labels are not primarily for features, those labels are primarily for targets. Person, location, and event for targets for named-entity recognition (NER).


Does reinforcement learning only work on grid world?

The short answer is no! Reinforcement Learning is not limited to discrete spaces. But most of the introductory literature does deal with discrete spaces.

As you might know by now that there are three important components in any Reinforcement Learning problem: Rewards, States and Actions. The first is a scalar quantity and theoretically the latter two can either be discrete or continuous. The convergence proofs and analyses of the various algorithms are easier to understand for the discrete case and also the corresponding algorithms are easier to code. That is one of the reasons, most introductory material focuses on them.

Having said that, it should be interesting to note that the early research on Reinforcement Learning actually focussed on continuous state representations. It was only in the the 90s since the literature started representing all the standard algorithms for discrete spaces as we had a lot of proofs for them.

Finally, if you noticed carefully, I said continuous states only. Mapping continuous states and continuous actions is hard. Nevertheless, we do have some solutions for now. But it is an active area of Research in RL.

This paper by Sutton from '98 should be a good start for your exploration!


How to use LSTM to make prediction with both feature from the past and the current ones?

Have a look at this question. There is a nice discussion about how to implement this. The idea in to create a separated Input to your model and concatenate it AFTER the recurrent layer(s). Also in the Keras documentation, there is an example on how to build such models with a few lines of code.


In my eyes, this is not a valid approach.

Note, that there is not one unique topic model (given some parameters like the number of topics and the algorithm for topic modelling) for a corpus. Different runs with different random seeds will give you different topic models for the same corpus.

So, any comparison comes down to a comparison of specific topic models, but not to a comparison of the corpora.

An approach with some better validity is to combine both corpora into one super-corpus, create a topic model of it, and than investigate the distribution of the topics with respect to the sub-corpora formed by the original corpora 1 and 2.


From analysing their previos transactions how can I predict for what type of product is the customer more likely to take take an EMI

This problem could also be formulated as a recommendation system for your marketing department.
Given the user's past purchases and the product information and payment methods, you could build a recommendation model to recommend products for your marketing department to push forward for the specific customer.


Try closing and restarting ipython. imblearn requires scikit-learn >= 0.20 and sometimes the runtime loads in an older version of scikit-learn.

If the issue still persists, then reinstall all packages together to make sure they are compatible.


How to properly train your Self-Organized Map?

An answer from Kohonen, inventor of the self-organized map himself:

"The true mathematical form of σ(t) is not crucial, as long as its
value is fairly large in the beginning of the process. Say, on the
order of half of the diameter of the grid, whereafter it is gradually
reduced to a fraction of it in about 1000 steps."

From: Kohonen, T., 2013. Essentials of the self-organizing map. Neural networks, 37, pp.52-65.


In layman terms, what is the meaning of fitting a model into a data set?

Your asking for a layman's explanation, and the phrasing of your question, suggests to me that you are getting tangled up in the semantics, and I totally get that.

So let's let go of the usual terms as much as we can, and speak in concepts.

You have a bunch of observations -- or maybe even only one observation. This is your data. Maybe you are one of the particularly inquisitive ancients, and you notice some sort of relationship between lunar patterns and tides. While tracking this data, you start getting a sense for what sort of forces might be at work that would explain the apparent cause-and-effect relationship. The set of rules that you eventually develop becomes your model. The more correct your rules, the better you will be able to predict the tide tomorrow, next month, and next year.

This concept you've developed -- probably something to do with gravity -- may be even more generally useful. If the general concept is re-used with different numbers, it can also help explain Earth's annual cycle. And why we see the Morning Star (Venus) with certain patterns.

In the cases where we are re-using the concept, we are "fitting the model to the data" -- we are testing whether the set of rules we've developed -- the model -- can explain the apparent forces at work in the observations -- the data -- that we have.

In the usual applications of regression (or any statistical method), we're not trying to formulate the laws of physics, we're trying to find rules that sufficiently govern complex systems -- often human behaviour -- and so we're unlikely to explain observations perfectly. But a model is useful to the extent that it successfully predicts future observations. Rather than waiting for the future, though, we typically perform analysis on part of a data set, and test for predictive power with the remainder of the data set.

A specific instance of regression analysis might be in trying to explain housing prices. There are many many possible factors -- square meterage (footage), number of floors, swimming pool or not, size of property, shape of property, age, likelihood of flooding, distance to neighbours, energy rating, economic conditions, quality of the local school, etc, etc, etc. Which factors are most important? Do one, two, three of the factors explain 95% of the variation in housing prices? Can we come up with an equation that will predict a house's selling price reasonably well, rather than relying on a real estate agent's recommendation -- and having to second-guess what his/her agenda is -- how they might be playing us?

And if you will humour a little editorializing:

There are countless examples of reliance on experts proving to be a mistake (Nate Silver's "The Signal and the Noise", Daniel Kahneman's "Thinking Fast, Thinking Slow", Philip Tetlock's "Superforecasting") -- Data Science is about finding ways to derive insight from the data, rather than experts' opinions. But many data scientists get caught up in the mechanics, and lose sight of the purpose. It's not just about razzle-dazzle infographics. As Hamming puts it: "the purpose of computing is insight, not numbers". So I applaud your effort to seek a layman's explanation, rather than technical speak, to understand regression.


how to modify sparse survey dataset with empty data points?

I would consider approaching this situation from the following two perspectives:

Missing data analysis. Despite formally the values in question are empty and not NA, I think that effectively incomplete data can (and should) be considered as missing. If that is the case, you need to automatically recode those values and then apply standard missing data handling approaches, such as multiple imputation. If you use R, you can use packages Amelia (if the data is multivariate normal), mice (supports non-normal data) or some others. For a nice overview of approaches, methods and software for multiple imputation of data with missing values, see the 2007 excellent article by Nicholas Horton and Ken Kleinman "Much ado about nothing: A comparison of missing data methods and software to fit incomplete data regression models".
Sparse data analysis, such as sparse regression. I'm not too sure how well this approach would work for variables with high levels of sparsity, but you can find a lot of corresponding information in my relevant answer.


This is a pretty massive question, so this is not intended to be a full answer, but hopefully this can help to inform general practice around determining the best tool for the job when it comes to data science. Generally, I have a relatively short list of qualifications I look for when it comes to any tool in this space. In no particular order they are:

Performance: Basically boils down to how quickly the language does matrix multiplication, as that is more or less the most important task in data science.
Scalability: At least for me personally, this comes down to ease of building a distributed system. This is somewhere where languages like Julia really shine.
Community: With any language, you're really looking for an active community that can help you when you get stuck using whichever tool you're using. This is where python pulls very far ahead of most other languages.
Flexibility: Nothing is worse than being limited by the language that you use. It doesn't happen very often, but trying to represent graph structures in haskell is a notorious pain, and Julia is filled with a lot of code architectures pains as a result of being such a young language.
Ease of Use: If you want to use something in a larger environment, you want to make sure that setup is a straightforward and it can be automated. Nothing is worse than having to set up a finnicky build on half a dozen machines.

There are a ton of articles out there about performance and scalability, but in general you're going to be looking at a performance differential of maybe 5-10x between languages, which may or may not matter depending on your specific application. As far as GPU acceleration goes, cudamat is a really seamless way of getting it working with python, and the cuda library in general has made GPU acceleration far more accessible than it used to be.

The two primary metrics I use for both community and flexibility are to look at the language's package manager, and the language questions on a site like SO. If there are a large number of high-quality questions and answers, it's a good sign that the community is active. Number of packages and the general activity on those packages can also be a good proxy for this metric.

As far as ease of use goes, I am a firm believer that the only way to actually know is to actually set it up yourself. There's a lot of superstition around a lot of Data Science tools, specifically things like databases and distributed computing architecture, but there's no way to really know if something is easy or hard to setup up and deploy without just building it yourself.


Model to predict coronavirus (covid19) spread

For a beginner, I would say the SIR model is a great place to start:

https://idmod.org/docs/general/model-sir.html

Numberphile did a great video on using SIR to predict Covid-19:

https://www.youtube.com/watch?v=k6nLfCbAzgo

Hopefully this can get you started on your journey!


Train-Test split for Time Series Data to be used for LSTM


I don't see any particular advantage in using linear (i.e.: none) activation. The power of Neural Network lies in their ability to "learn" non-linear patterns in your data. Moreover, the Tanh and sigmoid gates are thought to control for the stream of information that unrolls through time, they have been designed for that, and personally I'd be cautious in changing that.

If you want to do some multistep regression you have many options, but the most advanced architecture is the sequence-to-sequence, or seq2seq:

from tensorflow.keras import Sequential

from tensorflow.keras.layers import Bidirectional, LSTM
from tensorflow.keras.layers import RepeatVector, TimeDistributed
from tensorflow.keras.layers import Dense

from tensorflow.keras.activations import elu, relu

seq2seq = Sequential([

    Bidirectional(LSTM(len_input), input_shape = (len_input, no_vars)),

    RepeatVector(len_input),

    Bidirectional(LSTM(len_input, return_sequences = True)),

    TimeDistributed(Dense(hidden_size, activation = elu)),

    TimeDistributed(Dense(1, activation = relu))

])

where len_input is the length of the input sequence. The output is the same sequence shifted forward a number of steps equivalent to the lenght of your prediction. hidden_size is the size of Dense() layers, that is completely optional.

I used the Bidirectional() wrapper, but this is optional too. Not all tasks require bi-LSTM, feel free to remove it if you need.

The (combined) role of RepeatVector() and TimeDistributed() layers is to replicate the latent representation and the following Neural Network architecture for the number of steps necessary to reconstruct the output sequence. RepeatVector() generates this "multiplication", while TimeDistributed() repeats the application of on each of these repeated signals, generating the final sequence in practice.

Both input and output must be 3-dimensional numpy arrays of shape:

  ( number of observations , length of input sequence , number of variables )

Seq2seq models are harder to train (higher number of parameters, longer training times), but their performance is superior to other RNN architectures. At the present time, any state-of-the-art RNN is some kind of seq2seq.


Energy consumption time series forcasting

If you just want to get a forecast, use automatic forecasting software like Autobox, ForecastPro, SCA, etc.

Comparison of different software: https://stats.stackexchange.com/questions/68253/expert-forecasting-software-evaluation
Worked example: http://people.duke.edu/~rnau/autocomp.htm

If you are doing research on forecasting techniques, there are some Java packages, for example:

JMotif http://code.google.com/p/jmotif/
JTSA http://sourceforge.net/projects/jtsa/

However, Java is not very widely used for this; if you must use Java you will likely have a feeling of being out in the cold.   R and Python would have much more packages, examples, etc.  A tiny sample of what is available in R:

http://www.amazon.com/Introductory-Time-Series-Paul-Cowpertwait/dp/0387886974/
http://a-little-book-of-r-for-time-series.readthedocs.org/en/latest/
http://www.r-bloggers.com/time-series-analysis-and-mining-with-r/


count_vectorizer.vocabulary_.items() and count_vectorizer.vocabulary_ Inconsistent number of returns

For future readers: as Ben wrote in the comments, it is really hard in general that len(my_dict) != len(my_dict.items()). When these kind of strage behaviours happen, it is always a good practise to perform some routine checks:

Clean your environment from every variable, even better restart the kernel and then run again your code.
Check your code for variables with same name, be sure you're not overwriting stuff, it's easy also to end up defining functions with same name of other imported library functions.
Try your code on different datasets, to be sure the mistake is not due to some external issue with the data.

If none of the above checks solve the mystery, then you can start thinking about a bug, a corrupted installation of a library or some other more nested problems.


How to work with predictor variables which are all categorical but the target variable is an integer?

I think you need to create a dummy variables from your categorical variables. Almost all machine learning algorithms can't work with categorical variables.


Principle behind seq2seq model's example in keras?

The original Seq2Seq paper uses the technique of passing the time delayed output sequence with the encoded input, this technique is termed teacher forcing.

There exists a simplified architecture in which fixed length encoded input vector is passed to each time step in decoder (analogy-wise, we can say, decoder peeks the encoded input at each time step).

The paper "Learning Phrase Representations using RNN Encoder–Decoder
for Statistical Machine Translation" combines these both techniques (so it passes encoded input vector along with time delayed output sequence as inputs to decoder).


For this feasibility study, following will be high level steps :

For each feature perform PCA with rest of the features as train_x and feature as train_y. If you find a feature that can be predicted by other features; ML can be applied on Dataset
Can a Human solve it ? As a person; can you find patterns for a given feature, based on other features ?
Exploratory data analysis with Weka, Dataframe + Matplotlib or similar tools . https://datascienceguide.github.io/exploratory-data-analysis


Using an autoencoder for anomaly detection on categorical data

In general an autoencoder should perform well, when it comes to detect fraud examples. Fraud examples should have in theory a much higher reconstruction error.
When it comes to train the autoencoder on binary data, I agree with you that it can be quite challenging. I suggest to take a look at this blog:
https://blog.evjang.com/2016/11/tutorial-categorical-variational.html


Question about Similarity vs Dissimilarity Matrix

A similarity is larger if the objects are more similar.

A dissimilarity is larger if the objects are less similar.

This sounds trivial, but if you get the sign wrong, you suddenly search for the worst rather than the best solution...

It's easy to see that a distance is always a dissimilarity.

K-medoids could be implemented for similarities, but I am not aware of any implementation that does not expect the data to be a dissimilarity. It may be fine to simply pass -similarity to many implementations. Because all they care for is to minimize a sum of dissimilarities, which can trivially be shown then to be equivalent to maximizing the sum of similarities.


ValueError: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (142, 1)

def ReadImages(Path):
    LabelList = list()
    ImageCV = list()
    classes = ["nonPdr", "pdr"]

   # Get all subdirectories
   FolderList = [f for f in os.listdir(Path) if not f.startswith('.')]
   print(FolderList)

   # Loop over each directory
   for File in FolderList:
      for index, Image in enumerate(os.listdir(os.path.join(Path, File))):
          # Convert the path into a file
          ImageCV.append(cv2.resize(cv2.imread(os.path.join(Path, File) + os.path.sep + Image), (600,700)))
          LabelList.append(classes.index(os.path.splitext(File)[0]))

   return ImageCV, LabelList

A bit of change to your code after discussion in the chat room.


Python TimeSeries Error: ValueError: view limit minimum -36834.64916714945 is less than 1 and is an invalid Matplotlib date value

I assume an arbitrary df like:
df = pd.DataFrame(np.random.rand(720, 3), columns=list('abc'))

pyplot.plot([...]) doc expects a x and y, but you didn't specify what the x should be. If we include x as datetime values and y as the desired mean or std values, then the code looks like:
start = datetime.strptime('2016-01-01', '%Y-%m-%d')

df['day'] = [start + timedelta(x) for x in range(720)]

rolling_mean = df.rolling(window = 12, on="day").mean()
rolling_std = df.rolling(window = 12, on="day").std()

colors=['olive', 'gold', 'green', 'red', 'black', 'blue', 'purple', 'coral', 'teal']
for idx in df.columns.drop('day'):
    plt.plot(df['day'], df[idx], color = np.random.choice(colors), label = 'Original')
    plt.plot(rolling_mean['day'], rolling_mean[idx], color = np.random.choice(colors), label = 'Rolling Mean ' + str(idx))
    plt.plot(rolling_std['day'], rolling_std[idx], color = np.random.choice(colors), label = 'Rolling Std ' + str(idx))
plt.legend(loc = 'best')
plt.title('Rolling Mean & Rolling Standard Deviation')
plt.show()
plt.close()

This will give you this rainbow-coloured plot (if you wish other colors, check here)


How to group by ID, and output without repeating IDs?

This problem will by solved by group by function.

Whatever tool you are using (R or SQL), you need to just group the data by UserID.


Is this the correct way to apply a recommender system based on KNN and cosine similarity to predict continuous values?

The predicted rating is equal to the sum of each neighbors rating times similarity, then divided by 10 (number of neighbors).

You want to obtain a weighted-average of ratings, where the weight is the similarity score. Instead of dividing by 10 above, you should divide by the sum of similarities, to get a correct normalized score. Dividing by 10 is the likely reason for the predictions to be smaller than expected.

The formula you should use is:

  Final score = Sum_over_neighbours(Rating * Similarity) / Sum_over_neighbours(Similarity)


How to measure/rate the effect of a exogenous covariate in a ARIMAX Model?

Do this: create a series X, which is a multiple of your target variable say 3 or 5 times or a fraction of your target variable say 10% or 15%. Multiply your target variable values with the selected multiple or fraction but don't multiply all your target values by say a single multiple or fraction value, multiply target values by say numbers in a close range say between 10-13. example below

so Step 1: if my Y (target)= [10,20,30,40..], then I create X as [10*10=100,20*13=160,30*11=330,40*11.5=460...] (created random numbers between 10-13 and multiplied them by  target values) Same multiple will create perfect relation between X & Y, and different multiple values in a close range will overcome this to a good extent.

Step 2: To further ensure that  X is not a perfect predictor of Y but only reasonably good one and there are other predictors at play like TIME (since you are doing ARIMA), add random noise to X by creating some random numbers and adding them to series X creating a new series say Xnew.

random noise= [25,37,31..]

Xnew= [100+25, 160+37, 330+31..so on]

thus you can now add Xnew to your ARIMAX model.

Xnew is definitely a significant predictor, that you will see in your ARIMAX results.

Best Vik


Why is PyTorch's DataLoader not deterministic?

If you want to shuffle the data in a deterministic way, how about shuffling the dataset beforehand e.g. in a simple list of filenames, then simply reading that list deterministically in a single-processed loop, with shuffle = False in the DataLoader??

Another things that may cause non-deterministic behaviour is using multiple processes - then there are operations that are passed out and managed by the operating system, which doesn't pay attention to any of the random seeds you set. Performance is dependent on available resources i.e. affected by other activities running on your host machine.

In addition to that, any interaction between CPU and GPU could be causing non-deterministic behaviour, as data transfer is non-deterministic (related Nvidia thread). Data packets can be split differently every time, but there are apparent CUDA-level solutions in the pipeline.


K-Nearest Neighbor Classifier Best K Value

There's several ways that you can choose your k value for kNN -

You can use the common formula k = sqrt(n) where n is the number of data points in your training set or you can try choosing k where there is a good balance between computation expense vs noise.

Consider your what fits your problem:
Do you care about runtime? The higher the k, the more expensive computationally it is to run.

It looks like around  k= 6-10 you get some diminishing returns - you could set it there to get a good balance between noise and computation cost, but ultimately it is a very arbitrary selection, so pick what suits your use case best.


This type of spam is called keyword stuffing and it is a widely used SEO technique.
There might be different approaches to deal with this.
One is to use pre-trained classifier that will provide a "spam" score to each document. One such example is Waterloo spam classifier, covered in paper:

"Efficient and Effective Spam Filtering and Re-ranking for Large Web
Datasets"

, by: Gordon V. Cormack, Mark D. Smucker and Charles L. A. Clarke.
In addition, several predictors of document (specifically, Web document) quality were proposed. One of the most effective ones is the entropy of the unigram language model of a document. Specifically, when there is a large amount of repeated keywords in a document, the entropy of the language model is unusually low, since these keywords have unusually high probabilities. Another signal of document quality is the percent of stopwords in it, since natural language has more stopwords than a spam document.
More on Web documents quality measures can be found in: "Quality-Biased Ranking of Web Documents", by Michael Bendersky, W. Bruce Croft,Yanlei Diao.


What does an Input layer of shape=(None,) or (None,12) actually mean?

In tf.keras, a None dimension means that it can be any scalar number, so that you use this model to infer on an arbitrarily long input. This dimension does not affect the size of the network, it just denotes that you are free to select the length (number of samples) of your input during testing.


Looking for a 'CITY, STATE' within a body of text (from a CITY-STATE database)

The only thing I can see would be to separate both city and state lists and treat the problem as an automaton: parse your text, run through the n-grams, whenever you detect a CITY token (meaning a n-gram present in your list of cities or close to it in a similarity sense, as there might be misspellings) then look for a STATE token in its neighbourhood (similarly by looking into a list of states, using an edit distance metric to allow for misspellings). If you find one, then you can tag your text with that geographical location.

Of course, allowing for misspellings will bring some false positives but you could easily bypass that by doing a quick lookup through your corpus to see that "SALAMI, OREGANO" is different from "SALEM, OREGON" (because the frequency of the latter will hopefully be higher than the former)


How to apply StandardScaler and OneHotEncoder simultaneously in Spark Machine learning?

Okay, I came up with a solution by myself. Applied OneHotEncoderEstimator that was added recently instead of OneHotEncode on every categorical feature. Then I used MinMaxScaler on features from VectorAssembler.

val cutIndexer = new StringIndexer().setInputCol("cut").setOutputCol("cutIndex")
val colorIndexer = new StringIndexer().setInputCol("color").setOutputCol("colorIndex")
val clarityIndexer = new StringIndexer().setInputCol("clarity").setOutputCol("clarityIndex")

import org.apache.spark.ml.feature.OneHotEncoderEstimator

val encoder = new OneHotEncoderEstimator().setInputCols(Array("cutIndex", "colorIndex", "clarityIndex")).setOutputCols(Array("cutIndexEnc", "colorIndexEnc", "clarityIndexEnc"))

val assembler = (new VectorAssembler()
                    .setInputCols(Array("carat", "cutIndexEnc", "colorIndexEnc", "clarityIndexEnc", "depth", "table", "x", "y", "z"))
                    .setOutputCol("features_assem") )

import org.apache.spark.ml.feature.MinMaxScaler
val scaler = new MinMaxScaler().setInputCol("features_assem").setOutputCol("features")

val Array(training, test) = df_label.randomSplit(Array(0.75, 0.25))

import org.apache.spark.ml.Pipeline

val lr = new LinearRegression()

val pipeline = new Pipeline().setStages(Array(cutIndexer,colorIndexer, clarityIndexer,encoder, assembler,scaler, lr))

val model = pipeline.fit(training)

val predictions = model.transform(test)


Reasoning for temporal difference update rule

First of all the weights update is derived using gradient descent. So the proper form of update is the first one you have. This is derived using math and satisfies that the updates are towards minimizing the Mean Square Error between true value and approximated one. For the true value we use a biased sample of the true value at the next time step plus the reward obtained at the current timestep: $r_t + \gamma \hat{v}_{t+1}$ which is your TD target (what you try to approximate). So the tricky part in RL with FA is that you try to approximate something which is also an approximation of a true quantity.

I am not very sure what do you mean by making the value of γVt+1 closer to Vt−rt. For linear approximation, as you stated at the beginning of your question, the update needs to have the following form in order to reduce the MSE of the true value function: $\Delta w=\eta(v_\pi-\hat{v}_w)\phi(s)$. As I mentioned above because in RL we don't have knowledge of the true value, but only reward signals, we substitute the target (true value) with the target form I described in the previous paragraph.

Your first proposed update, first of all, eliminates the discount factor which ensures that in an infinite horizon scenario the sum of rewards converges into a real value and not to infinity. Even if you consider episodic tasks the sign change makes your weights updating into a direction that doesn't follow the gradient of the MSE. In your second update again you use a an arbitrary rule for updating and not derived through gradient descent. By the way cutting in half the learning rate does not help in any sense as again you can define a new learning rate.

I refer you to these documents in which you can clarify how the update rule is defined and why it has that particular form and what are the optimality criteria and the role of the discount factor in RL:

RL with FA,
2.3 Optimality Criteria and Discounting

Hope this helps!


You can use distance metrics that are better suited to boolean values such as Jaccard, or Manhattan. Usually K-means uses mean as the metric to re-adjust the centroid of the current cluster, but I think you could change this to majority vote.
I think if you one-hot encoded the categorical features, you could use Jaccard for similarity and then majority vote for updating the centroid. Jaccard would be good if your data is quite sparse. Can you reform your data so that it is one-hot encoded? What does the distribution look like of the count per row?


Using Keras to Predict a Function Following a Normal Distribution

My guess is that the problem is with your first layer: model.add(Dense(1, input_dim=1, activation='tanh',kernel_initializer = 'normal')). With a single hidden unit, a lot of information from the input layer will not be available to subsequent layers.


Apache Spark currently has no Deep Learning libraries. However, you can run TensorFlow models on clusters. According to this article "The TensorFlow library can be installed on Spark clusters as a regular Python library".

Additionally, I just found the SparkFlow module, that should be meant to interface Spark and TensorFlow. The GitHub repo is here.


When to use different Word2Vec training approaches?

The advantage of using pre-trained vectors is being able to inject knowledge from a larger corpus than you might have access to: word2vec has a vocabulary of 3 million words and phrases trained on the google news dataset comprising ~100 billion tokens, and there's no cost to you in training time.

In addition, they are fast and easy to use, just load the embeddings and look them up. It's straightforward to substitute different sets of pre-trained vectors (fastText, GloVe etc) as one might be more suited to a particular use case.

However, when your vocabulary does not have an entry in word2vec, by default you'll end up with a null entry in your embedding layer (depending on how you handle it). You'll need to consider the scale/impact and how to address it (keep/discard/consider online training). As yazhi says a decision must be made about how to handle out of vocabulary words.

The advantage of learning word vectors from your own corpus is that they would be derived from your dataset, so if you have reason to believe that the composition of your data is significantly different from the corpus used for the pre-trained vectors then that may result in better downstream performance. However, that comes at a cost in time taken to train your own vector representations.


Imbalanced dataset with 3 classes xgboost scale_pos_weight parameter

For my multiclass classification problem with similar unbalanced data I used the output from sklearn compute_class_weight function:

https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html

sklearn.utils.class_weight.compute_class_weight(class_weight, classes, y)


Any differences in regularisation in MLP between batch and individual updates?

Regularization is relevant in per-item learning as well. I would suggest to start with a basic validation approach for finding out lambda, whether you are doing batch or per-item learning. This is the easiest and safest approach. Try manually with a number of different values. e.g. 0.001. 0.003, 0.01, 0.03, 0.1 etc. and see how your validation set behaves. Later on you may automate this process by introducing a linear or local search method.

As a side note, I believe the value of lambda should be considered in relation to the updates of the parameter vector, rather than the training set size. For batch training you have one parameter update per dataset pass, while for online one update per sample (regardless of the training set size).

I recently stumbled upon this Crossvalidated Question, which seems quite similar to yours. There is a link to a paper about a new SGD algorithm, with some relevant content. It might be useful to take a look (especially pages 1742-1743).


Dropping attributes leads to better classifier accuracy? (Titanic Set)

I can't say for sure why this happens for this particular data set, but, presumably "Fare" and "Class" are highly correlated. The classifier may do better reasoning about one of them rather than splitting its reasoning across both, as they have mostly the same information.
It's also possible indeed that just knowing whether the person had any family members is most of the useful info, and trying to reason about how many is unhelpful as it is mostly noise that the classifier will overfit.
It's also possible that, given the tiny data set size, you can find lots of ideas that happen to fit the data better, but wouldn't necessarily generalize.


Difference between multinomia logistic regression and N 1 vs all binary

Quote from Alan, Agresti. "Categorical data analysis." A John Wiley and Sons, Inc. Publication, Hoboken, New Jersey, USA (2002), page 273:

  The separate-fitting estimates differ from the ML estimates for simultaneous fitting of the J-1 logits. They are less efficient, tending to have larger standard errors. However, Begg and Gray 1984 showed that the efficiency loss is minor when the response category having highest prevalence is the baseline.


Feature importance with scikit-learn Random Forest shows very high Standard Deviation

You are using RandomForest with the default number of trees, which is 10.
For around 30 features this is too few. Therefore standard deviation is large. Try at least 100 or even 1000 trees, like

clf = RandomForestClassifier(n_estimators=1000)

For a more refined analysis you can also check how large the correlation between your features is.


After having performed

X=np.array([[1,0,0,0,0],
           [0,1,0,0,0],
           [0,0,1,0,0],
           [1,0,0,1,0],
           [1,0,0,0,1]])

y=np.array([0,1,1,1,0])

you have to do the following:

y = y.reshape(1, -1)

model=svm.SVC()
model.fit(X,y)
test = np.array([1,0,1,0,0])
test = test.reshape(1,-1)
print(model.predict(test))

In future you have to scale your dataset. You can use either Standard Scaler (suggested) or MinMax Scaler.


How to determine the correct target for classification probability when the observed samples are probabilities of each class?

Imho the "cleanest" option would be to train a probabilistic model on the original categorical target, then obtain the predicted probabilities for every category as the final "predictions". By "training on the original target" I mean designing each instance as an event, e.g. in order to represent that 7/10 people select category A there would be 7 instances where the target is category A out of 10 instances in total.

The most simple option is Naive Bayes, but depending on the data it tends to always predict extreme probabilities, which would defeat the purpose.
An ad hoc Bayesian model could give very good results but it's probably more work to design it, depending on the features.


Linear Regression in Python using gradient descent

That could be due to many different reasons. The most important one is that your cost function might be stuck in local minima. To solve this issue, you can use a different learning rate or change your initialization for the coefficients.
There might be a problem in your code for updating weights or calculating the gradient.
However, I used both methods for a simple linear regression and got the same results as follows:
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import make_regression

# generate regression dataset
X, y = make_regression(n_samples=100, n_features=1, noise=30)

def cost_MSE(y_true, y_pred):
    '''
    Cost function
    '''
    # Shape of the dataset
    n = y_true.shape[0]

    # Error
    error = y_true - y_pred
    # Cost
    mse = np.dot(error, error) / n
    return mse

def cost_derivative(X, y_true, y_pred):
    '''
    Compute the derivative of the loss function
    '''
    # Shape of the dataset
    n = y_true.shape[0]

    # Error
    error = y_true - y_pred

    # Derivative
    der = -2 / n * np.dot(X, error)

    return der

# Lets run an example

X_new = np.concatenate((np.ones(X.shape), X), axis = 1)
learning_rate = 0.1
X_new_T = X_new.T
n_iters = 100
mse = []

#initialize the weight vector
alpha = np.array([0, np.random.rand()])

for _ in range(n_iters):

    # Compute the predicted y
    y_pred = np.dot(X_new, alpha)

    # Compute the MSE
    mse.append(cost_MSE(y, y_pred))

    # Compute the derivative
    der = cost_derivative(X_new_T, y, y_pred)

    # Update the weight
    alpha  -= learning_rate * der
alpha

for the gradient descent the coefficients were:
array([-3.36575322, 28.06370831])

Here is the code for closed-form solution:
np.dot(np.linalg.inv(np.dot(X_new_T,X_new)), np.dot(X_new_T, y))

And the coefficients for the closed-form solution:
array([-3.36575322, 28.06370831])

As the coefficients are equal, the RMSE,  MSE, R2 are equal.


How to choose an optimal threshold for binary discretization

Inspecting the histogram chart of that feature by different numbers of bins can be helpful. the maximum width of the bins in which the distribution of the data does not change noticeably can be a suitable choice.


LSTM not learning with extra nontemporal data added after LSTM layer - Keras

if anyone stumbles upon this - it seemed like my dimension of nontemporal data was too high and sparse. After filtering only for the most common/important variables in the nontemporal data, the model learned.


The following approach works perfectly,

l <- list(a=A,b=B)
dataset <- do.call(rbind, l)[order(sequence(sapply(l, nrow))), ]


Is it possible to implement a Recommender System without having a ratings/previous purchases similar data?

In the complete absence of user feedback, you can still build a (non-personalized) recommender system based on the products' descriptive features, e.g. by employing some distance metric.
In your case, there are at least two behavioral signals: clicks and search terms. Both can be used to personalize recommendations. I would start with getting familiar with session-based recommendations, which take into account user's actions in the context of the current session, and refine the recommendations incrementally (a kNN algorithm would be a good starting point).


You could compare the relative values over time quite easily with a "stacked" graph, where each of the processes is a layer in the stack. In Pandas' plotting, it is referred to as an area plot.

Here is an example with some dummy data:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

p1, p2, p3 = np.random.random(100), np.random.random(100), np.random.random(100)

df = pd.DataFrame([p1, p2, p3]).T
df.head(2)
# output
#              0         1         2
#    0  0.542490  0.099974  0.831589
#    1  0.988922  0.035026  0.752813

df.plot(kind="area", stacked=True); plt.show()

I think this might allow you to see the relative swings in each of the processes quite clearly - you could also shift the slightly to be in phase, which will make things easier.


Best practice for computing week-over-week change in MAU metric

Both of these options have value if interpreted properly. You are trying to answer  the question (I assume): 'are we doing better this week than last week?'

I don't know what industry you're in or how variable (or volatile) your metric is, but I'd advise caution in interpreting changes week to week. I might suggest that alongside your weekly (or whatever) metrics, you include some moving average. That way, you capture the overall trend while tempering your reaction to occasional ups and downs.


I have an answer now for my question.
I will share briefly the main steps / technologies I used to deploy the model in production.

I am using Python programming language.
After training and generating valid models I wrote a restful api using Python programming language and flask.

Using flask you can write a restful api. Three important points:

1- It is very important to give attention to where you will define the model architecture/initialize the parameters/ define the session.
Avoid doing this each time you call the restful api. this will be very expensive.

2- Flask provide a very good mechanism to run servers in production environment.
Read about flask + wcgi
Avoid runing the server code (the resful api) directly, in this case you will not have direct and full control.

3- Watch the memory and the cpu usage, make sure to limit the maximum number of instances that can run in parallel. Remember these models can take a lot from the memory.

unfortunately, I can not share codes with public. But hopefully my answer can give an idea about how to do it in production.


Training on accurate data versus noisy data

The answer to this severely depends on what you mean by 'noisy' data. Are the labels noisy i.e. wrong? Or are the features noisy? Or both? If only the features are noisy, definitely use the noisy data and probably also the clean data. If only the labels, definitely do not use the noisy data. If both, is it possible to correct the labels? At the very least, can you get a reliable test set (representative with correct labels)? You could try training on both the noisy and clean data and see which gives better performance. Important methods to consider here would be regularization and early stopping.

It also depends on what algorithm you are using. Linear regression, for example, is unlikely to over-fit, while neural networks are extremely sensitive to noise, and there are a variety of approaches in between those two extremes.


Can we make two separate models vs one for classification?

That doesn't make much sense, no. If you provide all the data to both models, they should perform exactly the same. If you only provide positive data to each model, you're making two incomplete models that (supposedly) won't learn as well as using all the available data. Making multiple models to a single response variable can make sense, but not when the response is binary.

If you happen to also have unlabeled then it's a different thing, because you can use techniques such as the two-view models. But in practice, it's often more useful to spend resources labeling data than developing a semi-supervised model.


This will always happen since in multi-output classification (OnevsOne or OnevsRest) classes are mutually exclusive i.e probabilities will always sum up to one.
If you want independent probabilities you could try rewrite your target so that this is a multi-output classification (not a multi class) problem
toy example with iris data:
from sklearn.datasets import load_iris
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.multioutput import MultiOutputClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder

X, y = load_iris(return_X_y= True)

X_train, X_test, y_train, y_test = train_test_split(X,y,random_state= 42, stratify= y)

model = RandomForestClassifier(random_state= 42).fit(X_train,y_train)

pd.DataFrame(model.predict_proba(X_test)).head(5)

Output:

Rewriting in a multi-output classification:
y_train = y_train.reshape(-1, 1)
y_test = y_test.reshape(-1, 1)

encoder = OneHotEncoder(handle_unknown= "ignore").fit(y_train)

y_train_multiclass = np.array(encoder.transform(y_train).todense())
y_test_multiclass = np.array(encoder.transform(y_test).todense())

model = MultiOutputClassifier(estimator= RandomForestClassifier(random_state= 42)).fit(X_train,y_train_multiclass)

pred = np.array(model.predict_proba(X_test))

pred[:,:,1].shape

prob0 = pred[:,:, 0].T
prob1 = pred[:,:, 1].T

pd.DataFrame(prob0).head(5)

Output:

Yo can see that in this approach the probabilities might not sum up to one, so it may happen that if you pass a document that does not belong to any category I should return 0 for all the classes
Other option would be to use a neural net model and changing the utput via the activation function of the output layer as mention on this answer:
Should estimated probabilities from multi class classification sum to 1


Probability distribution in input-output pairs

Some notes:

In the case of supervised learning you assume there is a function $f:\mathcal{X} \rightarrow \mathcal{Y}$, which means there is a connection somehow between inputs and outputs and you want to use it in order to predict. But how this function might look? Without considering the model itself, usually what happens is that there is some noise. And this noise can be in $Y$ and also can be in $X$. This noise gives use the probabilistic setup of the problem, because without it we would have only to solve some equations.
Now is important to understand that the noise defines the distribution. So a random variable can be imagined as a function having something fixed and well defined and something not fixed, but taking values according with a distribution. If the variable part does not exist, that you would not have a random variable, right, it would be a simple formula. But its not. Now the $P(X)$ incorporates what happens in $X$ alone, and $P(Y)$ what is in $Y$ alone. When you predict the decision theory says that you are interested in saying which is the most probable value $y_i$ given some input value $x_i$. So you are interested in finding $P(Y|X)$.
A joint probability is not always completely described by marginal probabilities. In fact, is completely described only when marginal probabilities are independent. Which means for r.v. $X, Y$ knowing $P(X)$ and $P(Y)$ does not put you in position to know the joint density $P(X, Y)$ (thought for independence you have  $P(X,Y)=P(X)P(Y)$).
From here you can go directly and try to estimate $P(Y|X)$. In fact if your only interest in only in prediction, this might be a fair bet. A lot of supervised learning algorithms tries to estimate directly this probability. They are called discriminant classifiers. The reason is because they are used to classify something to  the class with maximal conditional probability, you discriminate (choose) the maximum value.
Now arriving at your question. Notice the obvious $P(X,Y) = P(Y|X)P(X)$ than you see that by trying learning the joint probability, you also learn the conditional (what you need for prediction). This kind of approach is called generative, because knowing the joint probability you can not only predict, but you can generate new data for your problem. More than that, knowing the join probability can give you more insights related with how what are you model works. You can find such kind of information which is not contained only in marginal distributions.

Some final notes:

Technically you do not minimize the error function, but it's expectation. The error function remains as it is.
$\mathcal{Z}$ is only a domain it's impossible to describe a probability only by it's domain.

It's late, I hope I was not completely incoherent.


My CNN produces volatile validation_loss and does not converge in python(Keras)

It seems that there is some form of randomness or too many parameters in my model. I am aware that it is currently overfitting, but that should not be the cause of the volatility(?)

It is quite plausible that this behavior is caused by a combination of the model's excessive capacity and the data that you are feeding into it.

As you have pointed out, overfitting is obvious here - the model cannot generalize well and fails on unseen data. If you reduce model capacity (better yet, start off with the bare minimum and incrementally build up from there), then the network will be forced to learn instead of performing memorization. Since you mention that you have already tried tuning rates/layers/sizes, the model alone cannot be the cause, although maybe someone with more expertise in convolutional layers can provide useful insight.

The lack of convergence can be an indicator that there is simply not enough data to learn from - neural networks aren't guaranteed to converge when there is insufficient data. The required amount depends greatly on the specifics of the task.

  my data set consist of about 1800 images...600 of each dog

That's quite a small dataset, considering that:

  Two dogs are very similar and the 3rd is very different.

So roughly two thirds of your images can cause confusion. The more they look alike, the more samples you will need to capture the details.

  The videos are taken on different backgrounds, but +- the same amount on each background for each dog.

The model could then erroneously try to discriminate between the two similar-looking dogs based on the background alone (think worst-case scenario).

In your case, I would look at the data more closely, investigate which images are consistently the easiest/hardest for the model to classify and take full advantage of ImageDataGenerator's image augmentation to improve generalization capabilities. See how the model responds if you place similar-looking dogs under the same label, or have one of them appear exclusively in training, the other in validation. Maybe there are some answers there.


Could you share the code? It is kind of hard to debug with out the code

When you do pipeline.predict it uses the transform method for all the steps.

Might be that in the One Hot Encoding you have not specified how to handle the unknown.

from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(handle_unknown='ignore')


How to preprocess different kinds of data (continuous, discrete, categorical) before Decision Tree learning

One of the benefits of decision trees is that ordinal (continuous or discrete) input data does not require any significant preprocessing. In fact, the results should be consistent regardless of any scaling or translational normalization, since the trees can choose equivalent splitting points. The best preprocessing for decision trees is typically whatever is easiest or whatever is best for visualization, as long as it doesn't change the relative order of values within each data dimension.

Categorical inputs, which have no sensible order, are a special case. If your random forest implementation doesn't have a built-in way to deal with categorical input, you should probably use a 1-hot encoding:

If a categorical value has $n$ categories, you encode the value using $n$ dimensions, one corresponding to each category.
For each data point, if it is in category $k$, the corresponding $k$th dimension is set to 1, while the rest are set to 0.

This 1-hot encoding allows decision trees to perform category equality tests in one split since inequality splits on non-ordinal data doesn't make much sense.


How are selected the features for a decision tree in CART?

At each split point, CART will choose the feature which "best" splits the observations. What qualifies as best varies, but generally the split is done so that the subsequent nodes are more homogenous/pure with respect to the target. There are different ways of measuring homogeneity, for example Gini, Entropy, Chi-square. If you are using software, it may allow you to choose the measure of homogenity that the tree algorithm will use.

Distance is not a factor with trees - what matters is whether the value is greater than or less than the split point, not the distance from the split point.


Add Custom Labels to NLTK Information Extractor

Once you have your own lists of named entities, and you're only interested in extracting the relations, I believe there are simpler solutions (although I never tried relation extraction in NLTK, so I might be wrong):

ReVerb - a tool written in Java. Once it produces the results, you can simply keep the rows, where your labels are present as objects of the relation.
OpenIE - the successor of ReVerb (also written in Java). The authors claim better perfomance, and the output might be more informative.
IEPY - a relation extraction tool in Python. You should be able to provide your own labels/named entities using gazetees.
MITIE - this library has bindings in Python, and it offers relation extraction functionality.


Perform Person Re-Identification on custom image set

I haven't tried testing images from a custom dataset on AlignedReID but I have used another implementation laid out in the paper In Defense of the Triplet Loss for Person Re-identification.

The model is a simple feed-forward CNN which returns a 128 dimensional feature vector for each image. You can find the feature vectors for test images from your custom dataset and associate images with identities based on Euclidean distance for instance.

You can find the Github repository here.


How to take advantage of variables whose values are available in the past but not in the future?

My generic answer to the title is to use the extra data for regularization in representation learning; a transformation of your features into a space conducive to your main task: regression (prediction, forecasting). Here's a survey [PDF].

For your example, you could build a model that takes the delay of the target time from the present as an input, so you can predict arbitrarily far into the future, though it probably would not predict as well as a simple regressor that has a fixed horizon since it is trying to learn a more complex function.


Dataset with some mislabeled data (around 1%)

It think it's a reasonable approach, but currently it seems that you have no way to check whether the new labels are correct or not. I think you should at least check that the new labels don't introduce more errors than they solve.
Ideally you would re-annotate a random sample of instances, keeping both the old (possibly erroneous) labels and the new ones. Then you can use this sample as a test set and evaluate the two following points:

most/all instances for which the new label is the same as the old label should be predicted with this label (otherwise it means your method changes correct labels)
most instances for which the new label is different from the old label should be predicted with the new label (otherwise it means your method doesn't fix the wrong labels)

The problem with this approach is that you need to annotate a large sample, since you need a reasonable number of wrong labels which are only present in 1% of the data.
If re-annotating a large sample is not possible, you could try a kind of boostrapping approach: run your method, then take a sample of instances which are predicted as different from the old label. Among these labels changes, count how many are correct. This approach requires less manual annotation effort since you don't need a large random sample, however it would miss the cases of wrong label which is not changed by the classifier.


If you want to estimate demand, no, you do not need to train the model with unsold items for which you have supply, since they do not affect it. Having an excess supply/capacity does not affect your customer behaviour.

If you have supply, but no demand, it needs to be there.

However, also take into account that if you have had a demand greater than the supply at any moment, you have to do proper demand unconstraining to have a good forecast of your demand.


ValueError: Shapes are incompatible when fitting using ImageDataGenerator

Have you tried setting class_mode='categorical' in your generators?

I believe class_mode='sparse' works when your loss function is defined as 'sparse_categorical_crossentropy' whereas 'categorical_crossentropy' works for class_mode='categorical'.


Value Function of Generative Adversarial Network

Your notation is a little confusing, but I suspect this is because you're not reading the original equation exactly right. $\mathbb{E}_{x \sim p_{data}(x)}$ means "the expectation over $x$ drawn from the distribution described by $p_{data}(x)$". It looks like you're trying to multiply the expected value of $x$ times $\log(D(x))$ for all $x$, which isn't what's going on here and doesn't quite make sense in this notation.

The way to understand this formula is to relate it to the goal of the GAN paradigm. The first term represents "the expected ability of the discriminator to tell that real data is real". The second term means "The expected ability of the discriminator to tell that the generated data is generated". The goal is to maximize with respect to $D$ and then minimize with respect to $G$; this means "we want our discriminator to be really good at telling fake (generated) data from real data" and then "we want our generator to be really good at fooling the really good discriminator".

Hope this helps, good luck!


How many images can be trained in Google Colab?

The real problem is that you should not try to fit all your images in memory. Instead, you should small groups of images, normally called "minibatches", which can fit in the GPU/CPU memory.
For that, tensorflow offers the function tf.keras.preprocessing.image_dataset_from_directory that loads images from a directory. I suggest you take a look at Tensorflow's tutorial for image loading, which guides you to the image loading process, as well as the training of a model with those images.


Xgboost in R: Dealing with extreme class imbalance

The drop from 64 to 50 doesn't seem like overfitting, if your test set was properly held back from model development.  (And from the confusion matrix it looks like you haven't made the fairly common mistake of upsampling before splitting.)

If the "completely different test data" is not identically distributed to the training data, that can account for the discrepancy.  There aren't universally good ways to account for failure of the iid data assumption; you could try to underfit the model more, on the assumption that there's some underlying large trend that's constant across datasets and the model is finding (and overfitting to) some additional, more localized, trend in the train/validation set.


Orange file import method for raw binaries

What is the problem exactly?

When you create Domain you have to specify each Feature eg:

from Orange.data import Domain, Table, ContinuousVariable, DiscreteVariable

d = Domain([ContinuousVariable.make('Attribute 1'), DiscreteVariable.make('Attribute 2')])

Here is a link to doc, regarding Domain (yes, it's not the best)


My view on your question, is that tiny networks seldom work. The above method uses a Neural Network to learn the function $y=\sin(x)$. Although this problems seems simple, it cannot be expected to be solved by a really tiny network (the above model uses a 5-layer MLP with hidden size [5,6,7], which is small).
Even if back-propagation is implemented correctly, would the model learn anything? No. I suppose Tensorflow implemented back-propagation correctly, here is the result using Tensorflow:

You see, it learns almost nothing. In fact, the MSE loss is very close to 0.5 as stated above.
My suggestion is to try a 3 layer MLP with hidden size 256. Here is the result:

You can see it's much better. MSE<0.1 now.
------------------code---------------------

x_ =np.atleast_2d(np.arange(0,360,1)).T
y_ = np.atleast_2d(np.sin(x_/180*np.pi))
g = tf.Graph()
with g.as_default():
    with tf.variable_scope("mlp"):
        input_x = tf.placeholder(shape=[None, 1], dtype=tf.float32)
        input_y = tf.placeholder(shape=[None,1], dtype=tf.float32)
        layer1 = tf.layers.dense(inputs=input_x, units=256, activation=tf.nn.sigmoid)
        #layer2 = tf.layers.dense(inputs=input_x, units=6, activation=tf.nn.sigmoid)
        #layer3 = tf.layers.dense(inputs=input_x, units=7, activation=tf.nn.sigmoid)
        output_y = tf.layers.dense(inputs=layer1, units=1) # inputs=layer1
        loss = tf.losses.mean_squared_error(input_y, output_y)
        train_op = tf.train.AdagradOptimizer(0.01).minimize(loss)
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        for epoch in range(300):
            _, loss_ = sess.run((train_op, loss), feed_dict={input_x:x_, input_y:y_})
        y_hat_ = sess.run(output_y, feed_dict={input_x:x_, input_y:y_})
        print(loss_, end='\t')
plt.plot(x_,y_, 'g', x_,y_hat_,'b')
plt.legend(['ground truth', 'predicted'])


Is a test set necessary after cross validation on training set?

In "The Elements of Statistical Learning" by Hastie et al the authors describe two tasks regarding model performance measurement:

Model selection: estimating the performance of different models in order to choose the best one.
Model assessment: having chosen a final model, estimating its prediction error (generalization error) on new data.

Validation with CV (or a seperate validation set) is used for model selection and a test set is usually used for model assessment. If you did not do model assessment seperately you would most likely overestimate the performance of your model on unseen data.


Interpreting Gradients and Partial Derivatives when training Neural Networks

My question is, when we calculate partial derivative with respect to one parameter (e.g. weight between input x1 and 1st hidden layer neuron) then we are treating all other weights and biases as constants and we are evaluating how will cost function change if we were to take a step in the direction that is represented by that particular weight. Is this correct?

Yes.

Long answer: that is exactly the meaning of partial derivative. It's the impact of a specific variable, while keeping all the others constant. It's an all things equal condition (or ceteris paribus, if you like Latin).

You need it in order to understand how much that specific parameter, at that current value, is contributing to the final Loss. You have to keep everything else equal in order to understand what is that parameter's responsibility of the final model's error.

  Also, what is the role of Jacobian matrix in NN training?

More generally, the Jacobian matrix of a Neural Network is a matrix of the partial derivatives of the y. It's size is:

( Number of observations, Number of parameters )

It's a way to store/represent your gradient information. The application of the chain rule to execute backprop is executed on those in practice.


It has learnt to recognize the digits, but it might have put too much weight on single pixels. Try to add different amounts of L2 regularization or dropout and compare the visualizations of the weights. Adding some kind of regularization should make the net rely less on single / independent pixels and more on the inherent structure of the digits, giving you smoother weights / visualization.


Scikit-Learn - Learned model description?

For the particular case of the MultinomialNB you can look here.

However if want you want is to determine which features are the most important, the SelectFromModel for selecting the most important features for the model.


Dropping features after final evaluation on test data

What you're doing is manual feature selection based on the test set. You're right that it's not correct to proceed this way: in theory, feature selection should be done using only the training set and a validation set, not the final test set. The risk is data leakage: you're modifying the model using information from the test set. Maybe the performance is better without these features on the test set because they happen to be bad for this particular current test set by chance. As a result the model could be overfit, and you wouldn't be able to detect this problem on this test set since it's the source of the overfitting.
So in principle it's always better to separate the data first, keep the test set aside until the final evaluation and use a validation set for intermediate evaluation until the final model (including the set of features) is determined.
In practice, it sometimes happens that we realize that we should have done something differently after applying the model on the final test set. It's a mistake but it's not the end of the world, usually the risk of bias is low. As you said, obviously we don't have to ignore an information which is important for the performance of the model. However if you know that you're going to repeat this procedure with several features, you should defintely do it using a separate validation set (taken from the training data), not the test set: the more you use the test set like this, the higher the risk of data leakage and bias.


What are the limits of capping ratios to [0-1]?

In essence, you risk biasing the model by making the values 0 and 1 more likely than they actually are. This may or may not be a significant problem; really only cross-validation will tell you.

I would go ahead and try it, but compare your results to a different method of filling invalid values, like using the mean() of the valid values plus an extra feature flagging the samples that were invalid. Make sure you cross-validate using a hold out dataset that's been transformed in the same way; you can then select the method that results in the best results of the cross-validation process.


Does binning a time series with pd.qcut (using quantiles) create data leakage?

I think that indeed you may have leakage by using pd.qcut. A solution to avoid that leakage is to do it in a time-series cross-validation fashion. The idea is to derive the quantile values in a training fold, and, with those values, do the cut in its validation fold.
This is a little complex and if you want it simpler you can use pd.cut, which will have no leakage issues as long as you don't derive the cuts from the full distribution. If you use some default cuts defined by you, the risk of leakage will be very low.


You can have that but I recommend something else for your case. Suppose that you have 10 classes of digits that they are mutually exclusive. Consequently, your output vector will always be on-hot encoded. In a simple case, your output vector should contain ten entries for each class. To make your model distinguish between those inputs which are digits and those which are not, do the following:

Increase the size of output to eleven. Provide data-set that contains eleven-entries labels. the first entry corresponds to the existence of a digit. If it is a digit the first entry should be one and next 10 entries should be activated for the according to the value of the digit. So, your labels will not be one-hot encoded. If the input is not a digit the first entry of the output should be zero and other entries are don't cares.


Best way to evaluate performance for my case

It's not possible to obtain a single f1-score when there are several classes, probably you are using micro or macro f1-score?

First don't use the micro F1-score, since it gives the same weight to every instance and therefore gives a lot of weight to the "X" class.

Macro F1-score should be closer to what you want to evaluate, since it gives every class equal weight. If you want you could even calculate the macro F1-score over only classes A,B,C (i.e. excluding X), but that seems unnecessary to me.


Using a GAN discriminator as a standalone classifier

Yes, we can use the Discriminator of the GAN to classify images. But we should make sure that the images produced by the Generator are real looking.

If you have trained your GAN on a large number of images and it is performing pretty well on the dataset then I insist you to treat the Discriminator model as a pretrained model ( like we do in transfer learning ) and again train this model on images which were not used to train the GAN earlier. Thus the model is fine tuned on the dataset on the GAN wasn't trained before.
Another similar way could be to only use the weights of the CNN layers and load them in our new model.

  Suppose I train a GAN until the generator is good enough to fool the
  discriminator much of the time. Could I then use the discriminator as
  a classifier that tests whether an input belongs to a single class?

You should definitely try reusing a Discriminator from a GAN and share your results too :-).


Combining neural networks with different variables

This process is called fine tuning frequently in the neural network literature. If you already have the trained network, you simply run additional training steps on the new data.

You'll need to balance the weight of the prior training with that of the new observations to some degree, and that decision is parameterized by the number of free variables in the network, the learning rate of the new training steps, and the number of new training steps.

However, your particular case is very complicated by the presence of new variables. Maintaining the form of the prior network is difficult in this case. You may want to simply ensemble a newly trained network on the new data and variables with the old network to combine the predictions.The simplest ensemble (but maybe not the best) would be averaging the results of the two models.


What is the difference between regplot and lmplot in seaborn?

regplot() performs a simple linear regression model fit and plot. lmplot() combines regplot() and FacetGrid.

The FacetGrid class helps in visualizing the distribution of one variable as well as the relationship between multiple variables separately within subsets of your dataset using multiple panels.

lmplot() is more computationally intensive and is intended as a convenient interface to fit regression models across conditional subsets of a dataset.

You can research this comparison for yourself in the seaborn docs for regplot(), lmplot(), and FacetGrid to see which function will meet your needs.


Why does a filter need to be applied to the output of the input gate before cell state is added to?

Here is my hypothesis : The $i_t$ can add explainability to the model, as the value of the sigmoid function can give an idea as to how important a particular word is to altering the cell state $C$. This is because $i_t$ lies between 0 and 1. Having single $W$ do both filtering as well as a feature transform of $[h_{t-1},x_t]$ not only puts more stress on the matrix (has to do two things at once), but also no longer has this explainability factor.

Example: 2 vectors may require the same transformation $W*v$, but unless you allow seperate sigmoid function to give each an importance, their contribution to the cell state will remain same.


While I am not sure this is what you are looking for, here is what I think you are asking.

First, you are going to need an additional column. Two ways to get er done.

Calculated column

=CALCULATE (
AVERAGE ( Range[Value] ),
FILTER ( ALLSELECTED ( Range ), Range[Group] = EARLIER ( Range[Group] ) )
)
Measure

=CALCULATE (
AVERAGE ( Range[Value] ),
FILTER ( ALLSELECTED ( Range ), Range[Group] = MAX( Range[Group] ) )
)

Now edit your pivot table:

Now select your chart type:

For future reference you should look into Power BI.  It has features which do these sorts of calculations automatically.

If your looking for something different it would help is you provided the expected outcome.


Visualizing ConvNet filters using my own fine-tuned network resulting in a "NoneType" when running: K.gradients(loss, model.input)[0]

SOLVED:  I had to use: grads = K.gradients(loss, model.get_layer(my_Model_object).get_layer('input_1').input)[0]

instead of just grads = K.gradients(loss, model.input)[0]

which is confusing because both model.get_layer(my_Model_object).get_layer('input_1').input)[0]
and
model.input[0] print the same thing and are of the same type.


How to maintain CBOW dataset dimension and fit it in Neural Network?

This shows you did not properly flatten your output before the final dense layer. Your first model.add should be an embedding layer since that represents word vectors that'll be trained. The last layer should be the dense one. Hard to tell on the spot which hidden layers to consider, but try GlobalAveragePooling and Dense with the relu activation.


which classification model allows user to choose importance of data inputs?

The answer is that  you don't need to . Say you decide to go with some classification model, for instance a simple decision tree, the model in question will implicitly learn which are the importances (or weights) of the features of your training data, and will use these features as main nodes of the tree.

To better understand this idea, lets take as an example the ID3 algorithm, which is one of the several existing algorithms to generate decision trees from a dataset. This algorithm will build a tree by iteratively setting as decision nodes those features that maximise the information gain at each step, or in other words, those that are the best predictors.

So it will implicitly be giving more importance to the attributes that are better predictors, hence there is no need to assign weights to the features of the dataset.

So my suggestion is that you try using some classifier from scikit-learn such as RandomForestClassifier for instance.


There are three major assumptions (statistically strictly speaking):

There is a linear relationship between the dependent variables and the regressors (right figure below), meaning the model you are creating actually fits the data.

The errors or residuals of the data are normally distributed and independent from each other.

Homoscedasticity. This means the variance around the regression line is the same for all values of the predictor variable.

Update 2:: Multicollinearity is not an assumption, but it is rather a sanity check especially if interpretability of the model is important (thanks Ricardo Cruz for the comment). Multicollinearity occurs when the independent variables are not independent from each other. Multicollinearity between explanatory variables which can leads to less stable parameter fits (thanks KT. for pointing this out)). There are tests like correlation matrix (Pearson's Bivariate Correlation), Variance Inflation Factor that can be used to verify this.