Passing Dependency/Constituency trees to a Neural Machine Translator

First of all, I would like you to discourage you from using structured input in NMT. In most cases, the best thing you can do is just do some subword input and output segmentation and learn simple sequence-to-sequence conversion.
You can certainly pass the parsed input in the format that you showed above and the worse thing you can expect the model will to ignore the mark-up. Some modest improvements from using syntax in this way are shown in this WTM19 paper from Edinburgh.
People were more successful when using graph convolutional networks for processing the structured input, AFAI used for the first time in 2017. In this case, you would need to add a custom encoder into FairSeq and specify a model with this encoder. There are several PyTorch implementations of graph convolutional networks that you can reuse in FairSeq, e.g.,

gcn-over-pruned-trees

pygcn

AllenNLP also has some tools for structure input.

Note, however, that once you change model architecture, you need to be careful about the learning rate schedule that is otherwise tuned for the standard Transformers.


Do Activation Functions map to Higher Dimensions?:

We explain AI with intuition rather than maths most of the time, so everyone has its own explanation and representation of things, here is how I would explain activation functions (I'll try to make is clearer instead of adding another version to the already 2 you know):
Just in case you don't know what a basis is, you should have a look at Wikipedia since it is the core concept of your problem (to my eyes at least).

You need to consider the whole input values, not just their values. All input values are reals, so their dimension is 1, yet if we consider them all together, their dimension is N (number of inputs). And the basis of this space is $(Input1, Input2, ..., InputN)$.
Neurons are made of 2 parts :

linear combination of the inputs.
activation function

Let's imagine we use neurons that are only a linear combination of inputs $(Output = WA + B)$. Then we project our initial points into a space that has exactly the same basis as the input space (since all outputs are linear combination of the input if we forget about biases). So this action is just a remapping of the inputs into the same space, but with a different basis. This may help if your problem is linearly separable, but doesn't suffice if your problem is not That's why we use activation function.
If we consider neurons with an activation function now. The particularity of activation functions is that they are non-linear. So the use of activation functions maps your inputs into a different space, basis needs to be different since all outputs are not anymore linear combination of the inputs. So this time the output space is different from the initial one.
The way I see it is that activation function do not generate information from nothing, but they allow to remap the available information into a higher dimension space where the problem is easier to solve.
Hope this helps, feel free to ask if you have any remaining question.


Suggestions for binary classification algorithm

I would say support vector machine, support vector classifier. This will allow you to have as many predictors as you want. Also, you can improve your accuracy by tuning your model parameters.

Here is another nice link for support vector classification

Since you did not mention any language, I would use python as there are so much online support and you resources available online.

I would also check xgboost classifier.

Let us know how did it go.


Using neural network for "features matching" binary classification

From what I understand, your dataset is of pairs of images and a binary classification of their pairing?

There are models using an architecture named Siamese Neural Networks which are used exactly for this task of determining similarity.

You can start by reading the following article: Learning to Compare Image Patches via Convolutional Neural Networks. They are using Siamese architecture to compare between different image patches (similar to what you are describing):


Problem with sort by and group by in pandas

All you need is a groupby operation + aggregation on the min/max values.

df.groupby('id').agg(('min','max'))['date_column']

The output should be like this : different dataframe with each line containing an id and the min/max dates.


In the third paragraph of the first section (page 1), they define $\#(w)$, $\#(w; c)$, and $\#(c)$. Quoting such paragraph:

  Word-context pairs are denoted as $(w; c)$, and $\#(w; c)$ counts all observations of $(w; c)$ from the corpus. We use $\#( w ) =  \sum_c \#(w; c)$ and $\#(c) = \sum_w \#(w; c)$ to refer to the count of occurrences of a word (context) in all word-context pairs. Either $\sum_c \#(c)$ or  $\sum_w \#(w)$ may represent the count of all word-context pairs.

So, considering the equation you cited:

$\#(w_i, c_j)$ counts the occurrences of a specific word $w_i$ over the context $c_j$;
and $\frac{\#(c_)}{\sum_w \#(w)}$ accounts for the the probability of having context $c_j$ in the dataset.


The answer could be anything according to your data! As you can not post your data here, I propose to spend some time on EDA to visualize your data from various POVs and see how it looks like. My suggestions:

Use only price and quantity for a 2-d scatter plot of your customers. In this task you may need feature scaling if the scale of prices and quantities are much different.
In the plot above, you may use different markers and/or colors to mark category or customer (as one customer can have several entries)
Convert "date" feature to 3 features, namely, year, month and day. (Using Python modules you may also get the weekday which might be meaningful). Then apply dimensionality reduction methods and visualize your data to get some insight about it.
Convert date to an ordinal feature (earliest date becomes 0 or 1 and it increases by 1 for each day) and plot total sale for each customer as a time-series and see it. You may do the same for categories. These can also be plotted as cumulative time-series. This can also be done according to year and month.

All above are just supposed to give you insight about the data (sometimes this insight can give you a proper hint for the number of clusters). This insight sometimes determines the analysis approach as well.

If your time-series become very sparse then time-series analysis might not be the best option (you can make it more dense by increasing time-stamp e.g. weekly, monthly, yearly, etc.)

The idea in your comment is pretty nice. You can use this cumulative features and apply dimensionality reduction methods to (again) see the nature of your data. Do not limit to linear ones. Try nonlinear ones as well.

You may create a graph out of your data and try graph analysis as well. Each customer is a node, so is each product when each edge shows a purchase (directed from customer to product) and the weight of that edge is the price and/or quantity. Then you end up with a bipartite graph. Try some analysis on this graph and see if it helps.

Hope it helps and good luck!


Should bias updates be porportional to overfitting?

I think you are mixing up the bias of a model as in here, with the bias terms of a neural network which are just the constant term of the linear model of each layer. Updating the biases for training will not reduce overfitting since each bias is an additional parameter of the model.
Remember that the weights (and the bias is also a weight) are updated proportionally to the negative gradient of the loss function. Therefore there must be an error in your implementation since the training error gets larger which is highly unlikely for gradient descent (unless your learning rate is far too high).


Making sense of a accuracy plot for a 5 fold training using random forest

Using all default values for the RandomForestClassifier class leads to overfitting. As stated in scikit-learn documentation,

  The default values for the parameters controlling the size of the trees (e.g. max_depth, min_samples_leaf, etc.) lead to fully grown and unpruned trees which can potentially be very large on some data sets. To reduce memory consumption, the complexity and size of the trees should be controlled by setting those parameter values.

Fully grown trees have a perfect score on train data, but as you can guess, that is not how random forests are supposed to be used. Try setting the various parameters of the classifier (start with max_depth for instance), and you will get more consistent results.


So why we do not use gram matrix in loss function calculation between content and generated?

"Same content" means same objects in same locations in the image. If the original content shows a face at the top, we want to see a stylised face at the top.
"Same style" means same combinations of textures and line style together in any location in the image. The gram matrix is a good proxy measure of this, because it is invariant to the locations at which the style correlations occur, and similar line shapes and textures will tend to trigger the same combinations of low-level feature map channels.

If you used gram matrix for the higher order content channels, you would probably find that larger objects would be treated like a style and would be moved, duplicated and distorted in the final output. Depending on the trained network, which layer(s) you picked as representing content etc, the effect might still be interesting or artistic. But it would no longer be style transfer, but perhaps some hybrid of style transfer and deep dream.


Very basic question: what is an accepted term for "linear order distance"

Actually this is the Manhattan / $L_1$ / Norm 1 distance, if you just multiply the row by $8$ (or $N$).

The $L_1$ distance between $[24, 2]$ and $[32, 5]$ is what you are looking for.


Best practice on count of manual annotations for building criminal detection from news articles?

It always depends on each specific problem. The amount of data depends on a number of factors including (but not limited to) the complexity of the problem, the number of features, the quality of the training data, the ratio of the training classes (i.e. class imbalance), etc.

If you have no idea where to start, sometimes the best thing is just to experiment. For example, I would create a model with as much data that is both readily available and reasonably easy to handle in a cross-validation framework. From there, you can test the effects of increasing the training set size versus tuning the hyper-parameters. While you're not necessarily trying to find the point of diminishing returns (it's usually best to err on the side of too much than not enough), this type of exercise at least gives a general idea of the complexity of your problem.

A lot of people will have their own rule-of-thumbs or metrics; there are plenty of reading out there. For example, there's the Vapnik–Chervonenkis dimension and the one in ten rule (though I think computer vision people's rule-of-thumb is one in 1000, which again illustrates how variable these rule-of-thumbs are).


Algorithm Suggestion For a Specific Problem

You can use a clustering algorithm such as k-means to divide the generators into groups.  You never know what kind of groups you'll get until you try it.  Try and assess the character of each group of generators as you increase the number of clusters.  At some point you should find a meaningful division of generators.  The inputs to your k-means algorithm will be the criteria you mentioned in your post: the number of times it was activated, the number of activation problems, and so forth.  When you are finished, the group a generator belongs to is its ranking.  This method will not generate a ranking of 1-1000 if you have 1000 generators.  Rather it will give you, for example with k=3: a group of 243 outstanding generators, 320 average generators, and 446 terrible generators.


Transcription of Hindi text to English text

seq2seq RNN models:

Basically, we need to transform the script of the text from Hindi to English. We can do this with the help of seq2seq RNN models. These models are widely used in Neural Machine Translation. Know more from here.

  seq2seq RNN models can take the sequence of Hindi words as input and output their corresponding English words.

Note: As these models have done their best in translation between languages, I assume that they can transcript between languages in a pretty well manner.

The above shows how a seq2seq model is used for language translation. Such a model can be used for transcription as well. See this notebook for creating such a system in TensorFlow Keras. Refer to these links for more:

https://towardsdatascience.com/how-to-implement-seq2seq-lstm-model-in-keras-shortcutnlp-6f355f3e5639
https://www.geeksforgeeks.org/seq2seq-model-in-machine-learning/
https://www.analyticsvidhya.com/blog/2018/03/essentials-of-deep-learning-sequence-to-sequence-modelling-with-attention-part-i/

Limitations:

You can get training data for translation easily. But, there is no data available for transcription between Hindi and English.
For the data, you can try posting a question on OpenData StackExchange.

Tip:

There's a Google Cloud API for transcription. See here for more details.


How to compare different similarity measurements in text clustering?

I got the results of top n most similar titles, but all the results seem very reasonable, how can I compare them to see which method performs best?

I'm afraid there's no meaningful definition of "best" unless you specify some concrete metric.  If the results in each case seem reasonable, then all you can do is be educated about the various distance metrics and decide which makes the most sense for your use-case.  (For example, Euclidean distance measures spatial difference, while cosine similarity measures directional difference.)

  I also tried to do k-means, when I implement K-means clustering, it used Euclidean distance by default, how to use other distance to implement K-means?

Keep in mind that k-means is designed for Euclidean distance, which is why many implementations (such as scikit-learn) do not allow the use of any other distance function.  For many other distance metrics, such as cosine similarity, taking the mean of the points in the cluster is not appropriate, so you'd also need to replace the centroid-estimation function as well.

If you really want to, then the K-Means implementation from NLTK allows you to specify a custom distance metric.  Just keep in mind the GIGO principle.  If you violate the assumptions of the model, don't be surprised if you get poor results.


Different methods for clustering skills in text

You may want to consider preprocessing - convert different wording for the same type of talent to same wording. For example, a talent in machine learning is called data scientist at Coursera job site, data engineer at Udacity job site, or data analyst. This preprocessing is similar to stemming in concept.


What are the benefits of having ML in js?

There are a lot of services that offer free or very cheap hosting of static websites. If you are able to implement your ML model in JS this allows you to deploy your product/app/whatever easily and with low cost. In comparison, requiring a backend server running your model is harder to setup and maintain, in addition to costing more.


Classification problem in pytorch with loss function CrossEntropyLoss returns negative output in prediction

The numbers to which you are referring are probably the logits of the NN. The logits have to be transformed using a activation function which could be :

Softmax ( which you shown ) for classification tasks.
Linear activation function for regression tasks.

Softmax activation function :

The mathematical notion is as follows :

$\Large S( y_i ) = \frac{ e^{y_i}}{ \sum e^{y_i} }$

It converts the logits to class probabilities which sum upto 1. It is widely used in multiclass classification problems.

CNNs for classification need to produce class probabilities which are produced by a softmax activation function at the output layer.


Some suggestion for career in data science or predictive modeling

First of all I should say you question probably is an off-topic and will be closed soon.

Discussed at this SE site

Anyway I can target you to similar questions discussed at this SE site already:

Statistics + Computer Science = Data Science?
Starting my career as Data Scientist, is Software Engineering experience required?

Cross Validated SE

A set of relevant questions at Cross Validated Stack Exchange:

https://stats.stackexchange.com/questions/tagged/careers

Data scientist map

This is good infographics of data science knowledge you might need to start a career (Link to image):

Careers SE

Also, simple "data scientist" querying of Careers SE site
http://careers.stackoverflow.com/jobs?searchTerm=data+scientist&location= will lead you to the following knowlege "tags":

R
bigdata
data-visualization
hadoop
mapreduce
scala
python
matlab

etc.


You are right: decent journals/conferences are unlikely to accept a paper without a proper evaluation. Moreover the model is handcrafted, which probably means that it's hardly reproducible, right? And also that you can't do cross-validation I guess?

I think one would need pretty strong arguments to justify a contribution which has no scientific validation and cannot be reproduced. The only ways I can think of would be:

if the method solves a problem never solved before (or has specific characteristics which make it likely to solve such problem in the future)
if some sort of qualitative analysis demonstrates that the method is much better than state of the art approaches


Let me try to answer your questions point by point. Perhaps you already solved your problem, but your questions are interesting and so perhaps other people can benefit from this discussion.

Is Naive Bayes overfitting to the training set? If Naive Bayes is implemented correctly, I don't think it should be overfitting like this on a task that it's considered appropriate for (text classification).

  Naive Bayes has shown to perform well on document classification, but that doesn't mean that it cannot overfit data. There is a difference between the task, document classification, and the data. Overfitting can happen even if Naive Bayes is implemented properly.

Is my train/test split bad? I've tried splitting the data in different ways, but it does not seem to make a difference. Right now I'm splitting the data by placing a random 90% sample of documents into the train set and the rest into the test set - separately for each label.

  It's good that you're keeping the class distribution the same in your training and test set. Are you using cross-validation? Perhaps try it because, even though it's rare, it might happen that you just get unlucky with your splitting due to some seed.

Problems with numerical accuracy? I've implemented the calculations in log probabilities, but in any case, I would expect that problem to manifest in train set as well.

  You're correct, if it was an issue, it would show in the training set as well.

Problems with unseen words? The fraction of unseen words relative to a particular newsgroup is the same, 20%, for both train and test sets.

  This doesn't seem to be the issue then. You could perhaps reduce that percentage by using stemming or lemmatization.

Problems with Laplace smoothing? I wasn't sure what was the appropriate way to implement Laplace smoothing for this task, so I tried a variety of ways. To my surprise they all yielded very similar results.

  Laplace smoothing is useful especially when you do not have a lot of data and you need to account for some uncertainty. For this dataset, this doesn't seem to be an issue, as shown by the similar results you've experienced.

Imbalanced dataset? Doesn't look like it. There are approximately as many unique words inside documents of label 16 as there are in other classes. Also the number of documents per each label is roughly even.

  Are the documents of the same lengths? Because perhaps label 16 simply contains documents that are larger and therefore have bigger word frequencies. They might also contain very common words. It would be interesting to look at a histogram of the words in each class. This could be very informative in understanding whether label 16 is very different from the others.


From an algebra point of view PCA is a base change. You can write the transformation as:
$$ T = XW $$
Where $X$ is an nxp matrix (n instance, p features) and $W$ is a pxp 'weight' matrix (whose columns are eigenvectors of $X^TX$ - translating one base to another). Hence the general PCA has $p^2$ coefficients.
You can also reduce the space to m dimensions and use :
$$ T = XW_m $$
Where $W_m$ is a weight matrix constituted of m first columns of $W$. $W_m$ is a pxm weight matrix and has pxm coefficients (or easier to understand : mxp coefficients, that is number of dimension kept and parameters needed to express the new base vector in terms of the initial base). In the case of a projection onto 2 dimensions you will have 2p coefficients.


The most used loss function in tensorflow for a binary classification?

I think there is some confusion here. Softmax is usually an activation function which you will use in your output layer, and cross-entropy is the loss function that you will use.

Softmax

This activation function outputs the probability for each class, it is defined as

$\sigma(\bf{y_i}) = \frac{e^{\bf{y_i}}}{\sum_{j}{e^{\bf{y_j}}}}$.

For example, in a problem with 2 class labels. Then if we have some outputs from our neural network like $y$=[3, 4] then we can get the probability of each output classes by using the softmax function on these outputs.

import numpy as np

x = [3, 4]
np.exp(x)/np.sum(np.exp(x))

  [0.26894142, 0.73105858]

You will see that these are probabilities and do sum to 1.

Then we can get the class of the input by seeing which probability is higher. In this case class 2 has a probability of 73.11%, so the predicted class label, $\hat{y} = 1$.

Cross-entropy

Cross-entropy is a loss function that is used for classification tasks. For binary classification it is defined as

$H(p, q) = -y\log(p) - (1-y)\log(1-p)$.

Let's assume that the real class of the above example is 0, $y=0$. Then we made a mistake and you can see that

$H(p, q) = -0\log(0.26894142) - (1-0)\log(1-0.26894142) = 0.313$.

That is the loss that is used for backpropagation.


How to sample with guaranteed minimum spacing?

Let's say you have very frequent data across a period of time T and you want to sample N points.
Instead of sampling points directly you could sample the time that separate them. You then just need to add 12 hours to all gaps to enforce your constraints.
To do so you could sample N points uniformly in [0, T - (N-1) x 12h] and then compute the difference between consecutive points.
Sampling N + 1 points in [0, T - N x 12h] allows you to also sample the first timestep.
Afterwards you can match thoses times with the closest samples available in your data.
import numpy as np
T = 500
N = 200
min_diff = 0.5

t_diffs = min_diff + np.diff(np.sort(np.random.uniform(0, T - N * min_diff, N+1)))

times = np.cumsum(t_diffs)

import matplotlib.pyplot as plt
plt.figure()
plt.scatter(np.arange(len(times)), times)
plt.ylabel("Time")
plt.xlabel("Point index")
plt.savefig("fig.png")
plt.show()


gpu pytorch code way slower than cpu code?

TL;DR
GPU runs faster than CPU (31.8ms < 422ms).
Your results basically say: "The average run time of your CPU statement is 422ms and the average run time of your GPU statement is 31.8ms".
The second experiment runs 1000 times because you didn't specify it at all. If you check the documentation, it says:

-n: execute the given statement  times in a loop. If  is not provided,  is determined so as to get sufficient accuracy.

Number of runs
The %timeit function estimates the run time of a statement.
It does that by executing the statement multiple times and measuring the average run time of it.
The number of times it runs the experiment is $N\times{R}$, where $N$ is the number of loops and $R$ is the number of times it repeats it.
If you do not specify $N$ and $R$, the default $R$ is $7$ and $N$ may vary.
In your case, you did not specify any. So, in the first experiment, you had $R=7$ and $N=1$. The second experiment was $R=7$ and $N=1000$.
$N$ varies depending on how long it takes to run a single loop. If your statement is time-consuming, %timeit will execute fewer loops, whereas fast statements are executed more often. This is the case because it will give a better accuracy of the actual run time of your statement.
Results
You should not consider the time to run a statement based on the time that it takes to run the %timeit function. As you noticed, the number of loops may vary. Instead, try to interpret the printout.
For example, you had:
422 ms ± 3.93 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

422 ms ± 3.93 ms per loop: this is the mean average time and standard deviation it takes to run your statement one time. A loop is a single execution of your statement. Your statement is what you have defined when calling the function (in your case it is t_cpu ).
of 7 runs, 1 loop each: this is the number of times that %timeit ran your statement to calculated the average execution time. If you think that running the t_cpu only $7$ times is not reliable enough to estimate the $422ms$ then you may increase $N$ and $R$.
Documentation
I really don't mind answering your questions here. However, if you want faster clarifications to your answers, a good resource to read is usually the documentation of the function itself.
The documentation of the %timeit function can be found here.


Yes, you can do that by add them to the existing NLTK Stop-word dictionary for all such words/Creating a Custom Stop-word dictionary.

for Custom Stop-word dictionary you need to include all the key words in the dictionary before processing, make sure to check if the words are present in the stop-word dictionary. If yes, remove them from the text else nothing needs to be done.

set of code is for updating new words to the NLTK Stop-words dictionary:

#Custom function to remove the stop-words
    def removeStopWords(str):
    #select english stopwords from NLTK
    cachedStopWords = set(stopwords.words("english"))
    #add custom words, mentioned above
    cachedStopWords.update(("viewed", "flutter", "function", "k", "neighbour"))
    #remove stop words
    new_str = ' '.join([word for word in str.split() if word not in cachedStopWords])
    return new_str
set of code is for making a Custom stop-word dictionary:

stopwordList = ["viewed", "flutter", "function", "k", "neighbour"]
StopWordsRemover(inputCol="words", outputCol="filtered" ,stopWords=stopwordList)


How to generate synthetic dataset using machine learning model learnt with original dataset?

The general approach is to do traditional statistical analysis on your data set to define a multidimensional random process that will generate data with the same statistical characteristics.  The virtue of this approach is that your synthetic data is independent of your ML model, but statistically "close" to your data. (see below for discussion of your alternative)
In essence, you are estimating the multivariate probability distribution associated with the process.  Once you have estimated the distribution, you can generate synthetic data through the Monte Carlo method or similar repeated sampling methods.  If your data resembles some parametric distribution (e.g. lognormal) then this approach is straightforward and reliable.  The tricky part is to estimate the dependence between variables. See: https://www.encyclopediaofmath.org/index.php/Multi-dimensional_statistical_analysis.
If your data is irregular, then non-parametric methods are easier and probably more robust.  Multivariate kernal density estimation is a method that is accessible and appealing to people with ML background. For a general introduction and links to specific methods, see: https://en.wikipedia.org/wiki/Nonparametric_statistics .
To validate that this process worked for you, you go through the machine learning process again with the synthesized data, and you should end up with a model that is fairly close to your original.  Likewise, if you put the synthesized data into your ML model, you should get outputs that have similar distribution as your original outputs.
In contrast, you are proposing this:

[original data --> build machine learning model --> use ml model to generate synthetic data....!!!]

This accomplishes something different that the method I just described.  This would solve the inverse problem: "what inputs could generate any given set of model outputs". Unless your ML model is over-fitted to your original data, this synthesized data will not look like your original data in every respect, or even most.
Consider a linear regression model. The same linear regression model can have identical fit to data that have very different characteristics. A famous demonstration of this is through Anscombe's quartet.

Thought I don't have references, I believe this problem can also arise in logistic regression, generalized linear models, SVM, and K-means clustering.
There are some ML model types (e.g. decision tree) where it's possible to inverse them to generate synthetic data, though it takes some work.  See: Generating Synthetic Data to Match Data Mining Patterns.


Incorrectly applying random forest model?

I think you correctly identified the issue: if your model tries to classify whether an employee stays or leaves, by definition every employee "stays" as long as they are an employee in the company.

A possible direction would be to design the response variable as "does the employee leave within a year?" (or any specific period of time). This way you can use the records of past employees who have left and past employees who have stayed across time, i.e. you can have several instances corresponding to the same employee at different times. You could add features such as "had a raise or promotion in the past 2 year" for instance.


With such a heavy imbalance and two classes (it seems) you could treat this as more of an outlier detection problem. You should read up on models and algorithms in that direction!
If you go forward with a traditional classification you need to balance the data set, consider methods such as SMOTE.
Depending on the size of your data I would generally recommend downsampling the majority class which avoids producing "synthetic" cases but advanced methods such as SMOTE basically take care of this decision for you.
Can you elaborate what you mean with your first question as well? A classification algorithm needs 0/1 labels and therefore the output score cannot be ordered in the way that you mention. Some classification algorithms put out a probability score instead of predicted label so if this is what you mean I can tell you that the imbalance will still be a problem.


Classification/Prediction based on Multivariate Time Series

Train an LSTM-RNN to perform direct sequence classification. This essentially means that it will have multiple inputs and 1 output, i.e. the label (0 or 1). In Keras/Python this is very easy to implement, just make sure that you have a Dense layer in the end with sigmoid activation so that the output is between 0 and 1. You train the network based on your labeled data and then it outputs the label by itself. A useful tutorial on how to do this can be found here. The most important thing is that it inherently deals with linear/nonlinear cross-correlation between inputs so you don't have to explore them yourself. It is also capable of learning the dynamics of the input signals, because of its inherent memory.

Keep in mind that in overall it is a very convenient solution because it works like a black box that accepts time series and "spits" out their labels.

This approach has worked successfully for me for time series classification :)


I recall I was struggling for some time deriving the second equation. That constant keeps many of your missing elements. Let's break it down using your $(a-b)^{2}$ notation. We will have $a^{2}$, $b^{2}$, and $2ab$:

$a^{2}$: $y_{i}$ is constant since it is your true labels/values, thus $a^{2}$ i.e. $y_{i}^{2}$  goes to the constant.
$b^{2}$: $y\hat{}_{i}^{t-1}$+$f_{t}^{2}$+$2y\hat{}_{i}^{t-1}f_{t}$

$y\hat{}_{i}^{t-1}$ is constant as it is prediction in a step before $(t-1)$ that we know already, thus goes to the constant term. The other two terms remain as it is.

$2ab$: $2y_{i}(y\hat{}_{i}^{t-1}+f_{t}) = -2y_{i}y\hat{}_{i}^{t-1}-2y_{i}f_{t}$. Here also the first term is constant. Only the second term remains.

The rest should be pretty straightforward, just add things that are left and clean it up and you see the second equation comes out beautifully.


First thing first, when ever you use Time Series data you call it as Forecasting not Prediction as it is time dependent. To understand why you can go through this link
Metrics to compare models
When you are trying to compare between models you need to use AIC,BIC, AUC etc values. you can go though this link to understand better
Metrics to Access the Model
When you are accessing the performance of the model then you need to check for Error Rates(RMSE, MAE, MAPE, MSE etc). Yes, in this case you need to divide the data into Train and Test to access the model. You can go through this link for better understanding
Improve the Forecast
To take it to the next level, you can use an ensemble to get better the result. This might or might not decrease the error rate. In most of the cases it is helpful. You can combine 2/3 moderately performing hotels outcome to get best results i.e., Ensemble Model.


How will ADA Boost be used for solving regression problems?

Here are link to some famous boost of regressor.

Adaboost.R2: Improving Regressors using Boosting Techniques
Adaboost.RT: Experiments with AdaBoost.RT, an Improved Boosting
Scheme for Regression

Scikit-Learn have many implementations:

AdaBoostRegressor
Decision Tree Regression with AdaBoost


With a MLP (regression), is it appropriate to initialize bias in the final layer to be a value near the expected mean?

Cool idea! Looks like it really helps.

Training starts with smaller error, so you might be able to train the model in shorter time with this trick. You should be okay as long as you initialise the weights properly.

I trained model with different biases and plotted error at the start and the end of of the training. Expected mean of the data is 100. See code below.

from keras.models import Sequential
from keras.layers import Dense
from keras.initializers import Constant, Zeros, Ones
from keras.metrics import mean_squared_error
import matplotlib.pyplot as plt
import numpy as np

def getData():
    n = 200
    X = np.random.randn(n, 2)
    Y = 20 * X[:,0] + 10 * X[:,1] + 100
    return X, Y

def getModel(bias):
    m = Sequential()
    m.add(Dense(1,
        input_shape=(2,),
        bias_initializer=bias))
    m.compile('adam', loss='mse')
    return m

X, Y = getData()

constants = [0, 1, 10, 50, 100, 150, 200]
loss_at_start = []
loss_at_end = []

for c in constants:
    m = getModel(Constant(c))
    m.fit(X,Y,
        epochs=20,
        validation_split=0.2,
        validation_steps=20,
        steps_per_epoch=1000)
    loss_at_start.append(m.history.history['loss'][0])
    loss_at_end.append(m.history.history['loss'][-1])

plt.plot(constants, loss_at_start)
plt.plot(constants, loss_at_end)
plt.xlabel('Bias')
plt.ylabel('Loss')
plt.legend(['Loss at training start','Loss at training end'])
plt.title('Expected mean: 100')
plt.show()


The definition of $\alpha$-Approximate  Stochastic oracle has been given in the paper that you provide on Page $3$.

Given a distribution $D$ over $\mathcal{L}$, an $\alpha$-approximate stochastic oracle $M(D)$ computes $x^*$ such that such that
$$\mathbb{E}_{L \sim D}[L(x^*)] \le  \alpha \min_{x \in \mathcal{X}} \mathbb{E}_{L \sim D}[L(x)]$$

You  need not know the details of $M$, but if you provide it with a distribution $D$, then it returns you an $x^*$ such that if you evaluate the expected loss it is an most $\alpha$ times the objective value of the minimum of the expected loss.
Notice that different papers use different definitions and you should find the definition in the paper itself, as you can see, the first paper uses a multiplicative definition but the second paper uses an additive definition.


How to get the survival duration prediction for each individual in the data by using the Kaplan-Meier method?

The Kaplan Meier curve is a summary statistic, similar to the average. Therefore, it is an unconditional statistic. If you are interested in the conditional survival, we can use the Kaplan Meier curve to do that too. Some notation first. Let $T$ be the unknown time of death, and $S(t) = P(T > t)$ the unconditional survival curve.

Given a subject has lived past some time $s$, we'd like to know $P(T > t | T > s)$. We can expand this conditional probability:

$$
\begin{align}P(T > t | T > s) &= \frac{ P(T > t \text{ and } T > s)}{P(T > s)}\\
&= \frac{ P(T > t)}{P(T > s)} \text{   ,since t > s}\\
&= \frac{ S(t)}{S(s)}
\end{align}
$$

That is, we scale that entire survival curve by $S(s)$.

In practice, if you have a censored subject and you'd like to know their conditional survival after, you just scale by the survival function at their censored time.

In lifelines, there is the property KaplanMeierFitter.conditional_time_to_event_ which computes the median remaining survival time given survival up to time $t$. This can be used as a conditional prediction.

kmf = KaplanMeierFitter().fit(t, e)
censored_times = t[~e.astype(bool)]
predicted_life_remaining = kmf.conditional_time_to_event_.asof(censored_times)


In regression trees, sum of squared error (SSE) is the criterion for tree split. The first split is based on the feature/predictor and its values in your training set that yields the lowest SSE value. And then so on for the further splits.


Suppose (prior to one-hot-encoding) you have predictors/fields from a set $Z$ (say movie genre, user gender, and user race).  Suppose further, each predictor $z \in Z$ can take on one of $k_z$ values.  After one-hot encoding, you will have a new set of binary features $X$ of size $K := \sum_{z \in Z}k_z$.

In a model with all interactions, you must estimate a matrix of interaction coefficients $Q$, which has $K\times (K+1) / 2$ unique terms.

The factorization machine puts structure on the matrix $Q$, and assumes that $Q \equiv W^{T} W$, where $W$ is of dimension $l \times K$, with $1\le l \le K$ some number specified by the user.  We estimate $W$ instead of $Q$.

The field-aware factorization machine puts structure on $Q$ as well.  It partitions $Q$ into blocks based on $z$ (the original features).  If $q_{z_i, z_j}$ denotes the $z_i,z_j$ block of $Q$, we assume that $q_{z_i,z_j}$ comes from the $z_i,z_j$ block of $W_j^{T} W_i$, where $W_i$ is of dimension $l \times K$.  As with the FM, we estimate the $W_i$ instead of $Q$.

The FM factorization of $Q$ has $K \times l $ parameters.  The "feild-aware" FM has $K\times l\times |Z|$ parameters.  A model with all interactions has $K \times (K+1)/2$ parameters.


Welcome to our community!

If I understood correctly, you don't trust the labels on your training dataset.

  The problem is, that the original data wasn't generated but added by users.

That is not properly a problem, actually almost every dataset was created by human labeling. Your problem is that you don't trust the judgement of those users and you think that data might be incomplete.

It is true that this might affect your model's performance. But, you should try either way. Sometimes the model learns to label even better than the training dataset:

Recently I looked the master's dissertation of a friend (Wesley L. Passos,M.Sc. by UFRJ) which used deep learning to detect tires on drone images (for Aedes Aegyptis prevention procedures). The dataset was created by our group my manually annotating images with bounding boxes and while we missed some of the tires that were pretty well hidden the D-CNN model was capable of detecting those hard subjects.

Note: This dissertation was recently accepted and isn't available online yet. Once it does, I will update this answer with the proper reference.

Since wrangling with data is our everyday job this is a good opportunity for you to put in practice what you have learned:

Clean your data by either removing incomplete samples or filling missing values. This is a common part of our jobs as data scientists.
If you feel that the amount of work is  a bit overwhelming you can try clustering instead or semi-supervised methods to speed up the cleaning process.
Also you can try posting your data online for further contributing.

Curiosity Note:

Crowds are usually more intelligent than individual, so with problem statistical treatment data can actually be better annotated by crowds, using mean answers or by voting. Check for The Wisdom of Crowds (Not the TV Show).


Is there a difference between tf.nn.conv1d and tf.nn.convolution in Tensorflow?

In TensorFlow 2, they accept different arguments:

  tf.nn.convolution(input,...)

  Computes sums of N-D convolutions (actually cross-correlation).

  input: An (N+2)-D Tensor of type T, of shape [batch_size] + input_spatial_shape + [in_channels]

While the second one is more specific:

  tf.nn.conv1d(input,...)

  Computes a 1-D convolution given 3-D input and filter tensors.

  input: A 3D Tensor.input: A 3D Tensor.


This is an interesting question. So sorry for a long winded answer. The tl:dr; is it is a mix of some real applicability, theoretical basis, historical baggage (due to limited compute power) and obsession for analytically tractable models (instead of simulation/computational models). We should be very careful and discerning while using it in real problems.
Details
The importance of normal distribution comes from the following facts/observations,

Many naturally occurring phenomenon seem to follow normal distribution when sample size is large (more on this below).
In Bayesian statistics, if you assume a Normal distribution prior on parameters, then posterior distribution is also normal. This makes computations easier.
Somewhat related, central limit theorem tells us that average of a samples from any distribution (no fat tails) follows normal distribution. So normal distribution is useful and provides theoretical basis for doing population level parameter estimates from samples (think of election predictions). But again, this assumes the underlying data comes from a distribution which is well behaved and extreme values are very unlikely.

In short, normal distribution can be thought of as a good base case, which is analytically tractable, easy to code up and also seems to be applicable to many models of nature. Somewhat broken analogy, but in Physics we consider linear second order differential equations to study many systems. Now not all systems actually are linear second order, but it is a reasonable approximation under some constraints that is easier to analyze and code up.
And over-usage of normal distribution everywhere is actually controversial.

As we have to more computing power and access to Monte Carlo based simulation based method, we are no longer limited by using only analytically tractable distributions. We can use distributions which more accurately fit the reality.
Normal distributions are useful for natural phenomenon (heights of students in a class) but are way inaccurate to model mostly man made systems (income of people in a town, potential swings of stock indices during panic).
For example, many critics of probabilistic financial models observe that the underlying models use normal distribution. But real market swings are mostly fat tailed (distributions where extreme outcomes are more likely than normal distributions). If you want to go down on deeper into this, start with statistical consequences of fat tails by Nassim Nicholas Taleb. Fun fact, if you look at the wild swings in the price of GameStop stock from the r/wallstreeetbets saga, Taleb pointed out that the swings are not actually wild if you consider a fat tailed distribution.


Where can I find datasets with labeled duplicate text documents?

Instead of training a classifier, detecting duplicates may advantageously be done in a direct fashion: using a similarity metric like shingle signatures or a hashing/checksum function.

(A text to representation function F should preserve formal/meta elements that you already know to be identifiers or partial-identifiers, and it should also incorporate a compact representation of the content -- like sparse excerpts, checksum, or any other hashing key or canonized form. The comparison function G should then compare these text representations provided by F and define a similarity score.)

You'd still need an evaluation corpus, so looking for a corpus is still important. I don't know of any open one, but job-boards and academic databases contain lots of duplicates so you could start there.


I want to know which machine learning algorithms can be used for trajectory classifications?

Depending on the amount of data, any classification algorithm can be suitable. LSTM, however, are likely to be an overkill, considering that you probably won't be having much variation in the time series for each object.

Instead of pondering about the algorithm, you'd better think of useful features you can extract from your data. My guess would be that speed, accelerations, and altitude would be most informative.


What's an efficient way to compare and group millions of store names?

This is an entity resolution aka record linkage aka data matching problem.

I would solve this by removing all of the non-alphabetical characters including numbers, casting into all uppercase and then employing a hierarchical match.  First match up the exact cases and then move to a Levenshtein scoring between the fields.  Make some sort of a decision about how large you will allow the Levenshtein or normalized Levenshtein score to get before you declare something a non-match.

Assign every row an id and when you have a match, reassign the lower of the IDs to both members of the match.

The Levenshtein distance algorithm is simple but brilliant (taken from here):

def levenshtein(a,b):
    "Calculates the Levenshtein distance between a and b."
    n, m = len(a), len(b)
    if n > m:
        # Make sure n <= m, to use O(min(n,m)) space
        a,b = b,a
        n,m = m,n

    current = range(n+1)
    for i in range(1,m+1):
        previous, current = current, [i]+[0]*n
        for j in range(1,n+1):
            add, delete = previous[j]+1, current[j-1]+1
            change = previous[j-1]
            if a[j-1] != b[i-1]:
                change = change + 1
            current[j] = min(add, delete, change)

    return current[n]

This Data Matching book is a good resource and is free for seven days on Amazon.

Nominally, this is an $n^2$ algorithm without exploiting some sorting efficiencies, so I would expect to have to use multiple cores on $2\times10^7$ rows.  But this should run just fine on an 8 core AWS instance.  It will eventually finish on a single core, but might take several hours.

Hope this helps!


Units of mean decrease accuracy on a variable importance plot obtained from a Random Forests classifier?

Gini decrease is calculated based on the mean decrease in Gini i.e. $p_i(i-p_i)$ each time when the Tree is splitted on that Feature. Value is so high because the r package weight the impurities by the raw counts, not the proportions.
Accuracy decrease is calculated on OOB dataset by randomly shuffling the data for that particular feature in the OOB. Then subtracting the two accuracies i.e. before and after shuffling.
Unit -
Accuracy - Unitless - Ratio of correct classification to total records
Gini impurity - Unitless - Measurement of the cleanliness of split.Formal
These links have got good detail:
Stackoverflow
r-bloggers
Quora


Simple question related to an ANN diagram showing all of the matrices' dimensions: inputs, weights and output

Yes. reasoning presented here is correct. Mind that $x$ input vector is a row vector (not a column one): $[x_1, x_2, x_3]$. Multiplying $x\cdot w^1$ provides you an another row vector $y_1 = [y_{11}, y_{12}, y_{13}, y_{14}]$ which represents output of neurons 1 to 4 in 1st hidden layer.
Then you multiply $y_1\cdot w^2$ and gain another row vector and so on until the outlayer. At last performing an multiplying 1x4 $\cdot$ 4x1 provides you a scalar which will be a net output after using output's neuron's transfer function on that scalar.


How to classify/cluster clients of an application of transport like the Uber Model?

If you want to cluster people, you need to have one row per person.
To do this you will need to group by IDClient/IDChauffeur and compute features like meanHourRide, stdHourRide, ...

By feeding the algorithm with multiple records of the same person (with the same travels for example), the clustering algorithm might create a cluster with similar travels (time/location/distance-wise) even though it is made by only one customer.

This would result in travel clustering, which is also possible but not what you want here.


First of all, most clustering methods will be fine when used right. E.g. with correlation, or appropriate normalization.

Secondly, if you already have patterns in mind, use classification.

Third, in order to find outliers, clustering is not too helpful. Because it is not reliable enough. Rather use one of the clever outlier detection methods such as KNN, LOF, LOOP.

Don't forget detailed and problem oriented preprocessing to prepare your data as good as possible.


Don't use clustering at all.

Define a threshold, and theshold the data. Done.

Keep it simple and easy to understand.


Analysis of Split (A/B) tests using Poisson and/or Binomial Distribution

OK, so here's your data.

dd <- data.frame(position=rep(1:3, each=2),
                 variation=rep(c(1,3), 3),
                 impressions=rep(c(753, 767), 3),
                 clicks=c(26,7,16,13,2,7))

which is

  position variation impressions clicks
1        1         1         753     26
2        1         3         767      7
3        2         1         753     16
4        2         3         767     13
5        3         1         753      2
6        3         3         767      7

The two model assumptions you're thinking about are Binomial

mod.bin <- glm(cbind(clicks, impressions-clicks) ~ variation + position,
               family=binomial, data=dd)

where the dependent variable is constructed to have the count of the event of interest in the first column, and the Poisson

md.pois <- glm(clicks ~ variation + position + offset(log(impressions)),
               family=poisson, data=dd)

where the log(impressions) offset is necessary whenever the number of trials differs across observations.  This means coefficients are interpretable in terms of change in rate not change in count, which is what you want.

The first model generalises the binom.test to a setting with covariates, which is what you have.  That gets you a more direct answer to your question, and better (if not perfect) measurement of the relevant uncertainty.

Notes

Both models assume no interaction between variation and position ('independent effects').  This may or may not be reasonable.  You'd want more replications to investigate that properly.  Swap the + for a * to do so.

In this data summary confirms that the two models give rather similar results, so concerns about Poisson vs Binomial don't seem to matter much.

In the wild, count data is usually overdispersed, that is: more variable than you'd expect from a Poisson with a constant rate or a Binomial with constant click probability, often due to unmodeled determinants of click rate / probability.  If that's the case then prediction intervals from these models will be too narrow.


Filenotfounderror: [errno 2] no such file or directory: 'nsr001.ecg'

You just need to add the root directory in front of the filename so that the full filepath is correct. For this you can use os.path.join:
import os
import numpy as np

for root, dirs, files in os.walk('normal-sinus-rhythm-rr-interval-database-1.0.0'):
    for file in files:
        if file.endswith(".ecg"):
            print(os.path.join(root, file))
            averageArray = np.fromfile(os.path.join(root, file), dtype=float)


How to calculate classification accuracy with confusion matrix?

A confusion matrix is a cross tabulation of your predicted values against the true observed values, and (test) accuracy is emperical rate of correct predictions. So in this case you'll need to

Predict the 'play' attribute for your test set.  (Currently you don't have a method to predict your second test case, so for the sake of argument let's assume your model would predict yes for the sunny example.
The following method of keeping track of your predictions is reffered to as a confusion matrix.  The top labels are prediced

               Predicted
         +----------------+
         ¦     ¦ yes ¦ no ¦
Oserved  ¦ yes ¦ 1   ¦ 1  ¦
         ¦ no  ¦ 0   ¦ 0  ¦
         +----------------+

Here the first 1 is from your first test case and the second 1 is from the misclassified second test case.

Calculate accuracy,

Accuracy = (# correct predictions)/(# total predictions) = 1 / 2 = .50.


Is there a Hyperopt equivalent for optimization in R?

Give a try to DEoptim(). This package might solve your problem. For the documentation and more information visit-

Documentation
CRAN-R : DEoptim()

Hope it helps!


Elbow method on hundreds columns and rows

You are using different random starting points each time you run k-means (random_state=None) This means you may get different clusterings, different clustering metrics each time. That's expected. What you may wish to do is average the results over several different runs to get a more reliable estimate of the loss at each k.


Choosing regularization method in neural networks

There are not any strong, well-documented principles to help you decide between types of regularisation in neural networks. You can even combine regularisation techniques, you don't have to choose just one.

A workable approach can be based on experience, and following literature and other people's results to see what gave good results in different problem domains. Bearing this in mind, dropout has proved very successful for a broad range of problems, and you can probably consider it a good first choice almost regardless of what you are attempting.

Also sometimes just picking a option you are familiar with can help - working with techniques you understand and have experience with may get you better results than trying a whole grab bag of different options where you are not sure what order of magnitude to try for a parameter. A key issue is that the techniques can interplay with other network parameters - for instance, you may want to increase size of layers with dropout depending on the dropout percentage.

Finally, it may not matter hugely which regularisation techniques you are using, just that you understand your problem and model well enough to spot when it is overfitting and could do with more regularisation. Or vice-versa, spot when it is underfitting and that you should scale back the regularisation.


In Keras, with verbose=1 (default parameter of the fit method) will display the total number of samples, not the batch number.

If your batch size is 128, then the progress bar will jump by multiples of 128. You can try to change batch_size parameter to 13714, and you will see the progress bar jumping straight from 0/13714 to 13714/13714, since you would have defined a batch size of the size of your whole training dataset.

Finally, if you want to train your model only on one mini batch, you can extract it from your training set and fit your model on it.

import random
indices = random.sample(range(13714), 128)

X_train_seq_one_batch = X_train_seq[indices]
y_train_one_batch = y_train[indices]
LSTM_classifier.fit(X_train_seq_one_batch, y_train_one_batch, batch_size=128, epochs = 10)

The above code will train your network only on one random batch only. random library is used to draw a random batch of indices in your training set.


Why yolo4 pytorch re-training loss seems high as like first time training?

Apparently, only the lower layers weights of the pretrained model are being updated. to change it open models.py and scroll down to the class definition of Yolov4.
As one can see, only down1-down5 and neck layers are being updated by the pretrained model weights while the head is starting fresh. you should change this code section to something similar to:
if yolov4conv137weight:
        self.neck = Neck()
        **self.head = Yolov4Head(output_ch)**
        _model = nn.Sequential(self.down1, self.down2, self.down3, self.down4, self.down5, self.neck, **self.head**)
        pretrained_dict = torch.load(yolov4conv137weight)

        model_dict = _model.state_dict()
        # 1. filter out unnecessary keys
        pretrained_dict = {k1: v for (k, v), k1 in zip(pretrained_dict.items(), model_dict)}
        # 2. overwrite entries in the existing state dict
        #print("update weights")
        model_dict.update(pretrained_dict)
        _model.load_state_dict(model_dict)
    # head
    **else:
        self.head = Yolov4Head(output_ch)**

Run again and observe the expected loss (hopefully).
Good luck!


How to create a df that gets sum of columns based on a groupby column?

This question is more appropriate at Stackoverflow since it is more of a programming question.

However, I think you almost got it working. Just remove gender from your groupby:

df_new = df.groupby(['country', 'year', 'measure'])['value0', ... 'value12'].apply(lambda x : x.astype(float).sum())


How do I convert strings in CSV into integer in Pandas?

You cannot convert your string product names to integers and expect    it to work, if you convert it to integers the algorithm you use will    expect a linear relationship between these integers, a relationship    that in fact does not exist.

It is categorical data, and you should treat it like that, you have to one-hot encode it. If you have too many products to one-hot    encode all of them try grouping them in categories and then one-hot encoding the categories.


One of the most popular ways of doing classification on graphs is through graph convolutional networks. By running a convolution over the nodes of a graph, the neural net is able to learn the local neighborhoods of the graph. The seminal example is probably the paper Semi-Supervised Classification with Graph Convolutional Networks  by Kipf and Welling. https://arxiv.org/abs/1609.02907


How can I get an algorithm to have an evalutation metric based on aggregate predictions?

I don't care whether individual predictions are accurate

If this is true (which I don't think it is), then you could just take the group mean of the target for males and females to minimize the RMSE and make this your prediction for all male and female individuals, respectively. (Basically a model where Gender is the only feature if you insist on using xgboost.) One could argue that this one-feature model might not generalize as well to new data than a model with more features, but if Gender is not in any meaningful way related to the target you can't optimize the RMSE towards Gender anyway.

Based on your "Note" I assume you somehow want both, good predictions for individuals and on the group level. In this case minimizing the normal "individual-level" RMSE is already the best way to go.


Generate text using user-supplied keywords

Yes fine-tuning GPT2 could help you through this objective. But the only concern is regarding the size of input data you have. To get a better performing model, you must a have larger input set.


How does backpropagation differ from reverse-mode autodiff

They are more or less specify the same thing. I think the difference is that back-propagation refers to the updating of weights with respect to their gradient to minimize a function; "back-propagating the gradients" is a typical term used.

Conversely, reverse-mode diff merely means calculating the gradient of a function. Nothing here in this process implies that we aim to the weights/values of a function.


Neural net only works for small datasets

It could mean, that your data is too complex to be learned by the model using a certain number of observations (what is number of features do you have?). On the plot training loss is almost zero for small dataset, but you should rely on the validation loss, not on training one. And validation loss is lower for bigger number of samples, then learning process performes predictable normal


From an older version of the OpenNLP README:

  Training the Tools

  There are training tools for all components expect the coref component. Please consult the help message of the tool and the javadoc to figure out how to train the tools.

  The tutorials in our wiki might also be helpful.

  The following modules currently support training via the WordFreak opennlp.plugin v1.4 (http://wordfreak.sourceforge.net/plugins.html).

  coreference: org.annotation.opennlp.OpenNlpCoreferenceAnnotator (use opennlp 1.4.3 for training, models are compatible)

  Note: In order to train a model you need all the training data. There is not currently a mechanism to update the models distributed with the project with additional data.

As you can see, OpenNLP does not provide training tools for the coreference component. However, it seems at one point it was possible to train new models for OpenNLP's coref component using the third-party WordFreak plugin... however, it hasn't been updated in over a decade, so your mileage may vary.


How to find combined feature contributions in a random forest algorithm

My recommendation would be to use Partial Dependence Plots, which show the marginal effect one or two features have on the predicted outcome.  They plot the average predictions over a range of values of the predictor(s) you specify, with the influence of all other predictors "averaged out".  If you are familiar with linear regression, it is like plotting the predicted value of the target for different values of a predictor multiplied by the estimated weight/coefficient.

In your question, you are interested in two predictors (smoking and family history), so you would plot a two-way partial dependence plot showing the dependence of the target variable on joint values of smoking and family history.

There is a PDPBox package you can install that produces a variety of very nice plots and has some tutorials: https://github.com/SauceCat/PDPbox.  It claims to support all Scikit-Learn algorithms.

There is also a Cross Validated Q&A on interpreting PDP's for Random Forests (produced with R): https://stats.stackexchange.com/questions/121383/interpreting-y-axis-of-a-partial-dependence-plots


How to measure the performance of a domain adaptation /Transfer learning technique?

You can measure the divergence between the source and target domain using KL-Divergence (there are some ways to estimate k-l divergence e.g. depdended on k-nn algorithm). Then you can check if there is a correlation between the divergence and the accuracy of the models considering a few cases of source-target pairs of datasets. You can compare several algorithms of Transfer Learning/Domain Adaptation using the same source-target datasets.


ROC-AUC curve as metric for binary classifier without machine learning algorithm

Your intuition is correct: you don't necessarily need a machine learning algorithm to calculate a ROC curve and, from what you say, it seems you already have the necessary ingredients in place.

In fact, the ROC curve was historically developed under circumstances that had nothing to do with machine learning; from Wikipedia:

  The ROC curve was first developed by electrical engineers and radar engineers during World War II for detecting enemy objects in battlefields and was soon introduced to psychology to account for perceptual detection of stimuli. ROC analysis since then has been used in medicine, radiology, biometrics, forecasting of natural hazards, meteorology, model performance assessment, and other areas for many decades and is increasingly used in machine learning and data mining research.


Different models for different time durations of a day

This is a typical time series problem. The first step is to make sure your time series is stationary, see here for a good explanation why this is necessary.

So we first do,

dframe$power_diff <- c(NA, diff(dframe$power))
plot(dframe$power_diff, type = "l")

gives,

Next we want to consider an ARIMAX model. As part of this exercise we want to know whether there is a dependency in the time series, a so called auto regressive term. We can research this using,

acf(na.omit(dframe$power_diff))

which gives,

In this plot we see that there is a positive auto regressive term at lag 1 at a significance level of 95%, we conclude this based on the observation that the second line, which is lag 1, is above the blue dotted line.

Next we want to fit a ARIMAX model.

# Convert data to ts object
power_data <- ts(data = dframe$power_diff[-1], frequency = 8766, start = c(2016, 1, 1))
temp_data <- ts(data = dframe$temp_diff[-1], frequency = 8766, start = c(2016, 1, 1))

# Build the model.
m <- Arima(power_data, order = c(1, 0, 0), xreg = temp_data)

# Let's see how our fit is.
plot(power_data)
lines(fitted(m), col = "blue")

This gives this result,

You can play with the ARIMA(p, i, q) terms. We should leave $p=1$ since we found that the AR term is significant at lag 1. If we add a MA(1) we find a model that has a MAPE=260.46 versus a MAPE=272.62 for our initial model. So you can create the best model using,

m2 <- Arima(power_data, order = c(1, 1, 0), xreg = temp_data)
lines(fitted(m2), col = 'red')

Will lead to,

Of course we can automate the above step with auto.arima(). This function will search for the best arima model,

m3 <- auto.arima(power_data, xreg = temp_data)
lines(fitted(m3), col = 'green')

which will give you,

You see that each model has a poor fit in the spike. To improve this you could introduce a dummy variable,

xdata <- ts(data = as.matrix(cbind(temp_data, dframe$power_diff[-1] > 20)), frequency = 8766, start =c(2016,1,1))
m_dummy <- auto.arima(power_data, xreg = xdata)
lines(fitted(m_dummy), col = 'orange')

which gives us the following result,

We see that this model gives us a much better result. So if you can relate dummy to a point in time you could create a model that has a good fit.

According Occam's Razor you should try to keep your models simple. So I would suggest to build a single model for all your time slots instead of three separate models as you suggested in your question.

You can turn this back to the original time series with cumsum. Hope this helps.


Learning Resources for Recommendation system

If you have a technical background, I would certainly recommend the chapter 9 from Mining Massive Datasets by Jure Leskovec. Video lecture of the topic is also available there. Look for chapter 9. This blog could also give you some overall insight.

Please keep in mind that, ML-wise, recommendation is a more creativity-related topic than other classic questions like Classification or Regression and indeed all those classic problem could be a step in building a reasonable recommender engine.


Are there machine learning methods that can give captions to the New Yorker magazine cartoons?

Yes. Image captioning is a very active area of Deep Learning, at the intersection between Computer Vision and Natural Language Processing.

These models are based on an Encoder-Decoder structure. In its most classical form, the Encoder consists of Convolutional layers that process pixel data. Their representation is then fed to a Decoder based on Recurrent layers (usually LSTM or GRU) that generate the output sentence.

I suggest you to take a look at this tutorial on Image Captioning in TensorFlow 2. You can download the Notebook and run. That specific model is based on attention mechanism, which is a fancier version of seq2seq models. You can find tons of other models and tutorial simply googling them.


Abbreviation Classification using machine learning

Look into this package for Python:
https://pypi.python.org/pypi/Distance/

You can use this to generate a numeric value representing the similarity between word.

Here is a similar post that should help:
https://stats.stackexchange.com/questions/123060/clustering-a-long-list-of-strings-words-into-similarity-groups

Additionally, a level up in complexity would be to use t-SNE on an array generated using word2vec (this is word embedding).  Examples and resources for this are:
https://www.codeproject.com/tips/788739/visualization-of-high-dimensional-data-using-t-sne
http://sebastianruder.com/word-embeddings-1/


How can I classify text considering word order, instead of just using a bag-of-words approach?

There is a very simple hack to incorporate word order in an existing bag-of-words model implementation. Treat some of the phrases, such as the frequently occurring bi-grams (e.g. New York) as a unit, i.e. a single word instead of treating them as separate entities. This will ensure that "New York" is different from "York New". You could also define higher order word shingles such as for n=3,4 etc.

You could use the Lucene ShingleFilter to decompose your document text into shingles as a pre-processing step and then apply the classifier on this decomposed text.

import java.io.*;
import org.apache.lucene.analysis.core.*;
import org.apache.lucene.analysis.*;
import org.apache.lucene.analysis.shingle.ShingleFilter;
import org.apache.lucene.analysis.standard.*;
import org.apache.lucene.util.*;
import org.apache.lucene.analysis.util.*;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.charfilter.*;
import org.apache.lucene.analysis.core.WhitespaceTokenizer;

class TestAnalyzer extends Analyzer {

    TestAnalyzer() {
        super();
    }

    protected TokenStreamComponents createComponents( String fieldName, Reader reader ) {
        String token;
        TokenStream result = null;

        Tokenizer source = new WhitespaceTokenizer( Version.LUCENE_CURRENT, reader );
        result = new ShingleFilter(source, 2, 2);

        return new TokenStreamComponents( source, result );

    }
}

public class LuceneTest {

    public static void main(String[] args) throws Exception {

        TestAnalyzer analyzer = new TestAnalyzer();

        try {
            TokenStream stream = analyzer.tokenStream("field", new StringReader("This is a sample sentence."));
            CharTermAttribute termAtt = stream.addAttribute(CharTermAttribute.class);

            stream.reset();

            // print all tokens until stream is exhausted
            while (stream.incrementToken()) {
                System.out.println(termAtt.toString());
            }

            stream.end();
            stream.close();
         }
         catch (Exception ex) {
             ex.printStackTrace();
         }

    }
}


is there a way to check if i got a "good price" on something?

Yes, fit a regression equation ( or deep NN)  to predict price based on all other parameters. then calculate error ( actual- predicted) , positive error indicates higher price than expected( what is should be) and vice versa. get a basic idea of types of anomalies-

https://machinelearningstories.blogspot.com/2018/07/anomaly-detection-anomaly-detection-by.html


Fitting a pandas dataframe to a Poisson Distribution

I think the cause of the error is the np.math.factorial(k) function call, since curve_fit passes a numpy array as the first parameter to the poisson function, and if you try to run the code

np.math.factorial(np.array([1, 2, 3]))

You'll get the error

TypeError: only size-1 arrays can be converted to Python scalars

Try using scipy.special.factorial since it accepts a numpy array as input instead of only accepting scalers.

Thus, just change your poisson function to

def poisson(k, lamb):
    return (lamb**k/ scipy.special.factorial(k)) * np.exp(-lamb)

Hope this helps

EDIT:
Also I fixed the ^ to ** since that's how you use the exponential operator in python.


Large negative R2 or accuracy scores for random forest with GridSearchCV but not train_test_split

R2 can be negative if the model is arbitrarily worse according to the sklearn documentation
So the very negative train scores were indicative of an extremely bad performance.
Why was the test performance so colossally bad in GridSearchCV when it was decent in a simple test/train split?
The main problem is that train_test_split chooses observations randomly while GridSearchCV does not!
My problem was that the dataframe was sorted by the target variable!
The GridSearchCV and cross_val_score do not make random folds.  They literally take the first 20% of observations in the dataframe as fold 1, the next 20% as fold 2, etc.
Let's say my target is a range between 1-50.  If I sort my dataframe by target, then all observations are in order from 1 to 50.  The first fold of the cross-validation will take (for example) only observations with a target between 1-10, save this for the test, then train the model only on targets of 20-50.  This is why it was performing so badly!  The model was trained on a certain range, the test set only included a target range the model had never seen before!
The solution is simple. Shuffle the original dataframe before splitting into X, y for cross-validation.
df = df.sample(frac=1, random_state=0)

This solved my problem, now the test and train scores from GridSearchCV are both between 0-1, comparable to a simple train_test_split.
Lesson learned: Always shuffle a dataframe before a cross-validation - otherwise the folds will be subject to any biases in the order of how data was collected.


Really great question, and one that I find that most people don't really understand on an intuitive level. AUC is in fact often preferred over accuracy for binary classification for a number of different reasons. First though, let's talk about exactly what AUC is. Honestly, for being one of the most widely used efficacy metrics, it's surprisingly obtuse to figure out exactly how AUC works.

AUC stands for Area Under the Curve, which curve you ask? Well, that would be the ROC curve. ROC stands for Receiver Operating Characteristic, which is actually slightly non-intuitive. The implicit goal of AUC is to deal with situations where you have a very skewed sample distribution, and don't want to overfit to a single class.

A great example is in spam detection. Generally, spam datasets are STRONGLY biased towards ham, or not-spam. If your data set is 90% ham, you can get a pretty damn good accuracy by just saying that every single email is ham, which is obviously something that indicates a non-ideal classifier. Let's start with a couple of metrics that are a little more useful for us, specifically the true positive rate (TPR) and the false positive rate (FPR):

Now in this graph, TPR is specifically the ratio of true positive to all positives, and FPR is the ratio of false positives to all negatives. (Keep in mind, this is only for binary classification.) On a graph like this, it should be pretty straightforward to figure out that a prediction of all 0's or all 1's will result in the points of (0,0) and (1,1) respectively. If you draw a line through these lines you get something like this:

Which looks basically like a diagonal line (it is), and by some easy geometry, you can see that the AUC of such a model would be 0.5 (height and base are both 1). Similarly, if you predict a random assortment of 0's and 1's, let's say 90% 1's, you could get the point (0.9, 0.9), which again falls along that diagonal line.

Now comes the interesting part. What if we weren't only predicting 0's and 1's? What if instead, we wanted to say that, theoretically we were going to set a cutoff, above which every result was a 1, and below which every result were a 0. This would mean that at the extremes you get the original situation where you have all 0's and all 1's (at a cutoff of 0 and 1 respectively), but also a series of intermediate states that fall within the 1x1 graph that contains your ROC. In practice you get something like this:

So basically, what you're actually getting when you do an AUC over accuracy is something that will strongly discourage people going for models that are representative, but not discriminative, as this will only actually select for models that achieve false positive and true positive rates that are significantly above random chance, which is not guaranteed for accuracy.


Please enlighten me with Platt's SMO algorithm (for SVM)

Your understanding is correct. The point is that equation (8)
$$y_i(<\textbf{w}, \phi_i> + b) - 1 = 0$$
is not exactly an equation, but a system of equations, one for each $i$ index of the support vectors (those for each $0<\alpha_i<C$.

The point is that you cannot compute $b$ during the optimization of the dual problem since it does not matter for optimization, you have to go back and compute $b$ from all the other equations you have (one possible way is (8)).

Vapnick suggestion is to not use only one of those equations, but two of them, specifically one support vector for a negative observation and one for a positive observation. In other words two support vectors which have opposite signs for $y_i$.

Let's name $A$ the index of one support vector and $B$ the index of a suport vector of opposite side, baiscally you select from the system of equations at (8) only two of them. Evaluate both of them and take the mean.

From:
$$y_A(<\textbf{w},\phi_A>+b)=1$$
$$y_B(<\textbf{w}, \phi_B>+b)=1$$
We get:
$$b_A=\frac{1}{y_A}-<\textbf{w},\phi_A>$$
$$b_B=\frac{1}{y_B}-<\textbf{w},\phi_B>$$
Where $b_A$ and $b_B$ are two estimations, then the mean is
$$b = (b_A+b_B)/2 = -\frac{1}{2}(<\textbf{w},\phi_A>+<\textbf{w},\phi_B>)=-\frac{1}{2}\sum_{i=1}^{n}y_i\alpha_i(<\phi(x_i),\phi(x_A)>+<\phi(x_i),\phi(x_B)>)$$


I think more information on the project can help. For instance in your example, you just need to detect the flower frame in the whole image and crop it. Then you simply resize your images to be the same size before feeding your NN. In this case, detecting the target frame and scaling it is a part of pre-processing and the learning phase stays the same.

Another approach is to learn this scaling. It means that the learning algorithm needs scale-invariant features but NNs are supposed to actually create features in their hidden layers through learning. So if you have sufficiently big dataset in which all variants are presented enough, your NN will learn it (most probably with poor results). The process is intuitive ML-wise:

NN extract features from input images i.e. when the label of same object with different sizes are the same, it captures those features which discriminate these objects in a scale-invariant manner.
This was simplified! The trick is that CNN can learn resizing using a technique called Pooling. But you are the one who plays with parameters to make your NN working!

Probably the best is the situation in which your dataset is not that huge and rich or you do not want to play with many parameters or you want to increase the quality of results. Then focus on finding scale-invariant features. Either you extract them from your images and feed it to the NN, or you design NN in a way that it finds them itself. A more intuitive way is to train different models on different scales and ensemble them.

Hope it helps :) Good Luck!

PS: SIFT is patented. Be careful not to use it for commercial purposes.


How to transform dictionary data into a string vector?

If you want output in any order, do this-

>>> string_vector = ''
>>> for k,v in record.items():
        string_vector += k + ' ' + v + ' '

>>> string_vector = string_vector.rstrip(' ')
>>> string_vector
'first_name john last_name doe age 50'


Python strategies for handing categorical variables

The best option for encoding - OneHot, because if you use Label encoding you indicate that categorical values are comparable(for example label 1 < label 2), which most probably it's not true. One hot encoding create columns for each specific value in the column, moreover, these columns are linearly independent, so you don't create fake order between categorical values. Unfortunately, you got a lot of columns so learning of algorithms could be greedy for time and resource.

More details you will find there


What did Geoffrey Hinton mean when he said this?

Creating a model is having the computer write a small program. If you have a multi-layer network, then multiple nuggets might be interacting to produce complicated results.

That its pretty much how things like face detection work - you start with a series of really dumb "detectors" that look like basic rectangles and they combine in ways so that at the top, you're recognizing Brad Pitt.


Collinearity between continuous and categorical variable

You could just regress against any given variable.
You could also generate a matrix of correlation metrics. Depending on the variable type, you would need a different metric. Here are some common ones:

Numeric-Numeric: Pearson's Correlation Coefficient

For rank correlation, you can use Spearman's Rank Correlation Coefficient

Nominal-Nominal: Cramér's V

For two categories, this is equivalent to the Pearson Phi Coefficient

Numeric-Nominal: t-test for 2 variables or ANOVA for more than 2

You can also just regress against them. For example, logistic regression between age and sex could suffice.


Creating more than one worker nodes for local windows machine

There is no value in this. local[*] master will already use all cores and you can use as much local memory as you like already. Starting a worker just adds overhead.


How do we interpret the outputs of DBSCAN clustering?

The output from db_scan.labels_ is the assigned cluster value for each of the points that you provide as input to the algorithm.

You provided 20 points, so there are 20 labels. As explained in the relevant documentation, you will see:

  labels_ : array, shape = [n_samples]
  Cluster labels for each point in the dataset given to fit(). Noisy samples are given the label -1.

So each of your points was assigned to one of two clusters, and there were no points marked as noise (receiving a value of -1).

Interpreting the clusters

Your plot shows the original data, plotted on the axes using the original values; only the colour encodes the cluster to which each point was assigned.

As with many clustering algorithms, DBSCAN measures the distance between points in some n-dimensional space. It uses parameters you provide to decide if points belong to the same group, based on their distance from one another.

  Why do I have the negative scales both on the x-axis and y-axis?

The distance metrics used do not really care about the sign of the points (where they lie on an axis), simply the distances. In the case of DBSCAN, it is the Euclidean distance. There are other distances that you can pass to the metric parameter of the DBSCAN algorithm; here is the valid list.


Questions about Neural Network training (back propagation) in the book PRML (Pattern Recognition and Machine Learning)

As you point out, with $\delta_k = y_k - t_k$, the author is stating the relationship between the final units' output and the target. So Equation 5.54 is simply stating:

  the error on the $k^{th}$ output unit is the difference between its output and the target.

I believe that $\delta_k$ refers to the error simply at the output, whereas $\delta_j$ is the derivative of the output at unit $n$ with respect to any neuron back in the network, $j$. If this is the case, doing your derivative as you did, for $\delta_k$ (instead of $\delta_j$) means you are computing the error gradient between the output and the final layer. Over this layer, the activation must be linear (we do not apply a non-linearity at the final layers output). This would mean your
$\dfrac{\partial h(a_k)}{\partial a_k}$ term would actually fall away in this case (to a constant), using the $k$ subscript - and you have the answer from the author: $y_t - t_k$.

The network that is considered in your second question seems to only really discuss a single fully-connected layer, so yes, two actual layers of neurons - input and output only with no activation (so a linear activation, as you mention).


Feature Importance without Random Forest Feature Importances

There are many ways to try to estimate feature importance. Personally I think the random forest measures get overused simply due to the fact that they have “importance” in their name and many people have heard of them. However, what people don’t realize is that those features that the random forest deems important are important for the random forest. They are good at predicting your feature of interest in a random forest setting. Taking this and blindly applying it to non random forest problems is dangerous. What might be predictive for a random forest may not be for some other algorithm. Also these random forest importance measures are not without their faults, for example they are biased towards variables with wide ranges.
There are many other methods available for variable importance such as information gain and relief. I suggest you read this paper by Robnik-Sikonja
https://link.springer.com/content/pdf/10.1007%2F978-3-540-39857-8_30.pdf
It covers many different methods.


You should make the difference between the following concepts:

encryption = encoding information in such a way that only authorized parties can read it.
data anonymization = removing personally identifiable information from data sets, so that the people whom the data describe remain anonymous.
de-identification = preventing a person’s identity from being connected with information, while preserving identifying information which could only be re-linked by a trusted party in certain situations (unlike data anonymization which aims to be irreversible).

Your choice depends on your use case.


What is max aggregation on a set of word embeddings?

The second option is the right one. From the paper:

  In this case, we perform a component-wise max over the start
  embeddings of all instances of w to obtain the best features across
  all occurrences of w.

The previous quote can be found a few paragraphs before equation 17, second column, top of the page.


Interpretation of variable or feature importance in Random Forest

I would be reluctant to do too much analysis on the table alone as variable importances can be misleading, but there is something you can do.  The idea is to learn the statistical properties of the feature importances through simulation, and then determine how "significant" the observed importances are for each feature.  That is, could a large importance for a feature have arisen purely by chance, or is that feature legitimately predictive?

To do this you take the target of your algorithm $y$ and shuffle its values, so that there is no way to do genuine prediction and all of your features are effectively noise.  Then fit your chosen model $m$ times, observe the importances of your features for every iteration, and record the "null distribution" for each.  This is the distribution of the feature's importance when that feature has no predictive power.

Having obtained these distributions you can compare the importances that you actually observed without shuffling $y$ and start to make meaningful statements about which features are genuinely predictive and which are not.  That is, did the importance for a given feature fall into a large quantile (say the 99th percentile) of its null distribution?  In that case you can conclude that it contains genuine information about $y$.  If on the other hand the importance was somewhere in the middle of the distribution, then you can start to assume that the feature is not useful and perhaps start to do feature selection on these grounds.

Here is a simulation you can do in Python to try this idea out.  First we generate data under a linear regression model where only 3 of the 50 features are predictive, and then fit a random forest model to the data.  Now that we have our feature importances we fit 100 more models on permutations of $y$ and record the results.  Then all we have to do is compare the actual importances we saw to their null distributions using the helper function dist_func, which calculates what proportion of the null importances are less than the observed.  These numbers are essentially $p$-values in the classical statistical sense (only inverted so higher means better) and are much easier to interpret than the importance metrics reported by RandomForestRegressor.  Or, you can simply plot the null distributions and see where the actual importance values fall.  In this case it becomes very obvious that only the first three features matter where it may not have been by looking at the raw importances themselves.

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor

# number of samples
n = 100
# number of features
p = 50
# monte carlo sample size
m = 100

# simulate data under a linear regression model
# the first three coefficients are one and the rest zero
beta = np.ones(p)
beta[3:] = 0
X = pd.DataFrame(np.random.normal(size=(n, p)),
                 columns=["x" + str(i+1) for i in range(p)])
y = np.dot(X, beta) + np.random.randn(n)

# fit a random forest regression to the data
reg = RandomForestRegressor()
reg.fit(X, y)

# get the importances
var_imp = (pd.DataFrame({"feature": X.columns,
                        "beta": beta,
                       "importance": reg.feature_importances_}).
           sort_values(by="importance", ascending=False).
           reset_index(drop=True))

# fit many regressions on shuffled versions of y
sim_imp = pd.DataFrame({c: np.empty(m) for c in X.columns})

for i in range(m):
    reg.fit(X, np.random.permutation(y))
    sim_imp.iloc[i] = reg.feature_importances_

# null distribution function
def dist_func(var, x):
    return np.mean(sim_imp[var] < x)


Item based and user based recommendation difference in Mahout

Item Based Algorithm

for every item i that u has no preference for yet

  for every item j that u has a preference for

    compute a similarity s between i and j

    add u's preference for j, weighted by s, to a running average

 return the top items, ranked by weighted average

User Based Algorithm

for every item i that u has no preference for yet

 for every other user v that has a preference for i

   compute a similarity s between u and v

   add v's preference for i, weighted by s, to a running average

 return the top items, ranked by weighted average

Item vs User based:

1) Recommenders scale with the number of items or users they must deal with, so there are scenarios in which each type can perform better than the other

2) Similarity estimates between items are more likely to converge over time than similarities between users

3) We can compute and cache similarities that converge, which can give item based recommenders a performance advantage

4) Item based recommenders begin with a list of a user's preferred items and therefore do not need a nearest item neighborhood as user based recommenders do


If you use a single layer CNN, then each vector in the resulting activation maps would be related to the original 3x3 block. However, if you stack multiple CNN layers, you increase the receptive field of each resulting vector, as shown in the image below (taken from here):

After the CNNs, you can certainly compute an LSTM. There are, however, some design decisions you would need to take: disregarding the batch dimension, the LSTM takes as input a sequence of vectors (2D tensor) but you have a 3D tensor (height $\times$ width $\times$ num.channels) so, how are you going to make the 3D tensor fit as input to the LSTM? You could compute the LSTM of columns/rows of the image independently, in forward or reverse direction. You could also "collapse" one dimension (e.g. by averaging or taking the maximum value).


Can i pursue a career in data science with a business analytics degree?

You can. Nobody is going to demand a degree, but they will expect you to pass the interviews. I would recommend creating a blog and do some Kaggle competitions to show off your skills. I would also recommend taking more serious maths courses from the statistics department.


How to use a RBF kernel to create a "Kernel Space" using the similarity of each pair of point?

Let's take a simple example, a binary classification using only 2 dimensions with only 8 observations. Consider the first 8 rows of the dataset. It is impossible to linearly separate the data using a hyper-plane.

So, we can use a kernel transform. The books usually drop you off right here(no help). It does not tell us how to transform. Transform using what? How many dimensions? What constraints?

We could consider the simplest transformation, f(x,y)=xy. We could take this transformation and now consider it a new dimension. So we started with (D2+Class) features and we could add 1 dim (to get D3+Class) if we chose.

Q. Does this help?  I suggest either plotting this dataset out by hand. Using f(x,y)=xy is Not very helpful.

So let's try another, f'(x,y)=x^2 + y^2. We started with (D2+Class) features and added another dimension. We could conceivably plot D4+C, BUT using dimensions 1,2 & 4 are easier to visualize. I suggest plotting by hand (for effect) {d1,d2,d4} to your graphic or plot it using 3D software.

Now ask yourself, Is this new situation linearly separable?

As for the constraints, Do you remember LaGrange multipliers? Well if you want to use a gaussian like constraint we could use a Radial Basis Function (RBF).
Where:

RBF: K(x,y) = exp(-gamma * (||x−y||)^2)), gamma > 0

The first part is kernel 101. The LaGrangian is kernel 400. lol

dataset

        d1  d2      C          d3         d4
| row | x | y | class |f1(x,y)=xy|f'(x,y)=x^2 + y^2|
| ---:|--:|--:| -----:|---------:|----------------:|
|   1 |  1|  0|      0|         0|                1|
|   2 |  0|  1|      0|         0|                1|
|   3 | -1|  0|      0|         0|                1|
|   4 |  0| -1|      0|         0|                1|
|   5 |  2|  0|      1|         0|                4|
|   6 |  0|  2|      1|         0|                4|
|   7 | -2|  0|      1|         0|                4|
|   8 |  0| -2|      1|         0|                4|


Since you are only using one class for training, you can't expect the network to understand what the object is not.
For example, if you want to classify cats you have to pass to the network images that are not in your main class [dogs, cars, boats, random_noise, penguins]. If you try to pass any other image to your current model it will answer with great confidence that it is a cat.
Your validation accuracy will always be 100% if you have no other classes in your dataset.
For a classification problem you need at least two classes, and your train/val/test subsets should have enough images of each class so your metrics become meaningful.


Is a data set considered to be linearly separable if it can only be separated by multiple hyperplanes?

In simple terms: Linearly separable = a linear classifier could do the job. You could fit one straight line to correctly classify your data.

Technically, any problem can be broken down to a multitude of small linear decision surfaces; i.e. you approximate a non-linear function with a high number of small linear boundaries. That's what Neural Networks are doing with ReLU activations btw, but nobody would say that Neural Networks are linear models.


Theoretical and practical comparison of CTC and seq2seq loss in Tensorflow

From my own work, I give a few, sparse empirical reports:

bLSTMs are suitable for seq2seq loss in Tensorflow in case they are used within the encoder of an encoding-decoding scheme. I have not tried to use bLSTMs within the decoder.
--
seq2seq loss could be trained almost equally well with both RMSProp and ADAM on both G2P and P2G tasks.
--


You are going to have to do something - You can try combining them in different ways, multiply them together, divide them by each other, subtract one from another. Without the context around what these features actually relate to its difficult to say what would make sense. Ultimately to make a new derived feature you are going to have to combine them or transform them in some way.


how to handle different categorical values in train and test dataset?

Whether to remove the dimension or not would wholly depend on the kind of data you're working with and how important is the feature for your modelling task.
What you need to ask is are the labels predefined? And by that I mean is there any particular upper bound to the kind of labels you're receiving? If yes, you could simply use label encoding from sklearn. After which I'd recommending doing some feature analysis and getting the feature importance score for that label and then decide if you want to discard it or not.
Regarding your second question, you need to encode the labels before you train your model.


Finding linear transformation under which distance matrices are similar

I think what you are searching is related to metric learning. PCA is an unsupervised learning method but there are equivalent method in which you can introduce label information to learn the best linear transformation that maximize the distance between data from different labels.
http://contrib.scikit-learn.org/metric-learn/introduction.html#:~:text=Distance%20metric%20learning%20(or%20simply,%2C%20clustering%2C%20information%20retrieval).


What machine learning technique should I use in medical problem

Typically you need a much more exact phrasing (in more mathematical terms) of the question to ask.

Will a patient respond to medicine X? What is the likelihood? To what amount will patient repond to medicine X? Is patient in the group that is expected to respond to medicine X?
Are slightly diffent questions that may impact choice of technique.

Furthermore, your data plays an important role. Are you missing data? Have you normalized already? Do you expect 'Marital status' or 'Education level' to have an impact on medicine efficiency? (It might if certain groups take medicine at home, but it may be less likely when taking under doctor supervision)

A priori you determine how to measure or quantize the success of a model (typically prediction accuracy).

Then typically you try a few machine learning techniques, and make a model from the most successful one.