What does it mean by “t-SNE retains the structure of the data”?

You should break this down one step further: retaining local structure and retaining global structure.

Other well-understood methods, such as Principal Component Analysis are great at retaining global structure, because it looks at ways in which a dataset's variance is retained, globally, across the entire dataset.
t-SNE works differently, by looking at locally appearing datapoints. It does this by computing a metric between each datapoint and a given number of neighbours - modelling them as being within a t-distributed distribution (hence the name: t-distributed Stochastic Neighbourhood Embedding). It then tries to find an embedding, such that neighbours in the original n-dimensional space, are also found close together in the reduced (embedded) dimensional space. It does this by minimising the KL-divergence between the before and after datapoint distributions, $\mathbb{P}$ and $\mathbb{Q}$ respectively.

This method has the benefit of retaining local structure - so clusters in the low dimensional space should be interpretable as datapoints that were also very similar in the high dimensional space. t-SNE works remarkedly well on many problem, however there are a few things to watch out for:

Because we know have some useful local structure retained, we essentially trade that off for ability in retaining global structure. This equates to you not being able to really compare e.g. 3 clusters in the final embedding, where 2 are close together and 1 is far away. This does not mean they were also far away from each other in the original space.
t-SNE can be very sensitive to its perplexity parameter. In fact, you might get different results with the three-cluster example in point 1, using an only slightly different perplexity value. This value can indeed be roughly equated to "how many points shall we inlude in the t-distribution to find neighbours of a datapoint" - it essentially gives the area which is encompassed in the t-distribution.

I would recommend watching this lecture by the author of t-SNE, Laurens van der Maaten, as well as getting some intuition for t-SNE and it's parameters using this great visual explanation.

There are also some good answers here on CrossValidated with a little more technical information.


Combine two DataFrames column wise in Pandas

Use pandas.DataFrame.reindex()

df = pd.concat((df1, df2), axis=1)

#reorder columns
column_names=["A","C","B","D"]
df = df.reindex(columns=column_names)


How Do Machine Learning Models Work and Remember?

ML algorithms all learn a set of numbers associated with whatever model you're training. For a neural network, it's the weights on the network links. For regression, it's the coefficients. And so on. These numbers you can store and use later (most ML libraries have utilities for doing this). You use the coefficients to make predictions on data on which you have NOT trained the model.

The ML algorithm uses some sort of optimization for learning the coefficients. It might be gradient descent, or stochastic gradient descent, or xgboost, or any of a number of optimization algorithms. These are typically used to minimize a cost function. The definition of that cost function is model-dependent.


Not knowing the data in detail, this look like a linear model with "dummys".
A standard linear model looks like ($\beta_0$ is the intercept, $\beta_1$ is the slope):
$$ y = \beta_0 + \beta_1 x + u.$$
Now, when you have two distinct "groups" for which there is a "flag" in the data, you can assign a indicator variable or "dummy" (say $d$, a vector with =1 "true" or =0 otherwise). You can add this to your linear model:
$$ y = \beta_0 + \beta_1 x + \beta_2 d + u. $$
$d$ introduces a separate intercept for the assigned group ($d=1$).
You can also add "interaction terms" to allow for a separate slope for group $d$ by simply multiplying $x$ and $d$.
$$ y = \beta_0 + \beta_1 x + \beta_2 d + \beta_3 x d + u. $$
Note that since there already is one intercept in the model (the $\beta_0$), you can only add "contrasts" to the intercept. So when you have $i$ groups for which you want to have an individual intercept, you would add $i-1$ indicators/dummys to the model. For the "reference group", the intercept will be $\beta_0$ and for the group which is identified by the "dummy" ($d$ above), the intercept would be $\beta_0 + d$.


Performance difference between decision trees and logistic regression when one of the features is a string

String data can be either categorical (where you have e.g. more than 10 examples of each string) or free text.  If it's the former, a decision tree can deal with it no problem. You don't have to convert it into a numeric.

For regression you cannot directly use categorical variables.    If you want to use them in a regression, you will need to create dummy variables to encode the values. e.g. if your categories are "Red", "Yellow", "Blue" for the colour variable, you create variables "Red" (which will take a 1 or a 0) and "Yellow" (which takes a 1 or 0).  If both are 0, the colour must be "Blue".  There are functions in sklearn to do this automatically.

If your string is just free text then you will need a better way of grabbing information out of it.  You can use text mining such as tokenizing, TF-IDF etc. to convert it into numerical and categorical information that can be fed into a classifier.


Conditionally replace values in column with value from another column

for i in range(df.shape[0]):
    if(df['LastModified'][i].isnull()):
        df.loc[i,'LastModified']==df.loc[i,'YearBuilt']


Maybe you can modify my code:

   import numpy as np

   Duration_ = np.random.choice(range(0,56), 10000)

   Binary_Level = []

   b_ =["correct", "incorrect"]

   Binary_Level = np.random.choice(b_, 10000)

   import pandas as pd

   dd = pd.DataFrame({'Duration_': Duration_, 'Binary_Level':
      Binary_Level})

   xr_ = list(range(0,56,5))

   y_ = []

   for i in range(0,(len(xr_)-1)):

       a_ = np.logical_and(dd['Duration_'].values>=xr_[i],
       dd['Duration_'].values<xr_[i+1])
       b_ = np.logical_and(np.logical_and(dd['Duration_'].values>=xr_[i],
       dd['Duration_'].values<xr_[i+1]),
                               dd['Binary_Level'].values=='correct')
       y_.append(sum(b_)/sum(a_))

    import matplotlib
    matplotlib.pyplot.plot(xr_[1:12], y_)


How to map topic to a document after topic modeling is done with LDA

After training your LDA topic model you can input documents into the model and it will classify them into the pre defined number of topics. In gensim (python), this would look something like this:

ques_vec = dictionary.doc2bow(tokenized_document)
topic_vec = ldamodel[ques_vec]

The dictionary is something you should have created for training
ldamodel is the model that you trained.
The topic_vec will contain the classified topic number (class) and the probability that the document belongs to that class.

At this point, you will not know what is the meaning of each topic (class), because it is the result of unsupervised classification. To know what is the meaning of each topic that your lda model clusters your documents into, you have to look into the trained parameters like this:

words = ldamodel.show_topic(topic_number, topn = 200)

If you print that, you'll see the top 200 words that make up that topic number. Based on the meaning of the words in each topic, you name that topic as an appropriate class.


Yes, the linear assumption is incorrect since the graph is non linear


Diff. in P-value & F-Stat. Multiple linear regression

The overall F-stat says if your model is significantly better than the naive model that only has an intercept at the average of all response data pooled together. In other words, it measures if $R^2$ is significantly better than 0.

In your case, you can eyeball the $R^2$ and see that 0.897 is going to be better than 0. The F-stat and p-value will be more useful when the $R^2$ has a subtle difference from 0.

Another place where an F-stat is useful to regression is testing against a less naive model than intercept-only. In fact, the p-values on each parameter in the regression are equal to the p-value you'd get by doing the F-test comparing the full model to a model with that parameter excluded (assuming the typical nice properties that go along with doing parameter inference in linear regression).

The reason not to examine the individual p-values for each parameter is because of false discoveries, the usual business about multiple testing. Doing one F-test of the entire equation removes the possibility of false discovery of a significant parameter, in exchange for not being able to pin down which parameters are significant.


Dealing with missing data in several features at once

Imputing one or more Features at once you still Need to Diagnose.
Generic, mean or constant imputations rarely work (good).
To decide which strategy is appropriate, it is important to investigate the mechanism that led to the missing values to find out whether the missing data is missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR).

MCAR means that there is no relationship between the missingness of the data and any of the values.

MAR means that that there is a systematic relationship between the propensity of missing values and the observed data, but not the missing data.

MNAR means that there is a systematic relationship between the propensity of a value to be missing and its values.

Given what you have told its likely that its MCAR. (assumption is that you already tried to find this propensity yourself (domain knowledge) or build a model between the missing columns and other features and failed in doing so)
Some other techniques to impute the data, I would suggest looking at KNN imputation (from experience always solid results) but you should try different methods
fancy impute supports such kind of imputation, using the following API:
from fancyimpute import KNN

# Use 10 nearest rows which have a feature to fill in each row's missing features

X_fill_knn = KNN(k=10).fit_transform(X)

Here are different methods also supported by this package:

•SimpleFill: Replaces missing entries with the mean or median of each

column.

•KNN: Nearest neighbor imputations which weights samples using the

mean squared difference on features for which two rows both have

observed data.

•SoftImpute: Matrix completion by iterative soft thresholding of SVD

decompositions. Inspired by the softImpute package for R, which is

based on Spectral Regularization Algorithms for Learning Large

Incomplete Matrices by Mazumder et. al.

•IterativeSVD: Matrix completion by iterative low-rank SVD

decomposition. Should be similar to SVDimpute from Missing value

estimation methods for DNA microarrays by Troyanskaya et. al.

•MICE: Reimplementation of Multiple Imputation by Chained Equations.

•MatrixFactorization: Direct factorization of the incomplete matrix

into low-rank U and V, with an L1 sparsity penalty on the elements of

U and an L2 penalty on the elements of V. Solved by gradient descent.

•NuclearNormMinimization: Simple implementation of Exact Matrix

Completion via Convex Optimization by Emmanuel Candes and Benjamin

Recht using cvxpy. Too slow for large matrices.

•BiScaler: Iterative estimation of row/column means and standard

deviations to get doubly normalized matrix. Not guaranteed to converge

but works well in practice. Taken from Matrix Completion and Low-Rank

SVD via Fast Alternating Least Squares.

MICE was deprecated and they moved it to sklearn under iterative imputer
Another Option is old kaggle trick:
A trick I have seen on Kaggle.
Step 1: replace NAN with the mean or the median. The mean, if the data is normally distributed, otherwise the median.
In my case, I have NANs in Age.

Step 2: Add a new column "NAN_Age." 1 for NAN, 0 otherwise. If there's a pattern in NAN, you help the algorithm catch it. A nice bonus is that this strategy doesn't care if it's MAR or MNAR (see above).


I have 32k black and white images. Want to do clustering on them

As the dimension of your data is very high (40000), you can try using Spherical K-Means on your raw data. It uses the cosine similarity to compute the distance between 2 points. Indeed, euclidian distance is not so much informative in high dimensional vector spaces, because of the curse of dimensionality. see https://stats.stackexchange.com/questions/99171/why-is-euclidean-distance-not-a-good-metric-in-high-dimensions

Or you can apply a dimension reduction strategy like truncated SVD or ACP like you did, and then apply the classical k-means algorithm. So yeah you should try to apply k-means on your 100-dimensions data. Maybe try with k = 30 or something similar as 100 may already be too high.


How are clusters from DBSCAN sometimes non-convex?

A cluster in DBSCAN consists of multiple core points.

The radius is the area covered by a single core point, but together with neighbor core points the shape will be much more complex. In particular, they can be much larger than epsilon, so you should choose a small value, and rely on this "cover" functionality.

Wikipedia has an example of a non-convex cluster


How to determine whether a semantic concept is present in a string

This is essentially information retrieval: usually there is a collection of documents and the goal is to find the document which is the most similar to a given query (what you call the "semantic concept").
The traditional way to do that is to convert the collection of documents as vectors, typically with TFIDF weight but there are many options (I assume that recent approaches favor using word (or document) embeddings). Then a similarity measure (for example cosine) is used to measure the similarity between every document and the query, and the most similar document is selected. Since in this case there would be a single document, you could use a threshold on the similarity level in order to answer yes or no (concept found or not).


There are many approaches to this type of problem, which would be defined as text classification. You could create a Term Frequency or Term Frequency Inverse Document Frequency matrix based on the tokens (words) in the strings, then use standard machine learning algorithms such as SVM, Random Forest, Gradient Boosted Trees, Logistic Regression, Naive Bayes. You could also potentially use word embedding's and neural networks. In this case, each word would be represented as a vector of similar words. Finally, I have recently been experimenting with character level convolutional neural networks and this seems to work well for user generated text classification. There is link below.

https://papers.nips.cc/paper/5782-character-level-convolutional-networks-for-text-classification.pdf


You could treat your results as an undirected graph with weighted edges. Your nodes are A, B, C, etc. and your vertices are the connections between items, weighted by similarity value.

Here is a Python function that could be a great starting point. It draws thicker connections for vertices with a larger weight. I copied it and replaced the example data with your data points above:

[...]
node_list = ['A', 'B', 'C', 'D', 'J', 'R', 'Y']
[...]
G.add_edge(node_list[0],node_list[2],weight=0.12) # A - C
G.add_edge(node_list[0],node_list[5],weight=0.42) # A - R
G.add_edge(node_list[0],node_list[6],weight=0.34) # A - Y
G.add_edge(node_list[0],node_list[4],weight=0.62) # A - J
G.add_edge(node_list[0],node_list[1],weight=0.16) # A - B
G.add_edge(node_list[0],node_list[6],weight=0.83) # B - Y
G.add_edge(node_list[2],node_list[4],weight=0.21) # C - J

The result is this:

Of course, if you have 40,000 items, that would probably require additional work to make the plot comprehensible. You could begin by ignoring all connections between nodes with a similarity value of, say, 0.01 or less.


How were auto encoders used to intialize deep neural networks?

There were a few different techniques. One popular one was stacked autoencoders, where each layer was trained separately.

Essentially this was done by progressively growing the autoencoder, two layers at a time (one encode layer, plus equivalent decode layer), followed by complete training at each step of growth.

If learning from a fixed training set, you could store the encoded representation of the whole dataset so far as input into next stage of training, saving some computation when building up the layers.

After training each encoder layer separately you could use the weights of the encoder section of the autoencoder as the starting weights of the deep NN. Intuitively this made sense as you would have a representation of the input that you knew could be used to reconstruct it, and that typically was compressed, so should in theory have extracted salient details from the training data population. On top of these pre-trained layers, you may add one or two new layers that implemented whatever classification or regression task that you needed the final NN to perform. Then you would train with the labelled data - this is similar to fine-tuning networks and transfer learning that is still done nowadays.

The results from this pre-training stage could be worthwhile. It is still a valid technique if you have a lot of unlabelled data, and a relatively small amount of labelled data. However, the introduction of ReLU and careful controls on weight initialisation meant that deep networks could often be trained more directly. Recent additions such as skip connections and batch normalisation have further improved more direct training approaches.

Here is an example with code, using TensorFlow.


Creating categorical variable, without knowing true categories (through binning time series data)?

So, if I understood you correctly, you want to classify your data based on the frequency and amplitude of the temperature change, right?

I think your attempt with stdev is not so far away, but your interval is a little off, 6 hours might be to much. Think about Shannon, your sample rate must be at least twice the highest frequency in your dataset, if you want to reconstruct your signal completely. If your sample rate is 6 hours you might miss the big leaps at the end of the image.

My recommendation would be to use a sliding window and use Fast Fourier Transformation to get to the frequency spectrum. With those datapoints you can try to cluster them with SVM or kNN approaches. I think SVM is the way to go, since your class should be cleanly seperateble. And since you know how many classes you have there is no need for hypertuning this parameter.

I certainly don't know if this is best practice, but I think this would be a feasible approach.


AB testing : When AA testing doesn't work

I 'solved' the issue with Oracle's support, I still post an answer here because some of what I encountered can be useful :  According to Oracle's support, this bug in AB testing programs was known for the last 9 month, but corrected only this week-end.

So my problem is supposedly solved (I will re-launch a test and see if results are now consistent), but I still have some pointers if someone encounters the same problem I did :

Make sure this is not a statistical artifact : test your hypothesis on several independant periods. I used a Fischer test because it is exact for a binomial framework, it is what you should use if your population/conversion rate is too small for a normal approximation
Respect hypothesis testing framework : start your test after you formulated your hypothesis
Get a very very low p-value : No one will listen to your complaint about faulty AB testing if you come up with a 3% p-value, and they would be right. A bayesian analysis probably points to a false positive.
Make sure the random numbers that are used to split the populations are actually random (and not, say a 50/50 split)
Imagine all the possibilities that your reporting might be wrong. I thought I had imagined them all, but if you read below the real reason behind the bug, you will find that faulty reporting can be hidden very far away in parts of the environment you don't control or even have a view in.

I have the actual answer from Oracle that describes the bug that happened, in case someone has the exact same problem or wants to know how this kind of bug happens :

For programs triggered by event, their engine does not have the capability to associate the different open and clicks events to the correct email. In the case several emails have been sent through the same program, Responsys assumes the event concerns the earliest email that has been sent through this program.

In my case, the program could have been sent several times during the period.

In the period where I was not splitting my emails through two branches, the name of the campaign I used was already A  (the same than one of my branches in my testing period)

You can see where this is going : Every customer opening any email was marked as having read the "A" email, as long as they had been into the program in the 3 months before I started my AB testing, which caused a faulty reporting.

So actually both branches were sent to the correct customers, and it is just a reporting problem that comes from an approximative algorithm on Responsys side.

The workaround I will try now is simply to create two brand new "campaign" objects before starting a test rather than using a campaign that already existed.


Orange3 does not show Geo Map tool on OS X

Take a look if you have the widget "Map" in your Visualization group. It might be the same widget, just with the different name.

It looks "Geo Map" widget appears for Orange 3.4.2 for Linux, in Visualize group:

However, from this commit in Orange source code, it looks the previous name for this widget was just "Map".


Considering the output of a BLSTM in pytorch, what's the order of the elements?

Your answer is in the documentation of the code you linked in your comment:

  For the unpacked case, the directions can be separated using output.view(seq_len, batch, num_directions, hidden_size), with forward and backward being direction 0 and 1 respectively. Similarly, the directions can be separated in the packed case.

So you just need yo separate the directions as specified above and then take the element-wise maximum with torch.max.


Unless you normalize the MSE in scenario 1 (or denormalize the MSE in scenario 2), comparing two MSE with two different scales is irrelevant.

You can have data with values varying from 10 to 30 millions, centered then normalized to -1/+1. Suppose you have a MSE of 1000 in the first case, and 0.1 in the second, you will easily see that the second MSE is way more impacting than the first one after (de)normalization.

That said, if you want to retrieve the target from scenario 2, you need to apply the reverse operations to what has been done to get the "normalized" target. Assuming for instance that you centered / reduced the target.

$$
Z = \frac{Y-\bar{Y}}{\bar{Y}}
$$
$$
Z = \alpha X_1 + \beta X_2 + \gamma X_3
$$

Where $Y$ is your initial target, $\bar{Y}$ its average, $Z$ your normalized target and $X_i$ your predictors.

When you apply your model and get a prediction, say $\hat{z}$, you can calculate its corresponding value $\hat{y}$ after applying the reverse transformations to your normalization.

$$
\hat{z} = \alpha x_1 + \beta x_2 + \gamma x_3
$$
$$
\frac{\hat{y}-\bar{y}}{\bar{y}}  = \alpha x_1 + \beta x_2 + \gamma x_3
$$
$$
\hat{y} = \bar{y} *(\alpha x_1 + \beta x_2 + \gamma x_3 ) + \bar{y}
$$

Now in this particular case, as Ankit Seth pointed out, you need not normalizing your target. The linear model would have adjusted its coefficients automatically, because you have been using linear operands.

However, if you proceed to non linear operations for target normalization, the same logic should apply. Say for instance that your model predicts the logarithm of your target.

$$
log(Y) = \alpha X_1 + \beta X_2 + \gamma X_3
$$
The reverse operation would be to apply its reciprocal function, e.g. exponential.

$$
Y = exp(\alpha X_1 + \beta X_2 + \gamma X_3)
$$


There are some visual indicators of speed that should be present even in one frame. An easy example to think of is "blur", looking at a static image you could still determine "movement" (and therefore velocity) if a picture is blurred vs. absolutely sharp.

Additionally perspective, depth and shape of other objects might differ depending on velocity.

So it certainly could be possible to train a model based only on static images but it certainly should not outperform a model using both features of static images and differences between frames.

Edit:

I also looked at the repo and the presented method a bit. In the used method there could be a huge over-fitting problem if the video is only one single drive!

In this case the image detection would only identify "where" the picture was taken and since any point along the drive is associated with a single speed it could work from there i.e.: when the truck drove past a palm tree he was driving at 30 mph so identifying a palm in the picture should predict 30mph which works for a single drive data set but obviously is not able generalize.


Loss function to maximize sum of targets

Define $Z_i = 1[Y_i>0]$, i.e., $Z_i = 1$ if $Y_i > 0$, else $Z_i = -1$.  Now  the data set $(X_i,Z_i)$ defines a two-class boolean classification problem.  The solution to that boolean classification problem with highest accuracy solves your original problem (minimizes your loss function).

So, you are basically asking how to train a neural network to solve a two-class boolean classification problem.  A reasonable approach is to put a softmax layer at the output of the neural network and train it using the cross-entropy loss, as usual.  During test time, to pick a class, you pick whichever of the two classes has the higher likelihood output from the softmax stage.  In other words, at training time, you look at the actual outputs (continuous-valued) from the softmax layer and use the cross-entropy; at test time, you compare the two outputs and pick whichever is larger to choose the classification (and apart from that ignore their exact values).

The shortcoming of this approach is that it doesn't take into account the value of the $Y_i$'s, only their sign.  To improve this, use weights on the samples.  We're again going to train a boolean classifier on the training set $(X_i,Z_i)$, but this time we'll weight each sample in the training set differently.  Put the weight $Y_i$ on the sample $(X_i,Z_i)$ in your training set.  In other words, when training the boolean classifier, the loss function will be a weighted sum of the loss for each sample:

$$\text{Loss}(\theta) = \sum_i Y_i \cdot \ell(f_\theta(X_i),Z_i)$$

where $\theta$ are the model parameters, $f_\theta(X_i)$ is the output of the classifier on input $X_i$, and $\ell(\cdot,\cdot)$ is the cross-entropy loss.  Notice that errors in samples where $Y_i$ is large are penalized more, and errors in samples where $Y_i$ is small are penalized less; this is appropriate, as that's exactly what will happen when you use the classifier $f_\theta$ to compute $L(f_\theta)$.  This should improve on the approach you're taking.


solving quadratic equations using machine learning?

It is possible to predict the solution for arbitrary equations provided the at least one real root exists?

Yes.

You‘ll just need a sufficiently large training dataset and network that is flexible enough (start with one hidden layer).

You’re right your problem is not linear, thus you’ll need non-linear activation function.


Does the performance of GBM methods profit from feature scaling?

Scaling doesn't affect the performance of any tree-based method, not for lightgbm, xgboost, catboost or even a decision tree.
This post that elaborates on the topic, but mainly the issue is that decision trees split the feature space based on binary decisions like "is this feature bigger than this value?", and if you scale your data, the decisions might look different, as they are done on the scaled space, but the results should be the same.
As an example, a decision tree should split in the same way the data if you change units (a particular case of scaling). Let's say you want to use weight of a person to predict if someone is under 18 (as a binary classifier). If given the weight in grams, the decision tree might do something like: if weight < 5000 grams, then the person is under 18. If you change the units to kilograms, the decision tree will do: if weight < 5 kg, then the person is under 18.
To sum up, the splits will be equivalent under any scaling or, in general, any increasing linear transformation.


How to initialize a new word2vec model with pre-trained model weights?

Thank Abhishek. I've figure it out! Here are my experiments.

1). we plot a easy example:

from gensim.models import Word2Vec
from sklearn.decomposition import PCA
from matplotlib import pyplot
# define training data
sentences = [['this', 'is', 'the', 'first', 'sentence', 'for', 'word2vec'],
            ['this', 'is', 'the', 'second', 'sentence'],
            ['yet', 'another', 'sentence'],
            ['one', 'more', 'sentence'],
            ['and', 'the', 'final', 'sentence']]
# train model
model_1 = Word2Vec(sentences, size=300, min_count=1)

# fit a 2d PCA model to the vectors
X = model_1[model_1.wv.vocab]
pca = PCA(n_components=2)
result = pca.fit_transform(X)
# create a scatter plot of the projection
pyplot.scatter(result[:, 0], result[:, 1])
words = list(model_1.wv.vocab)
for i, word in enumerate(words):
    pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))
pyplot.show()

From the above plots, we can see that easy sentences cannot distinguish different words' meaning by distances.

2). Load pre-trained word embedding:

from gensim.models import KeyedVectors

model_2 = Word2Vec(size=300, min_count=1)
model_2.build_vocab(sentences)
total_examples = model_2.corpus_count
model = KeyedVectors.load_word2vec_format("glove.6B.300d.txt", binary=False)
model_2.build_vocab([list(model.vocab.keys())], update=True)
model_2.intersect_word2vec_format("glove.6B.300d.txt", binary=False, lockf=1.0)
model_2.train(sentences, total_examples=total_examples, epochs=model_2.iter)

# fit a 2d PCA model to the vectors
X = model_2[model_1.wv.vocab]
pca = PCA(n_components=2)
result = pca.fit_transform(X)
# create a scatter plot of the projection
pyplot.scatter(result[:, 0], result[:, 1])
words = list(model_1.wv.vocab)
for i, word in enumerate(words):
    pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))
pyplot.show()

From the above figure, we can see that word embeddings are more meaningful.
Hope this answer will be helpful.


Why do we need to handle data imbalance?

You need to deal with imbalanced data set when the value of finding the minority class is much higher than that of finding the majority.

Let say that 1% of the population have that rare disease. Suppose that you assign the same cost to saying that a healthy man is sick or saying that a sick man is healthy. Provide a model that say that everybody are healthy, get 99% accuracy and go home early.
The problem with such a model is that though it has high accuracy, it will probably not what you are looking for. Most of the time you want to find the people with the disease (giving high weigh to false negatives) much more than you are afraid to send an healthy person to unneeded test (low weight to false positives). In a real world health problem the ratio between the weight can easily be 1 to 1,000.

The imbalance in the distribution fails most algorithms from finding a proper solution.

You are correct that just balancing the distribution isn't the optimal solution. Indeed, an algorithm that is trained on a balanced distribution is not fitted to the natural distribution on which it will be evaluated.
My favorite method is adapting it back, as you can see here.
For a discussion, see here.

Just setting the ratio to some other problem won't work since you will have the same problem.

Smote is working in a different way, which didn't work as well when I tried it, but it might fit your problem.


Is cross-entropy a good cost function if I'm interested in the probabilities of a sample belonging to a certain class?

Cross entropy is indeed appropriate for multiclass classification.

When the tensorflow documentation states about cross_entropy that:

  "Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class)."

it is referring to the fact that, at a conceptual level, classes are not expected to overlap. An example of non-overlapping classes are "cat" and "dog". An example of overlapping classes are "Elephant" and "African Elephant"; this type of problem is called multilabel classification, because instead of classes, you have labels and each individual can be assigned many labels.

Update: with the new information about the problem, we can tell that the problem being faced is not a classification one, as the desired output are specific probability values. This implies that the problem is a regression one. As the outputs are probabilities, it is appropriate to use softmax to ensure they add up to 1. As it is regression, it would be appropriate to use MSE as loss function.


Calculate cosine similarity in Apache Spark

There's a related example to your problem in the Spark repo here.  The strategy is to represent the documents as a RowMatrix and then use its columnSimilarities() method.  That will get you a matrix of all the cosine similarities.  Extract the row which corresponds to your query document and sort.  That will give the indices of the most-similar documents.

Depending on your application, all of this work can be done pre-query.


How is PACF analysis output related to LSTM?

When used for time series, LSTM (or any Neural Network approach) fall under the category of auto-regressive models. The most general formulation of an auto-regressive model is:

$Y_t = f(Y_{t-1}, Y_{t-2},...,Y_{t-n})$ with n the number of past lags to include in your model.

When using a neural network for univariate time series modeling, n will determine the number of input neurons to use (It could be n, or could be n+k, with k additional causal features).

The PACF will help you determine n.

The key here is that neural networks are a generalization of AR(p) models.


Precision gives the percentage of correctly predicted positives out of total positives predicted by the model. Therefore, low conversion rate of your model is due to low precision.
Looking at the support, this looks like a class imbalance problem. 92% of your training examples belong to class '0'.Therefore, while training, your model has more examples of class '0' to learn from.
There are a few ways to deal with this problem:

Over-sampling - This when you randomly draw samples from the minority class with replacement and use it in the training set
Under-sampling - This is when you draw random samples from the majority class with or without replacement and decrease the number of examples of the majority class given to the model while learning.
Synthetic sampling - There are also techniques like SMOTE that generate samples of minority class synthetically.

These are only few techniques I have highlighted. You can check this link for further reading https://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/


Why is Reward Engineering considered "bad practice" in RL?

One argument is that we generally don't know, a priori, what will be the best solution to an RL problem. So by tweaking the reward function, we may bias the agent towards what we think is the best approach, while it is actually sub-optimal to solve the original problem.

This is the main issue.

  Am I missing anything?

I think you are missing the main point of setting a reward function. It should be a value that is maximised by an agent achieving the goals it has been set. Each time you change the reward function you may be explicitly setting new and different goals.

Changing a reward function should not be compared to feature engineering in supervised learning. Instead a change to the reward function is more similar to changing the objective function (e.g. from cross-entropy to least squares, or perhaps by weighting records differently) or selection metric (e.g. from accuracy to f1 score). Those kinds of changes may be valid, but have different motivations to feature engineering.

Tweaking a reward function is also called reward shaping, and sometimes it can have good results. It does come with the same risks as above, but if done carefully it can improve learning rate. For example if to achieve the main goal A, it is absolutely necessary to achieve B and C as interim steps, then it should be OK to reward achieving B and C - typically the things you have to worry about is whether it is possible for the agent to repeatedly achieve B or C via some loop through states - so you may need to add to the state vector whether it has achieved B or C enough times and only grant reward for first visit to B and C.

Feature engineering does still exist in RL, and is about how you represent state and action spaces. For instance, in a chase scenario where an agent needs to get close to a moving target, you should find that representing the state as polar co-ordinates of the target from the current agent's position is far easier for RL to learn optimal policy than if you represent state as cartesian co-ordinates of agent and target.


Can 2 dimensional input be applied to SVM?

If I understand your question correctly. Yes, SVM can take multiple inputs. My suggestion for handling a vector as a feature would be to expand it out. For example,

x0 = (1,2)       x0 = 1
x1 = .4  ----->  x1 = 2
x2 = 0           x2 = .4
                 x3 = 0

If this does not capture all of the characteristics of the vector that are important, then you may want to add other features (like magnitude of the vector) as well.


How do you avoid 'analysis paralysis' when choosing a method to implement?

This is a very broad question but in general you can define some criteria that you want the method to meet (eg low error, efficiency, scalability) and then you score the methods, which one solves your problem best.


Why does using Gradient descent over Stochatic gradient descent improve performance?

SGD has a regularization effect and finds the solution faster. GD on the other hand takes a look at whole data and finds the next best step.
SGD may be come to optimal global minima but GD can. But GD is not practical with large data.


There are a couple of good threads on Reddit right now (here and here).

I haven't used either of these frameworks, but from reading around and talking to users I gather that support for dynamic graphs in PyTorch is a 'top down design principle', whereas TensorFlow Fold is bolted on to the original Tensorflow framework, so if you're doing anything reasonably complicated with Tensorflow Fold you're probably going to end up doing a lot more hacking around than if you're using PyTorch.


How to filter Items in Recommender Systems?

Is there a reason you cannot filter NSFW articles as a post-processing step? For instance, if you need to recommend 10 articles, you find the top 50 recommendations and then apply the filter if it is needed and truncate down to top 10?
In my experience, it is pretty common to apply business rules before/after recommendations to account for things like this. Grabbing an example article on this topic for reference:
https://buildingrecommenders.wordpress.com/2015/11/10/the-components-of-a-recommender-system/


How to generate plot of reward and its variance?

To estimate the variance, you probably need to run your algorithm multiple times and keep track of the return for each of these runs. From these multiple returns, you can estimate the variance.

Once you have the standard deviation (or variance) of the return, you can plot something like your plots using matplotlib's fill_between function.


It really depends on your data. First, you should also keep in mind that you have to define what is a "drastic change" and if it is just a simple number or threshold then you might not even need an ML algorithm to solve this, you can easily have some conditions to check the values or do it threshold-based.

But, if you want to use an ML algorithm then I would suggest that you take a look at Novelty and Outlier Detection tools provided by Scikit-Learn here:

https://scikit-learn.org/stable/modules/outlier_detection.html

In particular, you can try "Isolation Forest" and "One-class SVM". For me in many cases "Isolation Forest" is the best fit, but try them out and see which one works the best for you.

Let me know if this is helpful.


How to read specific time for specific value row by row using python

One-liner:

result = data.loc[data.x == 4, "time"]    # assuming you expect just the time as output

Explanation

To describe your problem, first in English (then code):

  You want the rows of the data frame, where the value of the X column is 4, and all columns should be included in the result.

To check where coilumn X is 4, we can do this:

data.X == 4

This returns True or False for every single row. Now we can use this as a filter, by passing it to the pandas DataFrame's .loc accessor (loc = "locate").

result = data.loc[data.X == 4, :]
# Notation: dataframe[rows_we_want, columns_we_want]

Because we want all columns, we have left the colon : in there after the comma. Pandas assumes you want all columns by default, so we can actually leave it out to get the one-liner given above.

So if you just want to time to be given as output, you can filter for that by putting the name of the column after the comma:

result = data.loc[data.X == 4, "time"]


Correlation between Time Series Indicators ( Stock Prices )

For time series, correlation is a different. A variable might b related past N values of other variables.

This article explains theory behind finding relationships in time series (Skip to section "Stationarity in Time Series") : https://www.quantstart.com/articles/Serial-Correlation-in-Time-Series-Analysis

This is an implementation in Python : https://machinelearningmastery.com/gentle-introduction-autocorrelation-partial-autocorrelation/


extract classifier properties from pickled file

clf = SVC()
clf.fit(X, y)

print(clf.get_params())


Kernel Trick and Inner Product Preservation

It doesn't mean that $\langle x, y \rangle = \langle \phi(x), \phi(y)\rangle$.

Kernel method in general means that for an algorithm that involve $\langle x, y\rangle$, we can replace it with a function $K(x,y)=\langle \phi(x), \phi(y)\rangle$ where computing $K(x,y)$ is easy. It is known that for $K$ that satisfies Mercer's theorem, there is a corresponding $\phi$ which is the map to the higher dimensional space.

We do not need to know $\phi$ explicitly and $\phi$ can be very complicated.


Choropleth Map with plotly.express doesn't show map, just legend

You executed the code either inside a cell of a jupyter notebook or in jupyter lab. That explains why no graph was shown.
If you would ran the same code as a pure python script it shows the graph properly.
Plotly provides two different solutions for getting it running inside jupyter:
Plotting Inline
Please search for Plotting Inline on that page to find the paragraph that explains the solution en detail.
Proper running example purePython and Jupyter Notebook
I just let it ran as it is also as purePython and as Jupyter Notebook cell:
import plotly.express as px

df = px.data.election()
geojson = px.data.election_geojson()

fig = px.choropleth(df, geojson=geojson, color="Bergeron",
                    locations="district", featureidkey="properties.district",
                    projection="mercator"
                   )
fig.update_geos(fitbounds="locations", visible=False)
fig.update_layout(margin={"r":0,"t":0,"l":0,"b":0})
fig.show()


Have a look on HMM (Hidden Markov Model) also. A concrete example Of HMM is available in wikipedia. Decision tree is better
in generalising and applying learned data in another context and a Markov-model is better to recall the exact
learned machine state.


Binary classification to predict various targets

If you create 19 models, each model will learn weights corresponding to predictors and target variable values.

For example- When you create one model by taking col1 as target, the model will learn weights for giving output col1. When you take another column, lets say col2 as target, if you use the previous model, it won't identify the difference between col1 and col2 (i.e., target) as your data is binary. The model will continue to learn weights for each column, but it won't be the correct one as it will be result of learning all target columns.

You could have used multi-label classification if you wanted to predict different targets at same time using same predictors for each target. Since this is not the case, the only way for getting 19 predictions is to create 19 different models, one for each column as target.


How to increase model's prediction accuracy

It sounds like you’re overfitting the training data, and so you should research some approaches to regularisation of neural networks. I see you’ve used dropout, which is one of these, but may not be sufficient.
You could also consider dividing your training data into a training set and a validation set and halting training when performance on the validation set stops improving. This is called early stopping


How do I run SMOTE on image data using the packages available?

This might be helpful - https://github.com/tgsmith61591/smrt. Synthetic Minority Reconstruction Technique uses a variational auto encoder.


The United States Census Bureau has many free housing datasets (some of which are updated more than once every 10 years). There is an API for American Community Survey 1 Year Data that includes housing data. There are raw data sets at American Fact Finder.


Lower loss always better for Probabilistic loss functions?

Consider loss to be something akin to implied variability of your model. When loss is extremely high, your model's output could be just about anything. As the loss lowers, your model is becoming more confident in its output and will be able to give similar output classification/regressions even if the initial weights or input data is slightly different. Lower loss in the training set is always a good thing so long as you are seeing a subsequent decrease in the loss of the validation set. As soon as that validation loss starts to stagnate though, I like to save the model and quit. Usually at this point you could get a little better loss, but it tends to just be overfitting and doesn't help any on the test set.


I'll go through an example that will help you get started. It should get approximately 50% accuracy.

So I keep the code the same as yours for loading the data. The only difference is that I normalize the data to lie between 0 and 1. This is usually recommended to bound the weights more tightly.

import numpy as np
from keras.datasets import cifar10
from keras.layers import Dense, Activation
from keras.optimizers import SGD
from keras.utils import np_utils
from keras import backend as k
from keras.models import Sequential
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

(x_train, y_train), (x_test, y_test)=cifar10.load_data()

  Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz 170500096/170498071 [==============================] - 71s 0us/step

(x_train, y_train), (x_test, y_test)=cifar10.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.

x_train = x_train.reshape(50000, 3072)
x_test = x_test.reshape(10000, 3072)

# The known number of output classes.
num_classes = 10

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], x_train.shape[1],)
x_test_reshaped = x_test.reshape(x_test.shape[0], x_test.shape[1],)
input_shape = (x_train.shape[1],)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

Now let's make our model. Here I do make the hidden layers have more neurons per layer.

model = Sequential()
model.add(Dense(32,
                 activation='relu',
                 input_shape=input_shape))
model.add(Dense(128, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

Now we can train the model

epochs = 4
batch_size = 128
# Fit the model weights.
model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test_reshaped, y_test_binary))

  50000/50000 [==============================] - 3s 58us/step - loss: 1.6716 - acc: 0.4038 - val_loss: 1.6566 - val_acc: 0.4094


Linear machine learning algorithms "often" have high bias/low variance?

The "often" is the key here - the way that linear models are built, especially compared to other types of models, are more likely to favor certain types of errors.... in this case, they are more likely to produce bias-type errors rather than variance-type. Another way of thinking about it is that the way that most linear models will give you broadly correct predictions overall, instead of specifically accurate or wildly inaccurate predictions. Obviously that's a super simplistic reading, but I've found an intuitive way to approach. Essentially - the way that linear models are constructed will lead them to predict more generally, instead of reflecting every type of twist and turn in the data. This is partially a function of the models themselves, and partially due to linear models being the first line of attack with smaller datasets.

If you think about the formula, one way to interpret is that the intercept generalizes the model's predictions. That being said, with enough variables and certain types of data, it's very possible to overtrain a linear model (which will tip it towards variance-type errors).

I like to think of the bias/variance tradeoff question as being one with three factors, instead of just the two in the title. I'll use the short definitions below (for more information, I've found wikipedia's article on the topic to be relatively clear):

Bias - causing the types of errors frequently thought of as
"underfitting," high bias might generalize well at the cost of
not-capturing (losing out on) important nuances in the data

Variance - causing the types of errors frequently thought of as "overfitting," high variance will capture most nuances in the
training data including noise that will reduce its likelihood of
generalizing well to future like datasets
Data/Data Quality - While not usually included in the classic understanding of the tradeoff, data type and quality is integral to understanding why a certain model is tipping towards either the bias or variance sides of the tradeoff. This also includes any kinds of time effects or other quirks that in essence mean that a training/test split will not properly represent the model's future performance, especially as regards linear models.


How to use scikit-learn to extract features from text when I only have positive and unlabeled data?

So you should be clearer with what you are asking.

What are the classes of your classifier? Positive and unlabelled?

To create numeric feature from text you can use:
a) tf-idf, which works well with small datasets/ sentences.
b) Handmade features, like extracting sentence length, occurence of certain words..
c) Word embeddings, like word2vec or sentence2vec. Strong method when a does not work. This will convert every word or sentence/ document to a numeric vector of length n, usually between 100 and 500. See the gensim library for that
d) a combination of the above.

The main criteria of choosing the feature transformer should be that it captures your text well and that is something you can decide only by looking at the data yourself and trying out some experiments.
Good luck!


I'm not seeing where the relationship between hypotheses and research paper is. I know what the relationship is in reality, but your question is a bit unorganized. Perhaps we could think about it in a different way...

I would first create a definition of possible hypotheses (i.e., $H_{0}$ and $H_{1}$ are my two possible hypotheses, and every hypothesis will have both a null and an alternative), and then associate a $H_{0}$ with a $H_{1}$ such that a combination of both null and alternative hypotheses equate to one hypothesis.

With this, it's a matter of iteratively counting.

With $p \in P$, where $p$ is a paper in the set of papers $P$,
With $w \in W$, where $w$ is a word in a set of words $W$ that indicate whether $p$ is a research paper or not,
With $i$ as the number of times a $w$ occurred in a $p$,
count the number of times a $w$ occurs in each $p$ and iterate $i$ accordingly.


My answer can only be considered partial, I've not compiled a list, but I believe all algorithms implemented here, are, well, implemented for both offline and online mode. This one, can also be implemented for online mode.

Not trying to imply you should use that implementation, but this is kind of a living proof transcending deducing very analytically from articles. The thing to understand, is that certain CB algorithms are paired with rather benign algorithms for using offline-accumulated data for training them, in turn paired with mathematical proofs that the loss they incur in that offline training is a good predictor for the loss they'll incur in online mode (if the real world is still 'sufficiently similar' to the one logged from).

Some (other than mentioned above) algorithms may be only applicable to offline training, or at least I'm not aware of a theoretical refutation, that an algorithm may offline-train better in a way precluding direct use for online learning with the same algorithm. But many algorithms are encoded in software only for offline evaluations, as a lot of research dwells on offline. So I think it's a good question!

I think you should certainly email an author of any article that seems really helpful to you, to specifically ask them, if the article doesn't make that 100% clear, and they might even point you in rare cases at a solid online implementation! Do note online usage entails more production-readiness software considerations, and might be an extra mile in terms of the quality of the software expected as such ...


How much features are needed for Reinforcement learning?

It really depends on how much data (samples) you have instead of how many features each samples has. And more importantly it depends on how you plan to structure the problem into an Environment with States and Actions.


What is the difference between CountVectorizer() and Tokenizer() or are they the same?

I thought that they both use one-hot encoding

These are utility to preprocess your text. Like any other utility, it has multiple options to tweak your text. You should explore all the parameters using the official docs.
 I will explain one of these i.e. OHE vs Count
from sklearn.feature_extraction.text import CountVectorizer
corpus = [ 'This movie is bad.Too Bad', 'Awesome Movie. Too Awesome']
vectorizer = CountVectorizer(binary=True) #binary=False will make it Count
x = vectorizer.fit_transform(corpus)

import pandas as pd
df = pd.DataFrame(x.toarray(), columns=vectorizer.get_feature_names())
df

Our end goal is to create Features and each Feature has an indicator for its contextual meaning.

what I don't understand is why CountVectorizer is not used on Deep Learning models such as RNN and Tokenizer() is not used on ML Classifiers such as SVM,

When we are modeling a simple ML algorithm, we generally use scikit-learn.
So there is no point in adding Keras there.
Same is True for Deep learning.
Though, in this case, we have another reason too i.e. Deep Learning generally works on a large dataset. So, we mostly use the idea of embedding on our features. So it's better to use a Framework that provides an end-to-end solution


How is this tensorflow command interpreted?

This question is very much about code syntax of the Keras Input layer (Tensorflow backend)...

shape=(100,) meaning - The shape variable takes as input a tuple of integers, not including the batch size. For instance, shape=(100,) indicates that the expected input will be batches of 100-dimensional vectors.
It knows how to deal with "dtype='int32'" because that is how the Input class was written (I have no better answer). The data type variable takes as input a string (float32, float64, int32...).

The official documentation of the Input layer: tf.keras.layers.Input

This link has all the answers you are looking for.


Balanced Train set to predict Imbalanced Prediction set

It's possible. Consider KNN. When you train a model, it essentially "remembers" the training set. Now, when you use the model to predict an unbalanced data set, the model simply read off from the memory, so there's no problem.

For example, imagine you train a model with an uniformly distributed age group from 10 to 80. Now you have a test set where everybody is 70-80 years old. KNN simply find out the neighbors for your old people (say 60-80 old people in your training data) and then compute a weighted average.

Using a balance training set to predict a imbalanced test set is not super challenging. The other way around is much more challenging.


If you have enough data to train a machine learning model, I believe LSTMs are what you need. Long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in the field of deep learning.

LSTM networks are well-suited to classifying, processing and making predictions based on time series data, since there can be lags of unknown duration between important events in a time series.

Initially, you can use Pandas LSTM library, you design it from scratch by using Python Tensorflow library. Look at the following links:

Wikipedia

LSTM in python tutorial


high precision and recall but low cross validated accuracy

First of all, you should use cross_val_predict to get you predictions vector, so that you followed approximately the same validation scheme to get them :

Y_pred = cross_val_predict(clf, Y, cv=5)

Then, it is totally possible to have 1 for recall or precision given that these are the scores for the considered positive class for scikit. In fact scikit assigns by default one of your class as the 'positive' one and to metrics inherent to classes are computed according to it.

However, as you said you can't have 1 for both and 70% accuracy.

The fact is that your clf.predict gave you 100% accuracy while during cross validation it was not the case because in your second scheme:

clf.fit(X, Y.values.ravel()) and Y_pred = clf.predict(X)

you train and predict on the same data ! So you obtain 100% accuracy which is common, while during cross validation :

scores = cross_val_score(clf, X, Y.values.ravel(), cv=5, scoring="accuracy")

you trained and tested on different folds which logically will lead to worse results (but closer to real condition experiment).

Besides, try to use cross_val_predict and then precision_recall_fscore_support(Y, Y_pred) in order to have a detailed report of metrics for each class.


How to generate a normal distribution with variance $\sigma^2= 0.08$

To generate normal random variable, you can use numpy.random.normal where the scale parameter is the standard deviation. To generate multivariate normal random variable, you can use numpy.random.multivariate_normal.

If you are an R user, you might like to consider mvrnorm

However, I doubt that is what you really want. After all, it is possible to generate points that are beyond the square.

You might like to consider some bounded distribution instead. Consider truncated normal distribution or uniform distribution.


Which decision tree algorithm does H2O use?

I have heard H20 Driverless AI uses an ensemble of xgboost and lightgbm models. In some situations it uses tensorflow neural nets as well.


Convolutional Neural Networks layer sizes

Actually I guess you are making mistake about the second part. The point is that in CNNs, convolution operation is done over volume. Suppose the input image is in three channels and the next layer has 5 kernels, consequently the next layer will have five feature maps but the convolution operation consists of convolution over volume which has this property: each kernel will have its width and height, moreover, a depth. its depth is equal to the number of feature maps, here channels of the image, of the previous layer. Take a look at here.


df['date'] = pd.to_datetime(df['created_at']).dt.date

to_datetime: Convert argument to datetime. For example, if your column of "created_at" is a string column, it converts it to a datetime column

dt: Access object for datetime like properties of the Series values.


Difference between Factorization Machines and PCA?

The main difference is that PCA is a dimensionality reduction technique, while Factorization Machines are classifiers. You can use PCA to simplify/compress a given dataset, while you can use FMs to classify your observations.

The other difference is that PCA is a linear technique, while FMs are non-linear. PCA extracts later factors that are linearly associated with your input variables, while FMs, on the other side, can "learn" non-linear patters in your dataset.


feature importance via random forest and linear regression are different

So your query is a comparison of linear regression vs. random forest's model-derived importance of variables.

The lasso finds linear regression model coefficients by applying regularization. A popular approach to rank a variable's importance in a linear regression model is to decompose $R^2$ into contributions attributed to each variable. But variable importance is not straightforward in linear regression due to correlations between variables. Refer to the document describing the PMD method (Feldman, 2005) in the references below.

Another popular approach is averaging over orderings (LMG, 1980). The LMG works like this:

Find the semi-partial correlation of each predictor in the model, e.g. for variable a we have: $SS_a/SS_{total}$. It implies how much would $R^2$ increase if variable $a$ were added to the model.
Calculate this value for each variable for each order in which the variable gets introduced into the model, i.e. {$a,b,c$} ; {$b,a,c$} ; {$b,c,a$}
Find the average of the semi-partial correlations for each of these orders. This is the average over orderings.

The random forest algorithm fits multiple trees, each tree in the forest is built by randomly selecting different features from the dataset. The nodes of each tree are built up by choosing and splitting to achieve maximum variance reduction. While predicting on the test dataset, the individual trees output is averaged to obtain the final output. Each variable is permuted among all trees and the difference in out of sample error of before and after permutation is calculated. The variables with highest difference are considered most important, and ones with lower values are less important.

The method by which the model is fit on the training data is very different for a linear regression model as compared to random forest model.  But both models don't contain any structural relationships between the variables.

Regarding your query about non-linearity of the dependent variable: The lasso is essentially a linear model which will not be able to give good predictions for an underlying non-linear processes, as compared to tree based models. You should be able to check this by verifying the models performance over a set-aside test set, if the random forest performs better, the underlying process may be non-linear.  Alternatively, you could include variable interaction effects and higher order variables created using a, b, and c in the lasso model and verify if this model performs better as compared to a lasso with only a linear combination of a, b and c. If it does, then the underlying process might be non-linear.

References:

https://en.wikipedia.org/wiki/Decision_tree_learning
https://en.wikipedia.org/wiki/Random_forest
Relative importance of Linear Regressors in R
Relative Importance and Value, Barry Feldman (PMD method)


Proper way to make Train/test split on Time-Series

The problem here is that you're shuffling the time-series before splitting it.
This way, every time-step in the test set might have a time-step close to it in the train set.
To avoid this, you can set shuffle=False in train_test_split (so that the train set is before the test set), or use Group K-Fold with the date as the group (so whole days are either in the train or test set).
You can read more in this question in Cross Validated


Python Code to find the number of hapax legomena in a Text or Words_List

You can use nltk's freq dist, there is a built in method there for that.
from nltk.probability import FreqDist
from nltk import Text

self.text = Text(self.tokens)
self.fdist = FreqDist(self.text)
hapaxes = fdist1.hapaxes()

if by any chance you cannot use the library, feel free to manually calculate it:
text = 'your text ... '
local_hapax = list(set(text.split(' ')))
weighted_local_hapax = local_hapax/len(text)


I'll try and answer your question point-by-point from an architect's point-of-view:

  We would like to implement rule engine in one of our products, but
  when I read on the Internet, many are suggesting that rule engines are
  an old way of doing things

The 'people of the internet' don't know your scale and the use-case. Only you know. So, they might be suggesting ideas from a very high-scale point of view, when your system is not necessarily a high scale one. So, pick the right idea/tool for the job.

  suggesting that rule engines are an old way of doing things and it
  will be very hard to manage as we add more rules into it

True. Rule-based systems are not scalable, due to obvious reasons. And, that's why people consider ML as a highly scaled out rule-based engine, wherein the rules can no longer be understood by humans. Thus, the black-box name.

  So my Question is, should I learn Machine Learning or go on
  implementing Drools as rule engine in my application.

That is for you to decide. We don't know your scale. But, if you think the rules would be increasing at a huge rate in the future, then ML is the best way forward for your use-case.

  Or can you suggest any online resources achieving the same using
  Machine Learning?

For me, every recommender system is a complex rule-based engine under the hood. So, go through Amazon's and how they do their recommendation.


How do I crop faces with a neural network?

You are looking for a fully convolution neural network trained for image segmentation. The architecture is typically convolutions followed by de-convolutional layers with out pooling.

The forward inference serves to extract information with convolutions. The de-convolutions serve to resize the tensors to the size of the inputs height and width. A fully connected layer then classifies each pixel from the input. Deconvolutions demonstration.

Unfortunately you need pixel wise labels to train a network to do this. You could look for a pretrained model that already segments faces from images or you could grind out hand labeling for a 1000 images and use transfer learning (there are online tools for doing this such as Label box.


Word2Vec / Doc2Vec - map change of words overtime

Yes, you can check Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change


How to interpret statsmodel output - logit?

Very brief answer (no way to go into details here):

Aparently the two calls produce different tables containing slightly different statistics.
AIC and BIC compare nested models. So if you have some model and you add or remove some variables (for instance), you may compare AIC, BIC. There is no universal "okay" range in terms of overall figures. Even with a low(er) AIC, BIC, you can have a "bad" model. So AIC, BIC really is about comparing "similar" models against each other.
There are robust standard errors, which are computed in a different way than "normal" standard errors. I think this indicates that "normal" standard errors are calculated.
Not really. The pseudo R-squ. may give you some idea about model fit, but it is a little different from "normal" R-squared and I don't find it too useful.
The value of the coefficient is not directly related to significance (small values can be significant and vice versa). Significance is calculated from standard errors/t-/z-statistics. Note that your coefficients are log-odds (NOT marginal effects). In case you want to obtain marginal effects, you need to look for some package (like "margins" in R/Stata) or you do this by hand.

Overall I recommend to have a good read about logistic regression since you seem to be uncertain about basic concepts.


High-dimensional data: What are useful techniques to know?

This is very broad question, which I think it's impossible to cover comprehensively in a single answer. Therefore, I think that it would be more beneficial to provide some pointers to relevant answers and/or resources. This is exactly what I will do by providing the following information and thoughts of mine.

First of all, I should mention the excellent and comprehensive tutorial on dimensionality reduction by Burges (2009) from Microsoft Research. He touches on high-dimensional aspects of data frequently throughout the monograph. This work, referring to dimensionality reduction as dimension reduction, presents a theoretical introduction into the problem, suggests a taxonomy of dimensionality reduction methods, consisting of projective methods and manifold modeling methods, as well as provides an overview of multiple methods in each category.

The "projective pursuit" methods reviewed include independent component analysis (ICA), principal component analysis (PCA) and its variations, such as kernel PCA and probabilistic PCA, canonical correlation analysis (CCA) and its kernel CCA variation, linear discriminant analysis (LDA), kernel dimension reduction (KDR) and some others. The manifold methods reviewed include multidimensional scaling (MDS) and its landmark MDS variation, Isomap, Locally Linear Embedding and graphical methods, such as Laplacian eigenmaps and spectral clustering. I'm listing the most of the reviewed methods here in case, if the original publication is inaccessible for you, either online (link above), or offline (References).

There is a caveat for the term "comprehensive" that I've applied to the above-mentioned work. While it is indeed rather comprehensive, this is relative, as some of the approaches to dimensionality reduction are not discussed in the monograph, in particular, the ones, focused on unobservable (latent) variables. Some of them are mentioned, though, with references to another source - a book on dimensionality reduction.

Now, I will briefly cover several narrower aspects of the topic in question by referring to my relevant or related answers. In regard to nearest neighbors (NN)-type approaches to high-dimensional data, please see my answers here (I especially recommend to check the paper #4 in my list). One of the effects of the curse of dimensionality is that high-dimensional data is frequently sparse. Considering this fact, I believe that my relevant answers here and here on regression and PCA for sparse and high-dimensional data might be helpful.

References

Burges, C. J. C. (2010). Dimension reduction: A guided tour. Foundations and Trends® in Machine Learning, 2(4), 275-365. doi:10.1561/2200000002


Why imbalanced data-set will bias the prediction model towards the more common class?

Let us say you have a skewed class (which is linearly inseparable, otherwise the prediction is bound to be correct always) like this:

Image Credit: Learning from Imbalanced Classes

Now you can clearly see, if you move the boundary downwards you will be miss-classifying much more blue dots than red dots. And we know the cost function of a ML algorithm roughly tries to reduce the miss-classification (by converting it into a real valued loss optimisation problem, instead of discrete mis-classification reduction).

Think of it as democracy, each individual point tells the classifier what to do. The classifier will always listen to majority points. In this case the classifier will be miss-classifying much more blue points rather than red points if moved downwards. So the bias is set to high, so the line moves upwards.

Mathematically speaking: A classifier will always try to minimise its loss function, which roughly corresponds to making correct classifications (making correct classifications is discrete, optimising loss is not). So, it will position the boundary as such to make minimum number of mistakes disregarding class type. In general if a class has more points in the data-set, a classification boundary will tend to to make more miss-classifications of that class compared to class with less points. Hence optimising the class with higher points becomes more of a priority for th classifier, and it sets its bias accordingly.


Cluster algorithm which minimizes a distance while fulfilling a constraint

I found this article on "Clustering with Constraints" to be a helpful resource to what you're talking about:

Credit:
Clustering with Constraints
Incorporating Prior Knowledge
into Clustering
Adapted from a Tutorial of Sugato Basu and Ian Davidson (SDM 2005)


Why this single layer network does'nt work

I can't find the code in the link you posted to get more background information on what kind of data source it is that you are using.

However, a data source that is $60\times60\times3$ is described as being high dimensional. You have a total of 10,800 input features which need to be mapped down to 17 different classes. This is actually a very complex task and will not be successful using a single layer network. Such a simple network will not have enough parameters to capture the non-linearities between features in your input space.

That being said, if you insist on using a single layer network with a data of size $60\times60\times3$ with 17 input classes the code is as follows.

Let's first create some artificial data of the same dimension as your data

import numpy as np

n = 1000
x_train = np.zeros((n,64,64,3))
y_train = np.zeros((n,))
for i in range(n):
    x_train[i,:,:,:] = np.random.random((64,64,3))
    y_train[i] = np.random.randint(0,17)

x_train = x_train.reshape(n,64,64,3,)

n = 100
x_test = np.zeros((n,64,64,3))
y_test = np.zeros((n,))
for i in range(n):
    x_test[i,:,:,:] = np.random.random((64,64,3))
    y_test[i] = np.random.randint(0,17)

x_test = x_test.reshape(n,64,64,3,)

print('Training data: ', x_train.shape)
print('Training labels: ', y_train.shape)
print('Testing data: ', x_test.shape)
print('Testing labels: ', y_test.shape)

  (1000, 64, 64, 3)
  (1000,)
  (100, 64, 64, 3)
  (100,)

For a classification task we should convert our outputs to categorical vectors. Where we use one-hot encoding to identify the correct class.

import keras

# The known number of output classes.
num_classes = 17

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

We then build the model.

from __future__ import print_function
import keras
from keras.models import Sequential
from keras.layers import Dense, Flatten
from keras.models import model_from_json
from keras import backend as K

input_shape = (64,64,3,)

model = Sequential()
model.add(Flatten(input_shape=input_shape))
model.add(Dense(17, activation='softmax'))

model.compile(loss=keras.losses.mean_squared_error,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

You can see the summary of the model by using

model.summary()

Then we can train this model using

batch_size = 128
epochs = 10
model.fit(x_train, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test_binary))

This code works. However, the data is completely random thus the model cannot learn anything. However, even if your data is easily distinguishable, as I said above, I do not expect this model to be complex enough to distinguish such a large input space from 17 different possible classes.

If you post the data source you are using we can design a model to get a good result.


Building a content-based music recommendation system

If the features are identical, good start would be to use n-neighbors approach.

It would be something like that

from sklearn.neighbors import NearestNeighbors
all_songs_features = [[0, 0, 2], [1, 0, 0], [0, 0, 1], [100, 100, 100], [0, 0, 1.5]]
neigh = NearestNeighbors()
neigh.fit(all_songs_features)

my_song_features = [[0, 0, 1.3], [1.1, 0, 0]]
print(neigh.kneighbors(my_song_features, 2, return_distance=False))

These code returns the indexes of similar songs from the bigger dataset.

Note, if the features are with different range (i.e. one with values 0-1 and another 10000-1000000), then you need to scale them, for example with StandardScaler


Not sure if your model is too simple, I have used the same number of layers before and had very good results. Guess it depends on many factors, such as the training image dataset, etc. The high accuracy is coming from learning the training data quickly, but the low accuracy means the model you built doesn't apply to the test data, so the model is over-fitting. Some possible causes:

1) not enough training data

2) training images are too similar to each other, but different from the test data

3) model is too complex and does not generalize well
4) etc

Possible fixes:

1) If your training data is too small, you can use public pre-built models (VGGnn, etc) as a starting point. This will work best on models that were built with similar data.

2) If your images are too similar to each other, you can alleviate this by using image augmentation. The keras.preprocessing.image.ImageDataGenerator library is good for this.
Here is an example on how to use:

https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

3) If the model is too complex, you can try to simplify by changing filter sizes, or add regularization


My first machine learning experiment , model not converging , tips?

You may try Stochastic Gradient Descent optimizer with a learning rate decay and nesterov momentum. You can also try a different batch_size. Also you are missing drop out layers between the fully connected layers which the authors used.

Try

...
# flatten the conv layers
model.add(Flatten())
model.add(Dropout(0.5))

# fully connected 512
model.add(Dense(output_dim=512))
model.add(Activation("relu"))
model.add(Dropout(0.5))

# fully connected output layer
model.add(Dense(output_dim=2))
model.add(Activation("softmax"))

# compile
model.compile(loss='categorical_crossentropy',
              optimizer=SGD(lr=10e-4,
                            decay=10e-6,
                            momentum=0.99,
                            nesterov=True),
              metrics=['accuracy'])

# train
model.fit(x_train, y_train,
          nb_epoch=70,
          batch_size=64,
          verbose=1)

In my experience this usually helps a lot. If you get this setting to converge, you may try rmsprop and adam.


Does self driving technology gain more from the data or the state-of-the-art algorithm?

I'll give a high level answer.

Our Bias on Effort

Although self-driving technology is now on the verge of becoming mainstream, incredible effort has gone into its development. The DARPA Grand Challenge, which helped in this technology's advancement, started in 2004. Imagine the total effort that went into this technology over 13-14 years. Many labs were setup with sole aim of competing in this challenge.

So when you say that we can't replicate the success in our businesses directly, it is better to ask if the comparison is fair. In your business, have you worked on a single problem to such an extent that many bright minds have worked on it for many years? If not, the comparison is not fair (to your business!).

Self-driving cars are not overnight success due to a single algorithm.

No Single Secret Sauce

Media reporting makes us believe that there's one algorithm that rules the technology, reality is more complex. Check this review paper on the complexity of tasks. There are many modules involved (sensors, perception, planner and control module, feedback loops). Yes, probability and machine learning plays a large part in all modules, but there's no single oracle algorithm. Old workhorse algorithms like Kalman Filters also shoulder heavy workload, but wouldn't get mentioned in the deep learning hype.

Yes large data helps, reinforcement learning helps. But so do rules inferred from manual interventions and control theory and older algorithms.

Summary

Many areas from engineering (classic control theory, reinforcement learning, machine learning) form the basis of self driving cars. We can't copy over the success to other domain easily: there's no single secret sauce but tons of hard work and solid engineering over long time.


If in t-SNE digaram of binary classification both classes follow the similar curve what does t-SNE diagram show?

To have the intuition, let's make simplify the problem. Let's say that you are using PCA and that you are getting a linear function.

If you don´t get any cluster it means that it is not possible to separate them by linear combinations of the features that you have. And since it is a linear function, as you move in this direction there is a change in the variance.

t-SNE is similar to PCA but it does non linear combinatios of features. And the dimensionality reduction that you have mantains the variance in a non linear way.
So just change the world linear for non-linear in the second paragraph to answer your question.

By the way, this doesn't mean that if you make a classification problem with Machine Learning you are going to have bad results. ML can still work in this case.


Can I arbitrarily eliminate 20% of my training data if doing so significantly improves model accuracy?

One flaw in your procedure is the use of SMOTE before splitting in train/test. This should be avoided as you may have synthetic examples in the test data which generation depends on training data and that will be highly close to this data in your feature space (as SMOTE uses Euclidean distance).

Moreover, if most of the minority data belongs to the not-skewed region of your specific variables, these points will be also over sampled and so this reduction in the variables space will produce an overly optimistic performance which does not reflect the real distribution of the data.


Deep Learning: Does starting the training on a smaller subset of the data make sense?

It makes the most sense to train a deep learning model on the entire dataset. If you train on a subset of training, you are more likely to end up a local minimum.
One option is to adjust the size of the batch for stochastic gradient descent (SGD). The batch size is the number of data points given to the model before performing a learning update.


Extracting specific portions of text from poorly formatted document?

If I understand the problem, you have text that is pre-sliced into some unit of interest (your chapter), that requires pre-processing to convert into a form suitable for further work. You've done some of this pre-processing by hand and want to use it as a training set for a model to infer the rules that should be applied to accomplish the same result on raw text.

Because I don't know what you intend to do with the scrubbed text, I'm going to assume that it's destined for natural language processing to extract some semantic content. Here's what you'd normally do.

Convert unicode encoding to UTF8 to get rid of the diacritical marks and other characters that aren't part of your target language
Tokenize the text into a list of strings of words and punctuation marks
Strip (or optionally convert to words if short) numbers
Strip excess white space
Optionally strip stop words (such as the, of, them)
Lemminize (remove inflections, such as by changing plurals to singular)

This gets you to the point where you're ready to do linguistic analysis (which, as I say, I only assume is your goal).

Fortunately, there are packages, such as NLTK in Python that do this for you without the need for regex.

Writing a model to infer that from a training set of clean text might not be the best way to go, if getting to clean is your goal, rather than extending the frontiers of machine learning.

Could you post a MWE (Minimal Working Example) of a chunk of raw text and it's corresponding manually scrubbed version. It may turn out to be a quicker means to your end to apply a combination of off-the-shelf tools to get to where you want to be.


When to use GAN over conventional sampling methods?

Here are the Options:

Oversampling- sure, there are some possibly good ones like SMOTE etc. Just apply it after Train test split to avoid leakage.

Undersampling - reducing the 30000 to a certain number where what is left is representative of the Information you Need to classify this class. You could, for example, apply some unsupervised learning to see which clusters inside this 30k are available and then only sample from These clusters until you have 300 examples. Or apply other undersampling techniques.

GANS- finally even tough really powerfull in certain Scenarios they are also VERY expensive. I would advise you to try GANS as your final Resort since it will take time for the Network to generate good examples.

CONCLUSION: Maybe you expected a decisive yes or no for GANS, but the truth About it is, its an Experiment. It might work, and it might not. Just like there are situations where NN are terrible.
After update:
Theoretically you can apply most of them without any labels, just mark These 300 Points as one label and see what you get. Ofcourse without a clean Goal in mind you could justify any Output as reasonable.
SMOTE has many variants. SMOTE should be treated as a conservative density estimation of the data, which makes the conservative assumption that the line segments between close neighbors of some class belong to the same class. Sampling from this rough, conservative density estimation absolutely makes sense, but does not work necessarily, depending on the distribution of the data.
There are more advanced variants of SMOTE carrying out more proper density estimation. Here is a repo with a lot of smote variations .
Here is also git repo for tabular GANS data Augmentation, this should be easy to consume


Linear Regression bad results after log transformation

Make sure you transformed back your predictions and actual values before calculating MAPE.
You can check which observations contributed the most to high MAPE.
MAPE is very sensitive to prediction errors at small actual values. Most likely worst performing observations ("from MAPE perspective") are those with small actual values.
Depending on the goal of your analysis you could check other metrics as well (eg: MAE).
Sparsity: Yes, you might have a neighborhood category in your test set, that does not exist in your training set (or has only a few examples). In this case predictions for that category might be very bad. Though this does not explain why you don't have high MAPE when you don't transform area.


How to split train/test datasets having equal classes proportion

If what you want to do is keep the same proportions across the splits, what you are doing is right.
In order to validate properly your model, the class distribution should be constant along with the different splits (train, validation, test).
In the train test split documentation , you can find the argument:

stratifyarray-like, default=None
If not None, data is split in a stratified fashion, using this as the class labels.

One step beyond will be using Stratified K-Folds cross-validator.

This cross-validation object is a variation of KFold that returns stratified folds. The folds are made by preserving the percentage of samples for each class.

There are more splitting techniques in scikit learn that you can use, have a look.
In order to check what is the function doing just calculate the percentages in the splits:
 np.unique(y_train, return_counts=True)
 np.unique(y_val, return_counts=True)

But this will make you have the same proportions across the whole data, if your original label proportion is 1/5, then you will have 1/5 in train and 1/5 in test
If what you want is have the same proportion of classes 50% - 0 and 50% - 1. Then there is two techniques oversampling and undersampling. But I wont recommend you this for your problem. Your label seems fairly good balanced 1/5 its a great proportion.
Still this library has some implementations to do that
In this blog you can see an overview of imbalanced datasets but yours is not. Choosing a proper metric is more important.


How do convolutional layers get connected to each other?

There is more than one way to do this in principle, but most CNNs, and most CNN libraries will do the following:

Each layer has a target number of feature maps, which is what you have labelled nb_filters.
Each feature map is derived from all previous layer's feature maps.
The weights of the locally connected kernels do follow a width x height x nb_input_features x nb_output_features structure similar to your first example (with width and height being the convolutional kernel size), because each output feature in layer b is derived as the sum of convolutions over every feature in layer a.

So if your output feature map with (channel) index $j$ in layer $k+1$ is $I^{(j, k+1)}$, and you are looking at connection to all input feature maps $I^{(i, k)}_i$ in layer $k$ with filters connecting each map $F^{(i,j,k)}$, then the maths looks a bit like this:

$$I^{(j,k+1)} = f(b^{(j,k)} + \sum_i I^{(i,k)} \ast F^{(i,j,k)})$$

. . .where $f(x)$ is the transfer or activation function e.g. ReLU. The bias is $b^{(j,k)}$ and the $\ast$ represents the convolution function. Summing over $i$ means iterating over all the input channels. NB I have indexed the items all with a superscript like this to more clearly show that all the variables (apart from $b$) are matrices after indexing - i.e. this is not the equation for a specific pixel, but for a whole feature map.

You can see that the kernel $F$ is indexed by $(i,j,k)$ i.e. there are nb_input_features times nb_output_features filters connecting each layer.

In comparison, your expected scheme would keep the derived features separate layer-by-layer, and create an exponential growth in number of feature maps as the layers increased, with each feature "type" being handled separately. Whereas what happens in practice is each feature map is a "re-mix" of all the feature maps in the previous layer, convolved each with its own filter then summed up.

This cross-referencing of features between layers helps create richer complex feature maps - e.g. it can combine edge and corner detectors to detect corners with specific angles. Whilst a scheme that refined each feature into multiple separate sub-features would not be able to do this mixing between features as freely (until the later fully-connected layers) - although it might do well at other tasks - e.g. detecting variations of extended structures of the same type longer lines, curves etc (NB this is just a guess on my part, I don't know for certain).

  If it's not working that way, how do I know what filters are applied to what activation maps?

You cannot say something like "this feature map in layer N is derived only from this other feature map in layer N-1, using this single filter", because each feature map in layer N is derived from all feature maps in layer N-1 and results of the filters are combined in a way that you cannot easily reverse.

However, the weights array will be arranged so that you can find which filters are connecting each input feature map to a specific output feature map. Which dimension you would use to isolate those depends on the library. If your library stores weights in a [kernel_width, kernel_height, input_feature_map_id, output_feature_map_id] form, then your selection (using Numpy-style syntax might be) [:,:,:,n] where n is the id of the output feature map that you want to fetch the filters for.


Is there any way how to make samples balanced?

SMOTE is a python library popularly used with unbalanced datasets like yours - it applies resampling to create a new balanced dataset. You can find some example implementation here.

Maybe you will also find this question and answers relevant, as it contains advice on limiting bias during modeling and validation of unbalanced data classifier.


Image to image translation from image to sketch using GANs, do I need to annotated the face at all?

Transferring an original image to a different style of image is called style transfer. Although you can use a GAN for such a case, I am not convinced this is the best approach. Furthermore, there have been many papers recently published suggesting very interesting ways for which to train models for style transfer. An amazing implementation of a file transfer algorithm which can convert video in real time can be found here.

There is another method that I have come across more recently which uses the same approach but a different loss function. The paper can be found here.

The underlining method

These methods use a combination of two networks. The first network acts as a transformation function $f_W$ which maps your image $x$ to a transformed image $\hat{y}$. The purpose is to train this network such that the output $\hat{y}$ is the original input image in the new style, in your case as line art. To train this network we will use the second network on the right side which learns how to mix in the Style Target and the Content Target. We want the transformed image $\hat{y}$ is still similar in content, but we want the transformed image to have a similar style as the target.

To train this network we will use a massive amount of input images using some online database source. This training process will be very long. However, once it is complete then we can strip the network on the right side and you will only need the network which approximates $f_W$. You will then be able to put a new image into the input of this network and the output will be the line art version of the input image.

You will need to tune the extent at which the input image is line artified by tuning the weight on the style loss during the training.


How to speed up optimization using Differential Evolution?

Do you run your analysis algos in batch or live? Which programing language under which environment do you use?

At a first naive look, i would recommend to parallelize your code as each indicator calculation seems to be independent of the other's.


Orange 3 - Same prediction for all of my data when using impute widget

When imputing, certain characteristics of the data have to be recorded. E.g. when you impute missing data by mean value parameter $\mu$, you have to record that $\mu$ from training data in order to be able to reapply the exact same transformation on the test data.

Thus you only need to connect your training data to the Impute widget, while the testing data will be imputed with the same parameters when the two domains are matched (in Predictions).

If you connect testing data to its own Impute widget, it gets imputed with its own, testing-data-specific parameters, which is, as you have discovered, incorrect.

See also: Difference between fit and fit_transform in scikit_learn models?


What is a 1D Convolutional Layer in Deep Learning?

In short, there is nothing special about number of dimensions for convolution. Any dimensionality of convolution could be considered, if it fit a problem.

The number of dimensions is a property of the problem being solved. For example, 1D for audio signals, 2D for images, 3D for movies . . .

Ignoring number of dimensions briefly, the following can be considered strengths of a convolutional neural network (CNN), compared to fully-connected models, when dealing with certain types of data:

The use of shared weights for each location that the convolution processes significantly reduces the number of parameters that need to be learned, compared to the same data processed through fully-connected network.
Shared weights is a form of regularisation.
The structure of a convolutional model makes strong assumptions about local relationships in the data, which when true make it a good fit to the problem.

3.1 Local patterns provide good predictive data (and/or can be usefully combined into more complex predictive patterns in higher layers)

3.2 The types of pattern found in the data can be found in multiple places. Finding the same pattern in a different set of data points is meaningful.

These properties of CNNs are independent of the number of dimensions. One-dimensional CNNs work with patterns in one dimension, and tend to be useful in signal analysis over fixed length signals. They work well for analysis of audio signals, for instance. Also for some natural language processing - although recurrent neural networks, which allow for different sequence lengths, may be a better fit there, especially ones with memory gate arrangements such as LSTM or GRU. Still a CNN can be easier to manage, and you could simply pad the input to be fixed length.


Create recommendation system to recommend products to a customer on any e-commerce website

To answer your 1st question,I am not sure if a recommendation system based on search history has been implemented or not.This approach does sound cool.

Secondly, yes there are various algorithms to extract phrases from text. Phrase extraction and Text Summarization are one of the most important aspects in development of chatbots for conversational AI.
To give you an example, there's a good library for this purpose, called PyTextRank.
Go through this link for better understanding of how this works.

Once you have a ranked summary of phrases for the given user's search history, you will be able to generate the list of item/s , that person is interested in.
This is a good start for building the person's unique feature vector.
After this process, you can incorporate normal recommendation algos like Collaborative Filtering , etc.

Hope this helps :)