Why vSGD-fd optimization algorithm isn't popular?

This is my answer at Cross Validated for the same question asked by the same person:

It's already 2019, still nobody answered this question. I don't understand why vSGD isn't popular either. But I do have some reasons from my own:

not real auto-adapting. Like v-SGD, ADAM is also a so-called auto-adapting algorithm, but that is not true. The learning rate, and window size in v-SGD, the \beta terms in ADAM all need tuning. The new-variants like AMSGrad and NosAdam seem to be more robust though.
too "complex". v-SGD uses a "bprop" term to estimate the Hessian diagonal,
and later there is also a finite-difference version. These are somehow "complex" methods to use in computer science and engineering. The numerical instability and  the inherent inaccuracy can cause a lot of trouble. That may be the reason why Tensorflow and Pytorch developers didn't include v-SGD in their package. And that also caused very few comparisons with v-SGD in the subsequent optimization papers.
speed. If the estimated Hessian is a good approximate, then it surely speeds up. However, first, the local landscape may not be the "noisy quadratic loss" as stated in the paper; second, the estimate is also very rough. Actually, there can be no exact estimate of Hessian diagonal. These factors all make the speed of v-SGD algorithm not so competitive.

That is only some understanding with not a lot of practice. Hope someone could point out probably more accurate points.


Creating a composite score from dataset with no target variable

If you dont have the outcome of the observation, it is not possible to find which variables are more important (important to predict what ? the « what » is the target value).

I encourage you to gather some records (20 times the number of features is a good starting point) with targets so the algorithm can understand what features are relevant to the specific outcome you want.

However it is already possible to calculate correlations between variables and do some data analysis to understand how much variance each feature contains.


how to generate sample dataset for classification problem

I know this isn't answering the question that you actually asked, but I suggest that you NOT generate data for your 'short text' categorization problem.

Generated data can work for certain cases when data scientists who are very familiar with an algorithm want to demonstrate a specific feature, but there is a hokeyness that may lead you astray as someone new to data science and machine learning.

I suggest you avail yourself of some free and open data here, or an entire stack exchange dedicated to open data.

Real data will give you experience with less contrived problems and you can often try out similar ML models on very different data sets in order to gain experience with the variations and pitfalls of data science. Be creative... you might not find exactly what you want, but you can subset the data by projecting out a few text fields and using them to classify another field.

Addendum update:

With your recent edit, it is now apparent that you are seeking to turn your unsupervised training data into supervised training data in order to train a supervised learning classification model.  The method that you suggested, "classify some dataset using regex or lucene rule based matches and manually verify them", is a deterministic unsupervised method without the human verification step.

Without the human verification of the target data that you have created, I would not consider feeding the derived targets back into the classification algortihm as your results will only be as strong as the derived target data and predictions will show similar errors. Instead, you should think about a semi-supervised learning method where you perhaps employ a clustering algorithm and then label the clusters with the target variable.

With human verification of the target data that you have created, this will work just fine as training data for a classification model. The only issue is that this can become tedious. There are mechanical turks (humans paid to perform repetitive tasks) that you can hire to perhaps perform the labeling for you, which may be a more scalable option.

Hope this helps!


By default, most loss functions only return one scalar value. Some will also give you a 'per-sample loss in the batch' value. If you want other loss categories, you can use the add_loss() to add, compute and keep track of different types of losses in different layers.

Refer to https://keras.io/api/losses/
for more details.


How to down-weight non-words in text classification?

Usually, those non-sensical words are not problematic because they appear in one or two documents, and people usually just filter words with such low frequencies.

Are you not able to do this? Or do your non-sensical words appear that much?

Anyhow, your suggestion is what people use:

  outright ignore tokens that aren't discernable by a dictionary. Is there?

Exactly. There is no magic way to know if a word is English or not. What word processors do is to use a dictionary, as you yourself suggested.

In python, before stemming, you could filter based on pyenchant.

import enchant
d = enchant.Dict("en_US")
d.check('hello')  # True
d.check('mmmmmmmm')  # False

I bet this would be good enough.

But, if there are a lot of false negatives, you could ask the dictionary for the most similar words and then apply a word distance measure. Here I will use Levenshtein distance  as implemented in python:

>>> r = d.suggest('helo')
>>> r
['hole', 'help', 'helot', 'hello', 'halo', 'hero', 'hell', 'held', 'helm', 'he lo', 'he-lo', 'heel', 'loathe', 'Helios', 'helicon']
>>> min([LD('helo', w) for w in r])
1

The distance will be higher for the non-sensical words you suggested:

>>> d.suggest('mmmmmmmm')
['mammogram', "Mammon's"]
>>> min([LD('mmmmmmmm', w) for w in d.suggest('mmmmmmmm')])
5
>>> min([LD('aaannnammmmrrr', w) for w in d.suggest('aaannnammmmrrr')])
12
>>> min([LD('hdhhdhhhhhjjj', w) for w in d.suggest('hdhhdhhhhhjjj')])
10

To recap:

filter words that appear in less than 3 documents
if that's not enough or it is not a possibility for your use-case, use a dictionary

I think this last suggestion of mine of using Levenshtein distance to the closest English words is overkill.

If your documents involve a lot of urban words or hip terms, you could also filter based on the number of Google results, instead of using a common dictionary. Your word hdhhdhhhhhjjj, for instance, gives me only two Google results, one of which is this stackexchange. Data science is about being creative. :)

There probably are heuristics or statistics based on the number of consonants in a word, and letter combinations, which you could use to probabilistic shrink your documents' dictionary, but I wouldn't go there. It will be too much work and very brittle.


Make sure to pass the model an "objective" parameter and also use "rmse" for the "eval_metric" parameter.
https://github.com/dmlc/xgboost/blob/master/doc/parameter.rst


What is the effect of oversampling on logistic regression?

The effect will be increasing the intercept. I don't recommend doing oversampling unless any other solution doesn't work. Besides, 10% is not such a big imbalance.
I've been in kaggle competitions with way more imbalance where no imbalance solutions were adopted, logistic regression and random forest work quite well without the need of these.

Edit
After @Ben Reininger input, here's a theoretical justification on how does the intercept change.
Also, a quick experiment showcasing that over-sampling doesn't help improve a metric like AUC, and that it indeed increases the intercept of the logistic regression model.


Time series - is it necessary to retrain the model when new time series data is present

It is good practice to retrain model once in a while. That "in a while" depends on the specific task, of course. This is due to the fact that countless unmeasurable things change continuously.

Think for example about a model to forecast companies' revenues: the rules of the game change continuously, the market today is not like the market one year ago, not to mention five years ago. This is the first example that I thought about, but it's true for pretty much any possible application of ML.

That is why it was observed that models' performance deteriorates in time. For that reason, it is good practice to keep your models updated with fresh data.


how to check the distribution of the training set and testing set are similar

It looks like the person that wrote the blog is combining the samples from the test set and train set into one dataframe and then predicting if each sample comes from the test set or training set (his y variable is called “is_train” which indicates whether the sample came from the training set or not). I think his point is that if you are able to accurately classify whether the sample comes from the test or training set then the predictor variables have different underlying distributions. That would signify that your original model will probably not work well on this test data. Also — he is using AUROC as the performance metric. A high AUROC means that the model is performing well, and in this case it means that there is a big difference in distributions of predictor variables between the training and test set. Ideally, the distribution of the predictors for the training and test set should be the same, so you would want to get an AUROC that is close to 0.5.

I think this situation would only be relevant in cases where you have your model deployed and you need to check if your model is still relevant over time. If you are building a new model you shouldn’t need to do something like this because the test data is randomly sampled from the dataset. Additionally, if you’re doing cross validation then there is even less reason to worry about something like that.


how to use multiple generator in keras fit_generator()

In my opinion, you should build the custom data generator or just use the fit function of Keras and make your input data as follows:

x = [[image1, feature1], [image2, feature1], [image3, feature1]]

y = [class1, class2, class1]


You did not mention which package you are using. If you using scikit-learn,
sklearn.pipeline.FeatureUnion concatenates results of multiple transformer objects.
Something like this:
from sklearn.cluster             import DBSCAN
from sklearn.pipeline            import FeatureUnion, Pipeline
from skearnsklearn.preprocessing import StandardScaler

pipeline = Pipeline([('feats', FeatureUnion([
                        ('doc2vec', d2v_model,        ['sentence1', 'sentence2']),
                        ('numeric', StandardScaler(), ['price'])
                    ])),
                    ('cluster', DBSCAN())
                    ])


How do I determine if variables are correlated? Is it simply a mathematical calculation?

Yes, correlation is a mathematical concept, and it as well known as Pearson correlation. This is probably the one you are obtaining in your analysis.

However, there are other correlation analysis you can try in order to be totally sure about your result. The most famous ones out of Pearson are Kendall correlation and Spearman correlation.

In addition, sometimes relation between variables is less evident than a correlation analysis. For example, if the variables are time-related, one may have a lag over the other. For example, in finances it is known that the GDP is correlated with a lag of some months of the Employment Rate. It is part of a good Data Science knowledge to have a business knowledge, an intuition of why the variables may be related in some way and try to find how to prove that this non evident dependence exists.

Edit: I noticed you put R as a tag. You can study as well libraries polycor and ggm to learn about polychoric correlations and partial correlations, although the may not apply in your case.


Memory error - Hierarchical Dirichlet Process, HDP gensim

you can use an alternative of HDP that is LDA. HDP won't give hierarchical output. HDP and LDA are both creating a flat hierarchy. The only difference is that HDP is generating topics based on topic generated in a pervious iteration. Online LDA is quite memory efficient as well as good at capturing topics.


At the expense of over-simplication, latent features are 'hidden' features to distinguish them from observed features. Latent features are computed from observed features using matrix factorization. An example would be text document analysis. 'words' extracted from the documents are features. If you factorize the data of words you can find 'topics', where 'topic' is a group of words with semantic relevance. Low-rank matrix factorization maps several rows (observed features) to a smaller set of rows (latent features).
To elaborate, the document could have observed features (words) like [sail-boat, schooner, yatch, steamer, cruiser] which would 'factorize' to latent feature (topic) like 'ship' and 'boat'.

[sail-boat, schooner, yatch, steamer, cruiser, ...] -> [ship, boat]

The underlying idea is that latent features are semantically relevant 'aggregates' of observered features. When you have large-scale, high-dimensional, and noisy observered features, it makes sense to build your classifier on latent features.

This is a of course a simplified description to elucidate the concept. You can read the details on Latent Dirichlet Allocation (LDA) or probabilistic Latent Semantic Analysis (pLSA) models for an accurate description.


Detect blur image using ssdmobilenet and tensorflowlite

One idea is to use a shallow (a handful of layers) CNN. Deep CNN's are good at detecting objects in images. Shallow networks focus on lower-level features, such as edges and textures and colors. So, my suggestion is to train a simpler network on a larger training set by training it on image patches rather than on full cards. If the majority of your patches are classifed as blurry, then you classify your whole card as blurry.


Why are optimization algorithms slower at critical points?

In that simulation, the movement speed is a proxy for step size. The step size is a function of learning rate (v) and the approximate gradient of the function at that point ($\frac{\partial L}{\partial w}$). The learning rate can be constant. However, the approximate gradient is not constant. The approximate gradient is typically smaller closer to a critical point (i.e., the valley starts to gradually level out). Thus, the computed update value is smaller and the movement speed slows down.


How to handle missing data data in dependent variable?

The questions you're asking are empirical questions. The only answer anyone can give is to try all of them and see which works better.

You have three options:

Impute data
Throw away data
Use a classifier that can handle missing data, e.g. xgboost. See this answer. xgboost is a powerful classifier. So, if you're not tuning very hard for performance, xgboost is a great way to get a good v0.

Some other points:

The pattern of missing values is important, and can influence the choice of algorithm.
If your dataset is noisy, imputing may only amplify the noise. If you can afford to drop those 2k rows, try that, or train both with and without that data and see if the combination performs better.

Regarding software, there are many options:

Scikit Learn has some imputation functions
MICE, Multiple Imputation through Chained Equations works well for random data. Available in fancyimpute, and also in statsmodels.

You will find many resources if you search


when will the incareful features harm the model?

In very simple words,
Those models which use all the features of the instances will suffer from irrelevant features e.g. Neural Nets, KNN etc.
While the models which have an internal strategy to compares the features to decide the best while training will not suffer(at least for this reason) e.g. Tree
 Here is a snap from "The Elements of Statistical Learning, by Trevor Hastie,
Robert Tibshirani, Jerome Friedman

You can also read this section 10.3 Effect of Irrelevant Features from "Feature Engineering and Selection: A Practical Approach for Predictive Models by Max Kuhn and Kjell Johnson"
You can try these approaches to filters the features -

Using correlation with the target
Using Feature Importance
Using L1 penalty

Also, keep in mind that you might face other issues because of this and the models may suffer indirectly e.g.

High dimensionality
Issue of Trees "default feature importance approach" with high Cardinal features etc.


How do I add a column to dataframe on condition that certain observations in my dataframe contain a target word in Python?

You can use str.contains on the DataFrame:

df = pd.DataFrame({'Age' : pd.Series([35,42,87], index=[1,2,3]),
        'Famous_for': pd.Series(['businessman chairman of IBM (1973–1981)',
                                 'musician (House of Freaks Gutterball)',
                                 'baseball player (Oakland Athletics)'],
        index=[1,2,3])
        })

df['entertaining'] = df['Famous_for'].str.contains('musician')
print (df)

   Age                               Famous_for entertaining
1   35  businessman chairman of IBM (1973–1981)        False
2   42    musician (House of Freaks Gutterball)         True
3   87      baseball player (Oakland Athletics)        False

Note that str.contains also accepts a regular expression, so if you wish to search for various words you might use

df['entertaining'] = df['Famous_for'].str.contains('musician|businessman')


Grouping probabilities of full model vs fitting model on subset of variables

A very thought provoking question.

Surprisingly, second approach (subset) is better in theory. The first one (full) is an unbiased estimator of second one (subset). That is, expectation of average of probabilities in full model is equal to the probability in subset model.

I said "in theory", because we assume that full and subset models estimate the true P(event | full) and P(event | subset) exactly respectively (zero error, perfect generalization, etc.). In practice, choice of learning algorithm and training set affects how accurate they can fulfill this assumption. A model could be Naive Bayes, Logistic Regression, Neural Network with a SoftMax layer, etc.

Proof

Let

$F_f$ denote the full features,
$F_s$ denote the subset features (that we group on),
$F_d=F_f-F_s$ denote the features that we average out,
$\boldsymbol{A}=\{x|F_s(x)=a\}$ denote the set of all possible instances with subset features equal to $a$, and
$A \subset \boldsymbol{A}$ denote the observed subset of $\boldsymbol{A}$ in training set.

Here is the average of probabilities in full model (first approach):
$$\begin{align*}
P_f(e|F_s=a)&=\frac{1}{N_A}\sum_{x\in A}P(e|F_d(x),a)
\end{align*}$$

Expectation of above average is:
$$\begin{align*}
E[P_f(e|F_s=a)]&=\frac{1}{N_A}\sum_{x\in A}E[P(e|F_d(x),a)]\\
&=\frac{1}{N_A} N_A E[P(e|F_f(X))|X \in \boldsymbol{A}]\\
&=\sum_{b}P(e|F_d=b,F_s=a)P(F_d=b|F_s=a)\\
&=\sum_{b}P(e,F_d=b|F_s=a)\\
&=P(e|F_s=a)
\end{align*}$$

where $P(e|F_s=a)$ is the output of second approach.

Example

To verify, here is an example:

$e=\text{has_job}$

$F_f=\{\text{sex}, \text{has_degree}\}$

$F_s=\{\text{sex}\}$

$F_d=\{\text{has_degree}\}$

Full model sees:

x     sex  has_degree  has_job
1     f    1           0
2     f    1           1
3     f    0           0

4     m    0           0
5     m    0           0
6     m    1           1

And builds

sex  has_degree  P(has_job=1|sex, has_degree)
f    0           0
f    1           0.5
m    0           0
m    1           1.0

And answers $P(\text{has_job}=1|\text{sex}=f)$ with

$$\frac{1}{3}\sum_{x \in \{1, 2, 3\}}P(\text{has_job}=1|x)=(0.5+0.5+0)/3=0.33,$$

and answers $P(\text{has_job}=1|\text{sex}=m)$ with

$$\frac{1}{3}\sum_{x \in \{4, 5, 6\}}P(\text{has_job}=1|x)=(0+0+1.0)/3=0.33$$

On the other hand, subset model sees:

x     sex  has_job
1     f    0
2     f    1
3     f    0

4     m    0
5     m    0
6     m    1

And builds

sex  P(has_job=1|sex)
f    0.33
m    0.33

And answers $P(\text{has_job}=1|\text{sex}=f)$ with 0.33, and $P(\text{has_job}=1|\text{sex}=m)$ with 0.33.


AttributeError: 'numpy.ndarray' object has no attribute 'nan_to_num'

You are using the right method but in a wrong way :)
nan_to_num is a method of numpy module, not numpy.ndarray. So instead of calling nan_to_num on you data, call it on numpy module giving your data as a paramter:
import numpy as np
data = np.array([1,2,3,np.nan,np.nan,5])
data_without_nan = np.nan_to_num(data)

prints:
array([1., 2., 3., 0., 0., 5.])

In your example:
import numpy as np
val = np.nan_to_num(setTo.ravel())


Standardization with positive and negatives

You can use Normalization. Normalization rescale your mean to 0 and standard deviation to 1 containing both positive and negative value.
$X_{Normalised} = \frac{X - \mu}{\sigma}$
Here $\mu$ is your original mean and $\sigma$ is your standard deviation.


What kind of statistical analyses can I do with my data?

This is what your post looks like to me:

"I have these data, what can I do with them?"

The answer is "A LOT". What do you want to do exactly? what is the goal?

From the topic, I assume your goal is to find the relations between "Keyword", "Average Position", "Clicks", "Cost", ... with "Impressions". If this is true, then you have multiple predictor variables and a target variable (Impressions) in which you would like to know the relation. This can be done through multiple statistical techniques such as:

Pearson Correlation
Spearman Correlation
Kendall Correlation
Mutual Information
Simple regression (and checking importance of each predictor variable)
RReliefF algorithm etc.


Neural network got a lucky guess. Can it be trusted?

The loss in the curve suggests that the training can be improved by tuning hyperparameters, especially the learning rate and/or the batch size. Therefore the optimal decision would be to keep refining the model instead of using the already trained model.

If tuning hyperparameters is not an option, at least you should re-split training and validation data or at very least repeat with a different random seed.

If none of those is an option, you should take the best validation loss. The reason is that we are assuming that there is no leak of the validation data to the training data and that the validation data is representative of the data the model is going to be tested with. Without any other evidence, we should assume that the "sweet spot" you found would also lead to better results in a piece of test data the model has not seen before.


Does the output of the Sequence-to-Sequence encoder model exist in the same semantic space as the inputs (Word2vec)?

No, assuming your input vectors are one-hot encodings. These input one-hot encodings are in an $n$-dimensional Euclidean vector space. The last hidden layer of an LSTM is not due to the non-linear activation functions across the encoder. Therefore, an average of the inputs will not necessarily align well in a vector space with the model output, nor are you guaranteed any similarity in cosine/Euclidean distance.


How to get vectors of memory cell & the last output of $LSTM$ in keras?

Self answering after reading this article. Actually, the paragraph is saying that $218$ new features are obtained from -

Memory cells or Memory units of 2nd LSTM layer: It will give a vector of $100$ cell states (cell state corresponding to each memory cell). Note that the dimension of the vector is equal to the number of hidden memory cells.
Last output of 2nd  LSTM layer: It will give a list of $100$ cell outputs (cell output of each memory cell/unit).
Final output: Final output would eventually comprise of an $18$ dimensional vector. Dimensions are - $2$ for (a), $6$ for (b), $4$ for (c), $3$ for (d), $2$ for (e), $1$ for (f).

From the same article, it can be achieved in keras using return_state=True parameter.

from keras.models import Model
from keras.layers import Input
from keras.layers import LSTM
from numpy import array
# define model
inputs1 = Input(shape=(3, 1))
lstm1, state_h, state_c = LSTM(2, return_sequences=True, return_state=True)(inputs1)
model = Model(inputs=inputs1, outputs=[lstm1, state_h, state_c])
# define input data
data = array([0.1, 0.2, 0.3]).reshape((1,3,1))
# make and show prediction
print(model.predict(data))

This ouputs,

[array([[[-0.00559822, -0.0127107 ],
    [-0.01547669, -0.03272599],
    [-0.02800457, -0.0555565 ]]], dtype=float32), array([[-0.02800457, -0.0555565 ]], dtype=float32), array([[-0.06466588, -0.12567174]], dtype=float32)]


How can I measure the similarity between 2 IP addresses? Is there any code to re-use?

In Python there is the ipaddress module which can convert an IP address to an integer, then simply subtract to find the difference.

from ipaddress import ip_address
int(ip_address(‘10.0.0.1’))


Pretrained word embeddings vs model weights

Pre-trained word embedding methods such as word2vec, are used in NLP for the initialization of the first layer of a neural network before training for a new task. It is a single layer of weights (known as embeddings) and any prior knowledge from the word embeddings is only present in the first layer of the network, so the entire network would still need to be trained for a new target task.


Using Predictions: Data does not have the same target as predictors

This is because we have to exclude the target column from the second document (testing data-set).

In your example remove the column 'Status' from second file.

Example:

In below example, in training dataset, 'Loan_Status' column is present which is the target field, whereas, in testing dataset, 'Loan_Status' column is not present, and will be computed by the 'Prediction' as shown below:

Prediction:


How to understand backpropagation using derivative

This is a continuation to this answer. The first part of that answer covered gradient descent. In short this algorithm finds a set of parameters which results in a minimum of a function. From the description of the algorithm you can see where the derivative of the function is needed.

How this applies to a neural network

Let's first consider a single neuron network. This is called the perceptron. We will use the sigmoid activation function. It has inputs $x$ and an associated output

$\hat{y} = \frac{1}{1+e^{w^Tx+b}}$.

The model is parametrized by the weights $w$ and the biases. These are the model parameters we need to tune. So this single neuron takes a vector $x$ and using the weights vector and biases will output some value $y$.

If the weights are randomly chosen then the output will also be random. This is of no use to us. We need to determine a way of calculating a measure of correctness or wrongness in order to know how we should change our weights. I will use a measure of how wrong we are and call this the cost

$C = \frac{1}{2N} \sum_{i=0}^{N}(\hat{y} - y)^2$.

This is takes the squared difference between the predicted output $\hat{y}$ and the ground truth $y$. We will take the sum of this wrongness for all the instances that we try to predict, hence the summation. And we divide by $N$ to normalize the error. The additional division by 2 is simply to make the derivative easier to compute you will see soon.

So we want to minimize the cost function using gradient decent. Thus we need to take the derivative of the cost function with respect to the parameters we can tune, these being the weights $w$. We must thus compute $\frac{\partial C}{\partial w}$. Using calculus we can use chain rule in order to simplify our derivative as

$\frac{\partial C}{\partial w} = \frac{\partial C}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial w}$

Our gradient descent calculation thus becomes

$w^{new} = w^{old} - \nu \frac{\partial C}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial w}$

where $\nu$ is the learning rate.

Let's explicitly spell out this derivative

$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y$

and, the second is the derivative of the activation function which we are using which is the sigmoid function.

$\frac{\partial \hat{y}}{\partial w} = \frac{1}{1+exp(w^Tx + b)} (1 - \frac{1}{1+exp(w^Tx + b)})$


Wow, this was bit challenging but I was able to make one of these plots in python.
The two main components are:

plotting multiple radial axes on a polar plot
remapping radial axes for variables with reversed scales

code:

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns # improves plot aesthetics

def _invert(x, limits):
    """inverts a value x on a scale from
    limits[0] to limits[1]"""
    return limits[1] - (x - limits[0])

def _scale_data(data, ranges):
    """scales data[1:] to ranges[0],
    inverts if the scale is reversed"""
    for d, (y1, y2) in zip(data[1:], ranges[1:]):
        assert (y1 <= d <= y2) or (y2 <= d <= y1)
    x1, x2 = ranges[0]
    d = data[0]
    if x1 > x2:
        d = _invert(d, (x1, x2))
        x1, x2 = x2, x1
    sdata = [d]
    for d, (y1, y2) in zip(data[1:], ranges[1:]):
        if y1 > y2:
            d = _invert(d, (y1, y2))
            y1, y2 = y2, y1
        sdata.append((d-y1) / (y2-y1)
                     * (x2 - x1) + x1)
    return sdata

class ComplexRadar():
    def __init__(self, fig, variables, ranges,
                 n_ordinate_levels=6):
        angles = np.arange(0, 360, 360./len(variables))

        axes = [fig.add_axes([0.1,0.1,0.9,0.9],polar=True,
                label = "axes{}".format(i))
                for i in range(len(variables))]
        l, text = axes[0].set_thetagrids(angles,
                                         labels=variables)
        [txt.set_rotation(angle-90) for txt, angle
             in zip(text, angles)]
        for ax in axes[1:]:
            ax.patch.set_visible(False)
            ax.grid("off")
            ax.xaxis.set_visible(False)
        for i, ax in enumerate(axes):
            grid = np.linspace(*ranges[i],
                               num=n_ordinate_levels)
            gridlabel = ["{}".format(round(x,2))
                         for x in grid]
            if ranges[i][0] > ranges[i][1]:
                grid = grid[::-1] # hack to invert grid
                          # gridlabels aren't reversed
            gridlabel[0] = "" # clean up origin
            ax.set_rgrids(grid, labels=gridlabel,
                         angle=angles[i])
            #ax.spines["polar"].set_visible(False)
            ax.set_ylim(*ranges[i])
        # variables for plotting
        self.angle = np.deg2rad(np.r_[angles, angles[0]])
        self.ranges = ranges
        self.ax = axes[0]
    def plot(self, data, *args, **kw):
        sdata = _scale_data(data, self.ranges)
        self.ax.plot(self.angle, np.r_[sdata, sdata[0]], *args, **kw)
    def fill(self, data, *args, **kw):
        sdata = _scale_data(data, self.ranges)
        self.ax.fill(self.angle, np.r_[sdata, sdata[0]], *args, **kw)

# example data
variables = ("Normal Scale", "Inverted Scale", "Inverted 2",
            "Normal Scale 2", "Normal 3", "Normal 4 %", "Inverted 3 %")
data = (1.76, 1.1, 1.2,
        4.4, 3.4, 86.8, 20)
ranges = [(0.1, 2.3), (1.5, 0.3), (1.3, 0.5),
         (1.7, 4.5), (1.5, 3.7), (70, 87), (100, 10)]
# plotting
fig1 = plt.figure(figsize=(6, 6))
radar = ComplexRadar(fig1, variables, ranges)
radar.plot(data)
radar.fill(data, alpha=0.2)
plt.show()


Calculating the Standard Deviation by category using Python

You can easily do this using pandas:

import pandas as pd
import numpy as np

df = pd.DataFrame([["AA", 1], ["AA", 3], ["BB", 3], ["CC", 5], ["BB", 2], ["AA", -1]])
df.columns = ["Category", "Score"]
print df.groupby("Category").apply(np.std)


Pandas dataframes have many many more high level functions integrated right into the base classes that store the data for you.

Some of the commandline tools can be pretty powerful for manipulating text efficiently (Perl in particular), but I would argue that the learning curve is quite steep and the interactive experience is not as friendly. For one thing, it isn't easy to simply get a glimpse of your data or create an attractive plot.

While I admit that I am not a pro awk/sed or Perl user, I am pretty sure it will be a little less intuitive in those tools/languages to do something like this hypothetical computation, which involves numerical data and text:

In [1]: import pandas as pd
In [2]: import numpy as np

# Create a DataFrame holding some data over a time range

In [3]: df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
                                  'foo', 'bar', 'foo', 'foo']*4,
                           'B' : ['one', 'one', 'two', 'three',
                                  'two', 'two', 'one', 'three']*4,
                           'C' : np.random.randn(32)},
                          index=pd.date_range('01.01.2018', periods=32))

In [4]: df.head()
Out[4]:
              A      B         C
2018-01-01  foo    one  0.965554
2018-01-02  bar    one  0.053814
2018-01-03  foo    two  1.075539
2018-01-04  bar  three -0.999941
2018-01-05  foo    two -1.940361

Now imagine we want to group the rows so we have just rows with column A contains foo in one table, and another with just the rows containing foo.

From those two tables, we only care about column C. We want to compute the moving average over a 5 day time-frame. The moving average will leave some NaN values at the beginning, so we want to drop those time-steps.

Oh, and we want to visualise that!

In[5]: df.groupby('A')['C'].rolling(5).mean().dropna().plot(grid=True, legend=True)

From that one line of code, we get this:

The above also highlights the abundance of other powerful and specialised packages avavilable within the Python environment - here I used numpy in conjunction with Pandas.

For manipulating text files, perhaps cleaning up scraped text and parsing large amounts of text using regular expressions, it might be faster to use one of the commandline options, but as soon as you want to do any data science, I would really recommend using some specialised tools, like Pandas.


Dealing with highly variable feature set size

Before thinking of what type of algorithm you could use, I would think of how to properly preprocess your data.
Depending on how many possible values you might have for each of your 8 possible types (if I understood correctly), you could construct a 0's and 1's dataset, it is, indicating the presence or absence of each possible value in each event.
This would lead you to having a sparse matrix, but that is something you can deal with with some tools, a possibly nice example is in this link
Conclusion:

I would first try to identify if there are fixed possible values for each one of your types (by creating groups of ips by region? are there a fixed number of possible .exe, urls...?)
preprocess your data, so that http://google.com and google.com are the same value (i.e., by deleting the http:// from urls)
if you think the number of possible values is not ridiculously big, you might try to go for constructing the sparse matrix

After this, you could think of wich kind of algorithm to apply, and not going crazy for a neural network from the beginning.


Visualizing items frequently purchased together

I think what you probably want is a discrete version of a heat map.  For example, see below.  The red colors indicate the most commonly purchased together, while green cells are never purchased together.

This is actually fairly easy to put together with Pandas DataFrames and matplotlib.

import numpy as np
from pandas import DataFrame
import matplotlib
matplotlib.use('agg') # Write figure to disk instead of displaying (for Windows Subsystem for Linux)
import matplotlib.pyplot as plt

####
# Get data into a data frame
####
data = [
  ['Banana', 'Water', 'Rice'],
  ['Rice', 'Water'],
  ['Bread', 'Banana', 'Juice'],
]

# Convert the input into a 2D dictionary
freqMap = {}
for line in data:
  for item in line:
    if not item in freqMap:
      freqMap[item] = {}

    for other_item in line:
      if not other_item in freqMap:
        freqMap[other_item] = {}

      freqMap[item][other_item] = freqMap[item].get(other_item, 0) + 1
      freqMap[other_item][item] = freqMap[other_item].get(item, 0) + 1

df = DataFrame(freqMap).T.fillna(0)
print (df)

#####
# Create the plot
#####
plt.pcolormesh(df, edgecolors='black')
plt.yticks(np.arange(0.5, len(df.index), 1), df.index)
plt.xticks(np.arange(0.5, len(df.columns), 1), df.columns)
plt.savefig('plot.png')


Handling a combined dataset of numerical and categorical features for Regression

One could approach this in two general ways:

1) bottom up: thinking about unifying the data somehow to begin with

2) top down: deciding how the data needs to look based on the final model you wish to use

Do you already know which model you will use? If that is fixed (for whatever reason), you already know you need to get your data into the correct form, be it numerical or categorical.

As you pinned your question with the tag regression, I can tell you that you need to make your data all numerical, so regression can work.

An example of making numerical data categorical would be to put it into bins. Imagine we have values ranging from zero to ten: [0.173, 7.88, 3.91, ...]. You could simply say that values between 0.00 and 0.99 are category A, values between 1.00 and 1.99 are category B, and so on.

[Edit:]

A slightly more sophisticated way of defining the bins to use would be to define the bins based on some characteristic statistics of your dataset. For example, have a look at the possible ways possible implemented within python's Numpy. Of the available methods there, I have found the Doane method to work best - it will depend on your data though, so read the descriptions.

Making categorical values numerical in a meaningful way depends a little more on you data. It is easy to make them numberic, but you should focus on doing it in such a way as to retain as much of the information each variable contains as well as the relative relationships between each of the categories that you started with. E.g. converting colours into integers would allow you to perform regression, but if yellow becomes 1 and purple 10, the model needs to be able to learn that purple isn't necessarily 10 times bigger than yellow, and that is difficult in the context of regression!


What's the best way to plot a bar graph with large numeric difference in values?

A log scale will make the difference before and after revaluation look less extreme, and a log transformation is usually pretty defensible for things measured in units of currency (especially when there are no 0-value entries). Other transformations exist, but since this is mostly for aesthetics I don't think that you'd need much that a log transformation couldn't offer.

Fiddling with multiple vertical axes and continuity breaks may help, but at an almost-guaranteed cost of making the graph very difficult to view and interpret correctly. If you need to show both periods, I would probably log transform.

You will know your needs far better than I do, but I do question a little bit how valuable seeing the before and after this way is. The revaluation was an arbitrary decision introduced at an arbitrary time, and behavior of the currency before the revaluation is probably not very informative of anything that happened afterwards (and vice-versa). It's far from clear to me that there is much value in showing both at once like this, at least from my naïve look at a single graph.

The main reason I can think of to show both at once this way is to display how extreme the inflation was, in which case the scale differences would be a feature and not a bug.


Convolution and Cross Correlation in CNN

A neural network takes every input into a neuron and then through an activation function the neuron produces an output. When applying this method to images we can quickly see that this is not an ideal solution. The similarity of a pixel is much more related to a neighboring pixel.

The convolution operation in deep learning was used for this exact purpose. It is better to focus on the neighborhood of inputs before considering the correlation of that pixel with those on the other side of the image. So we can instead apply a mask that will mix the neighborhood of pixels.

Before we get into some theory, it is important to note that in CNNs although we call it a convolution, it is actually cross-correlation. It is a technicality, but in a CNN we do not flip the filter as is required in typical convolutions. However except for this flip, both operations are identical.

Discrete convolutions

From the wikipedia page the convolution is described as

$(f * g)[n] = \sum_{m=-\inf}^{\inf} f[m]g[n-m]$

For example assuming $a$ is the function $f$ and $b$ is the convolution function $g$,

To solve this we can use the equation first we flip the function $b$ vertically, due to the $-m$ that appears in the equation. Then we will calculate the summation for each value of $n$. Whilst changing $n$, the original function does not move, however the convolution function is shifted accordingly. Starting at $n=0$,

$c[0] = \sum_m a[m]b[-m] = 0 * 0.25 + 0 * 0.5 + 1 * 1 + 0.5 * 0 + 1 * 0 + 1 * 0 = 1$

$c[1] = \sum_m a[m]b[-m] = 0 * 0.25 + 1 * 0.5 + 0.5 * 1 + 1 * 0 + 1 * 0 = 1$

$c[2] = \sum_m a[m]b[-m] = 1 * 0.25 + 0.5 * 0.5 + 1 * 1 + 1 * 0 + 1 * 0 = 1.5$

$c[3] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0.25 + 1 * 0.5 + 1 * 1 = 1.625$

$c[4] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0 + 1 * 0.25 + 1 * 0.5 + 0 * 1 = 0.75$

$c[5] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0 + 1 * 0 + 1 * 0.25 + 0 * 0.5 * 0 * 1 = 0.25$

As you can see that is exactly what we get on the plot $c[n]$. So we shifted around the function $b[n]$ over the function $a[n]$.

2D Discrete Convolution

For example, if we have the matrix in green

with the convolution filter

Then the resulting operation is a element-wise multiplication and addition of the terms as shown below. Very much like the wikipedia page shows, this kernel (orange matrix) $g$ is shifted across the entire function (green matrix) $f$.

You will notice that there is no flip of the kernel $g$ like we did for the explicit computation of the convolution above. This is a matter of notation. This should be called cross-correlation, it is not a true convolution. However, computationally this difference does not affect the performance of the algorithm because the kernel is being trained such that its weights are best suited for the operation, thus adding the flip operation would simply make the algorithm learn the weights in different cells of the kernel to accommodate the flip. So we can omit the flip.


If you know that the multiple prices on a single day are chronological (and you don't have the actual timestamps), I would suggest simply taking the last price. This is then as close as you can get to using the Closing Price, which is the most common. Often you have Open, High, Low, Close, for example, but just use the Close prices.

Interpolating, or imputing, the data can be done in many ways. One factor that might help decide on a method, is how many of the points are missing? You have ~500 data points: how many days in total? how many of those require imputing?

For a relatively low number of missing values, scattered more or less randomly over your time-series, I would suggest going for simpler imputation methods, rather than model-based one.

A model-based approach can introduce other biases/problems that would potentially be harder to debug and more difficult to understand, so I would hold off there initially.

Forward-filling

This is my go to method, to just get started. If only a few percent of the entire time-series are missing, something as simple as fill-forward might be acceptable. Even the gaps in data are randomly or evenly spaced along the timeline, this is also a reasonable approach.

A nice property of this method is that it agrees with the theory on stochastic price-paths. These are usually modelled as Gaussian random variables, following the principles or Brownian motion. This all culminates in the idea of a Martingale Process, which states:

  at a particular time in the realized sequence, the expectation of the next value in the sequence is equal to the present observed value even given knowledge of all prior observed values.

Random sampling

One way to fill a seemingly stochastic price-path, would be to compute the mean and variance of the dataset, then draw random samples from that distribution to fill missing values. This should preserve the mean and variance of the dataset's distribution quite well (and fairly).

Here is is important to realise that you may inadvertently be inducing a bias within that dataset, namely that you have use population statistics to alter the data. This means, depending on your model testing method later on, that you violate information flow through time. E.g. when predicting the 100th step, using the 99 preceding steps, it could be the case that the 99th step was one you imputed. Using the mean and variance of the entire dataset to do so has inherently encoded some information from the 100th step (and all following steps) into that 99th step. This is a subtle detail, but something to be aware of.

If you can put the time-series in a Pandas DataFrame, it might give you some more ideas by skimming the DataFrame's built-in methods to handle missing data.

Here is a more in-depth analysis of possibilities with some good explanations.


Is an image with 5 labels equivalent to 5 images with 1 label?

No. These are two separate problems. Multi-label classification and multi-class classification. In general, when we talk about classification we mean multi-class classification i.e. there are a certain number of categories and the input training samples fall into only one of these which is your case of 5 images with 1 label. In the case of 1 image with 5 labels, it is a multi-label classification, which is a generalization of multi-class classification. Both of these are different tasks and the use case depends on the problem that you are trying to solve!


Python Seaborn: How can one obtain a box plot (box and whisker plot) for the non-diagonal variable pairs in a pairplot()?

This wouldn't be possible in seaborn without changing the source code.
pairplot is designed with only two options for the non-diagonal plots: scatter or reg.  This can be seen in the source code:
if kind == "scatter":
    plot_kws.setdefault("edgecolor", "white")
    plotter(plt.scatter, **plot_kws)
elif kind == "reg":
    plotter(regplot, **plot_kws)

One option would be copy the source code and create a function similar to pairplot, but with the ability to plot boxplots.


How to prevent a neural network from outputing too clustered probabilities?

That's usually the most ideal case that you get in classification, but if you really want to, you can add a penalty on the predictions to force them to be away from zero or one, e.g. loss'(x) = loss(x) - param * prob_hat(x) * (1 - prob_hat(x)), or some different penalty.


How to split up my dataset in a train and testset, in order to prevent data leakage?

In multi-patient datasets a typical cross validation split is leave-subject-out, check this out: What is difference between leave one subject out and leave one out cross validation
Basically, each subject used as test subjects in different folds.
Your dataset has a few issues. Firstly, depending on your use case there might or not be data leakage. For example, if it is possible to firstly observe enough time series samples of a subject for training, say the first half, then it's ok to use the second half for testing.
Secondly, two of your classes appear on two single subjects. For these two, you can't design an algorithm on based on other subject's data.
Thirdly, and more importantly, most classes don't have enough samples. For example, the eat_soup class has only 3 samples.
In summary, you should use leave-subject-out but on the classes that appear on a substantial number of subjects and there is enough samples in total.


Merging multiple data frames row-wise in PySpark

Stolen from: https://stackoverflow.com/questions/33743978/spark-union-of-multiple-rdds

Outside of chaining unions this is the only way to do it for DataFrames.

from functools import reduce  # For Python 3.x
from pyspark.sql import DataFrame

def unionAll(*dfs):
    return reduce(DataFrame.unionAll, dfs)

unionAll(td2, td3, td4, td5, td6, td7, td8, td9, td10)

What happens is that it takes all the objects that you passed as parameters and reduces them using unionAll (this reduce is from Python, not the Spark reduce although they work similarly) which eventually reduces it to one DataFrame.

If instead of DataFrames they are normal RDDs you can pass a list of them to the union function of your SparkContext

EDIT: For your purpose I propose a different method, since you would have to repeat this whole union 10 times for your different folds for crossvalidation, I would add labels for which fold a row belongs to and just filter your DataFrame for every fold based on the label


Does feature selection removes highly corelated variables?

Lasso is a very common strategy as it is able to handle correlated variables.

https://stats.stackexchange.com/questions/7935/what-are-disadvantages-of-using-the-lasso-for-variable-selection-for-regression


Feature engineering - house price prediction (small dataset)

latitude, longitude could be used for clustering. Then you should set cluster_id to every house as a new feature.

Also it possible to use geo coordinates to calculate the distances to important points (to the city center for example)


Long range forecasting with sequence-to-sequence models

I am wondering why you chose a Neural Network approach in the first place. In general a recurrent Neural Network requires many more parameters to be fitted than for instance a SARIMA model. Therefore, they are more sensitive to spurious artefacts in the train sample. (This is the price you pay for being more flexible in the way the relationships between past and future values are modelled). Given the fact that you mentioned the strong periodicity in your data I think modelling the data using a SARIMA model or even a model with seasonal dummies may generalise much better out of sample, given the sample size that you have. In addition you could add a cyclical trend component to also incorporate other cyclical components in the data. Hope this helps. Both the SARIMA model and the addition of a cyclical component are often described in the econometric literature. Comparing your current model with a SARIMA with: {p=0; d=0; q=0; P=1; D=1; Q=1, s=365 and trend='c'} may be a good place to start (These arguments refer to the statsmodels SARIMA that can be used with Python "statsmodels.tsa.statespace.sarimax.SARIMAX").


Try changing the plot type on Plotly to line:
import plotly.express as px

fig = px.line(df, x="Date", y="Close", title='COMERCIAL INTERNATIONAL BANK')
fig.show()


Treatment and Control selection in A/B Testing

If I understand your question clearly(?), you are describing two different things, 1)AB testing and 2)case-controlled studies.

Think back to Stats class, you did not learn about AB testing BUT what you did learn WAS Hypothesis testing or Null hypothesis testing, Testing the null hypothesis, Hypothesis-alternative testing (whatever) maybe even sometimes called a one-tailed or two-tailed test. Give or take, these are all fairly synonymous concepts. However, like almost all ideas when people leave academia the names change.

One typical experiment is to take 300 stores and sell either A or B. After a day, week, month, you measure the amounts sold then ask, 'Did the two items sell statistically different volumes?'

The second idea I read in your question considers a different way to set up your experiments.

Another type of experiment is called a 'case-controlled study.'

So I want to test people this time. I can't in good conscience NOT give a person with a bad heart his medicine. I can't separate two groups of old folk and see who dies or not. So the tests are retrospective. I go to the hospital and check for all the patients that A)did or B)did not take the medication over the past x years.

For example, I find a 64 yr old male with heart problems who takes blood thinners. Then I look for a male approx. 64 yrs old with heart problems that did not take blood thinners then look at liver functions for both and test difference, let's say.  I look for similarity among my patients in my study. Alternatively, If I choose a large enough sample size A)took blood thinners or B)no thinners. I can assume that most factors will cancel each other out. I go case by case.


Role of Image Resolution in Deep Learning

The answer may depend on what kind of information you want to extract from the images. However, the general approach to the problem is to find a perfect balance so that your image is not too small which is hard to extract too much information or it is not high-resolution input which will unnecessarily complicate your model. The latter will also be hard to train in terms of space complexity and time complexity.
Thus, if your objective is not something like identifying and classifying minuscule objects in the image, or a similar detailed and complex task, then you can use the small size of images.
Having good architecture and a well-trained model can ensure your powerful outcome. Consider that most of the famous and powerful NN models in Computer Vision and Image Processing fields are using input sizes like 96x96, 128x128, 224x224, 256x256. Maybe I am going too further by saying this, but, the good challenge would be building a powerful model with a small size of pixels such as 224x224 or close to this because in that case, the model's usability and usefulness will be scaled up. As you also mentioned, high-quality images can be downscaled to low scales but it is not possible to enlarge the small-sized image (at least, without the help of AI). As your model reasonably small, it will address that many of the all available images (also, detecting or classifying cropped images, small objects in the images, etc. can be considered another reason for having a small input size). So if you build a model using a large pixel size, will also require a high-quality image to be able to work and would not accept on low-quality images. Consider that google vision API works superb even with the 64x64 images.
Briefly, if your aim is not unusual and it does not necessitate high-resolution images to work with small-sized objects in the images, then use small-sized images. With the help of good architectured convolution layers, you can extract lots of information from those small images and processes in the next layers. Also, consider that even small changes in the input image size may affect the training time drastically.


Doing a cox regression, how do I analyze a continuous variable in which a lower result is worse?

With a simple linear regression you have a dependent and independent variable, with an outcome model of:
y = - mx + c

If I understand your problem, y would be mortality, and x would be your clinical variable (i.e. when x increases y decreases). And it would be the same if you want to do your other example, because there is an inverse relationship between the two (i.e. when you decrease x, y increases).
It sounds like your hazard ratio is m (i.e. the relationship between the clinical variable and the mortality)
m = (y - c )/x

So to getting a HR>1 will depend on your data.


Pre-trained CNN model makes Poor Predictions on Test Images Dataset

Kaggle's Dataset is really noisy and it is really tough to get very good result with simple models. You have to do lots of preproccesing ,Data Augmentations and larger network for many more epoch to get a good result.
Things you can do :

Go to notebook section of your competetion and look out for some preprocessing and augmentations and apply them.
Use some regularizer like dropout, mixup,cutmix and many more.

3.Train your model for more epoch

Train a larger model.


Text mining in Amazon product review using R. I wasn't able to extract the particular product's review

Change your url to:
url <- "https://www.amazon.in/Apple-iPhone-11-128GB-Black/product-reviews/B07XVLW7YK/ref=cm_cr_arp_d_paging_btm_next_2?ie=UTF8&reviewerType=all_reviews&pageNumber"

Otherwise, as is what your current code is doing, you keep looping only the first page of reviews.


Prediction using words which were not in training in a CNN with pre-trained word embeddings

If you keep the FastText embeddings unchanged a do not finetune them during training, it does not really matter that the words were not in the training set as long as they are in the FastText embeddings. After all, this is the biggest advantage of using pre-trained word embeddings.
The important property of the embeddings is that similar words get similar embeddings. The CNN might not have seen the exact same embedding, but similar words probably were in the training data.
Words that are not covered by the pre-trained embeddings, got a common representation for an unknown (out-of-vocabulary, OOV) word. These are usually proper names. It is usually good if you make sure that CNN learns to deal with them already at the training time (you can randomly replace some infrequent words by some random strings) because if the unknown token embedding (that is typically dissimilar to all other embeddings) appears at the inference time and it was never seen at training, it can lead to unexpected behavior.


How does BERT and GPT-2 encoding deal with token such as ,

BERT is not trained with this kind of special tokens, so the tokenizer is not expecting them and therefore it splits them as any other piece of normal text, and they will probably harm the obtained representations if you keep them. You should remove these special tokens from the input text.

In the case of GPT-2, OpenAI trained it only with <|endoftext|>, but it has to be added after the tokenization. Some people mistakenly add it before tokenization, leading to problems.
<|startoftext|> is specific to the library gpt-2-simple.


Checking if english sentences have impact catpured in them using NLP

The approach you propose is ok but it can be improved in my opinion:

Find a corpus (or several corpora), chosen to be representative of the kind data you expect to process eventually. Ideally it would be directly a sample of the target data.
Label all the sentences in the corpus or a random subset. This is because you need to preserve the distribution as much as possible, especially the proportion of positive/negative instances. If you start from the set of filtered sentences you'll have two problems:

a proportion of positive cases higher than in regular text, and this is likely to make your model over-predict positive cases;
sentences containing only the selected trigger words, and this will cause your model to predict as negative any sentence which doesn't have a trigger word (that's a problem since your list of trigger words can't be exhaustive).

Ideally you would manually label all your corpus, but that's probably not realistic. That's why you could try to use your idea of trigger words in a slightly different way in order to label all the instances efficiently:

Filter the sentences which contain any trigger word and label these; this is your initial training set
Use a semi-supervised approach to label the rest of the sentences (maybe you could consider active learning).


Getting same result for all predictions in cnn

Ok so I changed your model to simplify the training for example's sake. I will go through the example in detail below. When you feed a value from a different distribution to your model you are always at risk of misclassification. For example, your model was trained using handwritten numbers, thus it is not surprising for the model to missclassify numbers that are typewritten. Of course a deeper more complex model could do this.

Getting the data

The MNIST data is of size 28 by 28. We will convert these values to floating point values and then normalize them to the range 0 to 1. We will also determine that we have 10 output classes.

from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K
from keras.datasets import mnist
import numpy as np

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.

# The known number of output classes.
num_classes = 10
# Input image dimensions
img_rows, img_cols = 28, 28

Next we need to reshape the data such that it matches with the Tensorflow framework which we will be using under the hood of Keras. This requires that the instances be the first dimension and it also requires a channels dimension as the last one. thus for the MNIST data we need to have $(6000, 28, 28, 1)$.

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
x_test_reshaped = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)

Then we will bin the outputs into one-hot-encoded vectors

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

Let's build our simple model, you can add more layers to this to make it more robust

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

Let's train our model

epochs = 4
batch_size = 128
# Fit the model weights.
model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test_reshaped, y_test_binary))

This should yield about 98.75% on the validation set. This is pretty good and should be enough to test some new data using this set.

Validating results using MNIST

Let's pass through a few random values from the MNIST dataset to see if they get rightfully classified by our model. Note how I set the dimensionality of the data. The instance dimension must exist even if we only have a single instance to predict as well as the channel dimension

ix = 0

plt.imshow(X_test[ix,:,:,0], 'gray')
plt.show()

temp = np.zeros((1,28,28,1))
temp[0,:,:,0] = X_test[ix,:,:,0]

model.predict_classes(temp)

  array([7], dtype=int64)

Ok so that worked! Let's try another one ix = 100!

  array([6], dtype=int64)

That worked as well!

Validating the model on novel data

The image you sue has to match stylistically with the ones in the MNIST data set. Ideally they should be of the same distribution. The image you provided luckily gets correctly classified but it should be noted that this may not be the case for other numbers that are typewritten. If you want your algorithm to detect these then you should really train with typewritten numbers as well as handwritten numbers.

from skimage.transform import resize

def rgb2gray(rgb):
    return np.dot(rgb[...,:3], [0.299, 0.587, 0.114])

im = plt.imread("7 type.jpg")
im = rgb2gray(im)
im = resize(im, (28, 28))

plt.imshow(im[:,:], 'gray')
plt.show()

temp = np.zeros((1,28,28,1))
temp[0,:,:,0] = im

model.predict_classes(temp)

  array([7], dtype=int64)

Validating on hand drawn numbers

More realistically would be to use hand drawn numbers since that is what we used to train. I just drew this one using Paint.

  array([8], dtype=int64)

  array(2, dtype=int64)


Should the data be shuffled on a translation dataset

Yes, in NMT data is always shuffled.

For training, if each batch contains the same "type of content" (e.g. domain, register), then your model will be biased toward the type of content of the last batches. You don't want that. Apart from that, in NMT data is normally bucketed by length to avoid wasting batch space with padding.

For the test data, you would like to have data that follows a similar distribution to the one to be used in production. If the test is biased to only one domain, then you won't be able to assess correctly the expected performance of the model.

Document-level translation has little to do with the shuffling of the data. In document-level MT you normally have context sentences that are either ingested with an auxiliary encoder or as part of a single, longer sentence. This may reduce the number of sentences you can fit in the batch, but each sentence (+context) is independent from the rest.


Python: How to plot time interval from a Dataframe in Pandas

Some suggested alternative plotting methods to visualise this data:

Histogram of the y-axis. Check the distribution of time intervals
df.plot.hist(by='interval', bins=10) #test varying the bin size

Plot smaller subsets of the data if the order is important e.g.
df[:100].plot() furthermore, if there is periodicity in the data, e.g. daily, hourly etc. you could plot each hour on top of each other (in a different colour) to compare the differences between periods.


ValueError: Error when checking input: expected embedding_1_input to have shape (4,) but got array with shape (1,) numpy related

The model excepts the input to be a 1D array with 4 items. The current input single_x_test is a Python list with 1 item. The model implicitly converts the Python list to a 1D array but it still only has 1 item.

The model needs to be introspected to see what 4 items it expects as inputs. Typically items are separate channels of information.


How to perform Logistic Regression with a large number of features?

In order to reduce your model down to 7 variables there are a few approaches you could take:

PCA (unsupervised): this creates "new" linear combinations of your data where each proceding component explains as much variance in the data as possible. So the first 7 components (out of 27) should be able to explain a good percentage of the variation in your data. You can then plug these seven components into your logistic regression equation. The disadvantage here is that because the components are combinations of your original variables you lose some interpretability with your regression model. It should however produce very good accuracy. This same technique applied to other dimension reduction methods such as
Another common method in regression is forward stepwise where you start with one variable and add on another each step, which is either kept or dropped based on some criteria (usually a BIC or AIC score). Backwards stepwise regression is the same thing but you start with all variables and remove one each time again based on some criteria. Based on a brief search it doesn't seem that python has a stepwise regression but they do a similar feature elimination algorithm described in this Data Science post.
Lasso Regression uses an $L_{1}$ penalization norm that shrinks the coefficients of features effectively eliminating some of them.You can include this $L_1$ norm into your logistic regression model. It seems sklearn's LogisticRegression allows you do assign the penalization you want in order to achieve this. Note: Lasso will not explicitly set variable coefficients to zero, but will shrink them allowing you to select the 7 largest coefficients.

As @E_net4 commented, your continuous question is addressed in another post.


What are kernel initializers and what is their significance?

The neural network needs to start with some weights and then iteratively update them to better values. The term kernel_initializer is a fancy term for which statistical distribution or function to use for initialising the weights. In case of statistical distribution, the library will generate numbers from that statistical distribution and use as starting weights.

For example in the above code, normal distribution will be used to initialise weights. You can use other functions (constants like 1s or 0s) and distributions (uniform) too. All possible options are documented here.

Additional explanation: The term kernel is a carryover from other classical methods like SVM. The idea is to transform data in a given input space to another space where the transformation is achieved using kernel functions. We can think of neural network layers as non-linear maps doing these transformations, so the term kernels is used.


Are DBSCAN and dbscan from the sklearn.cluster package different?

Such questions are easily answered if you check the source code yourself.

https://github.com/scikit-learn/scikit-learn/blob/1495f69242646d239d89a5713982946b8ffcf9d9/sklearn/cluster/dbscan_.py#L350

One is an object-oriented wrapper with a more common API, the other the underlying function that does the actual work. Since the wrapper just calls the function, the results are not substantially different, only the presentation.

Hence, usually you won't need both.


turn off parts of features in a neural network?

As far as I know, you cannot do that.
First :

  Obviously setting the mid-term grades to be all 0s would be be a good
  option

No, actually that is a really bad option, neural networks do not understand magical parameters, meaning, if you put a 0 there it will think that the mid-term grades is 0, therefore it will probably give you a very low final grade prediction for you.

My advise, if you are very interested in doing that, create boolean factors, training inputs:

working_hours
mid-term_grade
mid-term_grade_present

mid-term_grade_present should be a 1 when you have the mid-term_grade training data, and 0 when you do not have it. BTW, yes, generate training data without the mid-term_grade.

If you are working with non linear machine learning algorithms that should be enough, if you are using linear algorithms you should one hot encode the mid-term_grade_present and multiply it by the mid-term_grade, ending with something like this:

working_hours
mid-term_grade_present_1 * mid-term_grade
mid-term_grade_present_0 * mid-term_grade


gensim's Phrases module may also be helpful:

from gensim.models import Phrases
documents = [
    "the mayor of new york was there",
    "machine learning can be useful sometimes",
    "new york mayor was present"
]

sentence_stream = [doc.split(" ") for doc in documents]

bigram = Phrases(sentence_stream, min_count=1, threshold=2)

sent = [u'the', u'mayor', u'of', u'new', u'york', u'was', u'there']
print(bigram[sent])
# Expected output:
# [u'the', u'mayor', u'of', u'new_york', u'was', u'there']

That code is from this other answer (I've copy-pasted it above for convenience).

For more on the Phrases module, check this page out.


does index of my data which is of type "Date time index" plays a part in reggression?

You can take the date into account, but you should convert it into numeric value.
Suggestions:
1) Consider "datetime" package in python to change it to numeric, or some other techniques such as indexing with date-time, and converting the index value to numeric, float, etc. I don't have your exact case, so cannot say specifically.
or
2) If you have time series data, then in many cases ARMAX, ARIMAX, Seasonal ARIMAX models can do a really good job.


What is new population in genetic algorithm?

In the simplest implementations, you simply discard the old population and maintain a population of size $n$ in each generation.

There is lots of variation on this though, any of which could be useful in practice. As a start to show the range of possibilities, you can do some or all of the following:

Keep a top number (say $m$) of the original population, and only generate $n-m$ new offspring in each generation.
Generate offspring in small groups and assess them. They must beat a member of the current population on the fitness measure in order to find a place (this can work nicely in round-robin tournaments, where the fitness measure is winner of a game).
Keep snapshots every so many generations (e.g. every 10 generations) of the whole population or just the best performing individuals. You may use these snapshots to assess long-term improvement or as a "gene pool" to keep good performers around in some form for a while longer than a single generation.

Variations like these are hyperparameters for your GA. They might help in certain circumstances, or be a hindrance in others. So start with the simple "everything is replaced" is recommended initially, especially if you are learning genetic algorithms.


R - Order each matrix inside a list of matrices

Because I did not have your exact data (I don't have the csv file), I created a dummy dataframe (swiped from this tutorial):

numPeople = 10
sex=sample(c("male","female"),numPeople,replace=T)
age = sample(14:102, numPeople, replace=T)
income = sample(20:150, numPeople, replace=T)
minor = age<18

population = data.frame(sex=sex, age=age, income=income, minor=minor)
population

> population
      sex age income minor
1    male  34    130 FALSE
2    male  48     86 FALSE
3  female 101     36 FALSE
4    male  64     78 FALSE
5  female  97     92 FALSE
6    male  19     45 FALSE
7  female  42    124 FALSE
8  female  24     68 FALSE
9  female  24     96 FALSE
10 female  91     30 FALSE

This means you will have to adjust the variable names and so on to match your example, but can learn how to order a list of matrices. More specifically, you really have a list of data.frame objects, after using the split function. it is important to notice the difference between a data.frame and a matrix, as they can behave differently.

I will split the data.frame by the sex column:

l = split(population, population$sex)

You can test that each element of the resulting list is indeed a data.frame (and not a matrix) by running:

> is.data.frame(l[[1]])
[1] TRUE

Notice that you have to use the doublesquare brackets to access the actual element of the list. Single square brackets will return something else:

> is.data.frame(l[1])
[1] FALSE

Tt is a named list-element and needs to be referenced by using (in my case) one of the names of the group that we split on: male or female
Below I comment the code heavily to explain what each step does.

> l[1]
$female                          # this is the name of the list element
     sex age income minor
1 female  20     42 FALSE
6 female  75    103 FALSE
7 female  30    148 FALSE

So we can access the actual element two ways:

by name:

> l[1]$female
     sex age income minor
1 female  20     42 FALSE
6 female  75    103 FALSE
7 female  30    148 FALSE

by double square brackets:

> l[[1]]
 sex age income minor
1 female  20     42 FALSE
6 female  75    103 FALSE
7 female  30    148 FALSE

That all being said, he is a working example of sorting a list (vector, holding lists) containing data.frames:

l = split(population, population$sex)    # population is defined as shown above
N = length(l)

results <- vector("list", N)

for(i in 1:N) {

    results[[i]] <- l[[i]][order(l[[i]]$age),]
}

I specify the length of the vector when I create it, as we know how long the resulting list will be in advance.
... and check the results:

> results[[1]]
    sex age income minor
8 female  66     90 FALSE
5 female  69    131 FALSE
2 female  70     52 FALSE
9 female  95     43 FALSE

> results[[2]]
    sex age income minor
6  male  18     87 FALSE
4  male  61     97 FALSE
3  male  64    131 FALSE
7  male  72     93 FALSE
1  male  82    107 FALSE
10 male  92     42 FALSE

The results depend on random choice when creating population at the beginning, using sample().


Terminology: SOMs, batch learning, online learning, and stochastic gradient descent

SOMs

Self-organizing maps (SOMs) are one specific type of neural networks. In contrast to multilayer perceptrons (MLPs; they are used much more often) the SOMs neurons have a position on a regular grid.

Training of SOMs

SOMs are usually trained in a stochastic way (source: [1]). This means one training example is used at a time. After every single training example the network learns something.

In contrast, batch learning means the network is presented the complete batch of all training-examples (or a big part of usually 128 examples; then you call it mini-batch). The parameters are only updated after all of the examples were presented to the network.

In contrast to MLPs, you don't apply gradient descent (as far as I know; I've never used them, but only read about SOMs). I wouldn't even know to which function you would apply the gradient descent algorithm, because there is no error function in SOMs. (At least no standard error function).

For MLPs, I know that the advantage of batch training compared to stochastic training is that the weights are not jumping as much. So you rather go in the right direction. However, the advantage of stochastic training is that you make updates much more often. You want to get the best compromise, so you make mini-batch training.

Online Learning

"Online" has quite a lot of different meanings:

"online" as in "in the internet".
"online" as in "online algorithms" which get input piece-by-piece and have to make a decision before knowing all of the input (e.g. when you have severe time constraints or when you can't store all the data like in CERN where you have to throw most of the data away right after you get it)
"online" in handwriting recognition in contrast to the offline OCR methods (see On-line Recognition of Handwritten Mathematical Symbols if you're interested). Online methods use the way how symbols were written, offline methods only use the image.

According to Wikipedia:

  Online machine learning is used when data becomes available in a sequential order to determine a mapping from data set corresponding labels. The difference between online learning and batch learning (or "offline" learning) techniques, is that in online learning the mapping is updated after the arrival of every new data point in a scale fashion, whereas batch techniques are used when one has access to the entire training data set at once.

Sources

[1] Andries P. Engelbrecht: Computational Intelligence: An Introduction. Page 62.


Ways to create simulated numerical data based on small sample (using Python)?

You can find what you need in

numpy.random

For example, generating a 100x100 matrix with normal data:

data = numpy.random.normal(mean, sigma, shape=(100, 100))

see here

[EDIT]: If you want more advanced way of generating data similar to an existing dataset, you can look into GANs.


Negative Rewards and Activation Functions

This would lead me to believe that the only appropriate activation functions would either be linear or tanh. However, I see any many RL papers the use of Relu.

Generally you want a linear output, unless you can guarantee scaling total possible reward to within a limited range such as $[-1,1]$ for $\text{tanh}$. Reminder this is not for estimating individual rewards but for total expected reward when following the policy you want to predict (typically the optimal policy eventually, but you will want the function to be able to estimate returns for other policies visited during optimisation).

Check carefully in the papers you mention, whether the activation function is applied in the output. If all rewards are positive, there should be no problem using ReLU for regression, and it may in fact help stabilise the network in one direction if the output is capped at a realistic minimum. However, you should not find in the literature a network with ReLU on output layer that needs to predict a negative return.

  If you do want to have both negative and positive outputs, are you limited to just tanh and linear?

There are likely others, but linear will be far the most common.

  Is it a better strategy (if possible) to scale rewards up so that they are all in the positive domain (i.e. instead of [-1,0,1], [0, 1, 2]) in order for the model to leverage alternative activation functions?

It may sometimes be worth considering scaling rewards by a factor, or normalising them, to limit gradients, so that learning is stable. This was used in the Atari-games-playing DQN network to help the same algorithm tackle multiple games with different ranges of scoring.

In continuous problems, the absolute value of reward is usually flexible, you are generally interested in getting the best mean reward per time step. So in that case you could scale so that minimum reward is 0, and use ReLU or other range limited transform in output - as above that might help with numeric stability.

In episodic problems without a fixed length, you typically don't have a such a free choice, because the agent is encouraged to end the episode quickly when rewards are negative. This is something you might want for instance if the goal is to complete a task as quickly or energy-efficiently as possible. A good example of this is "Mountain Car" - granting only positive rewards in that scenario would be counter-productive, although you might still get acceptable results with positive reward only at the end and discounting.

The general case is that rewards can be arbitrarily scaled and centred for continuous problems without changing the agent's goal meaningfully, but only arbitrarily scaled for episodic problems.


Evaluation methods for multi-class classification

In https://www.sciencedirect.com/science/article/pii/S004896971831163X?via%3Dihub, we have used the product of the sensitivity of class i (i.e., the ratio of data correctly classified for class i) obtained for each class. This summarizes these values into a single index, which ranges between 0 and 1 and to some extent is irrespective to data imbalance. This approach induces a good balance between the error comitted to each category because any value appreciably below 1 reduces significantly the performance. To relax this allowing a greater error in 1 o more categories you can simply use (in increasing order): the min, the geometric mean or the arithmetic mean. In https://ieeexplore.ieee.org/document/6940273/ and https://ieeexplore.ieee.org/abstract/document/5428802/ you can find these alternatives.
Good luck.
Rafael.


gradient descent for non convex function like $-x^2$

The calculation of the gradient is the same no matter the optimization algorithm you use. Gradient descent is a method to find (local) minima of a function. If no such minimum exists, any algorithm to find it will render wrong results or never converge. $-x^2$ only has a maximum, so looking for a minimum does not make much sense and to find the maximum you would need to run gradient ascent, i.e. going uphill.

Getting stuck in a saddle point is a different issue. For $f(x,y) = x^2 - y^2$ the gradient is $\nabla f(x,y) = 2[x, -y]^T$ and thus equals zero at $x=y=0$, so gradient ascent/descent would just sit there and not move. But $f(0,0) = 0$ is neither a minimum nor a maximum ($f$ actually doesn't have either of these).


Neural net without hidden layers should be a simple linear model: why do I get so different results?

I would say your way of comparing this is rather incorrect. There are two main problem from what you did those are :

R linear model (lm) is solved using deterministic solution whereas
keras model optimized with gradient descent. In the end you expect
the weights and result to be close but not similar. For OLS, it
attains the lowest theoretical training error(by definition)
averaged over all training sample. Your keras model will converge
towards this minima but it might have difficulty converging to
similar minima(due to the nature of GD).
you are not training your keras model with full data. Notice you put
validation split to be 0.2 i.e. you are training with 80 percent of
your data.

Finally, if you want to test whether single layer NN is similar to OLS You can try setting validation split to 0 (or simply don't declare it) and training it with the whole dataset as your batch.


Reinforcement Learning algorithm for Optimized Trade Execution

Why do we need n in the cost function update rule. Aren't we visiting each state exactly once?

The update is assuming a static distribution and estimating the average value. As each estimate is made available, it is weighted less of the total each time. The formula means that the first sample is weighted $1$, second $\frac{1}{2}$, third $\frac{1}{3}$ which is what you need to get the mean value when you apply the changes due to the samples serially whilst maintaining the best estimate of the mean at each step.

This is a little odd in my experience of RL, because it assumes the bootstrap values (the max over next step) come from a final distribution to weight everything equally like this. But I think it is OK due to working back from final step, hence each bootstrap value should be fully estimated before going backwards to previous time step.

  If I understand correctly, we should run this algorithm on every episode (in the experiment in the paper they had 45000 episodes)

This looks like an algorithm that you run on the whole data set, where each episode is the same length $T$. So you run each timestep (starting with the end time step and working backwards since the ultimate reward is established at the end of the episode, so this is more efficient), and sample from every episode at that timestep in the While (not end of data) loop. The values are therefore combined inside the loop at that stage, and there is no need to add anything to the algorithm to combine episodes.


Can somebody explain me the meaning of this sentence? (Color Similarity - Selective Search Algorithm)

If you have two images, you first start to make histograms of the values (0-255) in the three color channels (red, green, and blue). In the article 25 bins are used, meaning that the values are assigned to one of 25 ranges. The second step is to then concatenate the the three single channel histograms to one histogram for the full image. Since each image channel histogram has 25 bins, the concatenates histogram will have 75 bins (and thus is of length 75). When you do this for both images you want to compare you will now have two histograms. To get a similarity score you compute the intersection of the two histograms, i.e. the percentage of overlap. The following link from the article provides a more visual explanation on this.


Keras functional API Layer name not captured with TimeDistributed wrapper

The reason is you should name the TD instead of Dense. Be cautious with the bracket. Change it to:

  x = TimeDistributed(Dense(1, activation='relu'), name="t_output")(inputs)


Is it wrong to transform the target variable and test the model without dropping the column that was transformed? What's the disadvantage about it?

The problem is that, by definition, your target variable is not available at inference time, and that is why you want to predict it. If your target variable was available at inference time, then there is no point in predicting it.
Therefore, if you use the target variable (or a transformation of it) as input to your model, what data are you going to feed to that variable at inference time?


Which functions neural net can't approximate

A neural network can approximate any continuous function, provided it has at least one hidden layer and uses non-linear activations there. This has been proven by the universal approximation theorem.

So, there are no exceptions for specific functions. You ask:

  I read somewhere on the StackExchange that a neural network can't approximate the Pi number as a function of circles length and radii.

A neural network to approximate $\pi$ is very easy. Possibly what you read is that a neural network cannot generate new digits of $\pi$ that it has not already been shown. More on that later . . .

  What about the sum or multiplication of any arbitrary numbers?

Yes, a neural network can approximate that.

  Are there any other specific functions neural networks can't approximate or not?

No, there are no specific functions that a neural network cannot approximate.

However, there are some important caveats:

Neural networks do not encode the actual functions, only numeric approximations. This means there are practical limits on the ranges of inputs for which you can achieve a good approximation.
A neural network being able to approximate a function in theory is not the same thing as you or I being able to construct a neural network that approximates that function. There is no known method to construct a neural network by analysis of a function alone (it can be done for specific simple functions such as xor).
The usual way to achieve approximation is to train a neural network by giving example data. The network will approximate to data it has been shown. There is no guarantee that this will generalise to new inputs that it has not been trained on and approximate the correct outputs. In fact for certain types of input/output it cannot possibly do so. For instance, it will not learn how to generate the 4th digit of $pi$ if it has been shown digits 1,2,3,5,6,7,8,9. The best generalisation results occur for functions that have smooth transitions between the training examples.
Neural networks do not extrapolate well to inputs outside of the data they have used for training. They "fit" to the training data (imagine a rubber sheet draped over all the points in the training set).
Neural networks do not learn to copy algorithms, only functions. So if you take a complex algorithm, such as AES encryption, and attempt to train a neural network to perform this given lots of input examples, has no real chance of working. Now, AES encryption can be considered a function e.g. $output = encrypt( input, key )$. So the NN can approximate it. But it will only do so for the specific inputs and outputs it has been shown. In addition AES does not respond well to approximation - a single bit wrong will cause it to be a bad encryption. So you won't see NNs used to encrypt or decrypt in cryptography.
The capability of a neural network to approximate is limited by the number of neurons and connections it has. More complex functions require larger networks. In order to train a larger network on a more complex function takes more time and more training data. You could in theory train a neural network to learn a random number generator function. However, that would take an impossible amount of resources - memory to store the network, and time to train it against the whole output of the RNG.


How to treat the undefined values which make sense?

Recently had a discussion on the same topic at work. It boiled down to encoding missing values as impossible values (negatives, very high) or as information inferred from the dataset (mean, median). Some more sophisticated methods use models build on rest of data (non-missing columns) to predict missing ones.

If using tree-based approach setting it to -1 should be fine as a start, as as there might be observations where previous purchase was in same day (0).

With mean-trending models (linear regression), setting it to the mean might be also fine, but you need to calculate the mean on the train set only and propagate it to the test set separately.


Autoregressive Moving Average ARMA with statsmodels

Found the issue and I will post the answer in case anybody will get this issue with ARMA.

model = sm.tsa.ARMA(endog=dataset_period[col], order=(nlag,0), exog= dataset_period[dummy_variable_period]).fit()
dataset_period['predict_'+str(col[1])+'_'+str(col[2])] = model.predict()
dataset_period['ape_'+str(col[1])+'_'+str(col[2])] = abs(  (dataset_period[col] - dataset_period['predict_'+str(col[1])+'_'+str(col[2])]) / dataset_period[col])
valid_ape = dataset_period['ape_'+str(col[1])+'_'+str(col[2])][dataset_period['ape_'+str(col[1])+'_'+str(col[2])] < 50]

  issue was with the parameters. the model accepted 2 parameters for group_level and I was basically providing 1 so in this case automatically become zero and raise error.

  solved with creating a new def and the model will use single column as the parameter for group_level

model = sm.tsa.ARMA(endog=dataset_period[col], order=(nlag,0),exog= dataset_period[dummy_variable_period]).fit()
dataset_period['predict_'+str(col[1])] = model.predict()
dataset_period['ape_'+str(col[1])] = abs((dataset_period[col] - dataset_period['predict_'+str(col[1])]) / dataset_period[col])
valid_ape = dataset_period['ape_'+str(col[1])][dataset_period['ape_'+str(col[1])] < 50]`


VC dimension of half spaces over the real line

First you need to understand how to calculate the VC-dimension. There are two conditions for the VC-dimension to be $n$ (here $n=1$):

You need to find one set of $n$ points that can be shattered (i.e. classify all possible $|\{0,1\}|^n=2^n$ labelings correctly) and
It must be impossible to find any set of $n+1$ points that can be shattered.

Now to your questions:

1) $a$ defines the family of possible decision functions in your classification problem. In your case, $a$ defines all possible half spaces along the real axis. That means all points greater than $a$ a classified as $1$ and all points less than $a$ are classified as $0$.

First they show that one point can be shattered. And they choose that point to be $x=0$. They could choose another point but it is enough to show it for $x=0$.

To shatter $x=0$ you need to find an $a$ so that $h(0)=0$ and an $a$ such that $h(0)=1$. (Those are the $2^1=2$ possible labels.)

If the label is $0$ you can choose any $a>0$ to label $x$ correctly. $2$ is one example. With $a=2$ all $h(x)$ will return $0$ for all points less than $2$ (which includes $x=0$).

If the label is $1$ you can similarly choose any $a<0$. $-2$ is one example, but you could also choose $-0.01$ or $-1232132158932$. In the case $a=-2$, $h(x)$ will return 1 for all $x>-2$ (which again includes $x=0$).

So you fulfilled the first condition.

2) Now they show the second condition. But maybe they left out some of the details. Let me try to fill them in.

To show the second condition, you need to pick any two points and show that it is impossible to find $a$ to classify all possible labels correctly. Let's call the two points $x1$ and $x2$. The $2^2=4$ possible labels are: $\{0,0\},\{0,1\},\{1,0\},\{1,1\}$.

Clearly, if $x1=x2$ it follows that $h(x1)=h(x2)$ and therefore the labels must be the same and and only $\{0,0\}$ and $\{1,1\}$ are possible but not $\{0,1\},\{1,0\}$.

Therefore, we can move on to the case where $x1 \neq x2$. Since the naming of $x1$ and $x2$ is arbitrary we can assume that $x1 < x2$ ("without loss of generality" because if $x1 > x2$ we can just rename $x2$ as $x1$ and $x1$ as $x2$ and we would have $x1 <x2$ again).

Now, in this case it is also impossible to achieve all four possible labeling combinations. In particular they show that the labeling $\{1,0\}$ is impossible.
If the label of $x1$ is $1$ then it follows (similarly to 1) with $x=0$) that $a$ must be less than $x1$. But then $a < x1 < x2$ and therefore $h(x2)$ must also be $1$ and can't be $0$.

(You could now move on to the case $x1 > x2$ but as I tried to explain above that's basically the same idea, but now with the labeling $\{0,1\}$).

See this question for a very similar example with intervals instead of half lines.


Normalize Test and Control, or as a whole?

An experimental sample is a group that receives the variable being
  tested in an experiment. The control group is the group in an
  experiment that does not receive the variable you are testing using
  the same experimental procedure. Therefore the control group,
  receiving no intervention, is used as a baseline to compare groups and assess the effect of that intervention.

I gave this as an object lesson to show that test and control must be linked by a similar experimental procedure. So yes, if you were to normalize then use the entire matrix.

Additional point (bonus round)

The purpose of data transformations is to make data easier to model and understand. One purpose of transforming data is to find outliers, reduce the amount of skew, etc.

Looking at a small part of your data, it seems like the values are between 0-100,  Suggesting that normalization is not entirely necessary, IMHO.


When does it makes senses to use Dot-Product as similarity measure instead of Cosine?

Borrowing from this Quora answer -- which includes some concrete examples of when you might prefer one measure over the other -- it comes down to whether you care about taking the magnitude of the vectors into account. This is highly domain-specific, but the example used there is in information retrieval, where magnitude represents the length of the documents in question. The dot product will take the document length into account, whereas the cosine similarity will not.

In general, you will want to have good reasons specific to your problem space to use one over the other.


How to collect data which exists centre or border of the cluster?

You can use numpy.where documentation can be fouind here :
https://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.where.html

You can define a function to extract your data ID somthing like this

def ClusterIndicesNumpy(clustNum, labels_array):
    return np.where(labels == cluster)[0]

To get samples from cluster 3 for example:

ClusterIndicesNumpy(3, km.labels_)


How do I send the results of a convolutional layer and non-deep-learning features into a dense layer in Keras?

This can be done with Keras functional API.

In this example, "merge_1" layer gets input from :

Output of LSTM Layer (lstm_1)
aux_input layer

More info : https://keras.io/getting-started/functional-api-guide/


Train building classifier for imagerial data

This blog article might be a good starting point. From what you described and depending on your data, semantic segmentation might be overkill and classification will suffice.

Either way, the first step will be to get your hands on training data. If you do not have labels already, this might mean that you have to sit down for a while an label a bunch of images. This is much easier for classification than segmentation, but the information you end up with is not as rich.

For segmentation you should then check out U-Net or DeepLabv3. Classification is not my strong site, but VGG or a ResNet should do the trick.