non-binary nominal variable in linear regression

I've read about dummy variables, but as these three will depend on each other (if the value is "red", then it's not "black"), can I use dummy variables?

Yes, you can use dummy variables to represent a multilevel qualitative variable like Color.

ISLR Example

An example that may be helpful to you can be found in ISLR, chapter 3 on pages 85 and 86:

The results in the book can be recreated using the Credit data set.

In the example given in the book, balance is regressed onto ethnicity. That is, balance is the quantitative dependent variable (if it were qualitative this would be a classification problem rather than a regression problem), and ethnicity is a 3-level qualitative variable.

> Credit <- read.csv("http://www-bcf.usc.edu/~gareth/ISL/Credit.csv", header = TRUE)
> attach(Credit)
> summary(Balance)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
   0.00   68.75  459.50  520.01  863.00 1999.00

> summary(Ethnicity)
African American            Asian        Caucasian
              99              102              199

The task is to predict Balance based on Ethnicity. Therefore Ethnicity needs to be dummified.

As stated in the text pictured above,

  There will always be one fewer dummy variable than the number of levels. The level with no dummy variable—African American in this example—is known as the baseline.

is.Caucasian <- ifelse(Ethnicity=="Caucasian", 1, 0)
is.Asian <- ifelse(Ethnicity=="Asian", 1, 0)

The above code accomplishes creating dummy variables for Caucasian and Asian, with African American as the baseline. Now to create the linear model:

balance.lm <- lm(Balance ~ is.Caucasian + is.Asian, Credit)
summary(balance.lm)

Call:
lm(formula = Balance ~ is.Caucasian + is.Asian, data = Credit)

Residuals:
    Min      1Q  Median      3Q     Max
-531.00 -457.08  -63.25  339.25 1480.50

Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept)    531.00      46.32  11.464   <2e-16 ***
is.Caucasian   -12.50      56.68  -0.221    0.826
is.Asian       -18.69      65.02  -0.287    0.774
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 460.9 on 397 degrees of freedom
Multiple R-squared:  0.0002188, Adjusted R-squared:  -0.004818
F-statistic: 0.04344 on 2 and 397 DF,  p-value: 0.9575

The results are an exact match to the information displayed in Table 3.8 on page 86.

Example using 3 colors

Let's say we wanted to predict the mass of an ball using its color. In this case quantitative variable mass is the response (dependent variable) and color is the independent variable.

> color <- c("red", "black", "green", "red", "green", "red", "red")
> mass <- c(   12,      40,       1,    15,       2,    14,    14)
> ball.data <- data.frame(mass, color)
> head(ball.data)

  mass color
1   12   red
2   40 black
3    1 green
4   15   red
5    2 green
6   14   red

We can dummify color into two variables, one for red and one for black, with green as the baseline.

> ball.data$is.red <- ifelse(ball.data$color=="red", 1, 0)
> ball.data$is.black <- ifelse(ball.data$color=="black", 1, 0)
> head(ball.data)

  mass color is.red is.black
1   12   red      1        0
2   40 black      0        1
3    1 green      0        0
4   15   red      1        0
5    2 green      0        0
6   14   red      1        0

Then we fit a linear model like so:

> ball.lm <- lm(mass ~ is.red + is.black, ball.data)
> summary(ball.lm)

Call:
lm(formula = mass ~ is.red + is.black, data = ball.data)

Residuals:
         1          2          3          4          5          6          7
-1.750e+00  5.551e-17 -5.000e-01  1.250e+00  5.000e-01  2.500e-01  2.500e-01

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   1.5000     0.8101   1.852 0.137727
is.red       12.2500     0.9922  12.347 0.000247 ***
is.black     38.5000     1.4031  27.439 1.05e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.146 on 4 degrees of freedom
Multiple R-squared:  0.9947,    Adjusted R-squared:  0.9921
F-statistic: 376.7 on 2 and 4 DF,  p-value: 2.79e-05

When we compare the coefficient estimates in the linear model summary to the values in the data set, we see they are pretty close, as expected. That is about it.

More info

PSU STAT 501: Regression Methods, Lesson 8: Categorical Predictors

CODING FOR CATEGORICAL VARIABLES IN REGRESSION MODELS | R LEARNING MODULES


EDA for analysis of nominal variable with high cardinality

You can calculate mean target for each categorical variable and compare its values.
In pandas this can be done easily: df.groupby('categorical_feature').target.mean()

Then you can make a histogram to compare the approach. I also, seaborn has a catplot, where it do the same as above in a bar plot format, showing mean value for target variable based on each categorical one.


Data Science is all about creativity and do/see things what makes sense.

I would describe Data Scientist career in couple words: Take this path only and only if you are passionate about science, working with data and are curious. Do not take this way if you are thinking about money or job security ... this field is not so simple and you stay weeks and months experimenting and try to find something in that datasets ...
One example: as data scientist you are mostly starting to look at data and try to find something inside and solve problems, some dta does not provide any info and after weeks of "magic" is put on the side and moved to other project, after a while you returning and look into data to find information on other business problems.

Look at Data Science as pure science and only after that about money, also most of the data scientist's primary goal is not remuneration ... media is promoting salaries but not the real work what we doing ... how much we love data ...


Is this a Q-learning algorithm or just brute force?

Kudos for figuring out a working tic-tac-toe playing algorithm from scratch!

  Question 1: Can I successfully argue that I am estimating the reward based on history, and still claim the algorithm is reinforced learning or even Q-learning?

First things first, this is definitely not Q-learning.

However, I do think it classifies as Reinforcement Learning. You have implemented these key components of RL:

A state (the current board), used as input on each step.
An action (desired next board arrangement), used as output. When the action is effectively to choose the next state directly, this is sometimes called the afterstate representation. It is commonly used in RL for deterministic games.
Rewards generated by the environment, where the agent's goal is to maximise expected reward.
An algorithm that can take data about states, actions and rewards, and learn to optimise expected reward through gaining experience within the environment.

Your algorithm is closest IMO to Monte Carlo Control, which is a standard RL approach.

One of the big advantages of Q Learning is that it will learn an optimal policy even whilst exploring - this is known as off-policy learning, whilst your algorithm is on-policy, i.e. it learns about the values of how it is currently behaving. This is why you have to reduce the exploration rate over time - and that can be a problem because the exploration rate schedule is a hyper-parameter of your learning algorithm that may need careful tuning.

  Question 2: If I replace the reward lookup which is based on the board layout, with a neural network, where the board layout is the input and the reward is the output, could the algorithm be regarded as deep reinforcement learning?

Yes, I suppose it would be technically. However, it is unlikely to scale well to more complex problems just from adding a neural network to estimate action values, unless you add in some of the more sophisticated elements, such as using temporal-difference learning or policy gradients.

  Question 3: I'd don't think that I have either a learning rate or a discount factor. Is that important?

A discount factor is not important for episodic problems. It is only necessary for continuous problems, where you need to have some kind of time horizon otherwise the predicted reward would be infinite (although you could also replace the discount mechanism with an average reward approach in practice).

The learning rate is an important omission. You don't explain what you have in its stead. You have put update reward for board layout based on game outcome - that update step typically has the learning rate in it. However, for tic-tac-toe and Q-Learning, you can actually set the learning rate to 1.0, which I guess is the same as your approach, and it works. I have written example code that does exactly that - see this line which sets learning rate to 1.0. However, more complex scenarios, especially in non-deterministic environments, would learn badly with such a high learning rate.

  Question 4: Can tictactoe algorithms be classed as real learning rather than simply brute force?

Your algorithm is definitely learning something from experience, albeit inefficiently compared to a human. A lot of the more basic RL algorithms have similar issues though, and often need to see each possible state of a system multiple times before they will converge on an answer.

I would say that an exhaustive tree search from the current position during play was "brute force". In a simple game like tictactoe, this is probably more efficient than RL. However, as games get more and more sophisticated, the machine learning approach gets competitive with search. Often both RL and some form of search are used together.


How can I use data science / machine learning / statistics / data analytics techniques to find better customers in the future?

You can use linear regression to model the relationship between the profit each customer has given you and the other variables: staff, location, industry, and so on.

With this technique, you will by the end obtain the regression coefficients for each considered variable. These coefficients will tell how one variable affects the profit, and then you'll be able to check things such as: do customers with more staff generate more profit? how much this profit increases for each staff member? and so on.

Also, with these coefficients, you can easily predict, for the potential costumers, how much would be their profits. This prediction is not perfect, as it is an associated error, but it can be very good if you specify the model properly. As there is a loooooot of material about this on the internet, I won't write much more.

I don't know how much you know about math, or intend to dive in this subject, but this article can help as a first introduction: https://towardsdatascience.com/mathematics-for-machine-learning-linear-regression-least-square-regression-de09cf53757c

Linear regression is, for sure, not the only solution for your problem, but it is a simple one, somewhere to start. You may also want to look (later) for regression tree models and models that take in account the longitudinal dependency of your data. Longitudinal means you have repeated measures for the same costumers along the time.


Linear regression - LMS with gradient descent vs normal equations

Andrew Ng answers this question succinctly in his Coursera lecture about the normal equation.  I will summarize.

You have m training examples and n features.

Disadvantages of gradient descent:

you need to choose the learning rate, so you may need to run the algorithm at least a few times to figure that out.
it needs many more iterations, so, that could make it slower

Compared to the normal equation:

you don't need to choose any learning rate
you don't need to iterate

Disadvantages of the normal equation:

Normal Equation is computationally expensive when you have a very large number of features ( n features ), because you will ultimately need to take the inverse of a n x n matrix in order to solve for the parameters data.

Compared to gradient descent:

it will be reasonably efficient and will do something acceptable when you have a very large number ( millions ) of features.

So if n is large then use gradient descent.

If n is relatively small ( on the order of a hundred ~ ten thousand ), then the normal equation


Database/warehouse for storing 128D feature vectors

If you really need to do that (I will argue it is not a good idea), with Postgres you can store an array type and write a stored procedure for new item insertion.  This stored procedure can do whatever distance checking you wish, such as checking the distance of the new vector against all others in the database, before storage.

I would argue against this design because I suspect the criteria for uniqueness* could easily change over time.  I think it'd be a better idea to store all of the vectors except exact matches.  Then, create another table which uses the definition of uniqueness.  Creating this table would be handled on the application side.  If your definition of uniqueness changes, no problem, just make a new table.  You could even compare several different definitions to see how your results are sensitive to it.  If you do it this way, Cassandra is a great database choice.  It's specifically designed for denormalized data storage (you have the same data stored in different forms or variations, so that your application gets exactly what it needs without further computation).

*In your post you stated that a similarity of < 0.9 would result in storing a new vector.  That's what I mean by criteria for uniqueness.


Capturing movement importance - logistic regression output

You can calculate a distance in Euclidean space.

Given your specific goal, you can define a custom distance metric that would the equivalent of a hinge loss. For the most of the space, distance is not defined because the probabilities can be thresholded to zero. For a small section of space near (100, 50), there is a linear or nonlinear distance defined.

An alternative is to switch classifier to a support vector machine (SVM) with a hinge loss which would more directly model your problem.


Confusion Matrix of LSTM using Sklear getting error

A confusion matrix can be drawn for a classification problem, where the Machine Learning Model (in your case LSTM) predicts the target variable into one of the N classes.

Can you confirm that your problem is a Classification problem and not a Regression one?

If in case you have a regression problem, then you might want to use a different evaluation metric.


First, find out when the user switch and give a separate id to each message group:

df['group_id'] = ((df['user'] != df['user'].shift()).cumsum())

user              message  group_id
   A                  Hi.         1
   B               Hello.         2
   B         How are you?         2
   A          I am stuck.         3
   B  How can I help you?         4

Then groupby each group_id and aggregate a list of the messages for each id. By shifting these messages by -1 we recive the replies for each group_id:

df_reply = df.groupby('group_id')['message'].agg(list)
df_reply = df_reply.shift(-1).reset_index().rename(columns={'message': 'reply'})

 group_id                   reply
        1  [Hello., How are you?]
        2           [I am stuck.]
        3   [How can I help you?]
        4                     NaN

The replies can then by merged back into the original dataframe. The reply lists are exploded to ensure a single reply per row:

df.merge(df_reply, on='group').explode('reply').drop('group', axis=1).dropna()

The final result:

user       message                reply
   A           Hi.               Hello.
   A           Hi.         How are you?
   B        Hello.          I am stuck.
   B  How are you?          I am stuck.
   A   I am stuck.  How can I help you?


How can I get the max value and corresponding index of a tensor simultaneously in Tensorflow?

Before you worry about performance, I'd recommend profiling it. Finding the maximum is an inexpensive operation. Also, can you index into the element using the return value of argmax? That way you'd only take the max once.


Face recognition from images is still an open area of research. Many different techniques have been tried.

One approach is not to directly learn the identities of the people as classes, but to learn a multi-dimensional embedding for images of faces (e.g. 128 dimension vector) and train the network to make this embedding value closer for two different images of the same person than for two images of different people*.

You can use the network to generate feature vectors for each person who is registered, and store those values. Later when you want to identify someone, find the closest one (according to norm of the difference) and if the distance is less than a certain cutoff you can consider it to be a match.

To perform the training for the embeddings with this goal in mind, you can use a loss function called triplet loss, which is trained on 3 forward passes at a time. The network calculates feature vectors for an anchor image, a positive match (i.e. another image of the same face, so you need at least 2 for each person, and they should be taken from different contexts) and a negative match. Then it measures whether the distance between features from same person is less than distance between different people. The loss function to express that looks like:

$$\text{max}(0, |A - P|^2 - |A - N|^2 + \alpha)$$

. . . where $A$ is feature vector for the anchor image, $P$ for positive match image, and $N$ for negative match image. The parameter $\alpha$ is a hyper-parameter which creates a "margin" that you reward the model for maintaining between different identities. The idea being that if an image of different person is clearly further away in the feature space than a cutoff, the loss on that training example is zero. The model does not get rewarded for very large differences, just having "consistently enough distance".

The tricky thing with triplet loss is picking which samples to use for training. Purely random samples could make the network's task too easy. You need it to focus a bit on the more difficult cases to prevent mistaken identity. One way of doing this is to pick preferentially from closer faces, e.g. using k-means clusters and picking negative cases within own cluster.

The model can be trained on large numbers of faces, not necessarily the ones you will later want to classify. The point is to learn a feature vector that emphasises important differences between faces (and ideally ignores other variations such as lighting, background, pose). It's possible to start with any image classifier and adapt it for this task, including your original face classifier (just remove the final classifier layer and work with one of the later fully-connected layers as your output).

  How can we scale up the number of classes for deep learning after training a model?

This approach may not work well in all cases covered by your question title. I have recommended it for face recognition based on a Coursera course on CNNs where a few lectures and coding assignment are about face recognition specifically.

In some cases you may be better off re-building and re-training the model with new training set data, augmented by examples of your new class - you might start by fine-tuning only the last few layers.

* In fact this will be the case in many CNN classifiers, but is more likely to be reliably linked to a person's identity if you have trained the network explicitly for distinguishing between identities in general (as opposed to just classify specific identities). The triplet loss approach is one way to train for useful representations, and makes the goal of disambiguating identity explicit.


What is the difference between Latent and Explicit Semantic Analysis

The difference is that with ESA, the concepts are already known and labeled (hence, "manifest concepts"), whereas in LSA the concepts are latent (they are undefined and need to be discovered).

Note the following statement from the ESA Wikipedia page:

The name "explicit semantic analysis" contrasts with latent semantic analysis (LSA), because the use of a knowledge base makes it possible to assign human-readable labels to the concepts that make up the vector space.[3][1]

In short, ESA uses prior knowledge of relationships between words and concepts (as well as labels for those concepts), whereas LSA does not.

References cited by Wikipedia for the quote above:

[1] Ofer Egozi, Shaul Markovitch and Evgeniy Gabrilovich (2011). "Concept-Based Information Retrieval using Explicit Semantic Analysis" (PDF). ACM Transactions on Information Systems 29 (2). Retrieved January 3, 2015.

[3] Gabrilovich, Evgeniy; Markovitch, Shaul (2007). Computing semantic relatedness using Wikipedia-based Explicit Semantic Analysis (PDF). Proc. 20th Int'l Joint Conf. on Artificial Intelligence (IJCAI). pp. 1606–1611.


using movielens dataset build recommendation engine

There are plenty of ways to realize this. Originally everybodies approach to the Netflix Prize (quite similar to the movielense dataset) were nearest neighbor methods. Their intuition should be fairly clear. Matrix Factorization became the next big thing and was nicely described by Simon Funk in his journal: http://sifter.org/~simon/journal/20061211.html . If you are more interested in these two topics and don't mind a bit of bad english you may consider this thesis: http://patrick-ott.de/dl/diploma_ott.pdf .

The crowd then went on to Restricted Boltzmann Machines for the task, e.g. (cs.utoronto.ca/~hinton/absps/netflixICML.pdf) . I think it is beyond this answer to explain these for this particular task -- maybe worth another question,

All three approaches though (Nearest Neighbor, Matrix Factorization and RBMs) never we are able to reach to goal set by Netflix, which let to the development of more or less clever methods to combine their predictions, which finally was enough to get to the point Netflix set out.

These are all somewhat old approaches to the problem, their might be newer ones. However, the most recent machine learning hype, i.e. Deep Learning has so far not given many possible solutions although dominating in other fieds.


How to programmatically suggest changes to a text to match some other text?

Some time ago, I stumbled upon the Author Obfuscation task of PAN16 workshop. The task at hand is given a document, paraphrase it so that its writing style does not match that of its original author.

Although, it's not exactly what you want, you can take a look at their proceedings and find useful tips regarding authors style and how to "change" it (i.e. in your case, an "author" is a collection of documents from the same source).

For example, one approach
denotes specific metrics that define an author's style (average sentence word count, use of stopwords, pos-type frequencies). You could use the target's scores in these metrics as your gold standard and try to modify the source text, in ways discussed in the proceedings of the workshop, so as to mask your source text to match the target one (in terms of these metrics).


Understand the equations of quantile regression forest (Meinshausen)?

It's a good idea to remember what you're trying to predict and that is: $\mathbb{E}[Y | X=x]$. The simplest estimate is a single tree:
$$
\hat{\mu}(x) = \sum_{i \leq n} w_i(x, \theta) Y_i
$$
with:

$w_i(x, \theta)$ the single tree node weights (equation 4 in the paper)
$Y_i$ your observations

As we all know this is not a great estimate (high variance among others) so he defines a better estimator (random forest) as:
$$
\hat{\mu}(x) = \sum_{i \leq n} w_i(x) Y_i
$$
where the $w_i(x)$ are averages over the multiple trees (equation 5). So what does that mean? Your best estimator for $\mathbb{E}[Y | X=x]$ is $\hat{\mu}(x)$. What do you do if you want an estimation of a function $\phi$ of $Y$? You just transform your observations of $Y$ and use the same formula, i.e.:
$$
\hat{\mu_{\phi}}(x) = \sum_{i \leq n} w_i(x) \phi(Y_i)
$$
which would then be an estimator of $\mathbb{E}[\phi(Y)|X=x]$. If you define $\phi(Y) = \mathbb{1}_{Y \leq y}$ then you get that:
$$
\hat{\mu_{y}}(x) = \sum_{i \leq n} w_i(x) \mathbb{1}_{Y_i \leq y}
$$
is an estimator of $\mathbb{E}[\mathbb{1}_{Y \leq y}|X=x] = F(y|X=x)$.
I avoided a lot of low-level details here but to be clear this relates to the delta method i.e. convergence of a function of estimator. This kind of convergence is not trivial to prove (but I guess that's why they wrote a paper about it)


NLP techniques to label unlabeled data in a dataset

This is still a problem undergoing active research. I think it's commonly referred to as unsupervised labeling. If the dataset is not too large, it's probably more accurate and time efficient to either do it by hand, or hire someone on Mechanical Turk to do it.

If you're dead set on doing it automatically, trying some form of semi-supervised learning where you train it on a few examples, and then attempt to generalize is probably your best bet.


Binary classification model with time series as variables

I think one of your statements is simply a misunderstanding of the relationship of data structure and analysis. When you say "Melting the data, due to the structure, obviously wouldn't work too." you seem to failing to understand that melting the data properly would require construction of a supplemental covariate that would encode the "column location" of the values. In longitudinal data this location (in the "wide" version of the data) would become the "time" variable in the "long" format. Regression and neural network methods can handle such a format.

I intended that my comment only be a suggestion for improving a search strategy,  since you had not described the task or the inputs in any great detail. So I'm "answering" with some links resulting from several variations on a search strategy along the lines of ("binary outcome" OR "signal detection" OR classification) AND ("logistic regression" OR "neural networks") AND longitudinal AND missing:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5401893/

http://www.stats.ox.ac.uk/~snijders/WrightLondon2009.pdf

http://www.drizopoulos.com/pdf/Slides/jsm_2015.pdf

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3834930/

https://dl.acm.org/citation.cfm?id=3167596

https://mlhc17mit.github.io/slides/lecture7.pdf

https://www.cs.cmu.edu/~epxing/Class/10708-17/project-reports/project8.pdf

https://arxiv.org/pdf/1802.04664.pdf


Curious Georg if you ran across this article in your pursuit of trying to generate probabilities. It is worth noting that binary:logistic and multi:softprob return predicted probability of each data point belonging to each class.

You can look here to see how the following code is used:


Prove that the expected value of the training error is equal to generalization error for a class of binary clasiffiers

Let us call $L=L_{(D,f)}(h) = \mathbb{P}_{x \sim D} [h(x) \neq f(x)]$.

We know that $L_S(h)= \frac{1}{m}|\{x\in S : h(x) \neq f(x)\}|$, where $m$ is the size of $S$.
Then $L_S(h)$ has values of the form $\frac{i}{m}$ with $0 \leq i \leq m$.

Also $\mathbb{P}[L_S(h)=\frac{i}{m}] = {m \choose i} L^i(1-L)^{m-i} $.

(Not sure why, but make sense since the probablility of taken $i$ $x$'s such that $h(x) \neq f(x)$ is precisely $L^i$, same thing with $(1-L)^{m-i}$ when $h(x) = f(x)$. Additionaly,
\begin{equation}
\label{1}
1=\sum_{i=0}^m {m \choose i} L^i(1-L)^{m-i}
\end{equation}

In this way:
\begin{align}
\mathbb{E}[L_S(h)] &= \sum_{i=0}^m {m \choose i} L^i(1-L)^{m-i}*\frac{i}{m} \\
                   &= \sum_{i=1}^m {m-1 \choose i-1} L^i(1-L)^{m-i}\\
                   &= L \sum_{i=0}^{m-1} {m-1 \choose i} L^i(1-L)^{m-1-i}\\
&=L
\end{align}


You can augment videos in the temporal dimension through clipping, or taking random sequences of consecutive frames. You can also augment in the spatial dimension by cropping frames randomly to simulate camera jitter. Unfortunately, augmenting video data by extrapolating motion information or by producing a video taken from a different viewpoint seems extremely computationally intensive.


Is there an easy to use .net library for neural networks?

I have no personal experience with it, but I came across Accord, which has limited support for neural networks. Honestly, .net does not have a machine learning community, so you might be better off using something else if you do not want to DIY.


I am doing a forecasting for corona virus cases in NY, I have two models and not sure which one i should choose

I will suggest, first take some training data and make both models on that and compare the result on test data e.g

If you have data from 1st Jan 2020 to 28 Apr 2020.

Give the model, data from 1st Jan 2020 to 31st March(Or you can decide another range), make the model and forecast for April month.

And compare the results for both the models on April month(1st April to 28th April).

In this way, you will be able to argue better with results.


Sutton and Barto (2nd edition is out!). Also David's Silver slides for the RL course at UCL plus the youtube videos from his classes. You can start with the lecture videos and the slides and going back to the book for more details and exercises. Putterman is quite hard for a beginner, heavy on math and focusing on Dynamic Programming mostly.


Scatter plot for binary class dataset with two features in python

One approach is to plot the data as a scatter plot with a low alpha, so you can see the individual points as well as a rough measure of density.

from sklearn.datasets import load_iris
iris = load_iris()
features = iris.data.T

plt.scatter(features[0], features[1], alpha=0.2,
            s=100*features[3], c=iris.target, cmap='viridis')
plt.xlabel(iris.feature_names[0])
plt.ylabel(iris.feature_names[1]);

We can see that this scatter plot has given us the ability to simultaneously explore four different dimensions of the data:

the (x, y) location of each point corresponds to the sepal length and width,
the size of the point is related to the petal width, and
the color is related to the particular species of flower, i.e the Target Variable...

Multicolor and multifeature scatter plots like this can be useful for both exploration and presentation of data.


Stochastic gradient descent in logistic regression

Stochastic gradient descent is a method of setting the parameters of the regressor; since the objective for logistic regression is convex (has only one maximum), this won't be an issue and SGD is generally only needed to improve convergence speed with masses of training data.

What your numbers suggest to me is that your features are not adequate to separate the classes. Consider adding extra features if you can think any any that are useful. You might also consider interactions and quadratic features in your original feature space.


A robust metric in the presence of class imbalance

For evaluating the classification of a highly imbalanced, there are several measures that you may consider. Remember, that in such a problem we would prefer a measure that is not biased towards one of the classes but gives similar importance to both classes.

Methods for evaluating classifiers could be divided into two types: singular assessment measures where the evaluation is given for a specific threshold (i.e. trade-off) and area-under-curve (AUC) analyses where a range of thresholds is considered.

The F-measure combines precision and recall. Precision is sensitive to data distribution as it combines values from the two rows of the confusion-matrix hence, as the difference between the numbers of negative and positive instances gets larger, the influence of the positive class (minority) gets smaller. Recall, on the other hand, is not sensitive to data distribution since it does not refer to the negative class at all. Affected by the precision, the F-measure must be sensitive to class distribution as well. The F-measure gives higher importance to the correct classification of positive instances, a change in the TNR value will influence the F-measure value less than a change in TPR. This is explained by the impact of the recall, which considers only the correct classification of positive class instances.

The Geometric Mean (G-measure, GM) combines TPR and TNR, denoting the accuracy rates measured on both classes separately. The GM is independent of the class sizes, indicating the relative part of correctly classified instances in each class regardless of the degree of imbalance. That is, the GM is not influenced by the imbalance between the classes, The GM relates to the performance on both classes with the same importance since it is a harmonic mean of the TPR & TNR and their effect is symmetrical. Higher values of GM are achieved when the accuracies on both classes are high and similar. For example, a higher GM value is accomplished when accuracies on both classes are equal to 0.5 than when one is 0.6 and the other 0.4.

The ROC-AUC is based on the TPR and FPR, which is 1-TNR, and hence have the same reasoning for suiting such problems as the GM.

The PR-AUC, precision-recall curve, are also useful for highly imbalanced data and also is robust for changes in the balance of classes (addition of samples). I've found this article very useful for understanding AUC measures and the benefits of each one of them.

The AUC analysis provides a more generic evaluation however, in most cases it will not give the best solution when a specific trade-off is to be considered. The AUC is only relevant if your classifier output is not discrete but continues.

For more reading and comparisons you may use Prof. Haibo He lectures which are very comprehensive and useful.


Cost of greater than 1, is there an error?

The following piece of code does essentially what TF's softmax_cross_entropy_with_logits functions does (crossentropy on softmaxed y_ and y):

import scipy as sp
import numpy as np

def softmax(x):
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=0)

def crossentropy(true, pred):
    epsilon = 1e-15

    pred = sp.maximum(epsilon, pred)
    pred = sp.minimum(1-epsilon, pred)

    ll = -sum(
        true * sp.log(pred) + \
            sp.subtract(1,true) * \
            sp.log(sp.subtract(1, pred))
    ) / len(true)

    return ll

==

true = [1., 0.]
pred = [5.0, 0.5]

true = softmax(true)
pred = softmax(pred)

print true
print pred

print crossentropy(true, pred)

==

[ 0.73105858  0.26894142]
[ 0.98901306  0.01098694]
1.22128414101

As you can see there is no reason why crossentropy on binary classification cannot be > 1 and it's not hard to come up with such example.

** Crossentropy above is calculated as in https://www.kaggle.com/wiki/LogarithmicLoss, softmax as in https://en.wikipedia.org/wiki/Softmax_function

UPD: there is a great explanation of what it means when logloss is > 1 at SO: https://stackoverflow.com/a/35015188/1166478


Find the column(s) name where the value of the variable matches a regex

Would this work for you,

 names(iris)[apply(iris, 2, function(x) any(grepl("rsi", x)))]


The World Bank runs Enterprise Surveys that 'provide the world's most comprehensive company-level data in emerging markets and developing economies. Business data are available on 130,000 firms in 135 countries'. You can get access by requesting on the site subject to some confidentiality terms.

The data are fairly comprehensive and would give you a good starting point to form hypotheses as which areas (access to credit, regulation, corruption, etc) matter most that you could then try to test by mining news or social media.

The World Bank also collects data on Doing Business that, while not specific to small companies, highlight country-specific challenges.

Plenty of research has been published around the two programs above.


List of samples that each tree in a random forest is trained on in Scikit-Learn

I don't think it is possible to get it directly but we may utilize the random seed.

random_stateint, RandomState instance or None, default=None
Controls both the randomness of the bootstrapping of the samples used when building trees (if bootstrap=True)

This is from the RF Github code
def _generate_sample_indices(random_state, n_samples, n_samples_bootstrap):
    """
    Private function used to _parallel_build_trees function."""

    random_instance = check_random_state(random_state)
    sample_indices = random_instance.randint(0, n_samples, n_samples_bootstrap)

    return sample_indices

So, we can get these with a custom code if we fixed the seed above e.g. for 2 Tree,
import pandas as pd, numpy as np
num = 20 # max index
np.random.seed(0) # Fix the seed

sample_1 = np.random.randint(0,num,(1,num))
oob_1 = [elem for elem in np.arange(num) if elem not in sample_1 ]
sample_2 = np.random.randint(0,num,(1,num))
oob_2 = [elem for elem in np.arange(num) if elem not in sample_2 ]

Please verify it with a custom code. I have not verified it.


Can generic data sets be suitable for specific sentiment analysis

It depends.

You're basically asking if your sample (training data) is representative of the population (all written words).

Are you doing sentiment analysis on movie reviews? It'll work great.
Are you doing sentiment analysis on TV reviews? It'll probably work great.
Are you doing sentiment analysis on book reviews? I would give better than 50-50 odds it'll work.
Are you doing sentiment analysis on Twitter posts? Now we're getting shaky. People tend to write much less, use less formal language, and use more emojis which your movie review model wouldn't have seen.

That being said, there are definitely "generic" sentiment analysis services like here. Try out your model against Algorithmia on what you would consider a generic set of data (e.g. a bunch of tweets) and see how it does.


How to get the formulas used by seasonal_decompose for Trend and Seasonality

I do not know how seasonal decompose works, but there is another very similar method that is well described.

Triple exponential smoothing (other name Holt-Winters method) internally makes a decomposition into trend, seasonal and residual components. But you should always check the decomposition. It always finds the seasonal component even if it does not exist.

Also the decomposition is actually sensitive to the underlying model. If the data is linear in logarithms and you do not apply a logarithm you will get a much less precise decomposition.


Best common metric for comparing classic time series forecasting methods (ARIMA/Prophet) with ML approach?

In general the most common metrics mentioned in several articles (e.g. like this one) are the same we also commonly use for non-time series prediction:

MAE Mean absolute error
MSE Mean squared error
RMSE Root mean squared error

Outside of linear regressions I have not seen R² used that often to validate prediction models. In fact it isn't even one of the out-of-the-box metrics of xgboost for interval scale prediction tasks. So you should be fine using RMSE for everything.
Edit:
Prophet at least in the R package covers RMSE, MAE and MAPE so you should be fine.


Scikit-learn average_precision_score() vs. auc score of precision_recall_curve()

The metrics differ in the macro case because macro in scikit-learn ignores class imbalances. micro in scikit-learn looks at global rates. The data has class imbalances, thus the macro argument will return different values compared to micro.


To put it simply, it will give you a equation (less error relative to data-set presumed to be correct equation). Depending on data-set, equation has constants value for each variables which will provides the ratio of one variable with other variables.
You might find equation on similar format as this:

y = (b0 + b1.x1 + b2.x2 + b3.x3) + c

where x1  , x2 , x3 can be different characteristics (as variables which can be IQ, age, working momory, etc.) and b0 , b1, b2, b3 are the constant calculated from given data-set.

You can find exact value of y or x1 or x2 or x3 if you have exact value of other 3 variables.

Lets say, it was the data-set provided to an android/robot as in sci-fi movies; that android/robot will make it's decision based on this equation as law/fact/pattern and judge a person response time (if every other variables are known) or IQ (if every other variables are known) etc. by this equation.

There is also a lot of details in this.

Note: This logic will only practical with Multiple Linear Regression Method.


Have a look at this post. I found it quite useful when starting out with neural networks.

http://neuralnetworksanddeeplearning.com/chap2.html


Is addition of a random control variable a good idea, to ensure that results are correct?

This approach makes some sense but it's not the best approach for several reasons.
First, this control variable might not always be last in importance because it's possible that some other variable also don't have any impact at all on the target variable (outcome).
More importantly, the concept of a control group/control variable is useful in cases where one cannot evaluate objectively the effect of a method. Typically a drug trial needs a control group because of the placebo effect and various other biases. The control group plays the role of a baseline, and the performance of the method is measured relatively to this baseline.
In supervised classification, there is a much more direct way to objectively evaluate the effect of the method: this is exactly what we do when we evaluate the performance on a proper test set (fresh instances). Additionally there are simple ways to compare the performance of a system to a baseline: if there is no previous similar system for the task, one can simply use a random baseline or a majority baseline classifier.


The information in groups 'group_a_co_2' and 'group_b_co_2' are already redundant; they do not add more information to the model. Therefore they can be removed.
Adding even more redundant information will not improve your model further.


Does standardization result in normal distribution?

No, standardization does not change the shape of the distribution. It centers the distribution by subtracting the mean and scales it by dividing by the standard deviation.


Reinforcement Learning model always gives different output

Reinforcement learning in principle should not be the source of stochasiticy, i.e. given exactly the same inputs the dqn model and the rl agent should give exactly the same output.
The source of stochasticity in your setting may be the random seed. Make sure to set it once and share it with all libraries
that have a stochastic aspect (i.e. network initialization, batch selection, your simulator if you are using one).


Text extraction from documents using NLP or Deep Learning

Jurafsky and Martin's NLP textbook has a chapter about information extraction that should be a good starting point. For example, if you want to extract company names it will tell you how to do that.

  A paralegal would go through the entire document and highlight important points from the document.

What you need to do depends heavily on what your definition of "important" is here. It would help if you can give some specific examples.


Approaches to high dimension pattern matching problem

According to the following discussion on StackOverflow, a situation like that you've described can occur, when one of the variables in a data set is of unexpected type (for example, a factor instead of a character): https://stackoverflow.com/q/7246412/2872891.

Also, consider using package bigmemory, recommended in the accepted answer, or similar packages for big data analysis. For the latter, please see section "Large memory and out-of-memory data" in CRAN Task View High-Performance and Parallel Computing with R.

Finally, an additional note. There is an ecosystem of R packages, built around the arules package, which includes supporting packages for algorithms (arulesNBMiner), applications (arulesSequences, arulesClassify) and visualization (arulesViz). You are likely aware of that, but I have decided to include this good-to-know fact just in case and for the sake of completeness.


DataFrame.dtypes error when training model using xgboost

Check the values of train_new. You'll see that the columns mentionned are not of the expected types.

Another suggestion, i'm not sure of xgboost's handling of nulls. Might be that in those columns aswell.


Seems like the Keras .fit and .evaluate methods give different training accuracy (yet the same validation accuracy). Same thing on loss.?

training's loss/accuracy are not calculated for the whole dataset every time.
It is calculated for the current batch and averaged successively.
test's loss/accuracy is calculated at the end of an epoch for the whole test data.
Check these references
Keras FAQ
SO Answer
You may put the validation data same as the train data to check it.
ann=model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_data=(x_train,y_train))

# Evaluate the trained model
print("\n")
score = model.evaluate(x_train, y_train, verbose=0)

Or, you may write a custom callback to print appropriate logs as suggested in the SO answer.


Physical modelling with neural networks - single output + stack ensemble vs multi-output

My guess: I agree with your colleagues.  I see no reason to do anything other than a single neural network with multiple outputs.  If necessary, increase the capacity of that single neural network until you see no further improvement.

An stacking ensemble where you have a few neural nets whose inputs are fed as input into another neural network is itself equivalent to one bigger neural net.  I don't see any reason to expect the stacking approach to train more effectively or to be better for some other reason.

However, this is ultimately an empirical field.  The only way to find out what will work best is to try different approaches and see.  There's not a huge amount of theory that enables us to make predictions about what approaches will be most effective.

You might consider whether you can obtain more training data, particularly in the regimes of particular interest.  Often better training data sets offer higher gains than modifying the neural network architecture.


Identifying features which drives recommendations

This is original paper by Yehuda Koren et.al. on recommenders for implicit feedback datasets. It implements collaborative filtering method in a novel way for implicit feedback case. Section 5 of this paper gives a method for implementing explanations.

Though you will have to build the implementation yourself from their explanation. To the best of my knowledge no library implementation has explanation part inbuilt.


How to cluster/identify points away from a regression line

As already mentioned in a comment, there are some methods to detect outliers. The best known might be Cook‘s distance in (or after) regression. In essence you look for observations with a high residual.

Here is a short R tutorial. Here is a short Python approach.


Ordering of standardization, pca, and/or tfidf for neural network

I would go for this:
data -> tokenize -> tfidf* -> neural net

But in tfidf vectorizer, you could actually regularize the number of terms used, say for example restricting the minimum number of occurrences of a term and/or defining the max_number of features so that you only keep the ones that have the highest importance according to Tfidf.
If you want to reduce the number of features via some decomposition technique PCA won't be adequate since the term-frequency matrix is sparse, so you could, for example, using NMF (Non-negative matrix factorization instead)
So:
data -> tokenize -> tfidf->NMF -> neural net

This time the regularization on tfidf is not necessary since you have an additional step.
In the end, metrics on CV will guide you on what the best strategy is


What is this formula, related to simple linear regression, called?

I do not know other terms than the  average of inverse slope or the inverse of the harmonic mean of slopes. This is also the negative of the average slope of the perpendiculars.

It gives you the inverse of the average slope of lines passing through $(0,0)$ and $(x_i,y_i)$. If the $(x,y)$ are almost aligned with $(0,0)$, this is an estimate of the inverse of the slope of a line passing through the points.

Linear regression is quite different, as it involves cross-products of sums of $x$ or $y$.


How can I estimate user-item purchase probabilities of a e-commerce website?

If there are users' session available I'd recommend to go for a session-based recommendation - when we make recommendation of next click/buy item based on the current user's session. That seems the most appropriate for your data and also very attractive way of doing recommendation systems in the e-commerce, when still most of users while searching for purchase or window-shopping is not logged in.

While predicting the next action, few measures can be used. But from practical perspective and current publications, two are quite interesting:

REC@N - Recall at top-N best recommended items. This measure gives the information about the overall engagement of the users and many times it's said that is correlates with CTR on live(production) use.
MRR@N - Mean Reciprocal Rank at top-N ranked recommendation, which gives you the information how well is your personalized ranking for the current session prediction.

I'd recommend reading two recent papers about session-based recommendation - one is about making the next item recommendation based on sequences of items [1]. The other one [2] is about making recommendations based on the current user's session when you have rich context for recommended items and session events. Which is mostly true for the e-commerce system, where you have items described by: title, attributes, text description, images, etc. And also in users session there are additional events like interaction which search engines, which gives lots of users preferences in data for the query, sorting by which attribute, etc. This can be used as a current session context. In both works you can find references to related works, which can be helpful for you.

[1]: Hidasi, B., Karatzoglou, A., Baltrunas, L., & Tikk, D. (2015). Session-based Recommendations with Recurrent Neural Networks, 1–9. Retrieved from http://arxiv.org/abs/1511.06939

[2]: Twardowski, B. (2016). Modelling Contextual Information in Session-Aware Recommender Systems with Neural Networks, 0–3. http://doi.org/10.1145/2959100.2959162


In your article $\hat F$ is defined by Eq. (39). Although it's a bit abstract, it says that the partial dependency function $\hat F_j$ is obtained by averaging over all variables but a subset $s$.

As an example, the one-variable dependency function for the variable $1$ is given by $$\hat F_1(x_1) = \frac{1}{n}\sum_{i=1}^n \hat f \left (x_{1}, x_{i2}, \dots, x_{im} \right ) ,$$ where $\hat f$ is your model prediction function (Random Forest in your case), $n$ is the number of samples, $m$ is the number of features (I've included $\hat f$ instead of $\hat F$ for the predictor to avoid confusion). Here $x_1$ is a variable, and the $x_{i2}, \dots, x_{im}$ values are taken from your samples.

For example, if you have two variables and two samples $(x_1=0,x_2=0)$ and $(x_1=1, x_2=1$) in your dataset (this is a very dummy, pedagogical example), the partial dependence function for the first variable reads:
$$ \hat F_1 (x_1) = \frac{1}{2}\left [ \hat f(x_1, x_2=0) + \hat f(x_1, x_2=1) \right ]. $$

Similarly, you can compute the two-variables partial dependence function as
$$\hat F_{12}(x_1, x_2) = \frac{1}{n}\sum_{i=1}^n \hat f \left (x_{1}, x_{2}, \dots, x_{im} \right )$$ (same as before, but note that the $i$ index is not under $x_2$ anymore).

The two equation I've just made explicit are the only two terms that appear in the equation (44), thus you should be able to compute the $H$-statistic with them.

Let me know if it is still unclear, and check out this for more exegesis on the $H$-statistic.


Gaussian Process for Classification: How to do predictions using MCMC methods

In step 1, it's no need to use MCMC, instead, we can compute the posterior with some assumptions in closed form, it's $N(E(f_*),Var(f_*))$. In the second step, we use approximation method to compute the probs by sampling from the posterior distribution.


Label A records B times or label A*B records

In my opinion there's no correct answer between the two choices, and you are right about the arguments.

I would argue for a balanced compromise between the two: have a proportion of A (say 20%) labelled by multiple annotators (say at least 3), but the rest of A can be labelled by a single annotator. You could refine the proportion/number of annotators progressively based on the results: for instance if after the first batch one observes that annotators disagreement is very high, then it might be useful to increase the two parameters. This way you can still evaluate the level of disagreement and it effect on performance, while also maximizing the amount of data annotated.

An alternative is to use some form of active learning. For example, first use method 2 (each instance labelled once), then use cross-validation to tran a model and apply it to the corresponding test set. At the end of the process, the misclassified instances are the "hard ones" which need to be re-annotated by multiple annotators. This kind of process can be used iteratively to identify the most "ambiguous" instances.


Using time series to predict house prices vs. multiple linear regression

I agree, the simple version of this problem isn't really a time series problem. There are some ways to deal with irregular intervals, but, houses are not all the same and we do not have a series of sale prices for it in most cases.

What may of course be very predictive is time. Housing prices probably have a long-term trend that's linear over the space of a few years, at least. There is probably a seasonality component, so month might be meaningful if used correctly.


How does one go about feature extraction for training labelled tweets for sentiment analysis?

Feature Extraction is an important step when dealing with natural languages because the text you've collected isn't in a form understandable by a computer. If you have a tweet that goes something like

I do not like the views of @Candidate1 on #Topic1. Too conservative!! I can't stand it!

then we can't just feed this into a learning algorithm. We need to convert it into a proper format, so we perform pre-processing on our data.

To start, we might want to try to tokenize the tweet. To tokenize is something like segmentation. If you're familiar with Python, we can use ready made libraries (like NLTK) to aid us. Depending on how your tokenizer is made, you could transform the previous tweet into something like

['I', 'do', 'not', 'like', 'the', 'views', 'of', '@', 'Candidate1', 'on', '#', 'Topic1', '.', 'Too', 'conservative', '!', '!', 'I', 'can', "'", 't', 'stand', 'it', '!']

I manually segmented the tweet with the rule that words separated by a space should be tokens and punctuation should be tokenized separately. The way you build a tokenizer (or adjust the settings of a ready made one) will determine the output from a tweet. Notice how '@' and 'Candidate1' are separated? A tokenizer for regular text might not be able to identify that this is a social media entity -- a user mention. If you can adjust your tokenizer to account for social media identities and contractions (like "can't"), you could produce a list of tokens as such

['I', 'do', 'not', 'like', 'the', 'views', 'of', '@Candidate1', 'on', '#Topic1', '.', 'Too', 'conservative', '!', '!', 'I', "can't", 'stand', 'it', '!']

Now, you mentioned bigrams and unigrams. An n-grams (e.g. 1-gram == unigram) is just a sequence of tokens. So what we produced awhile ago was just a unigram list of tokens. If you want a bigram, then you'd want to take 2 tokens at a time. An example output of a bigram list of tokens would be

['I do', 'do not', 'not like', 'like the', 'the views', 'views of', 'of @Candaite1', '@Candidate1 on', 'on #Topic1', '#Topic1 .', '. Too', 'Too conservative', 'conservative !', '! !', '! I', "I can't", "can't stand", 'stand it', 'it !',]

Notice how words repeat? Imagine what a 3-gram or 5-gram would look like.

Before anything, why would we use bigrams over unigrams or those of higher n values? Well, the higher the n, the more about of order you're able to capture. Sometimes order is an important factor in learning. Playing around with how you'll represent your data might show you important features.

Now that we have our text tokenized, we can start extracting features! We can turning our example test, and other text samples, into a Bag-Of-Words (BOW) model. Think of a BOW as a table with column headers as the words/terms and rows as your text samples/tweets. A cell could then contain the number of words/terms for a given sample of text. You could start with counting each term in a sample, so based on the tweet, you'd come up with something like

tweet1: {
   'I': 2,
   'do': 1,
   'not': 1,
   'like': 1,
   'the': 1,
   ...
   '!': 3,
   ...
}

I don't want to manually write it all, but I hope you get the picture here. See how 'I' is 2 because it was mentioned twice in the sample. '!' was mentioned trice, so its value was 2. You'll find that there will be a lot of 1 values, specially in tweets, because there isn't room for much to be written.

You'd do this for each of your tweets and you'll come up with something like

       | 'I' | '!' | '#Candiate1' | ...
tweet1 |  2  |  3  |       1
tweet2 |  0  |  0  |       0
...
tweetn |  1  |  5  |       0

This is a BOW! Adjust your n-grams and you'll get different features with different values. This representation is actually a simple starting point for you because this is now understandable by a computer. You mentioned you labeled your tweets, so imagine that table, but with your label appended to it.

       | 'I' | '!' | '#Candiate1' | label
tweet1 |  2  |  3  |       1      | negative
tweet2 |  0  |  0  |       0      | negative
...
tweetn |  1  |  5  |       0      | positive

And with that, you can build a simple model with the appropriate steps after feature extraction.

There is a lot more to feature extraction! So many different combinations of methods to try out, but stick to things simple for now, if you don't feel so adventurous. Here are some things to consider:

Instead of counting terms, you could go with frequency (i.e. number of occurrence of a term divided by total number of terms)
Instead of extracting the words themselves, you can use part-of-speech (POS) tags (POS tags are language dependent, so you'll have to look for a POS reference for the specific language you're dealing with)
Consider extracting topic models or maybe meta information (avg length of words, avg length of tweets)
Is there an emotion lexicon for the language you're dealing with? Consider counting the number of emotion related words
Consider reducing the number of unique words or applying weighted schemes (like TFIDF)

There are so many ways to approach this, but it ultimately boils down to a design you'll finalize. I guess the best advice I can give you is to refer to related literature. Look at other papers/articles that process text written by Nigerians. What are the methods they use to process text? Is it language dependent or independent? Did they reduce the number of features? Can I create the language resources? Once you identify a good source, try modifying and experimenting from there.


Including identifier in machine learning model as feature vs separate model for every identifier

In my opinion including id as a feature will not make sense at all, because the model will treat the id as a numeric value which will decrease the model performance, because there should be no connection how big the id is and how many customers there are for that id.

Option 2 can make sense if you have enough data for every branch.

My suggestion will be to look deep into your features and try to find a feature which will replace the branch id. Let's say the number of supporting desks in a branch or the location of a branch as a categorical value. If you find enough features that can describe the specifics of branches, then no need to include ids or to do it separately.


What is your seperator in the csv- a comma? It would help if you could open the raw csv file in a text editor and post the result here. You could then check if this seperator can appear in the input data. You then have two options: Change the seperator (if possible) or create a small python script/ manually replace all cases in which this can happen.


Take a look at mixed effects models. These give you a natural way to deal with heiractical and nested data.


TypeError: Cannot cast array data from dtype('float64') to dtype('S32') according to the rule 'safe'

Spent some time with error and found that serial data consists a newline character"\n" in it and it caused error pitch data is a float and due added character roll value is a string.So, changed code as below a little and it worked


If you have already split your training and validation sets into separate directories then there is no need to technically do the splitting in your code. However, the problem with a pre-defined validation set is that it can lead to overfitting more easily: the primary purpose of a validation set is to detect overfitting and if you keep tuning your hyperparameters for your model using a fixed validation set every time you train, then your model hyperparameters may be overfitting to that specific validation set.
Another problem with a fixed validation set is that it prevents you from exploiting approaches such as K-fold cross validation, where you split your data into K groups and each each group takes its turn as a validation set during training (prevents hyperparameter overfitting to a specific validation set as discussed above)...
With regard to your question on overfitting, your list is very comprehensive. However, I would add Early Stopping to that list - this you may already be doing - but good to mention explicitly.


So you have collected data that shows which app is being used at any time, binned into hours in the day. And you have several apps. You mention other dimensions, like user state when using the app (walking, not-walking), (active, not-active - it appears to me that you are not collecting much usage 2-6. is that because the usage is from a self-directed ping from an app while the user is truly away?), location (is this going to be all possible values, or are you going to use something like the fact that this location has been seen often before?). Another interesting relationship could be pairing apps, i.e., mining for a relationship between App A being used after using App B or before App B.

Regardless, then you will definitely have many different dimensions upon which to measure the usage characteristics for any particular usage measurement, and so you are definitely going to have a multiple dimensional problem. You might try to visualize this as an N-space problem, with an axis of measurement for each of your characteristics. Each of your previous measurements represents vectors and you are producing a new vector with your next measurement.

From this, you want to predict future behavior based on measuring the input characteristics from your usage space. You could go for something that classifies as nearest neighbor, and you probably want to do this for your first stab at the problem. You might end up wanting to make the predictive model more sophisticated by adding probabilities to the classifier and acting on that. This means getting estimates of class membership probability rather than just simple classifications. But I would build the whole thing incrementally. Start simple and add complexity as you require it. The increased complexity will also have effects on performance, so why not baseline with something.

For the aging of data, are you wanting to reduce the predictive power of characteristics that are too long in the tooth? If so, be explicit with yourself about what that means, quantitively. Do I trust the usage data from last month less than yesterday's data? Perhaps so, but then why? is my usage different because I am different or because last month was special compared to yesterday, or vice-versa? Again, you might benefit from ignoring this at first, but then trying to search for "seasonal" or periodicity characteristics from the data. Once you determine if/how it changes, you can weight that contribution compared to your immediate usage in different ways. Perhaps you want to amplify the contribution of a similar period (same time of day && same location && same previous app usage). Perhaps you want to provide an exponential dampening on historical data because the usage is always adapting and changing, and recent usage seems to be a much better predictor than 3xcurrent.

For all of this, the proper data science perspective is to let the data lead you.


Plotly in Python provides a nice solution:
!pip install chart_studio
from chart_studio import plotly as py
import pandas as pd

df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/2011_us_ag_exports.csv')

for col in df.columns:
    df[col] = df[col].astype(str)

scl = [[0.0, 'rgb(242,240,247)'],[0.2, 'rgb(218,218,235)'],[0.4, 'rgb(188,189,220)'],\
            [0.6, 'rgb(158,154,200)'],[0.8, 'rgb(117,107,177)'],[1.0, 'rgb(84,39,143)']]

df['text'] = df['state'] + '
' +\
    'Beef '+df['beef']+' Dairy '+df['dairy']+'
'+\
    'Fruits '+df['total fruits']+' Veggies ' + df['total veggies']+'
'+\
    'Wheat '+df['wheat']+' Corn '+df['corn']

data = [ dict(
        type='choropleth',
        colorscale = scl,
        autocolorscale = False,
        locations = df['code'],
        z = df['total exports'].astype(float),
        locationmode = 'USA-states',
        text = df['text'],
        marker = dict(
            line = dict (
                color = 'rgb(255,255,255)',
                width = 2
            )
        ),
        colorbar = dict(
            title = "Millions USD"
        )
    ) ]

layout = dict(
        title = '2011 US Agriculture Exports by State
(Hover for breakdown)',
        geo = dict(
            scope='usa',
            projection=dict( type='albers usa' ),
            showlakes = True,
            lakecolor = 'rgb(255, 255, 255)',
        ),
    )

fig = dict( data=data, layout=layout )

url = py.plot( fig, filename='d3-cloropleth-map' )


(A*x)  Expression has dimensionality of (10, ) and b has shape of ( 10, 1) - this is why you see this error.

My fix solves the error, but you should double check the results

objective = Minimize(sum_squares(A*x - np.array(b).reshape(10,)) + gamma*norm(x, 1))

Hope this helps!


The problem is that when you merge two dataframes, you need enough memory for both of them, plus the merged one. There is a workaround from a stackoverflow answer

What you can do is to read the first dataframe only (the smaller one) and then read the second in batches.

def preprocess(x):
    df = pd.merge(df_train, x, on=['ncodpers','int_date'], how='left')
    df.to_csv("final.csv", mode="a", header=False, index=False)

reader = pd.read_csv("train_lag.csv", chunksize=1000)

for r in reader:
    preprocess(r)


You could approach this in two ways:

Association-rule learning

Intuitively, it's like trying to infer rules such that: if the reasons are X and Y, the diagnosis will likely be Z.

The most common algorithm is Apriori, which is easy to implement.

Classification/Supervised learning

Your features could be binary (0 if reason X is not applied, 1 if it is). You could have one for each reason, and even combination of reasons etc. You might run into some issues if you have many possible reasons (your features will be very sparse).

If this is the case, I'd look into dimensionality reduction to make your features denser.


Extracting individual emails from an email thread

First you can read about the format of ARPA internet text messages.

You're going to have to one up on Regex parsing if you are going to want to extract any meaningful info from the emails. I'd suggest the Oreilly book on regex, or reading over regular expressions.

There is also a tool to capture text from incoming emails and write to a text file.


Can I not call my project a "Machine Learning" project if it doesn't involve a prediction at the end?

Could you talk a bit more about your model training and evaluation?

If you are viewing correlations, it sounds like exploratory data analysis to me.

Typically, I believe a machine learning model will be trying to predict a data point/variable. This could be in the form of classification or regression (i.e. did you do one of these?)

Just curious, why is it important to label as a machine learning project? If a class project, maybe you would be interested in your model predicting points, points per game, or win/loss ratio. Or maybe based on your model, you would want to be clustering players into different segments.

Good luck!


Can a neural network be of variable depth?

"Wide and Deep network"  was used in a 2016 paper.

$\hspace{5cm}$Wide & Deep Learning for Recommender Systems
You may create using keras Concatenate layer
concat = keras.layers.Concatenate()([input, hidden2])

References to read -
The paper
Google blog
Another paper


What is wrong with a neural network model which is so dependent on the seed of initialization?

One thing you can do is train your model N times and report the average and standard deviation of the accuracy.

Is the train/test split fixed? That means, do you use the same train set for every evaluation? And do you have a stratified split? That means the class imbalance of your data is also present in the train and test data. Since your data is imbalanced imagine you have a split where most of your train data consists of class 0, your model will learn to achieve a low train-loss with predicting 0 all the time, but you will have a high test-loss since you have way more samples of class 1 in your test set.

It probably helps to check your predictions. Fix seeds and stratify your splits.


How to compute inner product between two networks'parameters

You can compute the Frobenius norm between the two matrices. Also look up this related article on distance measures between matrices/


Evaluating Language Model on specific topic

What you can do is to compare against a validation set of the same domain. First, you use your LM to generate many sentences, and, for each sentence, you compute the BLEU score against the whole validation set. This python script may be useful for that.
However, you should take into account that it is possible that your model generates very similar sentences always. For that reason, people normally evaluate not only quality but also diversity. For this, you can compute the "self BLEU" of your generated sentences, that is, you compute the BLEU of each generated sentence against the rest of the generated sentences. You can find a script for that here.
You should also take into account that you can trade quality for diversity and vice versa, by setting different values of the output softmax temperature. For this, you may want to evaluate your model at different values of the temperature, to understand different generation regimes, plotting them as a curve, like this:

You can find more about this kind of evaluation in this article, where I took the figure from.
You can find alternatives to the quality-diversity evaluation here.


Oscillating loss can be attributed to either of the following:

Learning rate: Reduce the learning rate so that the gradient descent doesn't overshoot the minima.
Optimizer: Choose ADAM optimizer over the others like SGD. It works well.


How to use random forest model to new data?

Yes, you can do that. However, how accurate your new labels are depends on the ability of your model to generalize to new data and the similarity of the new data to your training data. Therefore, it is something you need to test for in the first place by using a separate test dataset to assess model performance. In contrast to that, a validation dataset is used for model selection and hyperparameter tuning (e.g. max depth of your trees).

An in-depth coverage of the topic you can find in chapter 7 of "The Elements of statistical Learning" by Hastie et al.

With regards to your second question: If your training data does not contain the variables prop_C and prop_D then your model cannot make use of them. So either include these in the training data or ignore them in the new data.


difference between model-based boosting and gradient boosting

Gradient Boosting is fitting a base learner $f_{i}(X)$ to the gradient of the loss function of an existing model $F_{i-1}(X)$ i.e. find base learner $f_i$ which minimises $L(-g_i, f_t(x_i))$ where $g_i$ is the gradient of $L(y_i,\hat{y}_i)$ with respect to $\hat{y}=F_{i-1}(X)$ at the current iteration $i$. Effectively it's gradient descent in function space.

Component wise boosting schemes such as that used by mboost have a list of base learners of which one is selected at each step, i.e.

form2 <- y ~ bols(x1) + bols(x2) + bols(x1, by = x2, intercept = FALSE) +
               bspatial(x1, x2, knots = 12, center = TRUE, df = 1)

Specifies 4 possible base learners, bols(x1), bols(x2), bols(x1,by=x2) and bspatial(x1,x2), all of which are regression splines.

More generally gradient boosted decision trees fits a tree at each step. So the base learners are arguably more complex.

I believe the terms model based and functional are equivalent and both `mboost' and GBDT are examples.


overfitting of regression tree from the beginning

Complexity Parameter CP describes a threshold $T$. If branch provides improvement less than $T$ it is deleted from the tree. You are using cp=0 so you told the algorithm to DO NOT prune any branches independly on their results! CP tested by printcp are very small also.

The bigger value of CP is, the more branches would be pruned so your tree should be smaller and should generalize better. Therefore you can try with highers values of CP at start and then print cross-validation results for getting suggested CP value and finally use prune method to reduce your tree for example:

model_pruned <- prune(rpart_model, cp=0.01)

You can compare your trees' structures using rpart.plot(tree_object) method. I think the original tree is way too complex and that is why it overfitts at start.


For theory Tibshirani: The elements of statistical learning
https://web.stanford.edu/~hastie/Papers/ESLII.pdf
Also Andrew NG and other books from deeplearning.ai:
Machine Learning Yearning
https://d2wvfoqc9gyqzf.cloudfront.net/content/uploads/2018/09/Ng-MLY01-13.pdf
of course the applied machine learning books on computer languages:
An introduction into statistical learning in R
http://faculty.marshall.usc.edu/gareth-james/ISL/
or deep learning with python:
http://faculty.neu.edu.cn/yury/AAI/Textbook/Deep%20Learning%20with%20Python.pdf
Of course there are a lot of free public pdf books in internet. However in my opinion for the beginning Tibshirani and Andrew NG are good. (and of course take your own project and ask questions on stackoverflow. That helps me a lot).


There's no problem in having negative correlations, in fact, if you change C to -C the correlation will be positive and every machine learning model I know should be equivalent trained on a variable or the same one with the sign changed.
So no, in any supervised case, regression or classification, it doesn't matter if there are negative correlations among features, and it happens almost every time.


How to automatically mount my Google Drive to Google Colab

For some reason, Colab allows me to automatically mount if I'm the creator of the notebook. If that's not the case, such was when I create a copy, the from google.colab import drive code block automatically gets added, and I have to authorize again.

File > New notebook

Then navigate to the file browser. Click on "Mount Drive".

From then on, every time I connect, it automatically mounts Drive during the Initializing phase.


What stopping metric to chose to optimize 'sensitivity' for a GBM in H2O?

One solution to make the recall more important whitout getting to the problem mentionned by @Jurgy is to use $F_\beta$, a modified version of $F_1$ where recall is considered $\beta$ time more important. As seen here : https://en.wikipedia.org/wiki/F1_score, $F_\beta$ can be formulated both in terms of recall/precision, and in term of type I /type II error. You then need to select a $\beta$ based on how much recall is more important to you... this can be done by considerations on the costs of type I vs the cost of type II errors.


Is Pearson coefficient a good indicator of dependency between variables?

As you show, Pearson coefficient is clearly not a good measure of how variables depend on each other. A better measure is distance correlation. A nice property of distance correlation is that distance correlation 0 implies independence. An easy example that can happen in real life is that a variable is the square of another variable, as you show in your first picture. In this case, the Pearson correlation will be 0, but the distance correlation will be around 0.5. I think it is a mistake for Kagglers to choose variables in terms of Pearson correlations.


I don't think clustering is the right approach here. You want to merge them automatically, but clustering is too unpredictable for this; you need to manually examine clustering results.

I suggest you look at near duplicate detection instead. You want to merge documents that have a high overlap. There are some interesting - and scalable - algorithma for that. In particular, minhash (and later work) is worth looking at.


I don’t know about your method 3, but I do know about quaternions and Euler angles. I was discussing them over a beer last night, believe it or not.

The orientation (attitude) at each point has 3 degrees of freedom so 3 Euler angles describe it completely. A quaternion has one extra element which is superfluous. The point of quaternions is that some geometrical calculations are easier than with Euler angles.

The fact that the relation between input data and output data is nonlinear is not a problem, if you have sufficient data and you have a nonlinear model (basically, unless you insist on logistic regression).

But you can make it far easier for your model to learn the relation if you transform your input, or compute a different input from your raw data, that has a simpler relation.

For example, as input for your model you could provide the Euler angles of the orientation of each foot, the lower leg, the upper leg. But you can also compute the bend at the knee and the ankle, and the model will probably learn from that more easily.


how to calculate number of datapoints within a given time interval?

The answer given by @Aditya is very good and educational despite the fact that the how is deprecated. So the right answer would be as follows:
df.resample("15T").agg({'count':'sum'})


How Many Features Are Used in Random Forest When max_features is not an integer?

The argument for the number of features gets passed down to the BaseDecisionTree class. Looking at the code, you can see that the value that is used is calculated by first taking log with base two and then converting it to an integer (i.e. rounding it).
if self.max_features == "auto":
    if is_classification:
        max_features = max(1, int(np.sqrt(self.n_features_)))
    else:
        max_features = self.n_features_
elif self.max_features == "sqrt":
    max_features = max(1, int(np.sqrt(self.n_features_)))
elif self.max_features == "log2":
    max_features = max(1, int(np.log2(self.n_features_)))
else:
    raise ValueError("Invalid value for max_features. "
                     "Allowed string values are 'auto', "
                     "'sqrt' or 'log2'.")


How can I predict traffic based on previous time series data?

The problem with models like KNN is that they do not take into account seasonality (time-dependent variations in trend).  To take those into account, you should use Time Series analysis.

For count data, such as yours, you can use generalized linear auto-regressive moving average models (GLARMA).  Fortunately, there is an R package that implements them (glarma).

The vignette is a good resource for the theory behind the tool.


Dataframe looks the same but the structure is different when loop

I finally got what was wrong, it has to do with the type in the series, so when i printed the whole dataframe the values where there, but the grouping was not been apply as it seems they were not number

I apply

df.MissingColumn = df.MissingColumn.astype(np.float64)

And this fixed the issue


This XPath will look for a tag called <Text> inside a tag called <posthistory>:

XPath(x, 'posthistory/Text');

You want to find the Text attribute of the row tag in posthistory tags.

An XPath something like this will do that: /posthistory/row/@Text

See example here: http://www.xpathtester.com/xpath/bac9874ec344f9d8ebcfb250633aaf65 and click "Test" to see results set.

Learn up on XPath notation for more.


Similarity between bordermode and zero padding in Keras

I think I found the answer to this here https://github.com/fchollet/keras/issues/1984. If the stride is 1, border_mode = 'same' does the job of padding to ensure that the output feature maps are the same size as the input.


What you have in your data called imbalanced classes

From Datacamp

  Imbalanced data typically refers to classification tasks where the
  classes are not represented equally.

  For example, you may have a binary classification problem with 100
  instances out of which 80 instances are labeled with Class-1, and the
  remaining 20 instances are marked with Class-2.

In this link, you can find a nice article that explains more what it is and how you can handle it -> https://www.datacamp.com/community/tutorials/diving-deep-imbalanced-data

One of the solutions is to use over or under-sampling. You can achieve this with the SMOTE algorithm. Here is an example in Python.

from imblearn.over_sampling import SMOTE
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

sm = SMOTE(random_state=2)
X_train_res, y_train_res = sm.fit_sample(X_train, y_train.ravel())


Optimization technique for iterative distribution

It looks like a resource allocation problem. I can't think of any statistical method which would help here, but maybe there is.
I think the process can be simplified by first calculating the amount in excess and in "shortage". I don't know if there's a formal method for this but I would try something like the following:

Distribute $a$ based on the weights. Then calculate for every entity:

its excess amount: X has 0, Y has 15 (75-60)
its missing amount: X has 0, Y has 0.
its capacity for additional amount. X has 15 (40-25) and Y has 0.
its capacity for giving away some amount. X has 15 (25-10) and Y has 20 (60-40)

Calculate the sum of excess amount $S_{excess}$, sum of missing amount $S_{missing}$, and capacity $S_{plus}$, $S_{minus}$. Let $S = S_{excess} - S_{missing}$:

If $S>0$

If $S>S_{plus}$, then no solution
Otherwise (1) fill every entity which has amount missing to their minimum, (2) distribute amount S by filling any entity which has capacity for more.

If $S<0$

If $|S|>S_{minus}$, then no solution.
Otherwise (1) unload every entity which has amount in excess to their maximum, (2) take total amount |S| from any entity which capacity for giving away.

If I'm not mistaken this requires only two steps, each step going over all the entities.


Is there a possible intersection of mathematical logic and machine learning?

I think the reason why formal logic is not widely used is its strictness. Also, it has quite restricted area of application and requires the domain to be formalized first. In contrast to that, ML methods can be fuzzy, robust to noise and work well for real-life data.

Here is an interesting discussion on the topic:
https://www.quora.com/Whats-the-relationship-between-mathematical-Logic-and-Machine-Learning


Is it appropriate to use polar images as input to CNNs? Or must they be Cartesian transformed first?

At the basic level, CNNs work by finding spatially-linked correlations i.e. places in the input which often appear close together. For this reason, creating cartesian projections of your data from the polar information sounds like the natural way to go about it.

If there is some inherent structure to your raw data, it might be possible to use something like a CNN, but you'd have to think carefully about the architecture.

Here are a couple of other thoughts that cross my mind that might help you brainstorm or find related research papers:

Casting the polar data into images, as you describe your method, not only increases the dimensions of the data, but usually the sparsity of the data. This makes it very difficult to train something like a CNN, which generally works best for dense chunks of information, such as a photograph. If your data is sparse, you might consider some other pre-processing such as adding blur to the projections.
Polar coordinates with a distance metric is essentially the same as a point cloud. If you represent a series as scans, instead of a set of images, but rather as a 3d pointcloud, you could look into models such a PointNet, PointNet++, VoxelNet (see the example projects at the bottom of that linked webpage). There are examples of object detection or segmentation in 3d pointclouds, which might give you other ideas for your case - all available openly in Tensorflow or another DL framework.
How well do you understand your data? Perhaps some further exploration or visualisation might help spark some ideas or at least provide a better feeling for approaches that could work. Try plotting the cartesian coordinates of your data (if you can map many frames to a global origin), using something like PyntCloud (an example).


If you have the ground truth value of the documents (their topics) all you gotta do is pick a metric and compare results. For classification problems, as yours, a common metric would be f1_score; reference: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html


What is the difference between ensemble methods and hybrid methods, or is there none?

Here is a para that I found by searching What are hybrid methods in Machine Learning, on google.

  "In general, it is based on combining two different machine learning techniques. For example, a hybrid classification model can be composed of one unsupervised learner (or cluster) to pre-process the training data and one supervised learner (or classifier) to learn the clustering result or vice versa." Along with that example, let's consider an example of Ensemble Learning that is Random Forest. *

In classical ensemble learning, you have different or similar algorithms, working on different or the same data-sets (for example Random Forest Stratifies the data set and builds different Decision Trees for those data-sets, while at the same time you can build different models on the same unstratified data-set and create an ensemble method). So in essence, you have different machine learning models, working independently of each other to give a prediction and then there is a voting system (hard or soft voting) which determines the final prediction.

According to the example of the hybrid machine learning model that we saw, the models in hybrid machine learning models essentially feed their output to one another (one-way) in an effort to create an efficient and accurate machine learning model. So the difference in both is that ensemble methods work independently to vote on an outcome while hybrid methods work together to predict one single outcome, which no voting element present in it.

*https://www.sciencedirect.com/science/article/pii/S1568494609001215


LeNet-5 - combining feature maps in C3 layer


How to encode data with a feature having multidimensional features (colors)?

One-hot encoding will give you a sparse matrix, Try LabelEncoding, before converting them into RGB that is.

Also, you could try breaking the RGB values into three features (R, G, B) and try that approach as well.

Hope this helps.


Deep learning model gives random results

I identified several problems when looking through your codes :

The first column for in.csv is a dummy variable for bias term (all 1 for the whole column). For NN layers in Keras unless you put include_bias=False, weights for bias will be included automatically hence this column would not be necessary.
Your variable is not scaled/standardized, although sometimes it will work without normalization but it is always recommended to do this for NN since input like that may lead to unstable training. Consider using StandardScaler or MinMaxScaler for starter.
You mentioned on the tags that you are working on classification so this is what you should do. First, identify the number of unique classes, this should be the number of hidden unit on the last layer. Next, you have two choices, if you want to use OneHotEncoder then when compiling your model your loss should be 'categorical_crossentropy'. The other choice, you first make sure that your label index starts from 0 and then use 'sparse_categorical_crossentropy' as your loss.
Softmax activation function applies the function AFTER the weights and biases operation. So the output of your model will first be a set of probability of the sample being some class and to get one output simply call .argmax(axis=1) which will return the index with the most probability on each row. You also need to do + 1 since your class labels starts from 1 instead of 0.


Coordinate System's influence on $L$ distances (Manhattan and Euclidean)

For example, consider the green line. What is its length? In $L_2$, the answer is $1$, in $L_1$, the answer is $1$ as well.

Now, for the same line, let's rotate it $45^\circ$ counterclockwise. What is the length again? In $L_2$, its length remains to be $1$. However, in $L_1$, using Manhattan distance, it's length is now $\frac{1}{\sqrt{2}}+\frac{1}{\sqrt{2}}=\sqrt2$. As we can see, rotation can change $L_1$ distance.

Rather than rotating the line, we could also equivalently rotated our $x$ and $y$-axis instead.


Tensorflow Fine-tuning a model from my own checkpoint

Yes, you can. After the below lines everything else will be same as what you have done previously to save the checkpoint.

with tf.Session() as sess:
  saver = tf.train.Saver()
  saver.restore(sess, tf.train.latest_checkpoint('./')) # checkpoint file path

Please refer to the below links:

https://nathanbrixius.wordpress.com/2016/05/24/checkpointing-and-reusing-tensorflow-models/

https://github.com/tensorflow/nmt/issues/51

https://blog.metaflow.fr/tensorflow-saving-restoring-and-mixing-multiple-models-c4c94d5d7125


Naive Bayes implementation: why Laplace smoothing is different from theory?

I think the answer is just that in the case of the Bernoulli model, your variables are binary variables which means the only values they can take are 0 or 1. Therefore the "number of features" also called "size of the vocabulary" in the case of text classification is limited to 2.


How to calculate the fold number (k-fold) in cross validation?

The number of folds is usually determined by the number of instances contained in your dataset. For example, if you have 10 instances in your data, 10-fold cross-validation wouldn't make sense. $k$-fold cross validation is used for two main purposes, to tune hyper parameters and to better evaluate the performance of a model.

In both of these cases selecting $k$ depends on the same thing. You must ensure that the training set and testing set are drawn from the same distribution. And that both sets contain sufficient variation such that the underlining distribution is represented. In a 10-fold cross validation with only 10 instances, there would only be 1 instance in the testing set. This instance does not properly represent the variation of the underlying distribution.

That being said, selecting $k$ is not an exact science because it's hard to estimate how well your fold represents your overall dataset. I usually use 5-fold cross validation. This means that 20% of the data is used for testing, this is usually pretty accurate. However, if your dataset size increases dramatically, like if you have over 100,000 instances, it can be seen that a 10-fold cross validation would lead in folds of 10,000 instances. This should be sufficient to reliably test your model.

In short, yes the number of folds depends on the data size. I usually stick with 4- or 5-fold. Make sure to shuffle your data, such that your folds do not contain inherent bias.


A regression technique that allow to predict a multidimensional output can be the PLS ( partial least square ). I implemented it in scala and it will be soon available on Clustering4Ever repo. In fact we went a bit further by applying it with the clusterwise pattern which generate k-clusters driving by PLS regression which result with one regression model per cluster which can be much more accurate but also most costly due to approximatively 1000.n regressions. You can look on it with, A new micro batch approach for partial least square clusterwise regression.


How to make a gaussian distribution in python considering mean. variance. skewness and kurtosis?

The Gaussian distribution is fully described by its mean and variance.
Gaussians have fixed values for Skewness (0) and Kurtosis (3) - so you can't really change them if you have made the Gaussian assumption for your model.


Feature construction widget on Orange 3.13

As per Feature Constructor widget documentation, the expression field should return values from $[0, n]$, and the Values field should contain $n+1$ comma-separated labels.


Find most important features which differentiate two sets

A reformulation of your question would be:

  Which are the features that differentiate the two datasets more accurately, because they differ a lot across them?

A common way to answer this is by checking the information gain of the attributes, based on the reduction of the entropy of the dataset. The attributes with the highest information gain are the ones that separate the dataset more accurately.

Assuming you have m attributes, you need to:

Label your samples based on which dataset they belong (A or B)
Attempt to split the dataset based on the values of each attribute. This means you need to perform m splits.
The split that maximizes the information gain, thus minimizes the individual sub-entropy of the two splitted groups, shows the attribute that separates the data the best. This attribute is the one that differs the most in both datasets and is the answer to your question.

FYI this is exactly what is done when training decision trees, to cut out (prune) features that are not informative enough in order to:

Reduce the tree's variance
Reduce training / inferencing time


Stationary time series for clustering algorithms

Have you tried replacing the entire timeseries with some meta data of it - clustering on each point of a series seems to be a bit exzessive. Especially as then if your timeseries are not synchronized they might not be considered similar - eventhough they might be for the purpose of this exercise (Imagine two sinus curves but one is slightly phase shifted -- If you would want to cluster by frequency then these would be equal).

So you could try to do something similar to Fourier transformations Fourier Transformations, linear fits, etc and use this meta data as input for the clustering.


Why use gradient descent on Deep Nets / RNNs when cost function is not convex?

Even if your cost function is not convex, any minimization algorithm will do its job - including gradient descent. That said, a standard gradient descent is a local optimization procedure and can only ever guarantee a local minimum, not a global one. For quite a few applications, a good local minimum is good enough - for others, a lot of effort is put into using “global” optimization algorithms (such as genetic/evolutionary, simulated annealing, scatter search, multi start techniques, direct/multilevel searches etc...). Those algorithms are definitely more powerful but they are also significantly slower in converging to a good, potentially global, minimum. And none of them can guarantee global optimality unless you allow them a veeeeeeery large number of functions evaluations.


There is no shortcut syntax that goes as far as accepting [2,3,4,5] as a param and create a model. However, it would be very simple for you to create this as a Python function yourself, provided in your case you already have made the decisions about activation functions, the types of layer etc. The need to make those decisions and have them available in the Keras API means that Keras itself does not offer such a short form build function.

You can make the model building slightly less verbose by using a list of layers when you instantiate the model, instead of adding them afterwards:

model = Sequential([
  Dense(8, input_dim=2),
  Activation('tanh'),
  Dense(1),
  Activation('sigmoid')
])

However, if you want to try variations of a model, where the only things you change are the number and size of hidden layers, then you can write a short Python function to encapsulate that requirement. Here's an example (I'm sure you already know how to do this, just included for completeness):

def build_model(hidden_layer_sizes):
  model = Sequential()

  model.add(Dense(hidden_layer_sizes[0], input_dim=2))
  model.add(Activation('tanh'))

  for layer_size in hidden_layer_sizes[1:]:
    model.add(Dense(layer_size))
    model.add(Activation('tanh'))

  model.add(Dense(1))
  model.add(Activation('sigmoid'))

  return model

If you take this approach, you may find you end up parametrising other choices such as input size (as you try some feature engineering), hidden layer activation function, whether to use a Dropout layer etc. Again, it is this need to define all the other choices in a typical network that lead to Keras' design. The best you can do is compress down the choices for your case, with a custom function.

I'd like to address this comment in your question:

  Although the network above is quite small, the current implementation may become frustrating with deeper networks.

In practice, I have not found Keras' design difficult to use for deep networks. I typically write a separate build function, and parametrise a few things, such as input dimensions. However, I typically don't loop through a list of different layer sizes as the main param of the build function. Instead, I find the more verbose approach just fine, even when trying variations of network size/shape (I guess that might change if I wanted to grid search including layer sizes). I think that is because I find the function names very easy to read - even with a screen full of .add() functions, I can see quite quickly what the NN structure is.