When to choose character instead of factor in R?

Factors are stored as numbers and a table of levels. If you have categorical data, storing it as a factor may save lots of memory.

For example, if you have a vector of length 1,000 stored as character and the strings are all 100 characters long, it will take about 100,000 bytes. If you store it as a factor, it will take about 8,000 bytes plus the sum of the lengths of the different factors.

Comparisons with factors should be quicker too because equality is tested by comparing the numbers, not the character values.

The advantage of keeping it as character comes when you want to add new items, since you are now changing the levels.

Store them as whatever makes the most sense for what the data represent. If name is not categorical, and it sounds like it isn't, then use character.


What is the different between Fine-tuning and Transfer-learning?

Generally, I would refer to this as transfer learning or network adaptation. That is, taking a network that has learned useful features from one domain and adapting that network and its developed features to another domain.

That said, there appear to be many sources that closely conflate fine tuning with transfer learning. Therefore, I would say the difference in terminology is primarily opinion-based and suggest closure of this question on those grounds.


How to model a supervised recommender system with varying data

In this task you're missing something: you don't have any features to represent a specific visitor.
This means that the best movie that your model can predict is the one which is selected the most often by any visitor. As a consequence, the only thing that the model can learn from such a dataset is to associate every possible sequence with the most frequently selected movie given this sequence. Of course it would have to generalize a bit for sequences which don't appear in the training data, but that's the most ML there is in this task. In theory the task can be done just by counting the joint frequency sequence+selected-movie in order to calculate the conditional probability selected-movie given sequence.

[edit]
The standard way to represent a set would be one hot encoding. It's doable even with 2000 features, assuming there are enough instances in the data. In this basic option you could certainly reduce the number of features by removing the movies which are never/rarely proposed or never/rarely selected. This would be unlikely to achieve performance, because if there are no or few cases where a movie is proposed/selected, then the model cannot use this information (or if it does it's going to cause overfitting).
I could think of a couple alternative approaches:

Apply clustering on the sequences of movies as OHE (without the target variable), then train a model with for every cluster: this way there are less features to deal with for every model.

Just rank all the movies by order of preference based on how often the movie is selected. Given that there is no user preference, it can be assumed that there is a total order on the movies:

the top preferred movie is always selected, whatever the other movies proposed

the 2nd top preferred movie is always selected, unless the first one is proposed as well

... the $n^{th}$ top is always selected, unless another one higher in the top is proposed as well

To predict the selected movie, just pick the highest ranked among the 3-5 proposed.


The paper that you cited references two other papers that explain the method you are looking for here and here.  The second one is behind a pay wall.

But... these look like deterministic methods to me, and I suggest you use a machine learning method instead.  You basically need to separate events from each other using some sort of clustering algorithm.  I suggest you use DBSCAN and adjust the parameters based on whether you want every event to be assigned to an episode or not.

Another algorithm that might work well is simple k-means clustering with the addition of an elbow method or silhouette score to determine the optimal number of clusters.


Intuitive explanation of difference between PCA and SVD

PCA.
Suppose that you have n data points comprised of d numbers (or dimensions) each. If you center this data (subtract the mean data point $\mu$ from each data vector $x_i$) you can stack the data to make a matrix

$$
X =  \left(
 \begin{array}{ccccc}
  && x_1^T - \mu^T && \\
  \hline
  && x_2^T - \mu^T && \\
  \hline
  && \vdots && \\
  \hline
  && x_n^T - \mu^T &&
 \end{array}
\right)\,.
$$

The covariance matrix

$$
S = \frac{1}{n-1} \sum_{i=1}^n (x_i-\mu)(x_i-\mu)^T = \frac{1}{n-1} X^T X
$$

measures to which degree the different coordinates in which your data is given vary together. So, it's maybe not surprising that PCA -- which is designed to capture the variation of your data -- can be given in terms of the covariance matrix. In particular, the eigenvalue decomposition of $S$ turns out to be

$$
S = V \Lambda V^T = \sum_{i = 1}^r \lambda_i v_i v_i^T \,,
$$

where $v_i$ is the $i$-th Principal Component, or PC, and $\lambda_i$ is the $i$-th eigenvalue of $S$ and is also equal to the variance of the data along the $i$-th PC. This decomposition comes from a general theorem in linear algebra, and some work does have to be done to motivate the relatino to PCA.

SVD is a general way to understand a matrix in terms of its column-space and row-space. (It's a way to rewrite any matrix in terms of other matrices with an intuitive relation to the row and column space.) For example, for the matrix $A = \left( \begin{array}{cc}1&2\\0&1\end{array} \right)$ we can find directions $u_i$ and $v_i$ in the domain and range so that

You can find these by considering how $A$ as a linear transformation morphs a unit sphere $\mathbb S$ in its domain to an ellipse: the principal semi-axes of the ellipse align with the $u_i$ and the $v_i$ are their preimages.

In any case, for the data matrix $X$ above (really, just set $A = X$), SVD lets us write

$$
X = \sum_{i=1}^r \sigma_i u_i v_j^T\,,
$$

where $\{ u_i \}$ and $\{ v_i \}$ are orthonormal sets of vectors.A comparison with the eigenvalue decomposition of $S$ reveals that the "right singular vectors" $v_i$ are equal to the PCs, the "right singular vectors" are

$$
u_i = \frac{1}{\sqrt{(n-1)\lambda_i}} Xv_i\,,
$$

and the "singular values" $\sigma_i$ are related to the data matrix via

$$
\sigma_i^2 = (n-1) \lambda_i\,.
$$

It's a general fact that the right singular vectors $u_i$ span the column space of $X$. In this specific case, $u_i$ give us a scaled projection of the data $X$ onto the direction of the $i$-th principal component. The left singular vectors $v_i$ in general span the row space of $X$, which gives us a set of orthonormal vectors that spans the data much like PCs.

I go into some more details and benefits of the relationship between PCA and SVD in this longer article.

Original post on crossvalid


Similarity Measure of Simulated Time Series vs Observed time Series

in my opinion you can do the following to compare your simulated with your actual data.

Use usual measurements like  those you would use in predictions vs actual data:

RMSE
MAE (Mean Absolut Error)
MSE (Mean Squared Error) --> Gives greater weight to larger errors/gaps

Another statistical test that comes to my mind to test "prediction" quality between different models is the Diebold-Mariano test. It is fully implemented in R. As far as I know, not in any Python Library so far.

As a statistical test you could have a look at the distribution (but as you dealing with time-series you may face "memory", meaning that observations t and t+1 face autocorrelation (against most of statistical tests, as they assume independent observations). Autocorrelation seems to be the case for your data series. Therefore you should have a look on the % change of your sample data (from t to t1) and look if both series are drawn from the same distirbution - You can perform the kolmogorov smirnov test.

Hope this helps.

EDIT

Regarding your question concerning smoothing and interpolation. Empiricis (not statistics) is more a trial and error concept. It really depends on your data behavior.


Learning resources for data science to win political campaigns?

This is an interesting and relevant question. I think that from data science perspective, it should not be, in principle, any different from any other similar data science tasks, such as prediction, forecasting or other analyses. Similarly to any data science work, the quality of applying data science to politics very much depends on understanding not only data science approaches, methods and tools, but, first and foremost, the domain being analyzed, that is politics domain.

Rapidly rising popularity of data science and machine learning (ML), in general, certainly has a significant impact on particular verticals and politics is not an exception. This impact can be seen not only in increased research interest in applying data science and ML to political science (for example, see this presentation, this paper, this overview paper and this whole virtual/open issue in a prominent Oxford journal), but in practical applications. Moreover, a new term - political informatics or poliInformatics or poli-informatics - has been coined to name an interdisciplinary field, which stated goal is to study and use data science, big data and ML in the government and politics domains. As I've said earlier, the interest in applying data science to politics goes beyond research and often results in politics-focused startups, such as PoliticIt or Para Bellum Labs. Following the unfortunate, but established trend in startup ecosystem, many of those ventures fail. For example, read the story of one of such startups.

I am pretty sure that you will be able to find neither proprietary algorithms that political startups or election data science teams used and use, nor the their data sets. However, I am rather positive that you can get some understanding about typical data sets as well as data collection and analysis methods via the resources that I have referenced above. Hope this helps.


Python + Neural Nets + Large dimension Large Dataset

I experimented with PyBrain and PyLearn2
I rather find Keras [http://keras.io/ ] as an excellent choice. It way ahead of PyBrain for sure. PyBrain development has been stopped


How many words should be taken as features in a ML problem?

It depends on many factors: number of instances, number of classes, token/type ratio, etc. A common basic technique in order to avoid having too many words is to discard the ones which appear rarely, for example by discarding all the words which appear in less than 3 documents. Selecting only the top $N$ most frequent words is unusual, even though it's a similar idea. There is a risk to end up mostly with words which appear frequently and don't have a good discriminative power.
The proper way to determine an optimal threshold, for example the minimum frequency $N$, is to run a training/testing experiment for every value of $N$ and select the value corresponding to maximum performance. Note that this experiment should use a validation set different from the final test set in order to avoid data leakage.


Estimating forward velocity for a swimmer

Your most important question is not the algorithm. You must check the precision of the sensor you can buy. I've tried several sensors, after 50 seconds the cumulated error can be a magnitude higher then the velocity itself. If you can buy a 1000 USD sensor, then it's precision would be sufficient, but you still have the problem that there will be no 0 position and 0 velocity, ie. the sensor will be moving even if the swimmer has not yet started to swim. This movement of the sensor will be greater then the difference between the velocities of two swimmers.


Gradient descent and partial derivatives

What's the point?

First, it is good to understand what we are doing that leads us to need these tools. When we are trying to apply machine learning we want to infer some meaning from data. This means that given an instance we want to put it through our model and then we can have some output that tell us something about this data.

Let's look at the example where we try to label whether an image contains a cat or a dog. If we have a perfect model, we can give the model a picture and it will tell us if it is a cat or a dog. However, no model is perfect and it will make mistakes.

When we train our model to be able to infer meaning from input data we want to minimize the amount of mistakes it makes. So we use a training set, this data contains a lot of pictures of dogs and cats and we have the ground truth label associated with that image. Each time we run a training iteration of the model we calculate the cost (the amount of mistakes) of the model. We will want to minimize this cost.

Many cost functions exist each serving their own purpose. A common cost function that is used is the quadratic cost which is defined as

$C = \frac{1}{N} \sum_{i=0}^{N}(\hat{y} - y)^2$.

This is the square of the difference between the predicted label and the ground truth label for the $N$ images that we trained over. We will want to minimize this in some way. The cost is also commnly called the loss function.

So we have a minimization problem on our hands!

Indeed most of machine learning is simply a family of frameworks which are capable of determining a distribution by minimizing some cost function. The question we can ask is "how can we minimize a function"?

Let's minimize the following function

$y = x^2-4x+6$.

If we plot this we can see that there is a minimum at $x = 2$. To do this analytically we can take the derivative of this function as

$\frac{dy}{dx} = 2x - 4 = 0$

$x = 2$.

However, often times finding a global minimum analytically is not feasible. So instead we use some optimization techniques. Here as well many different ways exist such as : Newton-Raphson, grid search, etc. Among these is gradient descent.

How does gradient descent work?

Let's use a famously used analogy to understand this. Imagine a 2D minimization problem. This is equivalent of being on a mountainous hike in the wilderness. You want to get back down to the village which you know is at the lowest point. Even if you do not know the cardinal directions of the village. All you need to do is continuously take the steepest way down, and you will eventually get to the village. So we will descend down the surface based on the steepness of the slope.

Let's take our function

$y = x^2-4x+6$

we will determine the $x$ for which $y$ is minimized. Gradient descent algorithm first says we will pick a random value for $x$. Let us initialize at $x=8$. Then the algorithm will do the following iteratively until we reach convergence.

$x^{new} = x^{old} - \nu \frac{dy}{dx}$

where $\nu$ is the learning rate, we can set this to whatever value we will like. However there is a smart way to choose this. Too big and we will never reach our minimum value, and too large we will waste soooo much time before we get there. It is analogous to the size of the steps you want to take down the steep slope. Small steps and you will die on the mountain, you'll never get down. Too large of a step and you risk over shooting the village and ending up the other side of the mountain.

$\frac{dy}{dx} = 2x - 4$

$\nu = 0.1$

Iteration 1:

$x^{new} = 8 - 0.1(2 * 8 - 4) = 6.8 $
$x^{new} = 6.8 - 0.1(2 * 6.8 - 4) = 5.84 $
$x^{new} = 5.84 - 0.1(2 * 5.84 - 4) = 5.07 $
$x^{new} = 5.07 - 0.1(2 * 5.07 - 4) = 4.45 $
$x^{new} = 4.45 - 0.1(2 * 4.45 - 4) = 3.96 $
$x^{new} = 3.96 - 0.1(2 * 3.96 - 4) = 3.57 $
$x^{new} = 3.57 - 0.1(2 * 3.57 - 4) = 3.25 $
$x^{new} = 3.25 - 0.1(2 * 3.25 - 4) = 3.00 $
$x^{new} = 3.00 - 0.1(2 * 3.00 - 4) = 2.80 $
$x^{new} = 2.80 - 0.1(2 * 2.80 - 4) = 2.64 $
$x^{new} = 2.64 - 0.1(2 * 2.64 - 4) = 2.51 $
$x^{new} = 2.51 - 0.1(2 * 2.51 - 4) = 2.41 $
$x^{new} = 2.41 - 0.1(2 * 2.41 - 4) = 2.32 $
$x^{new} = 2.32 - 0.1(2 * 2.32 - 4) = 2.26 $
$x^{new} = 2.26 - 0.1(2 * 2.26 - 4) = 2.21 $
$x^{new} = 2.21 - 0.1(2 * 2.21 - 4) = 2.16 $
$x^{new} = 2.16 - 0.1(2 * 2.16 - 4) = 2.13 $
$x^{new} = 2.13 - 0.1(2 * 2.13 - 4) = 2.10 $
$x^{new} = 2.10 - 0.1(2 * 2.10 - 4) = 2.08 $
$x^{new} = 2.08 - 0.1(2 * 2.08 - 4) = 2.06 $
$x^{new} = 2.06 - 0.1(2 * 2.06 - 4) = 2.05 $
$x^{new} = 2.05 - 0.1(2 * 2.05 - 4) = 2.04 $
$x^{new} = 2.04 - 0.1(2 * 2.04 - 4) = 2.03 $
$x^{new} = 2.03 - 0.1(2 * 2.03 - 4) = 2.02 $
$x^{new} = 2.02 - 0.1(2 * 2.02 - 4) = 2.02 $
$x^{new} = 2.02 - 0.1(2 * 2.02 - 4) = 2.01 $
$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.01 $
$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.01 $
$x^{new} = 2.01 - 0.1(2 * 2.01 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $
$x^{new} = 2.00 - 0.1(2 * 2.00 - 4) = 2.00 $

And we see that the algorithm converges at $x = 2$!

Where do partial derivatives come into this?

We just minimized $y$ for $x$. But if we had a multi-dimensional $x$ then we will need to identify where is the steepest slope in each dimension. To do this we will take the derivative across each dimension. In 2 dimensions gradient descent is performed using the gradient of the function $\nabla y$.

For the function

$y = 2x_1^2 + 5x_1 - 4x_2^2$

we will take the gradient in both dimensions defined as

$x^{new} = x^{old} - \nu \nabla y$.

The operator $\nabla$ is called the gradient it is defined as

$\nabla y = \frac{\partial y}{\partial x_1} \hat{\textbf{x}_1} + \frac{\partial y}{\partial x_2} \hat{\textbf{x}_2}$.

$\hat{\text{x}_1}$ and $\hat{\text{x}_2}$ are the two dimensions that we are optimizing over. So in our analogy we will go down this 2D surface a bit in the $x_1$ dimension and the $x_2$ dimension. Kind of like going down the mountain in the North-West direction. A bit in front and a bit to our left. That's a 2D direction!

The total size of the step that we take across all dimensions is then described by

$\Delta y = \frac{\partial y}{\partial x_1} \Delta x_1 + \frac{\partial y}{\partial x_2} \Delta x_2$.

Relating all this to your question

Change our function $y(x_1, x_2)$ by $C(v_1, v_2)$. The size of the step we are taking is defined by

$\Delta C = \frac{\partial C}{\partial v_1} \Delta v_1 + \frac{\partial C}{\partial v_2} \Delta v_2$.

At each gradient descent iteration we will update $v$ as

$v^{new} = v^{old} - \nu \nabla C$

where

$\nabla C = \frac{\partial C}{\partial v_1} \hat{\textbf{v}_1} + \frac{\partial C}{\partial v_2} \hat{\textbf{v}_2} = (\frac{\partial C}{\partial v_1}, \frac{\partial C}{\partial v_2})^T$

Supplemental

2D gradient descent of a function

$y = x_1^2 - x_1 + x_2^2 - 3x_2$

The partial derivatives are

$\frac{\partial y}{\partial x_1} = 2x_1 - 1$

$\frac{\partial y}{\partial x_2} = 2x_2 - 3$

Thus,

$\nabla y = (2x_1 - 1, 2x_2 - 3)$.

Our update rule with a learning rate $\nu = 0.1$ will then be

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.0 \\
0.0
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.0 - 1 \\
2 * 0.0 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.1 \\
0.3
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.1 \\
0.3
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.1 - 1 \\
2 * 0.3 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.18 \\
0.54
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.18 \\
0.54
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.18 - 1 \\
2 * 0.54 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.24 \\
0.73
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.24 \\
0.73
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.24 - 1 \\
2 * 0.73 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.29 \\
0.88
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.29 \\
0.88
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.29 - 1 \\
2 * 0.88 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.33 \\
1.00
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.33 \\
1.00
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.33 - 1 \\
2 * 1.00 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.36 \\
1.10
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.36 \\
1.10
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.36 - 1 \\
2 * 1.10 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.39 \\
1.18
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.39 \\
1.18
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.39 - 1 \\
2 * 1.18 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.41 \\
1.24
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.41 \\
1.24
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.41 - 1 \\
2 * 1.24 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.43 \\
1.29
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.43 \\
1.29
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.43 - 1 \\
2 * 1.29 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.44 \\
1.33
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.44 \\
1.33
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.44 - 1 \\
2 * 1.33 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.45 \\
1.37
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.45 \\
1.37
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.45 - 1 \\
2 * 1.37 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.46 \\
1.39
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.46 \\
1.39
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.46 - 1 \\
2 * 1.39 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.47 \\
1.41
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.47 \\
1.41
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.47 - 1 \\
2 * 1.41 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.47 \\
1.43
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.47 \\
1.43
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.47 - 1 \\
2 * 1.43 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.48 \\
1.44
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.48 \\
1.44
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.48 - 1 \\
2 * 1.44 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.48 \\
1.45
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.48 \\
1.45
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.48 - 1 \\
2 * 1.45 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.48 \\
1.46
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.48 \\
1.46
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.48 - 1 \\
2 * 1.46 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.47
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.47
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.47 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.47
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.47
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.47 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.48
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.48
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.48 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.48
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.48
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.48 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.48
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.48
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.48 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.49 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.49 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.49 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)$

$
\left(\begin{array}{cc}
x_1^{new} \\
x_2^{new}
\end{array}\right) =
\left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)
- 0.1
\left(\begin{array}{cc}
2 * 0.49 - 1 \\
2 * 1.49 - 3
\end{array}\right)
 = \left(\begin{array}{cc}
0.49 \\
1.49
\end{array}\right)$

And we have convergence in 2D at those points above!


what's the difference between autoencoder and autoassociative neural networks?

The "autoassociative" paper is from 1992. The field had not settled on the term "autoencoder" for this concept.


Association Rule Mining across two market baskets

Option 1

What a good option can be is to group those purchases into individual itemsets and explore by customer.
Consider the following example:

Customer 001 goes to store A and buys: {Banana, Milk}. Then he goes to store B and buys {Eggs, Oranges}.
Next, call {Banana, Milk} as product 01 and {Eggs, Orange} as product 001.

Customer 002 goes to store A and buys: {Banana, Milk}. Then he goes to store B and buys {Apple, Watermelon}.
Now, using the previous customer's data, you see that {Banana, Milk} already exists and you also call it product 01, whereas {Apple, Watermelon} is a new product that you call product 002.

You perform this for all of the itemsets for all customers.

In the above example, your Customer 001 total receipt includes: product 01, product 001. Your Customer 002 total receipt includes: product 01, product 002.

Note: product with one leading zero (01) comes from store A, product with two leading zeros (001, 002) comes from store B.

Now you can perform the regular Market Basket Analysis using association rules in Python and get the rules you are looking for by identifying store products with leading zeros.

Option 2

Your other option is to encode products before hand. Using the above example, Banana and Milk can be coded as 1A, 2A (from store A). And Eggs, Orange, Apple, Watermelon can be coded 1B, 2B, 3B, 4B (from store B).

Then you combine it all in one dataset, and create a unique receipt number including all items so they are on one transaction.
For Customer 001, it will be {1A, 2A, 1B, 2B}, for Customer 002, it will be {1A, 2A, 3B, 4B}.

Now you can perform the regular Market Basket Analysis using association rules in Python and get the rules you are looking for by identifying store products by character A or B to find in which store a customer made a purchase.


That isn't easy to answer, as it highly depends on your taks. The only time where it is definitely recommended to remove a feature is when two attributes correlate perfectly (e.g. price and tax of a product).

Removing attributes that do not correlate perfectly could worsen your accuracy as information gets lost. But keeping them could affect the computation time. If you don't worry about computation time, I would highly recommend you to keep every bit of information about the data.

If you are concerned about overfitting there are way better techniques such as regularization to solve this problem.

Note: Dimensionality reduction is mostly used to visualize high dimensional data, so that a human eye can detect patterns. For example by shrinking the dimension to 3D or even 2D one could observe an exponential dependency.


Gramatically "Meta" comes from greek and means beyond something. "Meta-Feature" means beyond the feature.
Normally refers to how the feature was obtained, the uncertainty, error measurements...
For example, in atmospheric modeling, for the features "Temperature" you can have a lot of different meta features like, with which tool are you measuring the temperature? what is the error of the tool? in which conditions is the measurement happening?
I understand the term meta feature as a generalization of this concept but in Data Science.


You are multiplying matrices, so the dimensions have to match up.

The following line reshapes the embedding layer to a rank 1 matrix, hence causing the error.

embedded_chars = tf.reshape(embedded_chars, [-1])

You may want to rethink the embedding portion of the code (the function's docstring), as that is what needs to be understood perfectly in order to define the computational graph.


For the problem of overfitting, you could look train models that employ regularization. For instance this examples shows how to regularize an SVM.

Another thing I noted is that you have used the tag "unbalanced-classes". If that is the case, accuracy isn't a very good metric. While AUC is good at this, I've personally had trouble with this metric in the past. My suggestion would be to include a metirc like F1-score and most importantly in each case calculate the confusion matrix. This will show you if you are missing one class more than the other. If that is the case you might want to incorporate an oversampling method (e.g. SMOTE) into your pipeline.


Tips on how to preprocess data and outliers for churn analysis

1) when the variable follows the non normal distribution you can use different transformation to make it normal, if variable follows the skewed distribution(either positive or negative) you can use log transformation to make it normal.

2) and 3) Instead of removing the outlier always go through outlier capping method, it will help to improve the performance of the model.

4) In machine learning some algorithm need to scale the data before go to modeling. for ex. cluster Analysis, Principal component Analysis(PCA). we can use scaling when the some values of the variable is too high as compare to the values of the other variable. (1,2,3,56,900,100,34,22,9)
while using such type of algorithm always go to scaling instead of transformation.


Gumbel Softmax vs Vanilla Softmax for GAN training

Passing directly the output of the softmax is also common (among the few textual GANs out there), e.g. see the improved Wasserstein GANs (WGAN-GP).

With hard Gumbel-softmax (+ straight-through estimator), you pass one-hot encoded vectors, which is the same as what you have with real data. If you pass the output of the softmax, the discriminator should be able to more easily tell apart real data (one hot) from fake data (non-one hot).

That being said, in my opinion neither of the two approaches seems very promising nowadays. There seems to be far more REINFORCE-based textual GANs.


First, Huber loss only works in one-dimension as it requires $$\left\|\boldsymbol{a}\right\|_2=\left\|\boldsymbol{a}\right\|_1=\delta$$at the intersection of two functions, which only holds in one-dimension. Norms $L_2$ and $L_1$ are defined for vectors. Therefore, in my opinion, Huber loss better be compared with squared loss rather than $L_2$ loss, since "$L_2$" presumes a multi-dimensional input compared to "squared".

Huber loss is the same as squared loss for differences less than $\delta$, and the same as absolute loss for differences larger than $\delta$, i.e.
$$\begin{align*}
L_{\delta}(y_n, f_{\theta}(\boldsymbol{x}_n))
=\left\{
\begin{matrix}
\frac{1}{2}\left(y_n - f_{\theta}(\boldsymbol{x}_n)\right)^2 & \left|y_n - f(\boldsymbol{x}_n)\right| \leq \delta,\\
\delta\left|y_n - f_{\theta}(\boldsymbol{x}_n)\right| - \frac{1}{2}\delta^2, & \text{otherwise.}
\end{matrix}
\right.
\end{align*}$$

where $y_n$ is the target of data point $n$, and $f_{\theta}(\boldsymbol{x}_n)$ is model's prediction. Note that $L_{\delta}$ has nothing to do with $L_p$ norm, despite the similar notations.

Because of this definition, for large differences due to outliers, gradient of loss function remains constant $\pm \delta$, the same as absolute loss, i.e.
$$\frac{\partial \delta\left|y_n - f_{\theta}(\boldsymbol{x}_n)\right|}{\partial \theta_i} = \pm \delta \frac{\partial f_{\theta}(\boldsymbol{x}_n)}{\partial \theta_i}$$
compared to squared loss, where gradient increases with the difference, i.e.
$$\frac{\partial \frac{1}{2}\left(y_n - f_{\theta}(\boldsymbol{x}_n)\right)^2}{\partial \theta_i} = -\left(y_n - f_{\theta}(\boldsymbol{x}_n)\right)\frac{\partial f_{\theta}(\boldsymbol{x}_n)}{\partial \theta_i}$$

which leads to large contributions from outliers when we update a parameter solely based on squared loss as follows:
$$\begin{align*}
{\theta}'_i &=\theta_i + \lambda \sum_n \frac{\partial f_{\theta}(\boldsymbol{x}_n)}{\partial \theta_i}\left(y_n - f_{\theta}(\boldsymbol{x}_n)\right) \\
&= \theta_i + \lambda\sum_{n \notin \text{outliers}} \frac{\partial f_{\theta}(\boldsymbol{x}_n)}{\partial \theta_i}(\text{small}) +\lambda\sum_{n \in \text{outliers}} \frac{\partial f_{\theta}(\boldsymbol{x}_n)}{\partial \theta_i}(\text{large})
\end{align*}$$

It is worth noting that, here, outliers are irregularities in the joint input-output space $(\boldsymbol{x}_n, y_n)$, not necessarily just in the input space $\boldsymbol{x}_n$ as we usually visualize in unsupervised tasks. For example, in a linear trend, none of $(x, y)=\{(1, 2)$, $(5, 10)$, $(10, 20)\}$ are outliers, but $(1, 10)$ is, which leads to large difference $(10 - 2)$ when model expects (predicts) $f_{\theta}(1)=2$.

  When to use each of them?

Reminding that we are only talking about one-dimensional targets, Huber loss is a complete replacement for squared loss to deal with outliers. However, the challenge is the choice of $\delta$, which makes it a less favorable "first choice" when we are not yet familiar with the problem at hand. Therefore, we may start with squared loss (or other losses), and after a while try to experiment with Huber loss for different values of $\delta$.


AttributeError: module 'keras.backend' has no attribute 'backend'

Do you have multiple versions of keras installed? Did you try reinstalling? Sorry for posting as awnser but I don't have enough reputation to comment

pip uninstall keras
pip install keras


Unsurprisingly most programs in data science focus mostly on business or marketing applications. I too would like to see more programs focused on Biotechnology, Gaming or Energy for example but I think that in the future those kind of programs will become more and more available.
However there are a few I know of that offer a somewhat different approach.

Regarding health informatics there are many offers where you can find anything between Public health to Computational Biology and Quantitative Genetics. Take a look at the full list.

http://www.mastersindatascience.org/specialties/health-informatics/

On that list there are a few certification programs but there also are full fledged programs that lead to a M.Sc.

If you are more interested in Marketing Analytics you can find the same list but for marketing related programs.

http://www.mastersindatascience.org/specialties/marketing-analytics/

other options like Information Systems are also present, just browse that site and you might find something that interests you.


Let us say $K$ is an $m \times n$ matrix.
You want to find a vector $a$ that maximizes $\frac{||Ka||}{||a||}$.
Now maximizing $\frac{||Ka||}{||a||}$ implies maximizing its square:
$$\begin{eqnarray*}
\frac{||Ka||^2}{||a||^2} \\
= \frac{(Ka)^TKa}{a^Ta} \\
= \frac{a^TK^T Ka}{a^Ta} \\
= \frac{a^T(K^TK) a}{a^Ta} \\
= \frac{a^TSa}{a^Ta}
\end{eqnarray*}
$$
where $S = K^TK$ is always an $n \times n$ symmetric positive semidefinite matrix for all $K$
Now symmetric positive semidefinite matrices have several special properties one of which is that they always have REAL eigen values. Also the eigen vectors associated with these eigen values are always orthogonal to each other.
What this means is that if you locate the eigen vector for $S$ with the highest eigen value - you have basically found the vector that maximizes:
$\frac{a^TSa}{a^Ta}$
So in essence the eigen vector with the largest eigen value for $S$ or $K^TK$ maximizes both $\frac{a^TSa}{a^Ta}$ and the original $\frac{||Ka||}{||a||}$
Use the same analogy for the remainder eigen vectors - i.e the second best vector that maximizes $\frac{a^TSa}{a^Ta}$ is the eigen vector corresponding to the second highest eigen value.


Out of Memory Error when Selecting Data from Redshift Table

First, you are trying to access a big dataset with sqlalchemy, while specialized packages like bigquery would be a more suitable choice. I suggest learning about it on https://www.kaggle.com/learn/intro-to-sql
Also, I think that you get more data than your device can handle. Maybe setting up a limit on your data will help.
data_frame = pd.read_sql_query('SELECT * FROM table_name LIMIT 1000000;', engine)


How to achieve SHAP values for a CatBoost model in R?

catboost::catboost.get_feature_importance(model, pool = pool, type = "ShapValues")


Definition - center of the cluster with non-Euclidean distance

The easiest solution for a non-Euclidean cluster center is the medoid, as in the algorithm PAM. This works with arbitrary metrics, but unfortunately for a 2-element cluster the result by definition is random (by metric properties, each point is an equally good medoid).

Or you step back and rethink what the center is. It is the point which has the least squared error. So "all" you need to do is solve some math for your distance function, how to optimize this least-squares estimation. People have been solving such problems since Gauss. For example with the deviation abs(x-mu) the least-squares objective is minimized by the arithmetic mean, an the one dimensional result carries over to the n-dimensional case and thus Euclidean distance!

Obviously, this needs to be done for every distance function again.

So if you want to interpolate a center for an other distance function, solve that optimization problem (efficiently, but try to find an existing solution first, of course).


How to treat undefined input space in supervised learning algorithms?

It depends on the machine learning technique. The following link gives a really good example comparing many classifiers and their segregation area. The same happens with clustering methods.

Classifier comparison


word2vec: usefulness of context vectors in classification

So, from what I understand from the question, you want to get an idea of how word2vec works so you can assess how well the resulting context vectors from this model will help discriminate between words by their meaning.
Word2vec works on the premise of the distributional hypothesis which essentially states that words which appear in soimikar contexts  will have similar meanings (e.g. the dog ate the food/ the cat ate the food : both dog and cat appear in the same context so they are semantically close to each other)
So word2vec formulates this by a CBOW model which effectively is a feedforward neural network, which takes in the surrounding context of the target word as a series of one hot encoded vectors and aims to predict the target word (there is an assumption made of course where the contexts are treated as a bag of words and therefore assumes that a word’s meaning is not related to the word ordering of its context).
After training this, the models weights are then used to dorm the word embeddings, which represents a words meaning in semantic space. (For further reference)


What you could try is to make a transformation of the price data, i.e. log(price). By doing so, very large values become smaller. The data is more "harmonic" after the transformation.


What's the difference between transfer learning and feature extraction in CNN?

Transfer learning is, as you said, retraining the final layer of a deep network. Not only is this useful for solving problems with limited training examples, but also when you don't have adequate computing resources to train a network from scratch. Some models have hundreds of millions of parameters, which could take weeks to train if you only have modest equipment. However, if you have sufficient data, adapting weights via transfer learning is not preferable because the features that were extracted from the original training process are unlikely to be ideal for another application.
Feature extraction, in the context of CNNs, is not necessarily an explicit process, rather a sort of high-level product of the training process. Feature extraction refers to the portion of the training process by which a CNN learns to map input space to a latent space that can subsequently be used for classification via the final layer. In other words, the hidden layers learn discriminatory features in the form of weight-adjusted (usually by backpropagating the error) convolutional filters. Thus the term "feature extraction" generally refers to the portion of the training process that occurs before the final layer. So it is not part of transfer learning in which only the last layer is trained.


You will have to use a mix of text processing and one hot encoding. Text column should not be treated as one-hot encoded since it will try to create one new variable for every unique sentence in the dataset, which will be a lot (and not very helpful from learning). Text vectorizer will summarize text column based on type of words/tokens that appear in it.
So you should use a text vectorizer for processing only the text column first. This will give you one data-frame (say A). This data-frame will have columns corresponding to tokens/words in the dataset. So if text vectorizer picks up 100 unique words then you will have a 1000x100 size data-frame. Note that these 100 columns have been generated only by text vectorizer.
For symbols and notes, you can use one-hot encoding, which will get you another data-frame (say B). Then you should join A & B on common key to get your final data-frame if input. The common key here will be row ID (though read the following comment on aggregating data at user level).
It is not clear if the user name (Account) column unique in the data? If there are 1000 rows but only 400 users, that means there can be more than 1 row per user. In that case, you can consider aggregating the data at the user level (for text column, you can simply concat all strings for same user).


Techniques for trend extraction from unbalanced panel data

I would add a third column called month and then concatenate each list. So if you have a top 100 list for 5 months you will create one big table with 500 entries:

User-id | Threat_score | month
aaa       45             1
bbb       32             1
ccc       20             1
...       ...            ...
bbb       64             2
ccc       29             2
...       ...            ...

Then, to answer your first question, you could simply count the occurrences of each user-id. For example, if user bbb is in your concatenated table five times, then you know that person made your list all five months.

To answer you second question, you could do a group by operation to compute some aggregate function of the users. A group by operation with an average function is a little crude and sensitive to outliers, but it would probably get you close to what you are looking for.

One possibility for the third question is to compute the difference in threat score between month n-1 and month n. That is, for each month (not including the first month) you subtract the user's previous threat score from the current threat score. You can make this a new column so your table would now look like:

User-id | Threat_score | month | difference
aaa       45             1       null
bbb       32             1       null
ccc       20             1       null
...       ...            ...     ...
bbb       64             2       32
ccc       29             2       9
...       ...            ...

With this table, you could again do a group by operation to find people who consistently have a higher threat score than the previous month or you could simply find people with a large difference between the current month and the previous month.

As you suggest, visualizing this data is a really good idea. If you care about these threat scores over time (which I think you do), I strongly recommend a simple line chart, with month on the x-axis and threat score on the y-axis. It's not fancy, but it's extremely easy to interpret and should give you useful information about the trends.

Most of this stuff (not the visualization) can be done in SQL and all of it can be done in R or Python (and many other languages). Good luck!


Features impacting individual predictions after training a classification model

Depending on what model you are using, the way things are multiplied and added together at prediction time slightly varies, but in principal it's all the same.  You train a model that tries to learn the relationship of the feature vectors you give it to the associated targets.  At prediction time, the same mathematical operations are being applied to every input you are trying to make a prediction for.  So, the weights you learned during the training phase represent the feature importance across the entire training set, in which you hope generalizes to unseen "test" data.  If you suspect that there are other reasons behind why an employee decided to resign outside of what the model weights are telling you, this would indicate that your model does not have enough features to separate the two classes of "not resign" vs. "resign".  Essentially, during testing time, the feature importance is static and the exact same across any input you give.  If feature1 is the most important for input 1, its the most important for every input.  This is what the model learned.


Preparing for interview - Logistic regression question

They give you the current values of the model parameters $a$ and $b$, and a new data point $(x,y)$, and they request to perform one training step with gradient descent using the data point, and returning the updated values for $a$ and $b$.
The problem with your code is that the sign of the learning rate is wrong in the parameter update. If you change it to minus, the result is as expected:
new_weights = weights - learnrate * (y - y_hat) * X
new_bias = bias - learnrate*(y - y_hat)


If anyone's wondering, yes it is right. The weight-clipping WGAN is very sensitive to changes in the weight-clipping hyperparameter, which is probably causing this. Upgrading to the gradient descent version fixed it. Loss should look like this btw


How is dimensionality reduction achieved in Deep Belief Networks with Restricted Boltzmann Machines?

As you know, a deep belief network (DBN) is a stack of restricted Boltzmann machines (RBM), so let's look at the RBM: a restricted Boltzmann machines is a generative model, which means it is able to generate samples from the learned probability distribution at the visible units (the input). While training the RBM, you teach it how your input samples are distributed, and the RBM learns how it could generate such samples. It can do so by adjusting the visible and hidden biases, and the weights in between.

The choice of the number of hidden units is completely up to you: if you choose to give it less hidden than visible units, the RBM will try to recreate the probability distribution at the input with only the number of hidden units it has. An that is already the objective: $p(\mathbf{v})$, the probability distribution at the visible units, should be as close as possible to the probability distribution of your data $p(\text{data})$.

To do that, we assign an energy function (both equations taken from A Practical Guide to Training RBMs by G. Hinton)
$$E(\mathbf{v},\mathbf{h}) = -\sum_{i \in \text{visible}} a_i v_i - \sum_{j \in \text{hidden}} b_j h_j - \sum_{i,j} v_i h_j w_{ij}$$
to each configuration of visible units $\mathbf{v}$ and hidden units $\mathbf{h}$. Here, $a_i$ and $b_j$ are the biases, and $w_{ij}$ are the weights. Given this energy function, the probability of a visible vector $\mathbf{v}$ is
$$p(\mathbf{v}) = \frac 1Z \sum_{\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h})}$$
With that, we know that to increase the probability of the RBM generating a training sample $\mathbf{v}^{(k)}$ (denotes the $k$-th training sample), we need to change $a_i$, $b_j$ and $w_{ij}$ so that the energy $E$ for our given $\mathbf{v}^{(k)}$ and the corresponding $\mathbf{h}$ gets lower.


Difference between subgradient SVM and kernel SVM?

You are gravely misunderstanding SVM. Sub-gradient descent algorithm for SVM is a method to solve the underlying optimization problem of SVM.

SVM is always a linear classifier, which can yet by using kernels operate in a higher dimensional space. Therefore in input space, the separating hyperplane (linear!) computed in feature space (kernel!), seems non-linear.

Effectively you are thereby solving a non-linear classification task, but you are projecting into a higher dimensional feature space where the classification task is solved by a linear classifier.

Please read:

Wikipedia on svm

Please watch:

great lecture about SVM


error trying to build a histogram using r with the rStudio application

The following code :

data.frame(table(country$continent))

will create a dataframe where one column is the continent and the other is the number of countries in the continent.


Is the usage of the "momentum" significiantly superior to the conventional weight update

Using momentum is a noise reduction (noisy gradients) and signal amplification strategy.
Imagine a large hill with a rough terrain with lots of ups-and-downs. We are trying to navigate to the bottom of the hill by using purely local information. A bad strategy is course correct frequently every time we see a potential new direction with steeper descent.
The momentum term makes it difficult to course correct frequently by adding weight to the past directions chosen. It dampens (pun intended) the idea of enthusiastically switching directions.
The challenge with common consent: the exact nature of the terrain is going to change from problem to problem (dataset). So your mileage by choosing momentum based updates may vary. But in general, for non-toy problem using momentum based weight updates will be a good first strategy.


how can i leverage NLP features like SRL, LSA, POS, NER, entity type , relation type with deep learning to find semantic similarity of texts?

TL;DR:

Represent words as word vectors. Then add extra dimensions to the word vectors. In these extra dimensions, include POS, NER, etc. features in a numeric form.

Longer Version:

Say you have a word2vec/Glove model with each word represented by a 100 dimensional vector. Additionally, you have features like POS, NER, etc. for each word.
Instead of representing each word with just 100 dimensions, represent it with 100 + n dimensions and fill these n slots with the nlp features.

This will give the vectors an additional meaning which you have defined manually.

Suggestions

You could benefit from using other models like Doc2Vec/Sent2Vec which can represent a whole sentence as vector. With these vectors you could query similarity of sentences.

So say you've trained it on your data, just query :

model.most_similar_cosmul(SENT_42)

where SENT_42 represents "to whom i need to contact after i resign ?",
you will get a list of sentences similar to it.

I suggest you use gensim's doc2vec model in python.


Get the prediction probability using prediction function

I have followed scikit-learn https://github.com/scikit-learn/scikit-learn/blob/fd237278e/sklearn/linear_model/_base.py#L293 link and was able to get predicted probabilities.


Mxnet deepdog (hot dog not hot dog) example - how does the network know it is classifying a hotdog

You are on the right track. But you would have to realize that 713 is only, effectively, being used as an index lookup to get the weights just for that class and "seed" your new model. From there, model becomes a binary classification model where 0 is nothotdog and 1 is hotdog. Once the weights are picked up, 713 shouldn't be found anywhere else in your model.

So the output, initially, is going to be something [0.493, 0.507], each corresponding to the probability that the image belongs to one of the two classes. From there, the call with argmax() is looking for which probability is highest, according to the binary selection. So it would pick 0.507 and call it a hotdog since that's corresponds to the binary value of 1.

If you want to verify me on all this, go to your output code and do something like print(out.asnumpy()) and you'll see what I mean.


You can try this
wr.reset_index().plot(x='index', y=['a', 'b'], figsize=(10,5), grid=True)

The reset_index() will change the index back to column
The x = 'index' will use the date index as the x axis
The y=['a','b'] will plot both columns as two time series
figsize just tell the plot how big to be
grid=True will add a grid on the plot

You will need to give some extra 'love' to your styling to make it look exactly as your example.


In all the implementations for CNNs processing images that I have seen, the output in any layer is

Width x Height x Channels

or some permutation. This is the same number of dimensions as the input, no additional dimensions are added by the convolutional layers. Each feature map channel in the output of a CNN layer is a "flattened" 2D array created by adding the results of multiple 2D kernels (one for each channel in the input layer).

Usually even greyscale input images are expected to be represented as Width x Height x 1 so that they fit the same pattern and the same layer model can be used.

It is entirely feasible to build a layer design which converts a standard 2D+channels input layer into a 3D+channels layer. It is not something I have seen done before, but you can never rule out that it could be useful in a specific problem.

You may also see 3D+channels convolutions in CNNs applied to video, but in that case, the structure will be some variation of

Width x Height x Frames x Channels


solving multi-class imbalance classification using smote and OSS

If you convert your problem to a binary classification task, you do not need to worry about any conflict with re-sampling techniques. You can  then just use the imblearn.combine which combines Oversampling and under-sampling in one algorithm. There are classes available for this but I have found SMOTEENN to produce better results. SMOTEENN


library name should be defined along with libname statement and its path. try below

libname myprj1 'D:\Users\clkehl01\Documents\myprj1\';

data myprj1.autos;
set sashelp.cars;
run;


Comparing two time series data to find deviations between them

Your use-case looks solvable using dynamic time warping. DTW is an algorithm for comparing similarity between two temporal sequences.

Implementation of DTW are available in standard data-science programming language via libraries like this R implementation.


Correlations, p-values and features selection

The fact that a feature has low correlation with the target variable shows that it's not a good indicator on its own, but that doesn't mean that it can't be useful for the model when combined with the other features.
The only way to know if these features are useful is to use them to train a model, then evaluate on a validation set and see if it improves performance.

the p-value is less than 0.05

Is this the result of a correlation significance test? It depends on the test but in general a p-value lower than 0.05 means that there is a significant difference, i.e. in this case it probably means that the correlation is truly not zero. Anyway imho this wouldn't prove anything with respect to using these features or not.


Hyper-parameter tuning of NaiveBayes Classier

from sklearn.model_selection import GridSearchCV

hyper = {'C':[0.05,0.1,0.2,0.3,0.25,0.4,0.5,0.6,0.7,0.8,0.9,1],
         'gamma':[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0],
         'kernel':['rbf','linear']
        }

gd=GridSearchCV(estimator=svm.SVC(),param_grid=hyper,verbose=True)

gd.fit(X,Y)
print(gd.best_score_)
print(gd.best_estimator_)

Sources:

Hyperparameter Tuning of Machine Learning Model in Python

https://github.com/dataprofessor/code/blob/master/python/hyperparameter_tuning.ipynb
https://www.youtube.com/watch?v=AvWfL1Us3Kg


Transposing Every nth row to column in a large dataset

Hey is Python or other tools an option here? Since you mentioned it is a large dataset, you might want to iterate over it instead of loading all of them at once.

Here is a solution in Python:

import pandas as pd
from collections import defaultdict

inputs = [
'cat1/: Topic1_Variable1',
'cat2/: Topic1_Variable2',
'cat3/: Topic1_Variable3',
'cat4/: Topic1_Variable4',
'cat1/: Topic2_Variable1',
'cat2/: Topic2_Variable2',
'cat3/: Topic2_Variable3',
'cat4/: Topic2_Variable4',
'cat1/: Topic3_Variable1',
'cat2/: Topic3_Variable2',
'cat3/: Topic3_Variable3',
'cat4/: Topic3_Variable4',]

outputs = defaultdict(list)

for item in inputs:
    cat, topic = item.split('/: ')
    outputs[cat].append(topic)

print pd.DataFrame(outputs)

Output:

               cat1              cat2              cat3              cat4
0  Topic1_Variable1  Topic1_Variable2  Topic1_Variable3  Topic1_Variable4
1  Topic2_Variable1  Topic2_Variable2  Topic2_Variable3  Topic2_Variable4
2  Topic3_Variable1  Topic3_Variable2  Topic3_Variable3  Topic3_Variable4


Difficulty in choosing Hyperparameters for my CNN

I am going to comment on your ideas:

This is generally recommended. There is no proven reason not to do it but many reasons to do it, and you should initialize as much weights as possible. Typically it is done until the last layer.
Initialize everything that you can from pre-trained network.
I would freeze the first layers in the first epochs, and after that unfreeze them. If possible, I would rather use differential learning rates, as this post shows.
Totally, as shown above.
Adam is more widely used, I would use that.
Cosine scheduling with reestarts has been shown to be very effective, but any learning rate with reestarts should be similar.
It does make sense.
I wouldn't worry about that until you are overfitting, and you are not overfitting yet. If you overfit, I would recommend dropout instead of weight decay.


How does one fine-tune parameters and weights at the same time?

You can test each model with default parameters, and keep the model that seems best.

Or, for each model, you can tune the parameters for that model, to see what is the best that's achievable for each model (by separately tuning its parameters) -- then keep the best model.  This takes more computation power but might be slightly better.

Yes, parameter tuning depends on the model.  Each model may have a very different set of parameters, with entirely different meaning. You can't take parameters for a random forest and then try to use them as parameters for a SVM (for instance); that doesn't even make sense, as they have different parameters.

Yes, there are many possibilities on what order you do feature selection vs choosing a model vs parameter tuning.

I don't know what the "feature weights of FeatureUnion" are, so I can't answer that part.


The question,

whatever I'm doing is it correct or ...

will occur over and over again, until you have a firm understanding of statistics. See if you are looking for short-term learning, then you might pick up some fragments of information here and there, but, for long term learning, I'll suggest that you read a good text on introductory statistics. Try, this book, or this one on Applied Predictive Modeling
Now to answer the question, Yes what you are doing is correct. I've explained it below;
Outliers are observations that fall below the lower limit or above the upper limit. The five number summary of a continuous variable (or numerical variable) is Min, Q1, Q2, Q3, Max. Where the lower and upper limits are defined as, Lower Limit = Q1 - 1.5 x IQR, Upper Limit = Q3 + 1.5 x IQR and IQR is the Inter Quartile Range, which is the difference between the first and third quartile.
Q1 is the first quartile, Q2 is the second quartile or the median and Q3 is the third quartile.
Finally, please always plot your data.


What are the differences between Convolutional1D, Convolutional2D, and Convolutional3D?

Conv1D is used for input signals which are similar to the voice. By employing them you can find patterns across the signal. For instance, you have a voice signal and you have a convolutional layer. Each convolution traverses the voice to find meaningful patterns by employing a cost function.

Conv2D is used for images. This use case is very popular. The convolution method used for this layer is so called convolution over volume. This means you have a two-dimensional image which contains multiple channels, RGB as an example. In this case, each convolutional filter should be a three-dimensional filter to be convolved, cross-correlated actually, with the image to find appropriate patterns across the image.

Conv3D is usually used for videos where you have a frame for each time span. These layers usually have more parameters to be learnt than the previous layers. The reason we call them $3D$ is that other than images for each frame, there is another axis called time containing discrete values, and each of them corresponds to a particular frame.


In a SVD with user/video bias, why is the UV contribute so small?

I think that depends on the pattern of your data, its skewness and sparsity.
Because it would press these two bias term, which is designed to detect the difference with global mean.

For example, you can plot mean and std of the user's bias: ($R_u$ - global_mean) then make a histogram of std. Another way, just make up a rating matrix $R'$ according to the $\mu + B_u + B_v$ you've got. And run the solver again and see what happens. These parts play its own role, even your change on global_mean would make a difference to the final RMSE, I think the role of that latent factors are born to capture the REST non-linear pattern in user & item's interaction.

So, as for your question:

  Why does this part contributes so small?

Again, you can simply using $R = UV^T$, in this time, I think it contributes quite a lot, right? The contribution of each part is hard to say.


Order of normalization / augmentation for image classification

You can use both methods, in the vast majority of examples I have seen and worked with the usual order is.

Augmentate your data, rotations, zoomings, flips....
Normalize your images based on the new and augmented dataset.

Doing it like this you can also see the results of the augmentation phase, while if you normalize it first you cannot visualize it


I'm not sure about what's your question exactly. If you are interested in the model architecture time series are the typical example for using RNNs (recurrent neural networks) or some more advanced form of them i.e. LSTM or GRU. They are all implemented in keras (https://keras.io/layers/recurrent/).

You may also can experiment with CNNs (convolutional neural networks). Usually they are used for images but in certain cases they can even outperform RNNs. They are also implemented in keras. (https://keras.io/layers/convolutional/)


What is the use of additional column of 1s in normal equation?

The normal equations are designed such that each coefficient in the model has an input of some kind it's being multiplied against. The column of ones is the "input" to the intercept term.


How do we know Kernels are successful in making data linearly Separable?

You cannot guarantee this. Some data is not separable by any kernel because of duplicates.

By trying too hard, you will cause overfitting. Essentially, you force the implicit mapping to be so complex it contains a copy of your training data (which is exactly what happens if you choose a too small bandwidth with RBF).

If you want a good generalization performance, you will have to tolerate some errors, and use e.g. soft-margin and such techniques.

Perfect separation is not something to aim for. Such a guarantee is just a guarantee of being able to overfit! Use cross-validation to reduce the risk of overfitting and find the right balance between being optimal on training data and actual performance.


How can I get semantic word embneddings for compound terms?

If you want an exact answer, please provide a precise question i.e. define what data you have, and what you exactly wants.

This said, in a general manner, you need a dataset of texts that contain these compound terms. How to treat compound terms is a whole scientific field in itself, but since you're talking about semantic word embeddings, I suggest you take a look at the article Distributed Representations of Words and Phrases
and their Compositionality. The same guys who introduced word2vec describe here a simple method to go from word representation to phrase representation, giving btw a way to merge compound terms in single terms. The words "microsoft excel" become "microsoft_excel" and get their own unique embedding.

If you want a python implementation for that, take a look at the gensim.models.phrase class. This does the same work as presented in the previous article.


What is the best way to optimize the parameters in a Sklearn classifier, when I have little data?

You can't get around the test/train thing. I guess in your case, the problem is the choice of method. When I read your data description, I suspect that you might achieve better results with regularization, e.g. Lasso. See some examples here.

Basically you would apply a multiclass logit with lasso penalty in your case. Alternatively you could also check boosting with lasso (aka $l1$ penalty). Here is some playcode with the iris data.

Since your data has many columns and not so many rows, I might go for lasso (not ridge or elastic net), because lasso can shrink features to zero if they don't make a meaningful contributin to the classification. This is especially helpful with high dimensional data.


Does auto.arima of the forecast package deal with seasonality and trend automatically

Yes, the aim of auto.arima is for fitting ARIMA models automatically. You do not need to decompose your time series before hand. See how the algorithm works here https://otexts.com/fpp3/arima-r.html.
You may still want to look at arguments available in the auto.arima function, and you may want to change default maximum values for p, q, and d, etc.


Negative R2_score Bad predictions for my Sales prediction problem using LightGBM

I assume you are new to the field, thus, I would suggest using tutorials to achieve your goal. Because what you did is completely wrong and your approach is incorrect. I guess you want to model the sales as time series without using any predictor instead you want to model future values by looking at the past values. To achieve that, you need to use algorithms like ARIMA, exponential smoothing, etc. Here what you have done is trying to correlate the year, month, and day with the sales, which does not possess any information about the sale as expected (also you decoded it wrongly). Thus, your performance metric shows you a negative result. As a reference, check these which are similar to your problem. Source1, Source2, Source3. These will solve your issue.


Why not scale the values as the very first step? Import your values, scale the values and then do your splits for training/test/etc.


Your question needs a wide explanation.

  I suggest to go through this YouTube video DNN Backpropagation
  It will explain how backpropagation
  works in case of DNN.


You don't pass a string to requests.post's data argument, you pass a dictionary. So basically, instead of converting your dictionary to a string, you just do:
requests.post(url, data=data)

[EDIT]: From my experience, it is better to leave requests take care of everything, instead of using requests.post(url, data=data), you can use requests.post(url, json=data), which will format the header correctly.
See here for the doc.


You can refer to this answer here for a very detailed explanation and derivation of the backpropagation algorithm.

In general, backpropagation is based on the idea that it is possible to attribute an amount of blame to each parameter in your model for the resulting error. So this is what we do, first we will set all the parameters in our neural network (weights and biases) randomly. Then we will pass examples through the model, and at the output we will compute some loss function. Then using backpropagation we will tune our model parameters proportionally to their contribution to the incurred error.

Here are some other related answers that explain the complexities of backpropagation:

Derivative of the sigmoid
Quick run through of the equations dictating backpropagation
Gradient descent, backpropagation and the partial derivative
Computing the backpropagation
Backpropagation in CNN
Backpropagation with different activation functions


ValueError: Expected 2D array, got 1D array instead

selector.fit(X_rnd.reshape,Y_rnd) #select best features

X_rnd should be a 2D matrix.

I believe this error came when "fit" method was executed with X_rnd
but your code is showing X_rnd.reshape. In this case, error should be -

    ValueError: Expected 2D array, got scalar array instead:
    array=<built-in method reshape of numpy.ndarray object at 0x000000000C28EAD0>.
    Reshape your data either using array.reshape(-1, 1) if your data has a single                         feature or array.reshape(1, -1) if it contains a single sample.


This is a classical example of collaborative filtering, in particular you're talking about user-based collaborative filtering.

I don't think Amazon ML supports collaborative filtering (https://forums.aws.amazon.com/thread.jspa?messageID=712158), but you can do it in R and run your R code on the cloud.

It's easy in R, take a look at the recommenderlab package. The vignette has an example with the MovieLens movie rating data set.

You may also use the Microsoft Azure ML framework as it has better support for recommender systems. Google "Azure recommender systems" will give you useful links.


What is the reasonable max number of features for LSTM?

In fact, there is no maximum or reasonable amount of features for LSTM layers in general. Everything is extremely task specific.

Before training an LSTM model with that amount of variables you'd encounter a number of practical problems, for example:

Prohibitive computational costs. I don't think any machine could handle that without crashing.
Its need of massive amounts of data. Such a model would require a dataset of unimaginable size, in order to be trained properly.

But that said, practical problems apart, there is no theoretical threshold to the number of features you can feed into RNNs.


I had the same problem in my code:

c = max(
[np.dot(theta[j], X[i]) for j in range(theta.shape[0])]
) / temp_parameter

I ran debugger and figured out that theta[j], X[i] were not the same shape, so dot product gave an ndarray instead of scalar as a result. max() function was a bit confused about it, because it does not work with ndarrays.
So, the conclusion is:

if prob_ is an ndarray, please, use numpy.max(), it should definitely work:

[In]: f = np.array([[1, 1, 1], [2, 2, 2]])
[In]: max(f)
[Out]: Traceback (most recent call last):
  File "<input>", line 1, in <module>
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
[In]: np.max(f)
[Out]: 2
if prob_ is a list, check whether its components are scalars or lists:

[In]: f = [[1, 1, 1], [2, 2, 2]]
[In]: max(f)
[Out]: [2, 2, 2]
[In]: g = [numpy.array([1, 1, 1]), numpy.array([2, 2, 2])]
[In]: max(g)
[Out]: Traceback (most recent call last):
  File "<input>", line 1, in <module>
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()


how I get different array size in calculating tpr?

You get different array sizes for your different classes because the number of points in your ROC curve depends on the number of unique predictions.

If your model only predicts 0, .5, and 1 your ROC curve will have three points. If your model predicts 100 different probabilities, your ROC curve will have ~100 points. (In practice, the length of the ROC curve may have a couple extra points where the points (0,0) and (1,1) are appended.

If you look at np.unique(y_score[:,i]).size for different i you should see the differences.


Keras/Theano custom loss calculation - working with tensors

I think I might have finally just solved this myself.

1) I changed my Keras backend to use TensorFlow instead of Theano, so that I could use:

tf.boolean_mask

This command was not available under the Theano backend and thus giving me errors.

2) I had to change my code slightly to work with the correct dimensions. It now reads:

WherePredicted1 = K.greater( y_pred[:,1],0.5)
ExtraDataWherePredicted1 = tf.boolean_mask(MyExtraData, WherePredicted1)

Still finding it hard to debug/test a custom loss function, but it's looking like feasible values to might well be correct.


Will data science develop into a separate academic field?

At the risk of showing confirmation bias, yes, data science will become it's own field. The primary argument would be an economic one, data science enables a larger return on investment per resources expended than the many alternatives. It is one of the few profit levers left as business processes become more optimized. So where the money flows, the conferences, journals, et al. will follow.

The second argument is empirical, there are more data science education programs, articles, books, jobs, and salary. I have not modeled it, but this mirrors the indicators in Internet technology and enterprise resource planning waves that the industry has seen. Both have given rise to academic "fields". There appears to be strong correlation.

Unfortunately, large academic institutions and societies have an institutional momentum -- so that recognition may be slow in coming.  One could argue in the business world has already settled this with Chief
Data Scientist Officer
C-level positions.


How do i calculate prediction probability of a class in Java Weka Api?

Welcome to the community!
You can replace your code by the following code.

double[] prediction=classifier.distributionForInstance(instance);

    for (int k<prediction.length; k++){
    System.out.println("Probability of class "+
      trains.classAttribute().value(k)+
       " : "+Double.toString(prediction[k]));
      }

This loop prints all the four values.
Hope it will help you.


Clustering : handling categorical data, should we pivot and scale?

You need to create a dissimilarity matrix first and then apply the clustering technique. Below link has many answer referring your point. Hope below will suffice for this:

https://stats.stackexchange.com/questions/24540/clustering-of-mixed-type-data-with-r


Mapping between original feature space and an interpretable feature space

Welcome to the community @iaaml! I hope I understood the concept right by briefly going through your reference. This is my impression:

in 3.1, they say

  For example, a possible interpretable representation for text
  classiﬁcation is a binary vector indicating the presence or absence of
  a word.

So, I suppose the point is something like Sparse Representation (you may look for sparse representation to see the methods and examples). For instance, an image vector which is predicted as cat can be explained by a 3d explainer, namely, nose, glasses and mouth. Nose and mouth are evidence for cat while glasses is a human face feature (very naive example). So having a vocabulary for all explainers, you can come up with a sparse representation of each decision in which, significant criteria are represented with 0 or 1 to help validating prediction. It happens by examining the features which contribute the most to that class (that's why in Figure.1 they could understand that "No Fatigue" observation is against the prediction).

To obtain such representation, you can build the vocabulary (a set of features which are significant and all together cover whole or most of the space). Then you map your data on these space in which presence of each element of vocabulary is 0 or 1.

Example

I have three samples and their corresponding predictions:

a) Spanish is the main language in Buenos Aires         :Argentina
b) Apple released its new software                      :IT
c) Apple is the man agriculture product in Buenos Aires :Argentina

Using BoW for constructing the feature space, Apple becomes an important feature as it is a famous IT company, but in the last sentence it affects in a wrong way. Alongside, you can also have a map of which feature contributes the most to which class (let's say through Mutual Information or any other feature ranking method, specified on different classes) and construct the matrices below:

   BuenosAires  Apple
a     1           0
b     0           1
c     1           1

when you have what should happen according to the feature ranking for each class:

   BuenosAires  Apple
a     1           0
b     0           1
c     1           0

comparing these two gives you the probably-wrong Apple in last sentence (like "No Fatigue" in Figure.1). The first matrix is the mapping that you do, and the second is the mapping that feature ranking gives you.

Hope I understood it right!


Policy gradient - and auto-differentiation (Pytorch/Tensorflow)

Yes, just take the cross-entropy loss of the last layer and take the gradient with respect to it. The actual action will be the target.

For example, in PyTorch: apply CrossEntropyLoss on the last layer (no need of doing softmax, as it is done implicitly by this function)


What is non-decomposable and/or non-differentiable loss function?

Non-differentiable (loss) functions can be labelled according to the stardard definitions of differentiation: that the function has a derivative at all points in its domain. For example, the ReLU loss function is technically non-differentiable, because its gradient is not defined at zero (or wherever the two lines meet, if you're using a variant of the standard ReLU).

Non-decomposable functions must be looked at slightly differently. A loss function that is not decomposable is usually one that is composed of several statistics across the training metrics. Take the F1 score, for example:

$$
F1 = 2 \cdot \frac{precision \cdot recall}{precision + recall}
$$

Using this as a loss function means you would have to expand the terms for precision and recall in terms of your predictions over a batch, then work out the gradients of each function, finally combining them. It can be done (but it gets complicated), but I would imagine there are trade-offs when optimising at the level of such a metric, which really summarises your model's performance as an average over samples. Using a traditional loss function focusses on improving the model at the more granular sample level. This means the gradients are concentrated at the purest level and I would imagine better optimisation is possible. It could be that optimising at the more abstract level of the F1 score leads to less overfitting or even better generalisation. This is all just an idea though; I haven't seen any work making such comparisons directly.

Here is a short reddit thread with some interesting points and examples about when you might even consider trying to optimise such a non-decompasable function directly (i.e. using it as a loss function).

Have a look at this paper, which speak about non-decomposable loss functions and offers a way to deal with them. Reading through that and comparing the functions that the authors talk about should help you gain a better understanding of the key differences.


Correlation feature selection followed by regression

I think this is a case for linear regression with a lasso/ridge penalty. The lasso/ridge does „shrink“ features/variables, so that it is easy to see which features are important. Since you have 100 variables, you could opt for lasso, since lasso can also „automatically“ exclude features. Here is a lasso example in Python: https://datascience.stackexchange.com/a/53639/71442.


The number of rows in your training data is not part of the input shape of the network because the training process feeds the network one sample per batch (or, more precisely, batch_size samples per batch). And in input_shape, the batch dimension is not included for the first layer. You can read more on this here.

So, the input shape for your problem will be:

input_shape=(75, )


Construct word2vec (CBOW) training data from beginning of sentence

Here is a great answer to this question. I'll summarize:

The code example was taken from a "buggy" repository on GitHub and is not typical of robust solutions.
Robust solutions actually do use the first word as a target word. If the context window is length 10, then the method uses the next 5 words as the context and the first word as the target (it won't actually have a context of size 10 since the first half of the context doesn't exist).
Even though the first few words in the sentence are used as target words, they still will not appear in as many contexts. This issue is mitigated because they appear in smaller contexts. Since the contexts they appear in are smaller, they have more impact on the context and, therefore, more significance in back propagation.
Many of the more robust implementations will use an entire paragraph or document as opposed to a sentence (some even include punctuation). This make sense because the ending of one sentence may give context for the beginning of another sentence. When this approach is implemented, there are far fewer start/ending words, which reduces the issue.

The answer linked above has some other helpful details and is worth reading.


Can i use other regression types that arent based in decision trees to use it like a weak learners in gradient boosting?

The issue of using any linear model (a polynomial regression is a particular case of a linear model, with polynomial features), is that an ensemble of linear models is still a linear model. So, the family of models to optimize from given by the boosted polynomial regression and the single polynomial regression are the same.
This doesn't happen with trees, as an ensemble of trees cannot be expressed as a tree anymore, so that's why trees are used more in the context of gbms.


What is difference between text classification and topic models?

Text Classification

I give you a bunch of documents, each of which has a label attached. I ask you to learn why you think the contents of the documents have been given these labels based on their words. Then I give you new documents and ask what you think the label for each one should be. The labels have meaning to me, not to you necessarily.

Topic Modeling

I give you a bunch of documents, without labels. I ask you to explain why the documents have the words they do by identifying some topics that each is "about". You tell me the topics, by telling me how much of each is in each document, and I decide what the topics "mean" if anything.

You'd have to clarify what you me by "identify one topic" or "classify the text".


How do I replace NaN values using group by pivot_table in pandas DataFrame?

Consider using df.fillna(). That has a  variety of options to replace nans with values of choices. You can apply it on either the complete data frame or on specific columns (I.e Series).


Use all available data to build Logistic Regression model

Yes, you can(Atleast in most cases). Infact before going to production, you must do this, it will give you better accuracy provided you keep the cross validated parameters same.

When you participate in competitions this is a well followed strategy to increase the score.

However, sometime domain constraints prevent you to use the test/validation data.

Eg: When you have split your data based on time periods, you might not want to include the latest data just because those observations in real time might not have matured enough to be considered into a model.

A good use case is fraud identification model in Insurance. It takes time for a claim to be identified as fraud. Hence without accounting for this, if you put all your data into model, the most recent claims will have been marked as no fraud, but in near future they might actually turn out to be identified as fraud. Your model will have learnt from wrong samples.


Bayesian optimization is a principled way of sequentially finding the extremum of black-box functions. What's more, there a numerous software packages that make it easy, such as BayesOpt and MOE. Another flexible Bayesian framework that you can use for optimization is Gaussian processes: Global Optimisation with Gaussian Processes


What are Laplacian scores and how do they effect feature selection?

As @Spacedman said, the paper provides a very clear explanation of the algorithm on page 2.

If you're a little less comfortable with the math of the notation, here's the intuition/explanation in words.

Make a k-nearest neighbor's graph. That is, for each observation, define an edge in the Graph for that observation if another observation is one of its k-nearest neighbor's. If you're using a supervised algorithm you can define an edge if they share the same label.
If any two nodes (observations) are connected, define a weight matrix S measuring the similarity between those two nodes (using some distance measure).
For each feature define the Laplacian graph.
Compute the Laplacian score based on their equation.

Intuitively, you're using KNNs to define a network graph and assessing how similar the features are according to your distance metric. This is just as good of a measure of feature importance as any other but will also has its pitfalls, just like all of the others.


Relationship between two continuous variables in time series data

You could look at the cross-correlation between the two series for different lags. In short, you would look at the values from one series and see how they correlate with previous (or lagged) values from the other series. Auto-correlation is a special case of cross-correlation where you correlate a series with itself (or its lagged values).

To capture non-linear relationships you could see if you can use one of the series to predict the other. You could phrase it as a forecasting problem and see if adding the other variable improves forecasting accuracy.

For more details, check out this chapter.


Data Augmentation for Regression ANN with low Sample Size

In theory it's possible, everything depends on your data, the model you're training and how you're going to augment your training data.


Do you apply outlier detection of numerical data in practical applications?

You'd have to think about where the outliers come from. Technical malfunction? Then they are probably safe to disregard. But when you write that your main task is

  to draw conclusions from the training data that can be generalized to future unseen data

you have to keep in mind that your future data will contain outliers as well and deal with them appropriately. Models like trees are usually somewhat robust against those while regression can go haywire. There are several well working methods like disregarding everything that is more than 4 standard deviations (or median deviations from the median) out or just disregarding the n-th top and bottom percentiles.

But then there is the case where outliers are not just flukes and can actually tell you something about the problem domain you are working on. If you work in fraud detection, the outliers are the points that interest you. If you work on optimizing network performance, the packets that don't arrive can be the interesting ones.

So my advice is to not blindly throw away data out of principle but always think about what the data represent and why outliers may be present.


Found out what the issue was:

model.load('filename')

Should actually be:

model.load('./filename')

Hopefully this helps out someone else.


What happened to the accuracy before and after 75th epoch. Why its unstable at first, then stepped up after the 75th epoch?

I think that it is a convergence of the gradient descent happened. While training,  we are looking for correct weights and that is why we see these fluctuations on the picture. When we are close to a global minimum (found good weights), the model starting to converge. That is what happened after the 75th epoch in this case.

By the way, you have an overfitting


how can I replicate working of Multi Label Binarizer from sklearn package in R?

One way to do this is to convert the source code from Python to R - see line 289 here.

You could also check for functionality in existing R packages - for instance, mldr, mlr or caret packages.


XGBoost: How to set the probability threshold for multi class classification

You don't set it in xgboost. Its job is to return probabilities in predict_proba. predict does the logical thing and tells you the most likely class.

If you want to interpret the probabilities differently, you'd have to write code to do so. It depends on what "does not much differ" means. For example if you simply mean "the most likely class is has probability < 0.5", in numpy it's something like np.argwhere(probabilities.max(axis=1) < 0.5). Those rows are what are undetermined. For the rest the prediction is the argmax of the row.

Of course, that's not the only definition of when it's undetermined, so you need to figure out what the rule is that you're meant to implement. You could base it on entropy too, not simply the max class probability.


My experience is the same. I think in my case at least it's largely down to the algorithms I would generally use, all of which have the capacity to ignore features or down-weight them to insignificance where they're not particularly useful to the model. For example, a Random Forest will simply not select particular features to split against. A neural network will just weight features into having no effect on the output and so on. My experience is that algorithms which take every feature into account (like a vanilla linear regression model) generally suffer far more.

Additionally, in a "production" rather than competitive environment I found that feature selection became much more important. This is generally due to covariate shift - the distribution of values for certain features changes over time and, where that change is significant between your training dataset and the live predictions you're making day-to-day, this can really trash your model's outputs completely. This kind of problem seems to be administered out of the datasets used for competitions, so I never experienced it until starting to use ML at work.


In linear regression overfitting occurs when the model is "too complex". This usually happens when there are a large number of parameters compared to the number of observations. Such a model will not generalise well to new data. That is, it will perform well on training data, but poorly on test data.
A simple simulation can show this. Here I use R:
> set.seed(2)
> N <- 4
> X <- 1:N
> Y <- X + rnorm(N, 0, 1)
>
> (m0 <- lm(Y ~ X)) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -0.2393     1.8568  -0.129    0.909
X             1.0703     0.6780   1.579    0.255

Residual standard error: 1.516 on 2 degrees of freedom
Multiple R-squared:  0.5548,    Adjusted R-squared:  0.3321
F-statistic: 2.492 on 1 and 2 DF,  p-value: 0.2552

Note that we obtain a good estimate of the true value for the coefficient of X. Note the Adjusted R-squared of 0.3321 which is an indication of the model fit.
Now we fit a quadratic model:
> (m1 <- lm(Y ~ X + I(X^2) )) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -4.9893     2.7654  -1.804    0.322
X             5.8202     2.5228   2.307    0.260
I(X^2)       -0.9500     0.4967  -1.913    0.307

Residual standard error: 0.9934 on 1 degrees of freedom
Multiple R-squared:  0.9044,    Adjusted R-squared:  0.7133
F-statistic: 4.731 on 2 and 1 DF,  p-value: 0.3092

Now we have a much higher Adjusted R-squared:  0.7133  which may lead us to think that the model is much better. Indeed if we plot the data and the predicted valus from both models we get :
> fun.linear <- function(x) { coef(m0)[1] + coef(m0)[2] * x  }
> fun.quadratic <- function(x) { coef(m1)[1] + coef(m1)[2] * x  + coef(m1)[3] * x^2}
>
> ggplot(data.frame(X,Y), aes(y = Y, x = X)) + geom_point()  + stat_function(fun = fun.linear) + stat_function(fun = fun.quadratic)

So on the face of it, the quadratic model looks much better.
Now, if we simulate new data, but use the same model to plot the predictions, we get
> set.seed(6)
> N <- 4
> X <- 1:N
> Y <- X + rnorm(N, 0, 1)
> ggplot(data.frame(X,Y), aes(y = Y, x = X)) + geom_point()  + stat_function(fun = fun.linear) + stat_function(fun = fun.quadratic)

Clearly the quadratic model is not doing well, whereas the linear model is still reasonable. However, if we simulate more data with an extended range, using the original seed, so that the initial data points are the same as in the first simulation we find:
> set.seed(2)
> N <- 10
> X <- 1:N
> Y <- X + rnorm(N, 0, 1)
> ggplot(data.frame(X,Y), aes(y = Y, x = X)) + geom_point()  + stat_function(fun = fun.linear) + stat_function(fun = fun.quadratic)

Clearly the linear model still performs well, but the quadratic model is hopeless outside the orriginal range. This is because when we fitted the models, we had too many parameters (3) compared to the number of observations (4).

Edit: To address the query in the comments to this answer, about a model that does not include higher order terms.
The situation is the same: If the number of parameters approaches the number of observations, the model will be overfitted. With no higher order terms, this will occur when the number of variables / features in the model approaches the number of observations.
Again we can demonstrate this easily with a simulation:
Here we simulate random data data from a normal distribution, such that we have 7 observations and 5 variables / features:
> set.seed(1)
> n.var <- 5
> n.obs <- 7
>
> dt <- as.data.frame(matrix(rnorm(n.var * n.obs), ncol = n.var))
> dt$Y <- rnorm(nrow(dt))
>
> lm(Y ~ . , dt) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -0.6607     0.2337  -2.827    0.216
V1            0.6999     0.1562   4.481    0.140
V2           -0.4751     0.3068  -1.549    0.365
V3            1.2683     0.3423   3.705    0.168
V4            0.3070     0.2823   1.087    0.473
V5            1.2154     0.3687   3.297    0.187

Residual standard error: 0.2227 on 1 degrees of freedom
Multiple R-squared:  0.9771,    Adjusted R-squared:  0.8627

We obtain an adjusted  R-squared of 0.86 which indicates excellent model fit. On purely random data. The model is severely overfitted. By comparison if we double the number of obervations to 14:
> set.seed(1)
> n.var <- 5
> n.obs <- 14
> dt <- as.data.frame(matrix(rnorm(n.var * n.obs), ncol = n.var))
> dt$Y <- rnorm(nrow(dt))
> lm(Y ~ . , dt) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.10391    0.23512  -0.442   0.6702
V1          -0.62357    0.32421  -1.923   0.0906 .
V2           0.39835    0.27693   1.438   0.1883
V3          -0.02789    0.31347  -0.089   0.9313
V4          -0.30869    0.30628  -1.008   0.3430
V5          -0.38959    0.20767  -1.876   0.0975 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.7376 on 8 degrees of freedom
Multiple R-squared:  0.4074,    Adjusted R-squared:  0.03707
F-statistic:   1.1 on 5 and 8 DF,  p-value: 0.4296

..adjusted R squared drops to just 0.037


Using DNN as the objective function for a multi-objective optimization algorithm

This nonlinear equation will again be approximated with the net. There is no point in introducing this much computation complexity, if it is not learned by than than it wont be learned. Ocam rasor


What does the KFold error mean and how to get confusion matrix from Kfold random forest implementation?

modified my code like below, works now

kf = KFold(n_splits=num_folds,random_state=seed,shuffle=False)
kf.get_n_splits(X)
i=1
print("confusion matrix:")
for train_index, test_index in kf.split(X):

    X_train, X_test = X.iloc[train_index,:], X.iloc[test_index,:]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index]

    rfc.fit(X_train, y_train)
    print (i,"\n",confusion_matrix(y_test, rfc.predict(X_test)))
    i=i+1
    print(10* '#')