How to find from which row and column the value belong?

You can use pandas .isin() function to compare all the values of your dataframe to a given value:

In [10]: df[df.isin([6.9])]
Out[10]:
   Height_1  Height_2  Height_3
0       NaN       NaN       NaN
1       NaN       6.9       NaN
2       NaN       NaN       NaN
3       NaN       NaN       NaN

Now, if you want to get rows and column directly from it use .stack() on it. So, it will be like:

In [11]: df[df.isin([6.9])].stack()
Out[11]:
1  Height_2    6.9
dtype: float64

The output is a series. This will work in case of multiple matches too where the output will be a dataframe.


Practical problems in anomaly detection where the number of normal data is extremely high compared to abnormal data

The presumption I feel you might be making is that a Machine Learning model is required for this task, but to be honest statistical approaches and less fashionable algorithms are available.
The reality is training a ML model on  your data is going to be a struggle because of the significant class imbalance and low volume of anomalies. That is why pretrained models are attractive, in particular Neural Networks.
To get a feel of the range of options, I find the Python library pyOD a great resource. It has about 30 different ways to detect outliers.


Notion of cluster centers and cluster comparison in Density Based Algorithms

You can either use the medoid, you can sometimes compute a centroid (and just ignore that it may be outside of the cluster), or you can do pairwise comparisons and take the average of that rather than comparing centers.


Compute specificity and sensitivity at certain thresholds

For threshold = $0.5$ we have:

Sensitivity = True Positive Rate

= (number of points with label $1$ and $P(T = 1)\geq 0.5$) divided by (number of points with label $1$)

= $\left|\{(1, 0.54), (1, 0.88), (1, 0.90)\}\right| / 4$ = $3/4$ = $0.75$

Specificity = 1 - False Positive Rate

= 1 - (number of points with label $0$ and $P(T = 1)\geq 0.5$) divided by (number of points with label $0$)

= $1 - \left|\{(0, 0.78)\}\right|/6$ =  $1 - 1/6$ = $0.833$


from keras.applications.resnet50 import ResNet50

resnet_model = ResNet50(weights='imagenet')

#resnet_model.count_params()
resnet_model.summary()

  Total params: 25,636,712
  Trainable params: 25,583,592
  Non-trainable params: 53,120

Check your code once to be sure that it is ResNet50


How to interpret partial dependence interaction plot for binary classification?

The partial dependence plot shows an increase in survival probability at ages between 0 and 10 with any fare range.

For more on how to interpret partial dependence plot:

https://christophm.github.io/interpretable-ml-book/pdp.html


Decision trees, categorizacion and oversampling

Do you use Python?
Python class DecisionTreeClassifier has an attribute class_weight for this purpose.
So you do not need to adjust it manually. Check here
https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html
Regarding discretizing or encoding - hard to say what is better without knowing your data.
Unless you are really sure one is the best choice, you can check the model by using encoding instead of discretizing and compare the quality.


How to derive the sum-of squares error function formula?

There are two interpretations of this formula that I explain one of them.

\begin{equation} Xw = y \end{equation}

\begin{equation} X^tXw = X^ty \end{equation}

The above is for making sure that you make a square matrix that it has an inverse. It is possible that $X^tX$ does not have any inverse but its chance for linear regression problems is not that much. The reason is that you have a matrix $X$ which belongs to $R^{m \times n}$ which $m$ represents the number of samples and $n$ represents the number of features. Usually, the number of samples is much more than the number of features. Next,

\begin{equation}(X^tX)^{-1}(X^tX)w = (X^tX)^{-1}X^ty\end{equation}

\begin{equation}w = (X^tX)^{-1}X^ty\end{equation}

Consequently, you have found a closed form for the $w$ linear regression problem which can be generalised to non-linear regression.

Be aware that $(X^tX)^{-1}X^t$ is called the pseudo-inverse of the matrix $X$. The reason is that $X$ is not a square matrix and it does not have inverse but by the mentioned formula you can find its pseudo-inverse. Just multiply it by $X$ and you will get Identity matrix.

There is another interpretation of this. You can find here.


Interpretability of log loss
Log loss isn't necessarily between the range [0; 1] - it only expects input to be in this range. Take a look at this example:
$$
y_{pred} = 0.1 \\
y_{true} = 1.0 \\
log\_loss = -(log(y_{pred}) * y_{true} + (1 - y_{true}) * log(1 - y_{pred})) = -(log(0.1) * 1.0) = 2.302
$$
In an extreme case log loss can even be equal to infinity. So there is nothing wrong with the code and also there aren't many interesting things you can derive from the fact that log_loss is lower or greater than 1. What you can do with it is the same as with any loss function - compare it to a similar model with different hyperparameters and choose the one with the lowest average loss as your best model (a process called hyperparameter optimization).
When to use loss and when f1 score?
Let's say you have a dataset and a classification problem which you want to solve. You know that you can create a statistical model which returns probabilities of a given class. You also know that there is (hypothetically) an algorithm which classifies based on some heuristics which requires no training. You would like to know which of these is the best for your problem. What you do, if we simplify a little, is:

Split your dataset to train, validation and test sets.
Use your train set to train the model
While training the model calculate loss for train and validation set in each epoch (if you're not using deep neural networks you can and should use cross validation).
Plot loss for train and validation set and see if your model is biased (high train loss and high validation loss) or overfitted (low train loss and high validation loss). The lower loss for validation set the better.
Do 3. and 4. multiple times for different hyperparameters and select one with the lowest validation set loss. You now have a trained statistical model.
Now use f1 score to compare your model to the algorithm you also know about. The higher score the better. Notice that assuming that the algorithm returns classes and not probabilities if it is incorrect for even one example its log loss function will be equal to infinity. This is why we can't use log loss as a metric to compare these two methods.

In short, you should use loss as a metric during training/validation process to optimize parameters and hyperparameters and f1 score (and possibly many more metrics for example Area Under Curve) during test process to select the best method to solve your problem. This way it's possible to compare different methods to solve the problem - even ones which don't use machine learning at all.


Can BRAT be used for text classification annotation?

According to their documentation , BRAT does a lot of things, but text classification isn't one of them. BRAT is "too powerful" for that. I'd recommend you use a tool like Prodigy instead.

This should do what you're looking for.


Growth function of a 6-dimensional linear classifier

Suppose we have $n$ points in $\chi$. We have $d$ choices of $i$, two choices of $a$, and at most $n+1$ effective choices of $b$ so the number of possible combinations of outputs on those $n$ points is at most $2d(n+1)$. I'm sure you can take it from here.


The general idea is to standardize the format of the data so that it can be used consistently across a wide range of methods for analysis and prediction.

The negative impact is the same for all 5 types of messy data: one would spend much more time implementing functions which do nothing more than converting from one very specific format to another. Additionally through these cumbersome conversions one is more likely to introduce errors or omissions in the data.

Basically following the "tidy data philosophy" can save a lot of time and makes your data readily usable with a wider range of methods.


How to think about - and sometimes impute - geographic distances

I don't think time alone is what you are after, although it would certainly be useful.

It sounds to me like you would also want to consider the cost of transport.

Yes the drive and flight may take the same amount of time, but you could be looking at orders of magnitude difference in the total cost. If you are only looking at extremely wealthy people this may not matter, but otherwise this could have as big an impact as time alone.

My advice would be to use the existing distance column to approximate the minimum cost of transportation, similar to how you would approximate the minimum time of transportation. I would then use both time and cost as inputs to my model.


class weights formula for imbalanced dataset

I really don't suggest Under/Oversampling as it would change the distribution of dataset. we should consider distribution as a useful feature of dataset. so I think the weighted loss would have better performance in most cases. if you're using TF/Keras, this link would be useful. you can use a variety of loss functions, like the below one, to apply the weight.
tf.nn.weighted_cross_entropy_with_logits(
    labels, logits, pos_weight, name=None
)

A value pos_weight > 1 decreases the false negative count, hence increasing the recall. Conversely setting pos_weight < 1 decreases the false positive count and increases the precision.


The way LSTMs are capable of learning long term dependencies is by keeping a cell state which serves as a memory of sorts.
This cell state is updated based on the values of different gates within the cell, forget gate being one of them.
The forget gate looks at the previous hidden state and the current input and outputs a number between 0 and 1 for each value in the cell state - deciding which values should be discarded and which saved.
During training, the cell learns which values in the cell state it should keep at different time steps in order to yield better results for the task at hand.
Now, saying exactly what are those values are is hard since it changes between samples and time steps. For example, let's assume that your objective is to predict the temperature for tomorrow based on the previous 10 days. If you had an outlier on the 3rd day of the previous 10 days, that might hinder the prediction, and the cell could decide to forget that value. In other cases, it might decide to keep the value of the 3rd day since it helps prediction.
If you want to understand LSTMs better I suggest this great post by Chris Olah.


Can we use median to replace all the missing values from a column

imputing with median/mean affects distibution of the data, essentially squeezes it, and it does not account for variance in the data, not to mention we'll be making up data. Of course, there're some cases that it may not matter as much and some cases where it will.

If it's worthwhile your time and in case it adds value to your work, you can look into other methods such as proper stochastic regression imputation. Using this method you can randomly add value using linear regression imputation method and create several models and aggregate the results. In this case you'll be accounting for variance in your data and not essentially makin up data, but it's more computationally consuming of course in comparison to just fillna.

There's new method to impute values based on missing patterns, where you'll break your data based on the pattern and create models for each and aggregate.
Of course if it's worth the time and effort, these methods should be more beneficial.
Also you can look into python's fancyimpute

Here are some good resources that cover various methods of imputing missing values:
1) http://www.stat.columbia.edu/~gelman/arm/missing.pdf
2) http://www.bu.edu/sph/files/2014/05/Marina-tech-report.pdf
3) https://stefvanbuuren.name/fimd/sec-nonnormal.html
4) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2727536/


What knowledge should I gain for developing a supervised image processing software that learns how to edit photos based on past behavior?

From the description of your problem, you need both Computer Vision and Deep Learning for a task like that. It is going to be extremely difficult, but you are more in advantage than anyone else, given you have a strong statistical and machine learning background. You don't have to worry a lot about the image processing part as there are libraries that will do that for you. You can look into PIL for that.The hard part is the learning from the edits part.A simpler way to solve this problem would be to focus on image processing and pick out the cell line clearly. The other way would be to train a Convnet on a large collection of 'labelled-images' of the cell-line, so that it is able to identify it in any picture. I do think it is quite a hard problem to solve but do give it a try. Cheers. All the best.


What will happen if the number of units and the time-steps are different numbers in LSTM?

It depends on how you are sending the data into the LSTM. Lets say you have one data point with t timestamps. It can be of shape of 1xt or tx1, which means you will be sending all the time inputs once into LSTM and while in the second case you will sending one timestamp of data t number of times to complete one data point.

Based on the number of units, the network computes what are the sizes U, W and V matrices. units just tells what should be the hidden dimensions the network should operate at. So it does not matter what is your input sequence length is. It is going to take in one input at a time and take the output of last input you put in and computes new output, when you unroll it and see.

Edit

  First scenario:

  LSTM(units = 5 , input_shape(time-steps = 5, No_of_features = 3))

If LSTM cell does not return the intermediate cell hidden states, it will output 1x5 output at the end else it will be giving out 5*5

  Second scenario:

  LSTM(units = 15 , input_shape(time-steps = 5, No_of_features = 3))

If LSTM cell does not return the intermediate cell hidden states, it will output 1x15 output at the end else it will be giving out 5*15

  Third scenario:

  LSTM(units = 5 , input_shape(time-steps = 20, No_of_features = 3))

If LSTM cell does not return the intermediate cell hidden states, it will output 1x5 output at the end else it will be giving out 20*5

  Fourth scenario:

  LSTM(units = 15 , input_shape(time-steps = 20, No_of_features = 3))

If LSTM cell does not return the intermediate cell hidden states, it will output 1x15 output at the end else it will be giving out 20*15


Space between an object and the ground truth bounding box

The first image is a better bounding box.  A perfect bounding box has no space in between the edges of the object and the box, but it also fully contains the object.


Combining Linear Regression and Time Series

You can use time as an additional attribute of the data. If you are having monthly data then your time feature could have 12 different values from 0 to 11. If you are having quarterly data your time attribute could have 4 different value from 0 to 3. You can refer Forecasting: Principles and Practice for more clarification. It's a good book.


NameError: name 'MyStreamListener' is not defined

You need to override StreamListener() class.

# API Authentication

# Importing TWEEPY Library
import tweepy

# Store OAuth authentication credentials in relevant variables
access_token = "provided"
access_token_secret = "provided"
consumer_key = "provided"
consumer_secret = "provided"

# Pass OAuth details to tweepy's OAuth handler
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

# Streaming Tweets
#override tweepy.StreamListener to add logic to on_status
class MyStreamListener(tweepy.StreamListener):
    def on_status(self, status):
        print(status.text)

# Initialize Stream listener
l = MyStreamListener()

# Create you Stream object with authentication
stream = tweepy.Stream(auth, l)

# Filter Twitter Streams to capture data by the keywords:
stream.filter(track = ['clinton', 'trump', 'sanders', 'cruz'])


How does your timestamp look like? Apparently there are too many dimensions.

When using pandas DataFrames you could pass the .index (in case it's no multiindex) or just np.arange(len(<your_data>)) as timestamps.


GELU function

We can expand the cumulative distribution of $\mathcal{N}(0, 1)$, i.e. $\Phi(x)$, as follows:
$$\text{GELU}(x):=x{\Bbb P}(X \le x)=x\Phi(x)=0.5x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)$$

Note that this is a definition, not an equation (or a relation). Authors have provided some justifications for this proposal, e.g. a stochastic analogy, however mathematically, this is just a definition.

Here is the plot of GELU:

Tanh approximation

For these type of numerical approximations, the key idea  is to find a similar function (primarily based on experience), parameterize it, and then fit it to a set of points from the original function.

Knowing that $\text{erf}(x)$ is very close to $\text{tanh}(x)$

and first derivative of $\text{erf}(\frac{x}{\sqrt{2}})$ coincides with that of $\text{tanh}(\sqrt{\frac{2}{\pi}}x)$ at $x=0$, which is $\sqrt{\frac{2}{\pi}}$, we proceed to fit
$$\text{tanh}\left(\sqrt{\frac{2}{\pi}}(x+ax^2+bx^3+cx^4+dx^5)\right)$$ (or with more terms) to  a set of points $\left(x_i, \text{erf}\left(\frac{x_i}{\sqrt{2}}\right)\right)$.

I have fitted this function to 20 samples between $(-1.5, 1.5)$ (using this site), and here are the coefficients:

By setting $a=c=d=0$, $b$ was estimated to be $0.04495641$. With more samples from a wider range (that site only allowed 20), coefficient $b$ will be closer to paper's $0.044715$. Finally we get

$\text{GELU}(x)=x\Phi(x)=0.5x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)\simeq 0.5x\left(1+\text{tanh}\left(\sqrt{\frac{2}{\pi}}(x+0.044715x^3)\right)\right)$

with mean squared error $\sim 10^{-8}$ for $x \in [-10, 10]$.

Note that if we did not utilize the relationship between the first derivatives, term $\sqrt{\frac{2}{\pi}}$ would have been included in the parameters as follows
$$0.5x\left(1+\text{tanh}\left(0.797885x+0.035677x^3\right)\right)$$
which is less beautiful (less analytical, more numerical)!

Utilizing the parity

As suggested by @BookYourLuck, we can utilize the parity of functions to restrict the space of polynomials in which we search. That is, since $\text{erf}$ is an odd function, i.e. $f(-x)=-f(x)$, and $\text{tanh}$ is also an odd function, polynomial function $\text{pol}(x)$ inside $\text{tanh}$ should also be odd (should only have odd powers of $x$) to have
$$\text{erf}(-x)\simeq\text{tanh}(\text{pol}(-x))=\text{tanh}(-\text{pol}(x))=-\text{tanh}(\text{pol}(x))\simeq-\text{erf}(x)$$

Previously, we were fortunate to end up with (almost) zero coefficients for even powers $x^2$ and $x^4$, however in general, this might lead to low quality approximations that, for example, have a term like $0.23x^2$ that is being cancelled out by extra terms (even or odd) instead of simply opting for $0x^2$.

Sigmoid approximation

A similar relationship holds between $\text{erf}(x)$ and $2\left(\sigma(x)-\frac{1}{2}\right)$ (sigmoid), which is proposed in the paper as another approximation, with mean squared error $\sim 10^{-4}$ for $x \in [-10, 10]$.

Here is a Python code for generating data points, fitting the functions, and calculating the mean squared errors:

import math
import numpy as np
import scipy.optimize as optimize

def tahn(xs, a):
    return [math.tanh(math.sqrt(2 / math.pi) * (x + a * x**3)) for x in xs]

def sigmoid(xs, a):
    return [2 * (1 / (1 + math.exp(-a * x)) - 0.5) for x in xs]

print_points = 0
np.random.seed(123)
# xs = [-2, -1, -.9, -.7, 0.6, -.5, -.4, -.3, -0.2, -.1, 0,
#       .1, 0.2, .3, .4, .5, 0.6, .7, .9, 2]
# xs = np.concatenate((np.arange(-1, 1, 0.2), np.arange(-4, 4, 0.8)))
# xs = np.concatenate((np.arange(-2, 2, 0.5), np.arange(-8, 8, 1.6)))
xs = np.arange(-10, 10, 0.001)
erfs = np.array([math.erf(x/math.sqrt(2)) for x in xs])
ys = np.array([0.5 * x * (1 + math.erf(x/math.sqrt(2))) for x in xs])

# Fit tanh and sigmoid curves to erf points
tanh_popt, _ = optimize.curve_fit(tahn, xs, erfs)
print('Tanh fit: a=%5.5f' % tuple(tanh_popt))

sig_popt, _ = optimize.curve_fit(sigmoid, xs, erfs)
print('Sigmoid fit: a=%5.5f' % tuple(sig_popt))

# curves used in https://mycurvefit.com:
# 1. sinh(sqrt(2/3.141593)*(x+a*x^2+b*x^3+c*x^4+d*x^5))/cosh(sqrt(2/3.141593)*(x+a*x^2+b*x^3+c*x^4+d*x^5))
# 2. sinh(sqrt(2/3.141593)*(x+b*x^3))/cosh(sqrt(2/3.141593)*(x+b*x^3))
y_paper_tanh = np.array([0.5 * x * (1 + math.tanh(math.sqrt(2/math.pi)*(x + 0.044715 * x**3))) for x in xs])
tanh_error_paper = (np.square(ys - y_paper_tanh)).mean()
y_alt_tanh = np.array([0.5 * x * (1 + math.tanh(math.sqrt(2/math.pi)*(x + tanh_popt[0] * x**3))) for x in xs])
tanh_error_alt = (np.square(ys - y_alt_tanh)).mean()

# curve used in https://mycurvefit.com:
# 1. 2*(1/(1+2.718281828459^(-(a*x))) - 0.5)
y_paper_sigmoid = np.array([x * (1 / (1 + math.exp(-1.702 * x))) for x in xs])
sigmoid_error_paper = (np.square(ys - y_paper_sigmoid)).mean()
y_alt_sigmoid = np.array([x * (1 / (1 + math.exp(-sig_popt[0] * x))) for x in xs])
sigmoid_error_alt = (np.square(ys - y_alt_sigmoid)).mean()

print('Paper tanh error:', tanh_error_paper)
print('Alternative tanh error:', tanh_error_alt)
print('Paper sigmoid error:', sigmoid_error_paper)
print('Alternative sigmoid error:', sigmoid_error_alt)

if print_points == 1:
    print(len(xs))
    for x, erf in zip(xs, erfs):
        print(x, erf)

Output:

Tanh fit: a=0.04485
Sigmoid fit: a=1.70099
Paper tanh error: 2.4329173471294176e-08
Alternative tanh error: 2.698034519269613e-08
Paper sigmoid error: 5.6479106346814546e-05
Alternative sigmoid error: 5.704246564663601e-05


How to use neural networks with large and variable number of inputs?

There exist many possibilities for populating empty gaps on data.

Most repeated value: Fill the gaps with the most common value.
Create a distribution: Make the histogram and drop values according to that distribution.
Create a new label: Since you do not have information, do not assume any value and create another label/category to indicate that value is empty.
Create a classifier: Make a relation among the variable with empty gaps and the rest of the data and create a simple classifier. With this, populate the rest of the data.

There exist many others, but these are the most common strategies. My suggestion is not to populate and to keep unknown what is unknown.


Adding document vector doc2ec to seq2seq model

The purpose of the encoder (green part) is to determine this document vector, so if you want to provide it yourself use the decoder and feed the embedding as the initial state.

I suppose you could use your embedding as a prior for the encoder output. One alternative to your suggestion, which might work, is to use the embeddings as an MSE regularizer to the encoder output.


When using Scikit Learn Grid Search, why are my train and cv scores high, but my test score is a lot lower?

Here are some ideas of things to try:

I would try to investigate the problem by not using the 'refit' option. Run the Grid Search CV yourself, obtain the best parameters and train a new model on all of the training data after you input those best parameters. This is just to  make sure there is nothing funky going on with SKlearn.
Try using another metric for classification like AUC and see if there is any difference.
Also, it might be helpful if you provide more information about your data, such as the type of outcome and the type of predictors you are using


K-fold cross validation of scikit-learn with confusion matrix of Keras

KFold() is meant for cross-validation purpose where multiple models are created over the subsets of the entire dataset and discarded after the validation procedure is over. So the model.fit() should be called explicitly to create the model for purpose. Both the tasks can be easily done through the wrapper named KerasClassifier() by packaging all the details of the model design.


Setting missing values to 0 does not work well. Because this causes huge deviations.

K-means cannot handle NaN values well, at least in the standard implementations. You could remove the all zero vectors, and then use a nan-toletant implementation, but this will likely also cause undesired effects. Because the absence of a transaction is not the same as an unknown timestamp.

In the end, I believe you have not understood your problem yet. K-means likely is the wrong tool, so it does not matter how you prepare the data - it's answering a different question, but you haven't understood what you are asking yet (and no, "what are the clusters" is not a well defined question - phrase it mathematically, then you have at least a formal version of the question).


Monthly trend with fb prophet-Interpreting the graph

As explained in the link you provided, you will encouter problems with monthly data when you try to see what happen at a timestep below. They do not exactly use the same graph as yours but they say that : "The seasonality has low uncertainty at the start of each month where there are data points". It seems to be what happen here : on your graph you can see that there is no seasonal effect on a monthly basis, as the multiplicative coefficient seems to be 1 at the start of each month :

Below the monthly time-step there seems to be oscilations. I am not 100% sure but it might be a practical exemple of Runge's phenomenon, where oscillations appears while doing interpolation. Below is a graphical exemple with the Runge function : some oscilations appears when you try to approximate it with polynomials, and quite counterintuitively, the size of those oscillations may grow as you try to improve the model by increasing the number of estimation knots.

This might be what happen here as you dont have a perfectly flat effect and the interpolation create oscilations between your monthly dates.
Generally speaking for the prophet framework the way to deal with this are mentionned in the link you provide :

use monthly regressor if you only want to get monthly effect. As said : "this approach would avoid the within-month unidentifiability seen above. Be sure to use yearly_seasonality=False if monthly extra regressors are being added."
only make monthly forecasts, which can be done by passing the frequency into make_future_dataframe (freq='MS')

Obviously the main alternative being to use another framework that might be more transparent on what it does exactly.


Xgboost : A variable specific Feature importance

If you use a per-observation explanation, you could just average (or aggregate in some other way) the importances of features across the samples for each Dealer.

For example, using shap to generate the per-observation explanation:
import pandas as pd
from sklearn.linear_model import LogisticRegression
import shap

data = [['Alex',10,13,1,0],['Bob',11,14,12,0],['Clarke',13,15,13,1],['Bob',12,15,1,1]]
df = pd.DataFrame(data, columns=["dealer","x","y","z","loss"])

lr = LogisticRegression()
lr.fit(df[['x', 'y', 'z']], df['loss'])

# Whatever explainer you prefer:
explainer = shap.explainers.Permutation(lr.predict_proba, df[['x', 'y', 'z']])
shap_values = explainer(df[['x', 'y', 'z']])

# get just the explanations for the positive class
shap_values = shap_values[...,1]

shap_df = pd.DataFrame(abs(shap_values.values))
shap_df.columns = ['x_shap', 'y_shap', 'z_shap']
shap_df['dealer'] = df['dealer']
shap_df.groupby('dealer').mean()

produces

dealer
x_shap
y_shap
z_shap

Alex
0.260427
0.140054
0.075176

Bob
0.106593
0.069035
0.091146

Clarke
0.268328
0.083706
0.085807


Should I use a cleaned labeled data for sentiment analysis?

The OP asks "Does cleaning of data before labeling make any difference?" - that's an empirical question... one that should be investigated by EDA of your data.

In some cases twitter-specific conventions can be highly indicative of a specific class (e.g., http:// associated with spam/advertising tweets) or sentiment (e.g., :-) emoticon associated with positive valence). Similarly, as I discuss in this post, stop words can be great features to keep in certain types of text models. My support for the above answer is based on an (unpublished) project I did that involved about 10 people hand-scoring 50k tweets stratified across major industries (food, travel, electronics, CPG, etc.) and brands (McDonalds, Southwest Airlines, iPhone, Tide PODs).

My advice, create feature extractors for any textual feature that you believe has a theoretical or logical justification for being good indicator of text polarity. Then empirically test each feature to determine whether it significantly improves classification accuracy. Keep those that are in your model; save the other feature extractors for a rainy day.


With your three labels: positive, neutral or negative - it seems you are talking more about sentiment analysis. This answer the question: what are the emotions of the person who wrote this piece of text?

Semantic analysis is a larger term, meaning to analyse the meaning contained within text, not just the sentiment. It looks for relationships among the words, how they are combined and how often certain words appear together.

To gain a deeper insight into your text, you could read about topics such as:

Semantic Analysis in general might refer to your starting point, where you parse a sentence to understand and label the various parts of speech (POS). A tool for this in Python is spaCy, which words very nicely and also provides visualisations to show to your boss.
Named Entity Recognition (NER) - finding parts of speech (POS) that refer to an entity and linking them to pronouns appearing later in the text. An example is to distinguish between Apple the company, and apple the fruit.
Embeddings - finding latent representation of individual words e.g. using Word2Vec. Text is processed to produce a single embedding for individual words in the form of an n-dimensional vector. You can then compute similarity measures (e.g. cosine similarity) between the vectors for certain words to analyse how they are related.
Lemmatisation - this method reduces many forms of words to their base forms, which means they appear more regularly and we don't consider e.g. verb conjugations as separate words. As an example, tracking, tracked, tracker, might all be reduced to the base form: track.

Your next step could be to search for blogs and introductions to any of those terms I mentioned.

Here is an example parse-tree from spaCy:

Reducing dimensions

This is something that would then refer to the vectors, which describe each of your words. Generally, the Word2Vec vectors are something like 300-dimensional. You might want to visualise the words, plotting them in 2d space. You can try a method like t-SNE, which will map the 300d vectors to 2d space, allowing nice plots showing relationships, while retaining as much of the original relationships described in the 300d space. There will, of couse, be some information loss, but you could not have visualised the 300d vectors in the first place!

Using the vectors for your words, you can compute things like the similarity (on a scale between 0 and 1) between president and CEO is something like 0.92 - meaning they are almost synonyms!


Some of the biggest companies in the world are 'data science' companies; facebook, for instance, performs targeted advertising based on analyzing users preference.  In essence, it's the most highly trained recommendation engine ever, as it knows a great deal about it's consumers.  At it's heart, Google is an advertisement firm, and similarly leverages user information to provide targeted messaging to consumers.  The same can be said for Amazon, Twitter, and most likely pokemon go.

If you're thinking of a more purely 'data science as a service' company, larger consultancy firms such as the Boston Consulting Group have always incorporated analytics in their services, and have certainly profited from the proliferation of data.  They don't say "we do data science", because people care more about getting the right answer, than how the answer is obtained.

As far as Kaggle goes, while it's an awesome resource, and it provides cool tutorials and data sets, if you're a professional data scientist, then you don't work for free ;-)  If a billion dollar company can harvest even a 1% gain in efficiency by offering a $50K prize, they can recoup that money very quickly.  If you're only asset is intellectual property, you need to be cautious about giving it away.


Unbalanced data set - how to optimize hyperparams via grid search?

Scikit-learn's GridSearchCV uses StratifiedKFold so all classes will be proportional represented in the splits. GridSearchCV can be used for hyperparameter search.
Imbalanced-learn's SMOTE can also be used. If there are fewer samples than k, it will only use available samples.


Logistic Regression in R models for 1 or 0?

I always learned to model for 1. See below for the impact of switching the encoding,

model <- glm(ans ~ x, data = simulated_data, family = binomial)
summary(model)

#
# Call:
# glm(formula = ans ~ x, family = binomial, data = simulated_data)
#
# Deviance Residuals:
#       1        2        3        4        5
#  1.6388  -0.6249  -1.2146  -0.8083   0.7389
#
# Coefficients:
#             Estimate Std. Error z value Pr(>|z|)
# (Intercept)  -2.4753     2.5006   -0.99    0.322
# x             0.5957     0.6543    0.91    0.363
#
# (Dispersion parameter for binomial family taken to be 1)
#
#     Null deviance: 6.7301  on 4  degrees of freedom
# Residual deviance: 5.7505  on 3  degrees of freedom
# AIC: 9.7505
#
# Number of Fisher Scoring iterations: 4
#

simulated_data$ans <- !simulated_data$ans
model_opp <- glm(ans ~ x, data = simulated_data, family = binomial)
summary(model_opp)

#
# Call:
# glm(formula = ans ~ x, family = binomial, data = simulated_data)
#
# Deviance Residuals:
#       1        2        3        4        5
# -1.6388   0.6249   1.2146   0.8083  -0.7389
#
# Coefficients:
#             Estimate Std. Error z value Pr(>|z|)
# (Intercept)   2.4753     2.5006    0.99    0.322
# x            -0.5957     0.6543   -0.91    0.363
#
# (Dispersion parameter for binomial family taken to be 1)
#
#     Null deviance: 6.7301  on 4  degrees of freedom
# Residual deviance: 5.7505  on 3  degrees of freedom
# AIC: 9.7505
#
# Number of Fisher Scoring iterations: 4

Hope this helps.


Several possibilities. For example, to add c for existing mon-year combinations in your data frame:

rbind(df, transform(df[!duplicated(df[, 3:4]), ], col1="c", col2=NA))
#     col1 col2 mon year
# 1      a    1 Jan 2016
# 2      a    2 Feb 2016
# 3      a    3 Mar 2016
# 4      a    4 Apr 2016
# 5      b    1 Jan 2016
# 6      b    2 Feb 2016
# 7      b    3 Mar 2016
# 8      b    4 Apr 2016
# 9      a    1 Jan 2015
# 10     a    2 Feb 2015
# 11     a    3 Mar 2015
# 12     a    4 Apr 2015
# 13     b    1 Jan 2015
# 14     b    2 Feb 2015
# 15     b    3 Mar 2015
# 16     b    4 Apr 2015
# 17     c <NA> Jan 2016
# 21     c <NA> Feb 2016
# 31     c <NA> Mar 2016
# 41     c <NA> Apr 2016
# 91     c <NA> Jan 2015
# 101    c <NA> Feb 2015
# 111    c <NA> Mar 2015
# 121    c <NA> Apr 2015

To add c for all possible combinations of existing mon values and existing year values:

rbind(df, data.frame(col1="c", col2=NA, expand.grid(mon=levels(df$mon), year=levels(df$year))))

To add c for all possible combinations of all months names and existing year values:

rbind(df, data.frame(col1="c", col2=NA, expand.grid(mon=month.abb, year=levels(df$year))))

and so on.


Why `max_features=n_features` does not make the Random Forest independent of number of trees?

Interesting puzzle indeed.

First things first. The DecisionTreeClassifier has some stochastic behavior. For instance, the splitter code iterates through the features at random:

        f_j = rand_int(n_drawn_constants, f_i - n_found_constants,
                       random_state)

Your data is small and comes from the same distribution. What this means is that you'll have a lot of identical purity scores depending on how iteration is done. If you (a) increase your data, or (b) make it more separable, you'll see the problem should ameliorate.

To clarify: if the algorithm computes the score for feature A and then computes the score for feature B and it gets score N. Or if it computes first the score for feature B and then for feature A and it gets the same score N, you can see how each decision tree will be different, and have different scores during test, even if the train test is the same (100% if max_depth=None of course). (You can confirm this.)

During my exploration of your question, I have produced the following code with my own implementation of a random forest. Since it took me some time, I figured I might as well paste it here. :) Seriously, it can be useful. You can try to disable random_state from my implementation to see what I mean.

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
import numpy as np

class MyRandomForestClassifier:
    def __init__(self, n_estimators):
        self.n_estimators = n_estimators

    def fit(self, X, y):
        self.trees = [DecisionTreeClassifier(random_state=1).fit(X, y)
                      for _ in range(self.n_estimators)]
        return self

    def predict(self, X):
        yp = [tree.predict(X) for tree in self.trees]
        return ((np.sum(yp, 0) / len(self.trees)) > 0.5).astype(int)

    def score(self, X, y):
        return accuracy_score(y, self.predict(X))

for alpha in (1, 0.1, 0.01):
    np.random.seed(1)
    print('# alpha: %s' % str(alpha))
    N = 1000
    X = np.random.random((N, 10))
    y = np.r_[np.zeros(N//2, int), np.ones(N//2, int)]
    X[y == 1] = X[y == 1]*alpha
    Xtr, Xts, ytr, yts = train_test_split(X, y)

    print('## sklearn forest')
    for n_estimators in (1, 10, 100, 200, 500):
        m = RandomForestClassifier(
            n_estimators, max_features=None, bootstrap=False)
        m.fit(Xtr, ytr)
        print('%3d: %.4f' % (n_estimators, m.score(Xts, yts)))

    print('## my forest')
    for n_estimators in (1, 10, 100, 200, 500):
        m = MyRandomForestClassifier(n_estimators)
        m.fit(Xtr, ytr)
        print('%3d: %.4f' % (n_estimators, m.score(Xts, yts)))
    print()

Summary: Each DecisionTreeClassifier is stochastic, data such as yours, which is small and comes from the same distribution, are bound to produce slightly different trees, even if the random forest itself is deterministic. You can fix this by passing the same seed to each DecisionTreeClassifier which you can do using random_state=something. RandomForestClassifier also has a random_state parameter which it passes along each DecisionTreeClassifier. (This is slightly incorrect, see the edit.)

EDIT2: While this removes the stochasticity component of the training, the decision trees would still be different. The thing is that sklearn ensembles generate a new random seed for each child based on the random state they are given. They do not pass along the same random_state.

You can see this is the case by checking the _set_random_states method from the ensemble base module, in particular this line, which propagates the random_state across the ensembles' children.


Desicion trees make no assumptions on relationships between features. It just constructs splits on single features that improves classification, based on an impurity measure like Gini or entropy. If features A, B are heavily correlated, no /little information can be gained from splitting on B after having split on A. So it would typically get ignored in favor of C.

Of course a single decision tree is very vulnerable to overfitting, so one must either limit depth, prune heavily or preferly average many using an ensemble. Such problems get worse with many features and possibly also with co-variance but this problem occurs independently from multicolinearity.


How to Precondition A Fully Convolutional Neural Network

I've been thinking about this problem before and while I have not researched this or even implemented this, I have a few ideas, from easy/obvious to more elaborate:

Don't make it fully convolutional but add some dense layers where you concat this information
If it's just one category, one-hot encode it and broadcast it over your picture as extra channels at the input or maybe halfway in the network
Similar to the previous suggestion but learn an embedding for your categories that you broadcast either at the beginning or the middle
This time, use one-hot encoding as input and learn an embedding that is for example 8x8 and upsample this to your image resolution
Learn an embedding again, but this time use some upconvolutions to scale it to the input (or middle) size

Let me know how much it helps!


Will images modification get me a better machine learning model?

One can never 100% say that a data preprocessing approach will yield positive results. So, if you are trying different things, always test and use the metrics to see what works best.
With that said, what you described is often referred to data augmentation, namely the generation of more data points, typically from existing data points. It is very common practice and so I encourage you to try it on your problem. Common techniques include flipping, shifting, or rotating the original images.
The problem of having few examples per class is called zero- or few-shot learning. Specific techniques such as siamese networks have been devised to tackle these. Here is a good blog to use as reference.


LSTM for time series - which window size to use

Three simple steps:

Make data stationarity (remove trends and seasonality).
Implement PACF analysis on the label data (For eg: Load).
Interpret the PACF plots to choose an optimal lag value for sliding
window

Concise tutorial for PACF analysis: http://rinterested.github.io/statistics/arima.html


Does scikit-learn have forward selection/stepwise regression algorithm?

No, sklearn doesn't seem to have a forward selection algorithm. However, it does provide recursive feature elimination, which is a greedy feature elimination algorithm similar to sequential backward selection. See the documentation here:

http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html


Correlation matrix with a dataset with many missing value

It is impossible to say whether or not you should trim your data, without doing some Exploratory Data Analysis first. Those missing values may be important to your understanding of the dataset, so you have to look at your data comparing variable to variable to determine if the missing values hold information.
For example, if your data is about vehicles, there could be a variable representing number of doors.  If your missing data is structural an observation of a motorcycle may have a missing value for that variable.  In which case, the observation for that variable could be discarded as meaningless.  Or, you could impute the missing value to something meaningful like the number 0. It depends on what insight you are ultimately trying extract from you data.  Handling missing values is quite a complex subject.

I would suggest that you "Tidy the data" first.  From the Wickham paper

  Tidy datasets are easy to manipulate, model and visualize, and have a specific structure:
  each variable is a column, each observation is a row, and each type of observational unit
  is a table.

Initially this can help you do some Exploratory Analysis to determine what is important and what is not.  Once you have made those distinction you can find and implement strategies to handle the missing values appropriately.


Should I perform customer segmentation before performing churn prediction?

There is no direct reason to segment your customers before estimating their churn probability, unless you know of business reasons for the segments to behave in a substantially different way with regard to churn, or the business users of the model for the different segments will have differing requirements (e.g. on model interpretability). Otherwise, there is nothing that a well-trained churn model could not do that two models (clustering and churn classification) would.
You can actually see for yourself if the approach brings any benefit - add the segmenting features to the churn model input and check whether they have important influence on its outcomes (they are high in the decision tree, have high feature importance, etc.) If they do, segmenting the customers and training separate churn models might bring benefits, especially if you have strict requirements on model interpretability.


Parameter Tuning by Cross Validation for Random Forest

Are you looking at the accuracy on your validation set, rather than your training set? (you should be). Are you making sure your gap between training and validation accuracy is low? (you should be!) Is there enough data to warrant 3-fold cross-validation, or should you do 10-fold (and use more data for training)?

In general, random search (where you sample randomly from the parameter space) will get you a good result faster than grid search.  It has the added benefit that you can specify how many models you want to build, as each model's parameters are sampled independently and not constrained to cover an entire space like grid search.


Pretraining a neural network to teach it general information

Considering two tasks, first, classifying the gender and second, whether its overhydration or dehydration. We can't create a model that perform these two tasks in a combined manner. Yes, a system could be developed for the same.

Discussing pre-trained models for facial recognition and related tasks, we have the FaceNet model as well as VGGFace. We can build a model on top of these highly trained models.

A pretrained model as discussed above will first, provide a bounding box for each face present in the image. The image could be a video frame from a real-time stream.
We will crop that part of the image, so we are only left with a smaller image which only contains the face of the athlete.
Another model, which will be trained on overhydrated vs hydrated faces, will classify the face into the two classes as needed.

For each step, we can train a model and place it in the pipeline. Also, try weight quantization for a faster inference speed.


shifting specific column to before/after specific column in dataframe

If I get the question correct, you just need to change the order of your columns. This can be simply done by reassigning the new order of your columns to the dataframe.

For example:

#['a', 'b', 'c'] <-given columns order
df = df[['c', 'b', 'a']]

You can also use the built-in function reindex to accomplish you task as follow:

cols = df.columns.tolist() #['a', 'b', 'c']
new_cols = [['c', 'b', 'a']]
df = df.reindex(columns=new_cols)

EDIT:

If you have a large number of columns, the problem will arise in how you get the new_cols list. To do this you can use list indexing and slicing. Firstly get the index of columns you wnat to replace by using:

df.columns.get_loc("b") #1

Now suppose you have 699 columns and want to place the 100th and 200th column after the 7th one, you can do this:

cols = cols[0:7] + [cols[100]] + [cols[200]] + cols[8:100] + cols[101:200] + cols[201:]

You can now use this column to change the order of your columns in the above mentioned way. The expression will vary depending on your use case.


Classification targets with heterogenous meanings

Will it be P(5+ and reawaken), or will it be something else?

The events are not independent, so one cannot assume that p(A and B) = p(A) * p(B).

Let's denote the events as follows:

A = reawakens (not A = remains dormant)
B = purchases at least 5 times

The event "reawakens and purchases more than 5 times" is "A and B". In general we have:

p(A and B) = p(A/B) p(B)
p(A and B) = p(A/B) (p(A and B) + p(not A and B))

But it's impossible for a customer to stay dormant and buy anything, so:

p(not A and B) = 0

Which gives us:

    p(A and B) = p(A/B) p(A and B)
    p(A/B) = 1

Then we obtain:

p(A and B) = p(B)

Note that this makes sense intuitively: the probability of buying at least 5 times is the same as awakening and buying at least 5 times.

Currently the labels of the two models overlap so I don't think you can infer much from combining their outputs. A way to make it usable would be to make the first model consider only awakening customers in order to avoid the overlap. But it might be more useful to train a single joint model to get a clear picture of your data. In general such a model would classify between 4 categories:

not A and not B
not A and B
A and not B
A and B

But since p(not A and B) = 0 there are actually only 3 labels corresponding to:

stays dormant
awakens and buys less than 5 times
awakens and buys at least 5 times


First of all, some basics of classification (and in general any supervised ML tasks), just to make sure we have same set of concepts in mind.

Any supervised ML algorithm consists of at least 2 components:

Dataset to train and test on.
Algorithm(s) to handle these data.

Training dataset consists of a set of pairs (x, y), where x is a vector of features and y is predicted variable. Predicted variable is just what you want to know, i.e. in your case it is exception type. Features are more tricky. You cannot just throw raw text into an algorithm, you need to extract meaningful parts of it and organize them as feature vectors first. You've already mentioned a couple of useful features - exception class name (e.g. com.acme.PrintException) and contained words ("Timeout"). All you need is to translate your row exceptions (and human-categorized exception types) into suitable dataset, e.g.:

ex_class                  contains_timeout  ...   | ex_type
-----------------------------------------------------------
[com.acme.PrintException, 1                , ...] | Availability
[java.lang.Exception    , 0                , ...] | Network
 ...

This representation is already much better for ML algorithms. But which one to take?

Taking into account nature of the task and your current approach natural choice is to use decision trees. This class of algorithms will compute optimal decision criteria for all your exception types and print out resulting tree. This is especially useful, because you will have possibility to manually inspect how decision is made and see how much it corresponds to your manually-crafted rules.

There's, however, possibility that some exceptions with exactly the same features will belong to different exception types. In this case probabilistic approach may work well. Despite its name, Naive Bayes classifier works pretty well in most cases. There's one issue with NB and our dataset representation, though: dataset contains categorical variables, and Naive Bayes can work with numerical attributes only*. Standard way to overcome this problem is to use dummy variables. In short, dummy variables are binary variables that simply indicate whether specific category presents or not. For example, single variable ex_class with values {com.acme.PrintException, java.lang.Exception, ...}, etc. may be split into several variables ex_class_printexception, ex_class_exception, etc. with values {0, 1}:

ex_class_printexception  ex_class_exception  contains_timeout | ex_type
-----------------------------------------------------------------------
[1,                    , 0                 , 1              ] | Availability
[0,                    , 1                 , 0              ] | Network

One last algorithm to try is Support Vector Machines (SVM). It neither provides helpful visualisation, nor is probabilistic, but often gives superior results.

* - in fact, neither Bayes theorem, nor Naive Bayes itself state anything about variable type, but most software packages that come to mind rely on numerical features.


Multiple time-series predictions with Random Forests (in Python)

Random forest (as well as most of supervised learning models) accepts a vector $x=(x_1,...x_k)$ for each observation and tries to correctly predict output $y$. So you need to convert your training data to this format. The following pandas-based function will help:

import pandas as pd

def table2lags(table, max_lag, min_lag=0, separator='_'):
    """ Given a dataframe, return a dataframe with different lags of all its columns """
    values=[]
    for i in range(min_lag, max_lag + 1):
        values.append(table.shift(i).copy())
        values[-1].columns = [c + separator + str(i) for c in table.columns]
    return pd.concat(values, axis=1)

For example, the following code:

df = pd.DataFrame({'y1':[1,2,3,4,5], 'y2':[10,20,40,50,30], 'z': [1,4,9,16,25]})
x = table2lags(df[['y1', 'y2']], 2)
print(x)

will produce output

   y1_0  y1_1  y1_2  y2_0  y2_1  y2_2
0   1.0   NaN   NaN  10.0   NaN   NaN
1   2.0   1.0   NaN  20.0  10.0   NaN
2   3.0   2.0   1.0  40.0  20.0  10.0
3   4.0   3.0   2.0  50.0  40.0  20.0
4   5.0   4.0   3.0  30.0  50.0  40.0

The first two rows have missing values, because lags 1 and 2 are undefined on them. You can fill them with what you find appropriate, or simply omit them.

When you have matrix of $x$ values, you can feed it, for example, to a scikit-learn regressor:

from sklearn.ensemble import RandomForestRegressor
rf = RandomForestRegressor().fit(x[2:], df['z'][2:])

Finally, a piece of advice. Your model could much improve if you used not only raw lagged values as features, but also their different aggregations: mean, other linear combinations (e.g. ewm), quantiles, etc. Including additional linear combinations into a linear model is useless, but for tree-based models it can be of much help.


Where to upload large (0.5Gb) weights anonymously?

file.io or transfer.sh are good choices. transfer.sh has a 14 day lifespan for all files, while you can increase or decrease the lifespan on file.io. Additionally, you can use both from a command line or terminal via curl, and they're anonymous.


Explaining GRU equations from two sources

The equations are almost the same. First of all, they are written in a slightly different form. In Cho, et al. (2014), it is written that $W_r$ and $U_r$ are weight matrices which are learned. While in Colah's blog, both matrices are referred to as $W$.

For example you could write these two equivalent forms:

$z_j = \sigma([W_z x]_j + [W_z h_{t-1}]_j)$

$z_j = \sigma(W_z[x,h_{t-1}]_j)$

Secondly, there is a small mistake in Colah's for $h_t$. Let's ignore the different notation $h_j^{(t)}$ used in the paper and use Colah's notation $h_t$ instead. Then the equation for $h$ in Cho, et al. (2014) can be written as:

$h_t = z_t \ast h_{t - 1} + (1 - z_t) \ast \tilde{h_t}$

and if we use the same order as Colah's equation:

Paper's equation: $h_t = (1 - z_t) \ast \tilde{h_t} + z_t \ast h_{t - 1}$

Colah's equation: $h_t = (1 - z_t) \ast h_{t - 1} + z_t \ast \tilde{h_t}$

So we can actually see that for Colah's equation to match the one in the paper, we have to swap $\tilde{h_t}$ and $h_{t - 1}$ in this equation.
If you check the comments next to the GRU example on Colah's blog, you can actually see that some other people found the same mistake. Well spotted, I didn't saw it at first! :)


How to count categorical values including zero occurrence?

Based on the short example DataFrame you provided, this block of code will include all of the months. It is based on using the Series.reindex method and creating a new MultiIndex with the additional values for the months:

import pandas as pd
# Load example data into DataFrame
df = pd.read_table("categorical_data.txt", delim_whitespace=True)

# Transform to a count
count = df.groupby(['id', 'code', 'month']).month.count()

# Re-create a new array of levels, now including all 12 months
levels = [count.index.levels[0].values, count.index.levels[1].values, range(12)]
new_index = pd.MultiIndex.from_product(levels, names=count.index.names)

# Reindex the count and fill empty values with zero (NaN by default)
count = count.reindex(new_index, fill_value=0)
print(count)

Printing the result, I get something like this (only showing the first entry for sally/s_A):

id     code  month
sally  s_A   0        4
             1        0
             2        0
             3        0
             4        0
             5        0
             6        0
             7        0
             8        0
             9        0
             10       0
             11       0


Functions in scikit that detect outliers automatically?

The scikit-learn implementations of Isolation Forest and Local Outlier Factor should help you out here. Details on both at this link:

https://scikit-learn.org/stable/modules/outlier_detection.html


Keras models break when I add batch normalization

This link describes the cause of the problem. Its because we try to feed a placeholder defined as an Input layer using another input layer. You can modify your code as follows:

def buildDDPGNets(actNum, obsSpace):

  actorObsInput = Input(shape = (1,) + obsSpace, name = "actor_obs_input")
  criticActInput = Input(shape = (actNum,), name = "critic_action_input")

  flattened_obs = Flatten()(actorObsInput)
  a = Dense(600, use_bias = False)(flattened_obs)
  a = BatchNormalization()(a)
  a = Activation("relu")(a)
  a = Dense(300, use_bias = False)(a)
  a = BatchNormalization()(a)
  a = Activation("relu")(a)
  a = Dense(actNum)(a)
  a = Activation("tanh")(a)   # Bipdeal walker applies torque (-1, 1).
  actor = Model(inputs = [actorObsInput], outputs = a)

  c = Dense(600, use_bias = False)(flattened_obs)
  c = BatchNormalization()(c)
  c = Activation("relu")(c)
  c = Concatenate()([c, criticActInput])
  c = Dense(300, use_bias = False)(c)
  c = BatchNormalization()(c)
  c = Activation("relu")(c)
  c = Dense(1)(c)
  c = Activation("linear")(c)
  critic = Model(inputs = [criticActInput, criticObsInput], outputs = c)

  return (criticActInput, actor, critic)


RL Policy Gradient: How to deal with rewards that are strictly positive?

Let's say we're playing a game where the reward is always positive (eg. accumulating a score), and there are never any negative rewards, the gradient will always be positive, hence θ will keep increasing! So how do we deal with rewards that never change sign?

This is true. However, in many policy functions and in most situations, the gradient part $\nabla_{\theta} log \pi_{\theta}(s_t,a_t)$ will tend to zero as you reach a deterministic policy. This happens for a softmax action selection based on "preferences" (a matrix of softmax weights per action for each state) or as the output layer of a neural network. And it will counteract the tendency for the final layer preferences (or the logits of the neural network last layer) to grow uncontrollably.

You have identified a true weakness of REINFORCE. For instance, using the softmax action selection as an example and your always-positive returns:

When the agent selects a non-maximising action, this will result in a positive return, the agent will add to its preference for that action.
When the agent selects a maximising action, this will result in a larger positive return, the agent will add to its preference for that action more
REINFORCE works by increasing the preferences of better actions faster than preferences of worse actions.
This leads to a feedback process where better actions are chosen more often, increasing their preference values even faster.
Ultimately the preference for best actions will be so much higher than the alternatives that the softmax function will saturate. The gradient  $\nabla_{\theta} log \pi_{\theta}(s_t,a_t)$ for all actions will be close to zero.

This does not always happen quickly, and basic REINFORCE implementations can be numerically unstable for exactly the reason in your question. To improve stability (and often learning speed), you can use REINFORCE with baseline, which starts to address your concern by using offsetting values ($v_t - \bar{v}_t$ or similar). You can also then take that idea further and use Actor-Critic.


Fastest way for 1 vs all lookup on embeddings

There are libraries that are specialized in exactly that task, for instance FAISS by Facebook AI Research:

  Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and parameter tuning. Faiss is written in C++ with complete wrappers for Python/numpy. Some of the most useful algorithms are implemented on the GPU. It is developed by Facebook AI Research.

For cosine similarity, you can use FAISS class IndexFlatIP having normalized the vectors first, as specified in FAISS documentation.

If you want to know more about the techniques behind FAISS, you can have a look at their research paper, and if you want to know more about the fundamentals of similarity search in general, you can have a look at this blog post by Flickr .

Some alternatives to FAISS are Annoy, NMSLib and Yahoo's NGT. You can find a couple of comparisons of these and other libraries here and here.


From your question it seems that you are easily able to identify the periods assets were down for maintenance and simply wonder how to best code that in the data?

If that is the case I would simply add a new column of information marking assets as either 1 - active or 0 - down for each period. If need be you could even code it into active/down/maintenance, etc.

This would be additional valuable data for your prediction model (maybe assets are more or less likely to go down after a maintenance period). Additional it would allow you to simply mark the sensor data as missing NA because you will be able to differentiate between a broken sensor and an asset on maintenance from your other data.

As @Tasty213 suggested any good classification algorithm might have been able to do this as well but it seems you are simply able to add this data yourself.


What visualization technique to best describe a recommendation dataset?

I think you're looking for a similarity matrix (see bottom of the page). If you don't have data on similarity between certain pairs, you can always leave them as grey or white. Also, this will only work for data sets small enough to actually make out what's going on. I'd say 25 rows / columns maximum.

In a similarity matrix, x, and y coordinates correspond to the two things you're comparing, while a colormap magnitude represents similarity

EDIT:
One thing you could do to replace the colormap is the insert, say, circles of different sizes according to the similarity metric. Or you could insert the numbers themselves, again, varying the size of the number as the magnitude of that number varies. Size usually works best is business visualizations.


Are there neural networks packages that use complex numbers?

Neural networks learn a function mapping from the input space to the output. This can be framed as a classification or a regression problem. In both of these cases the function determines the most information rich inputs which will lead to a minimized loss. We minimize the loss with respect to the input space.

If we have a complex valued input this is no problem. Let's assume two complimentary extreme cases. If we have a complex image, however, the complex part is completely randomly distributed. Then this problem would be equivalent to only using the real part of the image as the input. This is due to the complex part not having any pertinent information which would lead to the classification or regression. This is equally true, if the real part of the image part is completely random but the complex part contains all the information.

For the in between situation where we do not know the information distribution in the real and imaginary space then you can input of these components into the network.

For a 64*64 complex image. This will result in $2*64^2$ input nodes each accepting a floating point value, if the first layer is a densely connected layer. This is commonly used for MRI images. So the nodes can identify which real or complex values are needed to infer meaning and provide an output.

I have not seen any implementation that directly applies neural networks on complex values. But derivatives of complex values exist. You would need nodes which have complex functions such that the gradient is applied to both the real and imaginary part. From a guess, I suppose it could be possible. But the common method is just to split the real and imaginary part and feeding them to the network.


ID and name are probably not predictive of retention.
With joining and leaving date, you can model the distribution of time at the organization.
Unless you have other features, you will not be create a useful model.


Multivariable real time system for fraud detection

I think it makes sense to stick with Classification here, since you already have examples of fraudulent and non-fraudulent calls that you can train on.

It might also be beneficial to train several models for different regions based on IP, as well as your 'global' model, and apply the region specific and global models to incoming calls. Just a few ideas.

From what I understand, real-time learning would require immediate feedback, which most fraud detection systems can't provide. For example, it may take a few days or weeks to have a case of fraud resolved (labelled fraud/not-fraud), and therefore take some time for the learning system to receive the feedback on it's prediction.


Assuming that the rows are in same order that you wish to merge all of the dataframes, you can use the concat command specificying axis=1.

new_df = pd.concat([df1, df2, df3], axis=1)

If the row index for each of the data frames are different and you want to merge them in the current order, you can also apply ignore_index:

new_df = pd.concat([df1, df2, df3], ignore_index=True)

For details on the merge, join & concatenation operations, please refer to the pandas docs.


keras UserWarning: The input 1303 could not be retrieved. It could be because a worker has died

The solution for me was to copy all training data from the mounted Google Drive file system to the Google Colab file system. I didn't realize before that the mounted Drive file systems has a comparatively bad performance that leads to this error.


Dummy variable only for character value in a column (Neglecting float and integers)

You have 2000 different values available for a dataset of 3000. I don't think you should treat that as a Categorical column.
Treat "No_information" as a NaN and impute using the best-suited strategy using the relation with other columns
Edit post comment
If you want to treat it as Categorical, I will suggest
- Create "bins" of values e.g. <50- Very Low, 50-500 - Medium, etc.
-  Then create One-Hot encoded data
-  With this approach too, "No_information" should be treated as NaN. The reason being that when Info is not available it could have been anything i.e. Low, High, Medium. Treating it as 4th value will have lesser information.
 - Try different bins/approaches and see which produce the best result
Dealing with High cardinality of Categorical feature -
- Search internet/SE with "Encode feature with very high cardinality". This is a known challenge, you will get plenty of resources.
 - Try other encoding approaches. See these links -
Beyond OHE
Library
kaggle post


How to find correlation between time-series of different units?

I recommend you to scale all three time series to unitless values, with zero mean and standard deviation of 1. After that, you can search for correlation among normalized time series. Python has StandardScaler function that could help with it.


How to pass a sequence of 4 images into LSTM and CNN-LSTM

CNN-LSTM are used in text recognition. How is text structured? Characters are stacking in a horizontal manner. Do the same with your MNIST images and for the ground truth make proper labels (should be of 4 size).


How to control the amount of positives in classification?

as desertnaut was saying, I think you are talking about class imbalance.

If you meet class imbalance "in the wild" there are a couple of ways to tackle this. Your approach of creating a custom metric would fall under cost function based approaches. You can use cost functions that handle false negatives as worse than false positives (which is usually the case when we're talking about class imbalance).

Another way would be to use sampling based approaches. These are rather simple and I personally tend to use these more often since it's easier to play around with aka experiment with them. Two extremes that can be combined as you wish:

Boosting one class / Oversampling: Look at the distribution of Class A and Class B. Boost the class you want to have more effect to have more datasamples changing your algorithm.
Undersampling: Look at the distribution of Class A and Class B. Remove datasets of the class you want to have less effect of your algorithm.

In reality I usually go for a combination of both to have more variance in my data. For example boosting Class A by 110% and reducing Class B to 80% to achieve a balanced dataset (although it sounds to me that you want to have an imbalance just the other way round. Take care about bias with that though)


Selecting ML algorithm for music composition

Let us formulate this problem in such a way that it can be understood from a machine learning perspective. You have a set of instances $X$ where each instance $x_i \in \mathbb{R}^m$ where $m$ is the dimensionality of the instance. In other words $m$ is the number of features that describe the instance. Your problem intends to go from a set of features to a class label good or bad. Thus, this is a mapping from $\mathbb{R}^m$ to $y \in \{0, 1\}$.

How to achieve this mapping?

This is when we will use the machine learning algorithms. We will train a model to effectively approximate the function which gives the output label from a set of inputs. It is evident that sparse features (low information entropy) will complicate the mapping function and will thus provide worse results. This is why feature engineering is of upmost importance for machine learning. It is probably the hardest part of the machine learning pipeline, however it is the lead factor in dictating your results.

You can use some feature reduction techniques in order to remove features which are uninformative with respect to the output label. Some techniques that I use frequently are principle component analysis (PCA), linear discriminant analysis (LDA). Alternatively, you can use some projection methods to reduce the dimensionality of the data whilst maintaining separation between the classes. Such techniques are Isomap, MDS, Spectral Embeddings and TSNE. You can check to see which is best suited for your type of data.

How to choose a model?

Firstly, your problem is a supervised classification problem. This already narrows the types of models you can use. Furthermore, model selection is based on some key factors such as: the number of instances you have, the number of features per instance and the number of output nodes. You should also keep in consideration that the separability of the probability distribution between the output classes will impact the performance of the model directly. For example discriminating between cars and oranges is much easier than oranges and clementines.

In your case, you have 1,000 instances and around 13 features. This means that deep learning based techniques are possible but discouraged. You do not have enough data. You can then attempt the following popular classification models

Support Vector Classifier
Naive Bayes
K-Nearest Neighbors
Decision Trees
Random Forests

To evaluate which model performs the best you will use the accuracy attained with a trained model on a test set. This set should be drawn independently from the training set as to catch overfitting. This is when the model cannot generalize to new data.

In code

Assuming matrix $X$ contains the data where rows are the instances and columns are the features, and matrix $Y$ contains the labels.

First we split our data into a training and testing set

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33)

from sklearn.svm import SVC
clf = SVC()
clf.fit(X_train, y_train)
print('Score: ', clf.score(X_test, y_test))

from sklearn.neighbors import KNeighborsClassifier
neigh = KNeighborsClassifier(n_neighbors=3)
neigh.fit(X_train, y_train)
print('Score: ', neigh.score(X_test, y_test))

from sklearn import tree
clf = tree.DecisionTreeClassifier()
clf.fit(X_train, y_train)
print('Score: ', clf.score(X_test, y_test))

from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators = 100)
forest.fit(X_train, y_train)
print('Score: ', forest.score(X_test, y_test))

This should be a starting point. Let us know if you fall into any problems, and let us know what accuracy you are getting we can then look deeper into these models and better suit them to your data source.


is final fit with X,y or X_train , y_train?

Training dataset is something that you use to train your
dataset(data    which the model learns upon).
Validation dataset is that part of your data which is not used to
train your model, but to check the model's performance and optimize
it further. Here, as the model is optimized based on its performance
on the validation dataset, so according to experts, your model though
indirectly, has 'seen' the validation data as you might unknowingly
create a bias towards the validation data by optimizing the model to
perform better on the validation data only.
The stage of testing the models, comes after all the optimization(by
that time your model has processed (but not trained on) the validation
data). So, once you think your model is
optimized (you simply can't know it). Now you can and should use validation data also to train
the model along with the training data.
The testing data on the other hand is that part of your data which is
completely unknown to model(that is neither it is used to train nor
to validate the model). It is the completely unbiased performance
check of your model.

Usually in case of competitions, the data you are given, since it has labels, it should be used to train and optimize you model. But testing should always be done only after the model has been trained on all the labeled data, that includes your training(X_train, y_train) and validation data(X_test, y_test).

Hence you should submit the prediction after seeing whole labeled data :- Hence clf.fit(X, Y)

I know this long explanation was not necessary, but one should know why you do what you do.

Hope it helps, thanks!!


How do you calculate sample difference in terms of sensor signals?


Exceptionally high accuracy with Random Forest, is it possible?

In order to understand if the model is performing well, first would do the following:

Plot the distribution of the classes, to understand if sampling mechanisms are required.
In case the classes are not evenly distributed, would do stratified sampling during the test-train split.
After the prediction, would plot the confusion matrix that is supported by libraries such as matplotlib or seaborn
Based on the class distribution its also important to understand what sort of metrics are required, micro-averaging / weighted / macro-averaging of precision, recall and f1 score.

This should help you evaluate if you are model is truly learning the features or if one of the classes are imbalanced causing such a spike in the results.


How can I weigh observations differently that were provided for a time horizon?

Therefore, the year of the observation was considered as a predictor variable which had a positive impact on the dependent variable. However, the year itself doesn’t have any effect on the dependent variable per se; rather, it is the other factors occurring in the same time period which result in improvements.

This sounds like a challenge for trees with sufficient interaction depth, as you've found that the year interacts with other factors that results in improvements. Ordinary least squares regression here do not capture that type of interaction well.

I would suggest setting up the following regression model:

Transform the date column into (CurrentYear - YearOfDateStamp) AS NumOfYearsAway. I'd recommend leaving this as a numerical feature rather than a categorical one. This would allow the tree based model to select cuts like NumOfYearsAway >= 5.5 instead of NumOfYearsAway in (6,7,10). This could also be more helpful when NumOfYearsAway = 0 occurs in your scoring dataset, where you might not have training data for current year data set.
Fit a tree based model, I'd pick XGBoost, with the usual CV to tune the hyperparameters such as interaction depth.

The drawback of picking XGBoost in your application however, is that the interpretation of the impact of a particular variable on the target variable is not obvious. You'd need partial dependence plot to observe how the target variable vary with the bespoke input variable. If interpretability is very important, one could pick a single tree regression model like rpart.


Why bias is not considering in Regularization?

The point of regularization is to avoid overfitting, and overfitting happens when you have too many predictor variables (i.e. neurons) contributing to the outcome. So, by regularization you are essentially excluding some neurons.

However, bias is an intrinsic property of every neuron. It is not a connection transporting the result of a neuron from the previous layer. Maybe you can imagine all biases in the network being connections to a single, virtual neuron with a constant output of one, but still, as it is a constant, it is not a result of a computation and does not depend on the input.


Random Forest Classifier - KFold CV Tunes Very Deep Trees --> Overfitting?

There are signs of overfitting the data when the model's performance drops on the test set considerably.

Good feature engineering can also improves your model's performance. Considering that you have an imbalanced data set, you can look at resampling techniques for your data set such as random over/under-sampling and SMOTE. You can also extract feature importances from ensemble learners or regularized regression models like LASSO to see which features are contributing the most weight in your model's predictions. You could also apply various statistical tests such as ANOVA or chi-square square tests to gain further insights of your data. There is lots to do here.

You can also reconsider using the ROC AUC metric and choose something better tuned to a class imbalance in your data set. The ROC curve plots the True Positive Rates (TPR) on the x-axis and the False Positive Rates(FPR) on the y-axis whose formulas respectively correspond to $$TPR=\frac{TP}{TP+FN}$$ and $$FPR=\frac{FP}{FP+TP}$$ Since TPR is based only on the TP instances, the ROC curve won't measure the effects of negative instances. The AUC does does not consider the class imbalance when evaluating overall model performance, so we cannot trust it to weigh on the underrepresented class.

One way to resolve this is by using the Matthews Correlation Coefficient since it takes into account true and false positives and negatives. This helps represent the minority class a lot more during your model evaluation.

You could also look into random grid searched for your hyperparameter tuning; this method uses a fixed number of parameter settings sampled from your specified distribution. This allows you to get a general neighborhood of your parameter space where you can use a brute force grid search in a shorter parameter space. Then you can use cross validation to evaluate your model as you normally would.

This is just a handful of things to consider, there are many other things that you may find are suitable for your problem.


As you require it to be per week vs point of sale, you have to group them by those columns. Once grouped, you could run do the standard split-apply-combine, where based on the grouping, you get splits or groups, you can apply a function that normalizes data in this group and then you combine all the groups back.

You can read about it more here : https://pandas.pydata.org/pandas-docs/stable/groupby.html


How to plot an histogram from a dictionary data?


Definition of a model in machine learning

I was interested in the same question recently and came to the realization that there is no single definition of a "model" in machine learning. It's highly dependent on the sources you're consulting, which may be the documentation for a particular software program, the slang adopted by its user community, or the definitions used in published academic papers, which can vary widely from journal to journal. Moreover, I had to learn to keep in mind that such papers are written not just by specialists in machine learning, but by experts in other disciplines who have a need for applying machine learning techniques (such as imaging, various medical fields, etc.). Many of them do not explicitly define the term "model," which is often used loosely. Here are just a couple of different definition of "model" I've seen in all this widely scattered literature:

• Statistical models, particularly the stats related to probability distributions.

• Regression data and related statistics.

• Mathematical models as mentioned by Neil Slater above.

• The data models used in machine learning, such as the columns involved, their data types, the data sources and other metadata. This is particularly tricky because there's nothing mathematical about this definition at all, unlike the first three I listed. For an example, see all the documentation for SQL Server "mining models," which serve double-duty for machine learning purposes.

• Sometimes all of the definitions above are expanded to include machine learning structures built on top of the equations and the metadata, such as the specifications of neural nets. In other cases, these are considered separate entities.

All of the above are sometimes mixed and matched together, depending on the source. I'm sure there are other definitions of "model" I've left off this list, which will complicate the matter even further. To deal with this ambiguity, I'm trying to train myself to divine the intentions of the author whenever they use the term "model." Sometimes it's easy to determine based on the context or the field the author works in, but other times I have to read deeply into an article or documentation before figuring it out. I wish I could be more definitive about it, but it's really a naturally fuzzy term; there's never going to be a simple one-size-fits-all answer to this. I hope that helps.


If you change your for loop to for(i in 1:nrow(data)){... then it will work.

I would not call your variable min because that is the name of the function which you should be using instead such as:

tmp=min(data[,2])


How to adjust the hyperparameters of MLP classifier to get more perfect performance

If you are using SKlearn, you can use their hyper-parameter optimization tools.

For example, you can use:

GridSearchCV
RandomizedSearchCV

If you use GridSearchCV, you can do the following:

1) Choose your classifier

from sklearn.neural_network import MLPClassifier
mlp = MLPClassifier(max_iter=100)

2) Define a hyper-parameter space to search. (All the values that you want to try out.)

parameter_space = {
    'hidden_layer_sizes': [(50,50,50), (50,100,50), (100,)],
    'activation': ['tanh', 'relu'],
    'solver': ['sgd', 'adam'],
    'alpha': [0.0001, 0.05],
    'learning_rate': ['constant','adaptive'],
}

Note: the max_iter=100 that you defined on the initializer is not in the grid. So, that number will be constant, while the ones in the grid will be searched.

3) Run the search:

from sklearn.model_selection import GridSearchCV

clf = GridSearchCV(mlp, parameter_space, n_jobs=-1, cv=3)
clf.fit(DEAP_x_train, DEAP_y_train)

Note: the parameter n_jobs is to define how many CPU cores from your computer to use (-1 is for all the cores available). The cv is the number of splits for cross-validation.

4) See the best results:

# Best paramete set
print('Best parameters found:\n', clf.best_params_)

# All results
means = clf.cv_results_['mean_test_score']
stds = clf.cv_results_['std_test_score']
for mean, std, params in zip(means, stds, clf.cv_results_['params']):
    print("%0.3f (+/-%0.03f) for %r" % (mean, std * 2, params))

5) Now you can use the clf to make new predictions. For example, check the performance on your test set.

y_true, y_pred = DEAP_y_test , clf.predict(DEAP_x_test)

from sklearn.metrics import classification_report
print('Results on the test set:')
print(classification_report(y_true, y_pred))


Need equations for some of weight initializers in tensorflow?

2 - Not necessarily between 0 an 1. Example: If you see the normal glorot formulae, The sigma_i value will entirely depend on fan_in and fan_out. So, it need not be centred at 0.

EDIT 1:

EDIT 2:

Research paper - weight initialization by Xavier

Credits: Also, check out this and this
Hope it helps.


How do I go from number data to string data (NN)?

There are various ways to handle string inputs to neural networks, but since you are trying to predict spelling difficulty, I suggests representing your words as a sequence of characters. This will preserve information about the particular spelling of the words including the order of the letters.

To represent a sequence of characters you can one-hot encode each character, so each word will be represented as a sequence of one-hot length 26 vectors.

To handle this kind of input I suggest either a 1D Convolutional Neural Network or some flavor of a Recurrent Neural Network.

If you choose a 1D CNN you will have to feed it fixed sized inputs. To do this choose a max word length, k, and either cut off or pad with zeros each input word to fit into a k*26 input matrix.

Here is an example of a CNN architecture you might use in Keras:

model = Sequential()
model.add(Convolution1D(nb_filter=32, filter_length=5, activation='relu', input_shape=(k, 26))
model.add(MaxPooling1D())
model.add(Convolution1D(nb_filter=32, filter_length=5, activation='relu')
model.add(MaxPooling1D())
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(3, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])


time series forecasting - sliding window method

Try this:

Make the data stationary (remove trends and seasonality).
Implement PACF analysis on the label data (For eg: Load) and find out the optimal lag value. Usually, you need to know how to interpret PACF plots.
Apply the sliding window on the whole data (t+o, t-o) where o is the optimal lag value.
Apply walk forward validation to train and test the models.

The way to escape sliding window is to use Recurrent Neural Networks but believe me, the method I suggested is worth it.

If you want the code:
https://machinelearningmastery.com/time-series-forecasting-supervised-learning/

This can help too: Master Thesis research applicable to your question


Confused about description of YearPrediction Dataset

You are right, the covariance matrix should have n^2 elements. However, since cov_{i,j} = cov_{j,i}, there is no need to have a repeated feature cov_{j,i} if cov_{i,j} is already accounted for. Hence there will be only n*(n+1)/2 = 12*13/2 = 78 unique covariances and thus only 78 unique covariance based features (n of those will be variances).


Kmeans utilize the mean of your data points for clustering . If your dataset is made of plain text or other type of factors (i.e not numbers) then it wont work for you . You need to do another step of preprocessing your data before you can apply Kmean or most of the ML algorithms .

Categorical dataset : i.e your data is in the form of multiple categories like column of fruits  with values of Apple , orange ,banana etc. Then you can use "one hot encoding" method that will transform your category column into multiple columns that each indicate if the sample is belong to the relevant category (i.e for column with 3 fruit types you will get 3 new binary (1 or 0) columns - is apple ? is orange? is banana ?  read more about how to do it in R here : One hot encoding in R

Update: like some suggested in the comments , K means wont be the best approach for clustering categorical data and in some cases you can get much better results when using more suitable approaches .Here is a link to another (more advanced) method for clustering categorical data in R - ROCK algorithem (kaggle notebook) . Also ,you can read about "Kmode" which is similar to kmeans for categories and implemented in R

If your dataset is plain text (like tweets or stackexchange posts) :
One common method is using td-idf (but there are many more) , you can read more here:
Text clustering using R: an introduction for data scientists
and here in a nice kaggle R notebook:
R : cleaning data, and using TF-IDF


Welcome to the community Dalide!

There is a sequential pattern in your data. As a non-Bioinformatics guy, I simply googled and saw that biomarker are also vastly used for learning the Progression of a disease (e.g. Alzheimer). I hope I am right about example and if I am, then it is not a big surprise that you see those curves which might indicate the progress of something. If you need more precise answer you better give some more info about the origin of data (or a link to your data if it is publicly available).

From ML point of view, about which I fortunately know a bit, those curves in tSNE are caused by sequential patterns.

Fun: As an equally surprising example, once I found sequential pattern in characteristics of different Wines!! (each row a Wine and each column a wine property like Alcohole%, density, etc.)


Hadoop and input informations divided in splits

All bigdata eco system works on something called parallel processing.

We have to process 100gigs of file. If we didnt split the file, then all the 100 gigs should be processed by single JVM(single map).

If we split the file into 1000 parts each of 100mb, then we can process each part with different JVM and apply the map function in less time.

MPP: Massively parallel Processing


Python Time series: extracting features on a rolling window basis

Yes, there are easy ways to do this in Python. My favourite would be to put the data into a Pandas DataFrame, which has a convenient method called rolling that will cycle over your data in a given frame-size and compute whatever you like on that block.

Let me show you an example - say we start with the following column of data:

In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: df = pd.DataFrame({"A": np.random.randint(0, 100, (20,)),
                           "B": np.random.randn(20)})

Look at the first 10 rows:

In [4]: df.head(10)
Out[4]:
     A         B
0   63 -0.003947
1   55  0.442597
2    6  0.684125
3   17  0.968987
4   33 -0.018640
5   50 -0.579558
6   71  0.563125
7   31  1.417384
8    8  0.607813
9   36  0.186146

We can compute the rolling average over each column and save it back to the dataframe like this:

In [6]: df[["rolling_a", "rolling_b"]] = df.rolling(5).mean()
In [7]: df.head(10)
In [9]: df
Out[9]:
     A         B  rolling_a  rolling_b
0   63 -0.003947        NaN        NaN
1   55  0.442597        NaN        NaN
2    6  0.684125        NaN        NaN
3   17  0.968987        NaN        NaN
4   33 -0.018640       34.8   0.414624
5   50 -0.579558       32.2   0.299502
6   71  0.563125       35.4   0.323608
7   31  1.417384       40.4   0.470260
8    8  0.607813       38.6   0.398025
9   36  0.186146       39.2   0.438982

You might notice that the first 4 rows contain NaN values (Not a Number). This is because the rolling() method will let the mean() method work an a window-size smaller than 5 (in our example). There are a lot of options in the rolling() method that you can experiment with.

You can do the same above for single column of a large dataframe like this:

>>> df["rolling_some_column_name"] = df.some_column_name.rolling(5).mean()

You can also apply just about any function to the rolling frame - not just mean().


How are neural nets related to Fourier transforms?

They are not related in any meaningful sense. Sure, you can use them both to extract features, or do any number of things, but the same can be said about a many techniques. I would have asked "what kind of neural network?" to see if the interviewer had something specific in mind.


Question about train example code for TensorFlow

If I understood you correctly, you are asking about this line of code:

train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})

Here you only specify which part of batch is used for features and which for your predicted class.


Sensitivity analysis of a machine learning model

Without further knowledge of the domain, there is no simple answer.

Are the predictor variables discrete or continuous? If they are discrete, you could search all combinations that sum to 0, then all combinations that sum to 1, etc. until you find a combination that increases y by 10. If they are continuous, then:
Is the target variable monotonically increasing with each of the predictor variables? In other words, given A, B, C and D, when increasing A, B or C, will y always either increase or stay the same? If the predictors are continuous and the target variable is monotonically increasing, you can do a search similar to the one above by increasing each variable by a small step until y has increased by more than 10, this will give you an idea, within the range of that small constant, of where the answer to your question lies. You can then reduce the size of the step to find a more precise answer within that range.
If the target variable is continuous and not monotonically increasing with the predictor variables, this gets much more complicated. Suppose you've found two points (A1, B1, C1, D) and (A2, B2, C2, D) that do not increase y by 10: nothing guarantees that there is not a point between the two, (A3, B3, C3, D), that does. Your best bet then would probably be to explore the model. Depending on the number of trees in your ensemble, this may or may not be possible (computationally). I think you would have to find all possible combinations of leaves where D is unchanged and y is increased by more than 10, then find the minimum amongst these.


What's the relationship between an SVM and hinge loss?

They are both discriminative models, yes. The logistic regression loss function is conceptually a function of all points. Correctly classified points add very little to the loss function, adding more if they are close to the boundary. The points near the boundary are therefore more important to the loss and therefore deciding how good the boundary is.

SVM uses a hinge loss, which conceptually puts the emphasis on the boundary points. Anything farther than the closest points contributes nothing to the loss because of the "hinge" (the max) in the function. Those closest points are the support vectors, simply. Therefore it actually reduces to picking a boundary that creates the largest margin -- distance to closest point. The theory is that the boundary case is all that really matters to generalization.

The downside is that hinge loss is not differentiable, but that just means it takes more math to discover how to optimize it via Lagrange multipliers. It doesn't really handle the case where data isn't linearly separable. Slack variables are a trick that lets this possibility be incorporated cleanly into the optimization problem.

You can use hinge loss with "deep learning", e.g. http://arxiv.org/pdf/1306.0239.pdf


What is the difference between Okapi bm25 and NMSLIB?

Note that I don't know nmslib and I'm not familiar with search optimization in general. However I know Okapi BM25 weighting.

How do they both (bm25, nmslib) differ?

These are two completely different things:

Okapi BM25 is a weighting scheme which has a better theoretical basis than the well known TFIDF weighting scheme. Both methods are intended to score words according to how "important" they are in the context of a document collection, mostly by giving more weight to words which appear rarely. As a weighting scheme, Okapi BM25 only provides a representation of the documents/queries, what you do with it is up to you.
nmslib is an optimized similarity search library. I assume that it takes as input any set of vectors for the documents and query. So one could provide them with vectors made of raw frequencies, TFIDF or anything else. What it does is just computing (as fast as possible) the most similar documents to a query, using whatever representation of documents is provided.

How can I pass bm25 weights to nmslib to create a better and faster search engine?

Since you mention that the results based on BM25 are satisfying, it means that the loss of quality is due to the nmslib search optimizations. There's no magic, the only way to make things fast is to do less comparisons, and sometimes that means discarding a potentially good candidate by mistake. So the problem is not about passing the BM25 weights, it's about understanding and tuning the parameters of nmslib: there are certainly parameters which allow the user to select an appropriate trade off between speed and quality.


To deal with missing data you can use one of the following three options:

If there are not many instances with missing values, you can just
delete the ones with missing values.
If you have many features and it is affordable to lose some information, delete the entire feature with missing values.
The best method is to fill some value (mean, median) in place of missing value. You can calculate the mean of the rest of the training examples for that feature and fill all the missing values with the mean. This works out pretty well as the mean value stays in the distribution of your data.

Note: When you replace the missing values with the mean, calculate the mean only using training set. Also, store that value and use it to change the missing values in the test set also.

In python you can use the Imputer() class to fill up the missing values as follows:

from sklearn.preprocessing import Imputer
impute = Imputer(strategy="median")
impute.fit(df)

Also you can check the calculated median values for each column by using:

imputer.statistics_