To build a model to detect the old categories and your new ones, you need to re-train the model with your own dataset and the dataset used to pre-train the model. Fortunately, this dataset is available to the public for downloading. The dataset used to train the model is written in the name of model you choose. It's probably the COCO dataset.

Since the model as been pre-trained to detect the old categories, I think you don't need to download all the dataset. The data is just necessary to avoid the model to "forget" what it has learned.

I hope you understood me, and that I have answered well your question.


Assumptions/Limitations of Random Forest Models

Reproducing the accepted answer from CrossValidated here, as it is the most complete and the best answer to this question.

In order to understand this, remember the "ingredients" of random forest classifier (there are some modifications, but this is the general pipeline):

At each step of building individual tree we find the best split of data
While building a tree we use not the whole dataset, but bootstrap sample
We aggregate the individual tree outputs by averaging (actually 2 and 3 means together more general bagging procedure).

Assume first point. It is not always possible to find the best split. For example in the following dataset each split will give exactly one misclassified object.

And I think that exactly this point can be confusing: indeed, the behaviour of the individual split is somehow similar to the behaviour of Naive Bayes classifier: if the variables are dependent - there is no better split for Decision Trees and Naive Bayes classifier also fails (just to remind: independent variables is the main assumption that we make in Naive Bayes classifier; all other assumptions come from the probabilistic model that we choose).

But here comes the great advantage of decision trees: we take any split and continue splitting further. And for the following splits we will find a perfect separation (in red).

And as we have no probabilistic model, but just binary split, we don't need to make any assumption at all.

That was about Decision Tree, but it also applies for Random Forest. The difference is that for Random Forest we use Bootstrap Aggregation. It has no model underneath, and the only assumption that it relies is that sampling is representative. But this is usually a common assumption. For example, if one class consist of two components and in our dataset one component is represented by 100 samples, and another component is represented by 1 sample - probably most individual decision trees will see only the first component and Random Forest will misclassify the second one.


Will a larger data set in order to support your subset?

This is a common approach to address class-imbalance. You'll generally see it referred to as "upsampling" or "over-sampling". In addition to simply reusing existing data, you can "augment" your data by transforming it or adding noise, or generate synthetic data. A good demonstration of data augmentation is the winning solution to the kaggle Galaxy Zoo contest. A popular technique for generating new data is SMOTE. Keep in mind: if you change the class proportions in your dataset, your effectively changing the prior for those classes. Depending on your choice of algorithm and what you are trying to accomplish, you may want/need to apply a probability adjustment to the model's output.

Another technique you can use is to just pick a different decision threshold. For example, if you're performing a logistic regression and don't care about performing inference on parameters, duplicating the data for one class won't actually change the slope of the decision boundary, it will just shift its location. You could equivalently use the decision boundary learned on the unmodified data and then pick a cutoff other than $P(X=1) \ge 0.5$. A good heuristic is to use the threshold which maximizes Youden's J, which can be trivially calculated from a ROC curve. If you have a specific model evaluation metric you are concerned with, choose your threshold relative to that.


You can use LIME. It gives you the most relevant features to a certain prediction the model gave by approximating the model locally as a simpler model. Here is a good lecture on it: https://www.youtube.com/watch?v=HcaAKI1tVGM


Categorization of approaches to deal with imbalanced classes

The way I see it all three categorizations agree in many things. For example, all three have a category for pre-processing steps.

I would tend to mostly agree on the third categorization as its more generic and encompasses more things.

The data-level category includes any pre-processing steps dealing with class imbalance (e.g. over/under sampling).
The algorithm-level could be considered to include the second categories of the first two articles. Any change to the algorithm that deals with class imbalance would go here (e.g. class weighting).
Finally, a hybrid category for combining the two.

The only thing missing from the first two articles are the post-processing steps, which to be honest, aren't used in practice as often as the other.


algorithmic difference between image analysis and video analysis

This is a huge topic, so I will just give you a high-level overview and some pointers to more info.

Yes, there are definitely ways to work with video that are different from working with individual still images.

At the simplest level, it could be running an object detector such as HoG (or a sliding-window convnet) on each frame, and then some means of assigning nearby detections in adjacent frames to be the same object, and discarding detections which do not seem to have continuity in time.  Many of the algorithms in this field seem to look at a single frame as a building block to looking at the whole sequence, where perhaps the data from adjacent frames is combined, aggregated, and/or used to disambiguate the current frame.

Another approach is to first estimate object motion between frames (using optical flow, phase correlation, pyramid block matching or another method) and then treat areas of multiple frames that are collocated after accounting for motion as being the same object.  This is very powerful but limited by the accuracy of the motion estimation.

In newer research, there is a back-and-forth between finding where things are (detection) and how things move (tracking), where each task can help the other one, for example (Kalal 2010) or (Andriluka 2008), to the point that the two parts of the algorithm are no longer separable.  Kalal's TLD algorithm is a recently famous version of this.

There are also algorithms which work directly in the spatiotemporal (or sometimes just temporal) domain.  An example of purely temporal would be detecting vehicles by the periodicity of variation of their wheel spokes.

Some of the frequently studied model problems are:

People, vehicles or other objects detection and tracking throughout a video sequence.  For example, using the CalTech Pedestrians dataset or other standard benchmarks.
Hand gesture recognition, either for games, user interface, or sometimes sign language recognition.  Often using depth data, for example Kinect video.
Activity recognition, for example walking vs standing.  Due to the nature of this problem, it is much more common to see pure spatiotemporal algorithms here, for example (Sadanand and Corso 2012).
Simultaneous location and mapping (SLAM).  Often found in robotics, this problem is basically to build a 3D model of the environment from video of a single camera (or stereo or depth video) which is moving around; the environment is often assumed to be static.  This is often done by running some kind of feature detector such as SIFT or SURF, matching up features in successive frames, and then building point clouds from the relative 3D motion implied by features.

Here are some more references:

Shah, Mubarak, and Ramesh Jain, eds. Motion-based recognition. Vol. 9. Springer Science & Business Media, 2013.

Turk, Matthew. "Gesture recognition." Computer Vision: A Reference Guide (2014): 346-349.

Rosenfeld, Azriel, David Doermann, and Daniel DeMenthon, eds. Video mining. Vol. 6. Springer Science & Business Media, 2013.

Kalal, Zdenek, Krystian Mikolajczyk, and Jiri Matas. "Tracking-learning-detection." Pattern Analysis and Machine Intelligence, IEEE Transactions on 34.7 (2012): 1409-1422.

M. Andriluka, S. Roth, B. Schiele. People-Tracking-by-Detection and People-Detection-by-Tracking. Computer Vision and Pattern Recognition (CVPR) 2008

Sadanand, Sreemanananth, and Jason J. Corso. "Action bank: A high-level representation of activity in video." Computer Vision and Pattern Recognition (CVPR), 2012


Some libraries auto encode categorical features into numbers before running the model. Pre-processing the categorical features is done explicitly by the programmer or within framework. Can you post more details with your questions?


How to print out to a file using Standford Classifier

I don't have experience with Stanford Classifier, but if it is sending output to stdout that you would like to capture, you can simply pipe the output to a file. You didn't mention an OS, but on most the > operator is used to redirect stdout to a file.

For example, if you are running Standard Classifier like this:

java -cp stanford-classifier.jar [option...]

You can instead run it like this to output to a file:

java -cp stanford-classifier.jar [option...] > my-output.tsv


Libraries for Bayesian network inference with continuous data

You can use pymc3. I am pretty sure it works for all the 3 requirements.
http://pymc-devs.github.io/pymc3/


Predicting and clustering at the same time?

Your solution seems interesting. However, it implies that the customers’ value is assessed through whether they are active or not in next period. It might be interesting to look at the purchase by customer for, say future 8 weeks (or future 6 weeks or 12 weeks for that matter – the time frame depends a lot on the particularity of the business). This measurement reflects whether customers are active or not in the next period and also quantifies their 'activity’.

For a somewhat similar project I implemented the following solution:

Start by randomly selecting one breakpoint for each of the three dimensions (RFM); by breakpoint I mean a random value among the unique values of Recency, another randomly selected value for Frequency, and another one for Monetary.
Draw some initial segments based on these three breakpoints by selecting groups of customers that are relatively similar in terms of RFM according to the selected breakpoints. As there are 3 breakpoints (one per variable), you will get 2^3 segments because each variable will be broken down in 2 parts leading to 8 segments in total. In other words, you will have:

Segm 1: Recency < breakpoint_Recency & Frequency < breakpoint_Frequency & Monetary < breakpoint_Monetary

Segm 2: Recency < breakpoint_Recency & Frequency >= breakpoint_Frequency & Monetary < breakpoint_Monetary

Segm 3: Recency < breakpoint_Recency & Frequency < breakpoint_Frequency & Monetary >= breakpoint_Monetary

Segm 4: Recency < breakpoint_Recency & Frequency >= breakpoint_Frequency & Monetary >= breakpoint_Monetary

Segm 5: Recency >= breakpoint_Recency & Frequency < breakpoint_Frequency & Monetary < breakpoint_Monetary

Segm 6: Recency >= breakpoint_Recency & Frequency >= breakpoint_Frequency & Monetary < breakpoint_Monetary

Segm 7: Recency >= breakpoint_Recency & Frequency < breakpoint_Frequency & Monetary >= breakpoint_Monetary

Segm 8: Recency >= breakpoint_Recency & Frequency >= breakpoint_Frequency & Monetary >= breakpoint_Monetary

Compute the mean of future 8 weeks purchases for each of the resulted segments
Then, of course, this segmentation was random and is not the optimal one. To get the optimal segmentation you then implement a Genetic Algorithm to re-do the previous procedure and explore the space for better breakpoints. By this you will likely get the optimal segmentation of homogeneous customers in terms of their purchasing behavior (as characterized by RFM) – optimal with respect to their value (as you decide to define their value).

By the way, the fitness function for the GA aims to maximize the mean of future 8 weeks purchases per segment. Or you can also set your own objective on how to measure the value of customers and define the fitness function accordingly.

Hope this helps. Good luck!

Update: This solution allows you to find 'the homogenous groups of customers' and 'rank segments per value'. I am not sure if this is what you meant by 'rank customers per value'. In my view supervised clustering defeats the idea of individual ranking (but maybe I am wrong).
Moreover, customers change behaviors from one period to the other and it is perhaps better to cluster them in terms of purchasing behavior, not to rank individuals at a point in time. In this way, when a customer changes behavior (in terms of RFM), we know where s/he went (i.e. in which cluster), and thus we can assess whether s/he 'improved' or not. In the latter case, marketing people can take action to incentivize the concerned customer to move back into the better off segment.


How to merge all the data to have a final dataset

A slightly hacky way to get there maybe but you can do this to get what you want from the second table;
df2['count'] = 1
pivot = df.pivot_table(df, index='userid', columns='productid', values = 'count').reset_index()
pivot = pivot.fillna(0)

You would then want to merge this to the first dataset like this;
finaldf = pd.merge(df1, pivot, left_on='userid', right_on='userid')

another great thing to use for generating the dummies for categorical variables is
pd.get_dummies()

The approach seems ok to me and making some more features would also not be a bad idea.


Matrix factorization for like/dislike/unknown data

This is very similar to the netflix problem, most matrix factorization methods can be adapted so that the error function is only evaluated at known points. For instance, you can take the gradient descent approach to SVD (minimizing the frobenius norm) but only evaluate the error and calculate the gradient at known points. I believe you can easily find code for this.

Another option would be exploiting the binary nature of your matrix and adapting binary matrix factorization tools in order to enforce binary factors (if you require them). I'm sure you can adapt one of the methods described here to work with unknown data using a similar trick as the one above.


Fuzzy rule based system: Should rules contain all inputs and outputs?

Not really. It is possible to create fuzzy logic model without setting whole ruleset. But there is a risk, that model will perform worse in some unexpected situations which would be handled by missing rules.

However, it is possible to figure out fuzzy rules basing on data. I personally used Wang-Mendel method and it worked fine for me: still 6 inputs and 4 outputs will be many of rules to deduce :(. Wang - Mengel method is described in this paper: http://sipi.usc.edu/~mendel/publications/Wang%26Mendel%20SMC%2011-12-92.pdf


Understanding portfolio-level risk models

I think this is more of a finance question. Assuming portfolio here refers to an investment of two assets, then

To determine the portfolio risk therefore, you'd need to either predict, or use historicals for $\sigma_1$, $\sigma_2$, $Cov_{1,2}$

Formula from: http://www.investopedia.com/exam-guide/cfa-level-1/portfolio-management/portfolio-calculations.asp


How to identify the appropriate predictors in regression model

I assume by too many variables you mean the case where number of variables is greater than the number of observations ($p>n$) or close to that. You will need to either reduce the number of variables before feeding them in, use some kind of automatic feature selection methods as @M.Mashayekhi has pointed out or utilize learning models that are robust to high feature dimension. Keep in mind that feature selection/creation, in my opinion, is the most important part of model building. There is no point in using a complex machine learning algorithm if all your input variables are useless.

It is always a good idea to first reduce the number of input variables by business intuition or context. By that I mean answering the questions: "Do I really need all of these variables to be in the model?", "What variables just wouldn't make sense to include in the model as a predictor?" "Are there variables that has to be in the model even if it is not the strongest predictor, simply for business reasons? If these questions are not easy to answer, or you have already reduced the number of variables but are still in the case of too many variables, then you can try the following.

Principal Component Analysis (PCA)

One suggestion would be to use PCA to reduce your feature space then run learning algorithms like regression, decision trees on the reduced feature space. You can do that in R using pca. Pick the first however many principal components where the next PC has a decline in marginal variance explained (Since each addition principal component always increases variance explained).

Regression with Lasso ($\mathcal{L1}$) Regularization

An alternative would be to let the model do the feature selection for you. A good starter would be a regression with the lasso ($\mathcal{L1}$) penalty, this shrinks the estimated coefficients toward zero. It does that by shrinking large effect coefficients more slowly toward zero than small effect ones. This effectively removes small effect coefficients from the model by setting them equal to zero, hence, auto feature selection. The challenge is to pick a good shrinkage parameter $\lambda$, which governs how much you are shrinking the coefficients. You can do that with cross validation. glmnet in R let's you run lasso regressions, and glmnet.cv let's you pick $\lambda$ using CV. Also note that by shrinking coefficients, lasso regression reduces prediction errors by balancing between variance reduction and bias increase. You will have to decide which is more important, inference or making good predictions, since unlike OLS regression, estimates by lasso regression are no longer unbiased.

Random Forest

Random Forest is an ensemble tree learning algorithm (implemented in randomForest in R) which is robust to many useless variables. Particularly, it works well in the case of $p>n$. RF is very popular because of its robustness and simple tuning. The downside is this is what's called a Black Box algorithm. Meaning that there is no good way to interpret the relationship between the input and dependent variables. You can however use something like the "variable importance plot" to give you a rough idea of what variables were most useful during the splits on average. This is fine if you are only concerned about making good predictions.

Other techniques

There are certainly other techniques that goes well with high dimensional data like Support Vector Machines and Boosted Regression Trees (Difference between RF and Boosted Trees), but they might usually require a lot more tuning than the ones mentioned above.


Statistical comparison of 2 small data sets for 2X increase in the population mean

Yes, in principle, resampling can help answer this question.

incumbent <- c(7.3, 8.4, 8.4, 8.5, 8.7, 9.1, 9.8, 11.0, 11.1, 11.9)
proposed  <- c(17.3, 17.9, 19.2, 20.3, 20.5, 20.6, 21.1, 21.2, 21.3, 21.7)

set.seed(42)

M  <- 2000
rs <- double(M)

for (i in 1:M) {
    rs[i] <- mean(sample(proposed, replace=T)) - 2 * mean(sample(incumbent, replace=T))
}

To make the assessment, you should choose one (not both) of the following:

A. The (two-tailed) 90% confidence interval for the difference in the (weighted) means using Hall's method is:

ci.hall <- 2 * (mean(proposed)-2*mean(incumbent)) - rev(quantile(rs,prob=c(0.05, 0.95)))
names(ci.hall) <- rev(names(ci.hall))
ci.hall

   5%   95%
-0.29  2.95

This is appropriate if you have any concern about missing the possibility that mean(proposed) might actually be less than 2 * mean(incumbent).

B. The proportion of resample means >= 0 provides the (one-tailed) estimate that mean(proposed) is at least twice mean(incumbent):

sum(rs>=0)/M

[1] 0.8915

The problem is that the samples are really rather small and resampling estimates
can be unstable for small n. The same issue applies if you want to assess normality and go with parametric comparisons.

If you can get to, say, n >= 30, the approach described here should be fine.


Machine learning models for predicting good machine settings given bad machine settings

Can you explain the problem more precisely? What are inputs (besides control settings), is output same if the inputs are fixed? Are there any stochastic variables(external factors prone to probabilistic behaviour), what is the desired outcome?( Minimizing faulty outcome by choosing a setting or something similar)


GausianNB: Could not convert string to float: 'Thu Apr 16 23:58:58 2015'

To cast that column of your data-frame as type float try:

k_train = train['cols_to_use'].astype(float)
target = train['final_status'].astype(float)

More documentation can be found here or you can cast them when loading the csv file sep=',' maybe usful assuming that your data is separated by a , in your CSV file

train = pd.read_csv("datasets/train2.csv",
                    dtype={'cols_to_use': float , 'final_status': float})

Note that converting unix TimeStamp to readable date is very simple with the datetime library example:

import datetime

datetime.datetime.fromtimestamp(1526972723).strftime('%Y-%m-%d %H:%M:%S')

I hope it's helpful.


You can read the sklearn documentation here. If you click in source you can check the code.
If you want to do one hot encoding you can do
    >>> from sklearn.preprocessing import OneHotEncoder
    >>> enc = OneHotEncoder(handle_unknown='ignore')
    >>> X = [['Male', 1], ['Female', 3], ['Female', 2]]
    >>> enc.fit(X)
    OneHotEncoder(handle_unknown='ignore')
    >>> enc.categories_
    [array(['Female', 'Male'], dtype=object), array([1, 2, 3], dtype=object)]
    >>> enc.transform([['Female', 1], ['Male', 4]]).toarray()

You should only try to implement this if you are an experimented developer. A couple lines of code that implement OHE in a non optimal way is:
for col in columns_encoding:
    variables = df[col].unique()
    for v in variables:
        df[col + str(v)] = [1 if row == v else 0 for row in df[col].values]

where columns_encoding is a list of the columns you want to encode and df your dataframe


How does Q-Learning deal with mixed strategies?

One possibility is to use softmax and choose each action a randomly with probabiliy $p = \frac{\exp(Q(s,a))}{\sum_a \exp(Q(s,a))}$. I don't thinks it is still Q-learning though.


What to report in the build model, asses model and evaluate results steps of CRISP-DM?

The way you are trying to present the outcome is pretty good.

I cannot say that the following procedure is the standard procedure in my scenario I did something like this:

This is how I presented to my managers to make them understand the procedure which I followed.

I made a slide for each and every segment and highlighted the points which are necessary for them to be known.

While showing the results, I named the data(accuracies) differently

Train Set Accuracy: Which is 70% of the total dataset, derive the accuracy % also called model accuracy
Test Set Accuracy: Test set which is 30% of the total dataset, derive the accuracy called Test Accuracy
Blind Test Accuracy: Which is completely new data, here I trained the model with whole dataset and then tested using the new data(you can also call it as Validation set)

This is how I presented:

Since yours is also a classification problem to explain it better I gave them the breakdown:

Do let me know if you have any issues, would love to help you.


Approach to creating a user profile in music web application

The first question is: what to do you want to see in user profile?

Top-10 tracks, top-10 artist by user?
How many tracks/artists a user listens to in a day on average (may be, in last month)?

May be you want to get some general information related to the whole user base:

Which artist/track is the most popular among users from different countries (top-N of them)?

The second: you store and want to make aggregates for millions of records. It's not a text file deal. Make a database. Create a table id-user | country | id-artist | id-track. Create another tables with some aggregates from #1, update it regularly and diplay on the front end.


How to deal with multiple categorical data set

Simply yes. Before that you may want to check how correlated those features are, so you can simply deselect redundant features, but in general you are right. Starting with one-hot encoding is a good choice.

What may need more inspection later, is the number of different regions. Then you come up with many sparse features for which you need to reduce dimensionality. If you provide more information on the whole project I can provide more insight.


ScikitLearn - RandomForestRegressor score different in and out of grid search

I have found the answer to my question : there is randomness in RandomForestRegressor().

Using the random_state hyper parameter fix the problem :

rfr = RandomForestRegressor( random_state = 123 )

What is surprising is the difference of score obtained from different values of the random_state hyper parameter. It means my best hyper parameters n_estimators and min_samples_split are strongly depend on the random_state hyper parameter.

I guess it is something I will have to deal with.


Random Forest Model Giving Same Accuracy for different feature sets after tuning

The fact that their accuracy is the same but their AUC score is different is encouraging. First and foremost, it would be good for you to inspect what is learned by the various trees. A good way to do that is to follow the instructions on that link.

What I think is that all 3 algorithms essentially learned that the same features are useful and that these features are present in each feature set. That is what random forests do, given a lot of features, they inherently "decide" which features are worth using.


Interpretability of RMSE and R squared scores on cross validation

And one last thing, is my result on X_train indicative that my features are informative enough to learn the target? or is the R² train score somehow biased?

High scoring fits on training data does not necessarily indicate that your features are informative enough to learn the target in a general fashion. Only your cross validation scores can do so.

  Note : In this code i'm using 10 folds for CV. Used 3 folds gave me a better result on CV, if someone can also explain that , it would be great.

I wouldn't expect wildly significant differences so it depends on how much better they were, but remember that you're randomly selecting a subset of data for each fold. It's plausible that when you ran it for 3 folds the model was coincidentally better able to learn from the selected training sets to predict the validation sets. Generally speaking, the more folds you use the more confident you should be in the reliability of the score.

  My question: is this considered an overwhelming overfitting problem? or is it mild? and should I do more feature engineering or tune hyperparameters more? ( used GridSearchCV for the current hyperparameters )

I'm no professional, but I would consider that to be severely overfit. I would in the first instance return to hyperparameter tuning to try and bring them in line. I'll also  perform the obligatory plug of RandomizedSearchCV at this point: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf

EDIT:

Since you've asked for further details:

1) Does a high score against training data indicate that the features are informative enough to learn the target?

Consider the following example:

import numpy as np
from xgboost import XGBRegressor

x = np.random.rand(5000, 12)
y = np.random.randint(0, 200, 5000)

rfr = XGBRegressor(colsample_bytree=0.6,gamma=0.3,learning_rate=0.1,max_depth=8,min_child_weight=3,subsample=0.9,n_estimators=1000,eval_metric='rmse')
rfr.fit(x, y)
rfr.score(x, y)

  Out[26]: 0.999918392694166

Entirely random targets trained against entirely random inputs, and still scoring almost perfectly. That's a completely useless model; validating against your training data does not allow you to determine that your features are informative enough to allow you to predict the target against data that your model has not seen. Only cross-validation against unseen data can do this.

2) Used 3 folds gave me a better result on CV, if someone can also explain that , it would be great.

The RNG gods smiled upon you. Again, consider an example:

from xgboost import XGBClassifier, XGBRegressor
import pandas as pd
from sklearn.model_selection import KFold

df = pd.read_csv(r'https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv', index_col=None)

x = df.drop('species', axis=1)
y = df.species

xgb = XGBClassifier(max_depth=3)

kf = KFold(n_splits=10, shuffle=True)

for train_idx, test_idx in kf.split(x):
    xgb.fit(x.iloc[train_idx], y.iloc[train_idx])
    print(xgb.score(x.iloc[test_idx], y.iloc[test_idx]))

  Out[17]: 0.9333333333333333
  0.9333333333333333
  0.9333333333333333
  1.0
  0.8666666666666667
  1.0
  1.0
  0.9333333333333333
  0.8666666666666667
  1.0

That code splits the data 10 times and then trains and scores the model on each selected slice. As you can see, there's a 13.4% difference between the highest and worst scoring slices.

Your 3-Fold run coincidentally gave you 3 'good' folds. The more Folds you have, the more representative of the true result your cross-validation is.

3) is this considered an overwhelming overfitting problem? or is it mild? and should I do more feature engineering or tune hyperparameters more? ( used GridSearchCV for the current hyperparameters )

Well, whether or not it's overwhelming is something of a matter of opinion. In my mind however a loss against the validation set that is 300% the training score is very severely overfit indeed, but if your validation RMSE is still within the margin of error you're willing to accept then I suppose you could go ahead and use it anyway. It's really your call.

To reduce the overfitting, you need to tune your hyperparameters better. Reducing max_depth and increasing min_samples_split is my usual go-to with trees. If your revised model (exhibiting either no overfitting or at least significantly reduced overfitting) then has a cross-validation score that is too low for you, you should return at that point to feature engineering. We can highlight the effect of hyperparameters on overfitting quite easily by plotting the effect of tweaking them:

from xgboost import XGBClassifier, XGBRegressor
import pandas as pd
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
from sklearn.preprocessing import OneHotEncoder

df = pd.read_csv(r'https://raw.githubusercontent.com/mwaskom/seaborn-data/master/diamonds.csv', index_col=None)

ohe = OneHotEncoder(sparse=False)
x = pd.DataFrame(ohe.fit_transform(df[['cut', 'color', 'clarity']]))

df.drop(['cut', 'color', 'clarity'], axis=1, inplace=True)

df = df.merge(x, left_index=True, right_index=True)

x = df.drop('price', axis=1)
y = df.price

cv = {}
for i in range(20):
    xgb=XGBRegressor(max_depth=i+1)
    x_train, x_test, y_train,  y_test = train_test_split(x, y)
    xgb.fit(x_train, y_train)
    cv[i+1] = (xgb.score(x_train, y_train), xgb.score(x_test, y_test))

train = []
test = []
for i in cv.keys():
    train.append(cv[i][0])
    test.append(cv[i][1])

fig = plt.figure()
plt.plot(train)
plt.plot(test)
plt.legend(['train', 'test'])
plt.xlabel('max_depth')
plt.xlim(0, 20)
plt.ylabel('R^2 Score')
plt.show()

Apologies for the stupid x-axis labels :P. We can see the affect of the max_depth hyperparameter on the model's tendency to overfit easily; anything over 3-ish and the train/test scores begin to diverge.

Hope that those revisions are more helpful to you :)


TypeError: '<' not supported between instances of 'int' and 'str'

Based on our discussion, omit reverse = True and use list instead of sorted and print it. I guess you will see what you want.


anaconda navigator not start direct without source command

You can write script (.sh) for that.

STARTPATH=$(pwd)
cd ~/anaconda3/bin
case $1 in
    -j|--jupyter)
        shift
        source activate
        jupyter-lab
        ;;
    -n|--navigator)
        shift
        source activate
        anaconda-navigator
        ;;
esac
source deactivate
cd $STARTPATH

Then you can run the script by ./ or you can put the script in /usr/local/bin and make it executable chmod +x *yourfile*.sh.

After that you should be able to type in your console: *yourfile* -n and this will launch anaconda navigator.


Increase Accuracy on Keras Multiclass Classifier

You should understand if the model you built is able to learn from the data. Therefore, plot the accuracy and the loss for both training and validation set versus the number of epochs.
Then, if the model needs more epochs, give it more epochs. Otherwise, try different optimizers, activation functions, number of neurons, number of layers, and batch size.

From the (flawed) plots you provided (please, plot accuracy for training and validation on one graph, and loss for training and validation on another graph), you can see two clear phenomena:

accuracy stops to increase after a very few epochs;
loss curve is unstable.

which suggest that the model does not learn from the data (I would start off with a shallow architecture, tuning number of neurons, trying different optimizers, activation functions, and then number of layers) and the dataset could be unrepresentative.


Feature engineering decrease my cross validation

You might be dealing with overfitting, since your model does not generalize that well. It should be noted that Light GBM is sensitive to overfitting in small datasets.

To solve this, you can check for features with correlation and drop some of them. Another solution can be the implementation of a feature selection method. Lastly, since GBM is a tree algorithm, you can try an early termination.

If you are dealing with highly imbalance data, you can consider using a stratified method rather than CV. In sklearn, there are StratifiedKFold and StratifiedShuffleSplit, which ensure that relative class frequencies are approximately preserved in both train and validation phases.


Does recall has different interpretation when comes to classification and information retrieval

According to wikipedia, Recall is defined as-

In information retrieval, recall is the fraction of the relevant documents that are successfully retrieved.

In your second formula, recall = p/q, where q is total number of relevant documents. If you define a confusion matrix for binary classification, you will see that q is actually sum of TP and FN. And p is already (by definition) is TP.

Now lets get the definition of TP and FN. TP (True Positive) is number of predicted positive docs out of number of actual positive docs. FN (False Negative) is number of predicted negative docs out of number of actual positive docs. So, basically when you add TP and FN, it becomes total number of positive docs.

So, let say out of 50 positive (actual) docs, you get 35 positive and 15 negative predictions. So, recall by your first formula is

$35/(35+15) = 35/50$

and by your second formula, $p = 35$ and $q = 50$, so recall is

$p/q = 35/50$

Hope this helps. Please comment if I am wrong about something or missed anything.


Consider the formula for Entropy:

$E(S) = \sum_\limits{i=1}^{n}-p_{i}\log_2p_{i}$

Expanding that summation for the four concept decision attributes for your problem gives

$E(S) = -p_{cinema}\log_2\left(p_{cinema}\right) - p_{tennis}\log_2\left(p_{tennis}\right) - p_{stayin}\log_2\left(p_{stayin}\right) - p_{shopping}\log_2\left(p_{shopping}\right)$

There are three observations where Weather=Sunny. Of those three, one has Decision=Cinema and two have Decision=Tennis. So for Weather=Sunny you have $p_{cinema}=\frac{1}{3}$, $p_{tennis}=\frac{2}{3}$, and other decision probabilities are zero. Plugging those values into the equation above gives

$
\begin{align}
E(S_{Sun}) &= -\frac{1}{3}\log_2\left(\frac{1}{3}\right)-\frac{2}{3}\log_2\left(\frac{2}{3}\right) - \left(0\right)\cdot\log_2\left(0\right) - \left(0\right)\cdot\log_2\left(0\right) \\
           &= 0.38998 + 0.52832 + 0 + 0 \\
           &= 0.918
\end{align}$


The distance between objects of complex structure can be tricky. Consider a simpler example, when you got two sequences of words. How do you calculate the distance? In speech recognition, the distance between the groundtruth and recognized transcriptions is calculated as the word-error-rate based on the Levenstein distance between the aligned sentences.

GRAAL is an extension of the alignment algorithm to the networks. However, it only considers the network topology. You might want to extend the node similarity degree to encounter the type and attributes of the nodes.


How can I make big confusion matrices easier to read?

You can apply a technique I described in my masters thesis (page 48ff) and called Confusion Matrix Ordering (CMO):

Order the columns/rows in such a way, that most errors are along the diagonal.
Split the confusion matrix into multiple blocks such that the single blocks can easily printed / viewed - and such that you can remove some of the blocks because there are to few data points.

Nice side effect: This method also automatically clusters similar classes together. Figure 5.12 of my masters thesis shows that:

You can apply confusion matrix ordering with clana


How to combine dictionary of word-forms into a single list without duplicates

You can get a list of all possible words (including the nouns, verbs, adjectives and so on) by using a comprehension. This loops over all elements of a sequence, as with a for-loop, but can be faster.

In you case, you have a dictionary, so we loop over each key-value pair. The keys are n (noun), a (adjective), v (verb) and r (??). For each of these keys, we have a list of words, which we can append to your final desired output list, if they aren't already in the list.
Because we loop over a dictionary, and then over each of the lists, with becomes nested:

from word_forms.word_forms import get_word_forms
word_forms = get_word_forms("review")

desired_words = []
for word_type, word_list in word_forms.items():
    for word in word_list:
        if word not in desired_words:
            desired_words.append(word)

print(desired_words)
# ['reviewers', 'review', 'reviewer', 'reviews', 'reviewing', 'reviewed']

This can be compressed into a nested comprehension:

desired_words = []
{desired_words.append(word) for word_type, word_list in word_forms.items() for word in word_list if word not in desired_words}

print(desired_words)
# ['reviewers', 'review', 'reviewer', 'reviews', 'reviewing', 'reviewed']

We didn't need to save the resulting object, we only want to side-effect of appending to desired_words

Another way to do this would be to simply put all the words into a list, then convert it into a set() object, which would remove all duplicates and order the list alphabetically:

desired_words = []
# the 'if' statement is removed in the dictionary comprehension:
{desired.words.append(word) for word_type, word_list in word_forms.items() for word in word_list}
# Set removes duplicates - we then wrap it back into a list
desired_words = list(set(desired_words))


XGBoost Linear Regression output incorrect

It seems that XGBoost uses regression trees as base learners by default. XGBoost (or Gradient boosting in general) work by combining multiple of these base learners. Regression trees can not extrapolate the patterns in the training data, so any input above 3 or below 1 will not be predicted correctly in your case. Your model is trained to predict outputs for inputs in the interval [1,3], an input higher than 3 will be given the same output as 3, and an input less than 1 will be given the same output as 1.

Additionally, regression trees do not really see your data as a straight line as they are non-parametric models, which means they can theoretically fit any shape that is more complicated than a straight line. Roughly, a regression tree works by assigning your new input data to some of the training data points it have seen during training, and produce the output based on that.

This is in contrast to parametric regressors (like linear regression) which actually look for the best parameters of a hyperplane (straight line in your case) to fit your data. Linear regression does see your data as a straight line with a slope and an intercept.

You can change the base learner of your XGBoost model to a GLM (generalized linear model) by adding "booster":"gblinear" to your model params :

import pandas as pd
import xgboost as xgb

df = pd.DataFrame({'x':[1,2,3], 'y':[10,20,30]})
X_train = df.drop('y',axis=1)
Y_train = df['y']
T_train_xgb = xgb.DMatrix(X_train, Y_train)

params = {"objective": "reg:linear", "booster":"gblinear"}
gbm = xgb.train(dtrain=T_train_xgb,params=params)
Y_pred = gbm.predict(xgb.DMatrix(pd.DataFrame({'x':[4,5]})))
print Y_pred

In general, to debug why your XGBoost model is behaving in a particular way, see the model parameters :

gbm.get_dump()

If your base learner is linear model, the get_dump output is :

['bias:\n4.49469\nweight:\n7.85942\n']

In your code above, since you tree base learners, the output will be :

['0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=2.85\n\t\t4:leaf=5.85\n\t2:leaf=8.85\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=1.995\n\t\t4:leaf=4.095\n\t2:leaf=6.195\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=1.3965\n\t\t4:leaf=2.8665\n\t2:leaf=4.3365\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=0.97755\n\t\t4:leaf=2.00655\n\t2:leaf=3.03555\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=0.684285\n\t\t4:leaf=1.40458\n\t2:leaf=2.12489\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=0.478999\n\t\t4:leaf=0.983209\n\t2:leaf=1.48742\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=0.3353\n\t\t4:leaf=0.688247\n\t2:leaf=1.04119\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=0.23471\n\t\t4:leaf=0.481773\n\t2:leaf=0.728836\n',
 '0:[x<3] yes=1,no=2,missing=1\n\t1:[x<2] yes=3,no=4,missing=3\n\t\t3:leaf=0.164297\n\t\t4:leaf=0.337241\n\t2:leaf=0.510185\n',
 '0:[x<2] yes=1,no=2,missing=1\n\t1:leaf=0.115008\n\t2:[x<3] yes=3,no=4,missing=3\n\t\t3:leaf=0.236069\n\t\t4:leaf=0.357129\n']

Tip : I actually prefer to use xgb.XGBRegressor or xgb.XGBClassifier classes, since they follow the sci-kit learn API. And because sci-kit learn has so many machine learning algorithm implementations, using XGB as an additional library does not disturb my workflow only when I use the sci-kit interface of XGBoost.


You can modify the matrix used to compute the distances between points. By default, KNN with quantitative features is used with Euclidean distances, this is, the identity matrix in the equation (x-y)' I (x-y) (it returns the squared Euclidean distance between point X and Y).

Instead of the identity matrix, you can use other positive definite matrices and compute Mahalanobis distances.

Other modifications you can do is manipulate each feature. This also affects the output of KNN, aside of standardizing, you can try non linear modifications of the features, reducing kurtosis or skewness, for example.


What is the best Keras model for multi-class classification?

Your choices of activation='softmax' in the last layer and compile choice of loss='categorical_crossentropy' are good for a model to predict multiple mutually-exclusive classes.

Regarding more general choices, there is rarely a "right" way to construct the architecture. Instead that should be something you test with different meta-params (such as layer sizes, number of layers, amount of drop-out), and should be results-driven (including any limits you might have on resource use for training time/memory use etc).

Use a cross-validation set to help choose a suitable architecture. Once done, to get a more accurate measure of your model's general performance, you should use a separate test set. Data held out from your training set separate to the CV set should be used for this. A reasonable split might be 60/20/20 train/cv/test, depending on how much data you have, and how much you need to report an accurate final figure.

For Question #2, you can either just have two outputs with a softmax final similar to now, or you can have final layer with one output, activation='sigmoid' and loss='binary_crossentropy'.

Purely from a gut feel from what might work with this data, I would suggest trying with 'tanh' or 'sigmoid' activations in the hidden layer, instead of 'relu', and I would also suggest increasing the number of hidden neurons (e.g. 100) and reducing the amount of dropout (e.g. 0.2). Caveat: Gut feeling on neural network architecture is not scientific. Try it, and test it.


I think the closest you can get is with either the warm_start parameter or the partial_fit call. They are available in some models and allow you to train a model several times without losing progress.

From sklearn docs:

  warm_start : bool, optional, default False

  When set to True, reuse the solution of the previous call to fit as
  initialization, otherwise, just erase the previous solution.

and:

  partial_fit(X, y, classes=None, sample_weight=None)[source]

  Perform one epoch of stochastic gradient descent on given samples.

  Internally, this method uses max_iter = 1. Therefore, it is not guaranteed that a minimum of the cost function is reached after
  calling it once. Matters such as objective convergence and early
  stopping should be handled by the user.

Which one to use depends on what model you are using and when you want to make your checkpoints. But if you for example use a RandomForestClassifier which has warm_start you could do the following:

# set warm_start so model to avoid erasing model between fits
clf = RandomForestClassifier(warm_start=True)
number_of_checkpoints = 10

for checkpoint in range(number_of_checkpoints):

    # Load only a subset of the data and train on it
    X, y = load_data_batch(batches=number_of_checkpoints, current_batch=checkpoint)
    clf.fit(X, y)

    # Save model checkpoint for each fit
    with open('path/to/models/random_forest_ckp_{}.p'.format(checkpoint), 'wb') as f:
        pickle.dump(clf, f)


Evaluating a Multi-Label Classification model

You are correct for using sigmoid+binary CE for multi-label classification problem. On the other hand, try to think about how accuracy would be defined for this problem? I would use categorical accuracy!


When should one use L1, L2 regularization instead of dropout layer, given that both serve same purpose of reducing overfitting?

I am unsure there will be a formal way to show which is best in which situations - simply trying out different combinations is likely best!

It is worth noting that Dropout actually does a little bit more than just provide a form of regularisation, in that it is really adding robustness to the network, allowing it to try out many many different networks. This is true because the randomly deactivated neurons are essentially removed for that forward/backward pass, thereby giving the same effect as if you had used a totally different network! Have a look at this post for a few more pointers regarding the beauty of dropout layers.

$L_1$ versus $L_2$ is easier to explain, simply by noting that $L_2$ treats outliers a little more thoroughly - returning a larger error for those points. Have a look here for more detailed comparisons.


What are some good sources to learn fraud/anomaly detection in normal/time-series data?

Chandola et al's survey is probably the best and most-widely cited survey in the anomaly detection field.


Expected 2D array, got scalar array instead

The error itself solves your problem. Just follow what it says. The predict() method takes a 2d array of values you want to predict on. Each item in the array is a "point" you want your model to predict on. So try,

lin_reg_2.predict(poly_reg.fit_transform([[6.5]]))

Here the input is a 2D array of shape (1,1).

Or as the error suggests:

lin_reg_2.predict(np.array([6.5]).reshape(1, 1))


Quick start using python and sklearn kmeans?

This has been answered in other places e.g. here.

You could run Principal Component Analysis (or other dimensionality reduction techniques) and plot the cluster for the first two principal components.
You could plot the results for two variables at a time.
You could encode third or fourth variables using standard visualization techniques like color coding, symbols or facetting.
There are ways to visualize the quality of the fit e.g. silhouette analysis or elbow test for determining the number of cluster etc.
Have a quick look at this link


I suggest this tutorial on ordered logit: http://www.ats.ucla.edu/stat/r/dae/ologit.htm

It showcases the use of polr in the MASS package, and also explains the assumptions and how to interpret the results.


Last cell in recurrent network always the most accurate

I am using a recurrent network for time series forecasting.

Interesting would be especially the type of the network you chose. How many layers? How many neurons per layer?

  The prediction from the last cell in the network always seems to be the most accurate.

I assume by cells you refer to your samples. Always, if you learn in datastreams (online), the network in later iterations has seen more data than the network in early iterations. Thus, with having seen more samples, your predictions become more accurate. Of course, this assumes that you do online learning on your datastream.

  At present I would input data x(t=0, t+1, t+2) to obtain predictions for t=0, t+1, t+2.Is this the wrong approach?

This depends on how your RNN is set up. A Jordan Network for example has context neurons, which should do exactly what you propose. The reason for recurrency in the network is typically that you want to simulate short term memory - meaning, in a mathematical view, you want to feed information from previous iterations into the front of the network. Thus, since this feature should be taken care of in your RNN, I do not think it is good to extend your input, since you introduce additional, potentially useless weights that have to be learned by the network.


What is the correct way to apply KNN to a time-series using a rolling window?

I never used KNN on time series. I didn't know it was possible before reading your question. But by googling it found this tutorial that feel pretty clear.

And if understand you correctly what you do with KNeighborsRegressor, by repeating 4 - 5 - 6 you will based the later predictions on what you predicted for the previous row.

If this is not intended you can just predict every row following the 200 used for training.

Hope it helps :)


Find the longest palindrome substring of a piece of DNA

Longest palindromic substring is a computer science problem.

One common solution is Manacher's algorithm.


In general, Machine Learning algorithms handle volumes data. This doesn't mean that you cannot extract information from "small" data.

Keep in mind:

Overfitting. With only a few data, the risk to overfit your model is far higher.
Outliers. Those become nastier. If you have millions of data, a couple of outliers will not be a problem. But with only a few, they will definitely skew your results.

Models to use

To avoid overfitting, you need to avoid complexity. As a result, you need a model that has as less of parameters as you can. For example, linear/logistic regression could be what you need. Naive Bayes and SVM might work well, but this is not a generalization.

Cleaning up your data

Together with the outliers, you need to be careful with the rest of the cleaning process. Noise can skew a lot your data.

This is a nice blog post about "small data" and what you can do when you are working with them. Highly recommended.

What to do with “small” data? (7 min read)

About Outliers

I recommend the answer from this Data Science Stack Exchange question. Short story. You need to examine the outliers one by one. There isn't any rule written in stone. But you need to identify if those outliers are just wrong input data or data that makes sense to be there, even as an outlier. For example, if you have the sales of an eshop per day and in your data you have a specific day where the sales are 10x more, you need to understand if that was an error or something happened that day to increase the sales (maybe a discount campaign or email advertisement).

Also, it depends on what kind of problem you have. If your problem is an anomaly detection, then you don't want to remove the outliers.

Another suggestion (I don't have any source to link) is to update the values of those outliers with the mean, mode or another metric from your data. If you have only a few rows of them, removing a dozen will make it more difficult.

In general, this should be part of your model process. You try once with outliers, a few times without or with updated values and try to understand if it is underperformed or overfitted.


How to get p-value and confident interval in LogisticRegression with sklearn?

One way to get confidence intervals is to bootstrap your data, say, $B$ times and fit logistic regression models $m_i$ to the dataset $B_i$ for $i = 1, 2, ..., B$. This gives you a distribution for the parameters you are estimating, from which you can find the confidence intervals.


One result for MovieLens 20M using Factorization Machine can be found here. They got MAE: 0.60 and RMSE: 0.80.

Another result for MovieLens 20M using Autoencoders can be found here. They got RMSE: 0.81.


Whats an explanation of PCA that is intuitive for someone in senior leadership who doesn't have a technical background?

Try to make your explanation simple and brief. Maybe something like "PCA distils a dataset's features into as few attributes as possible; these attributes explain as much of the observed trend as possible while being as different from each other as possible". If you can do this using visual aids, even better.

As far as analogies go, maybe use an example of a motor accident. There are many predictors of an accident, but PCA may combine the majority of them to form 2 features - perhaps vehicle attributes and driver attributes. These are different but arguably are both highly predictive of a motor accident. [I realise this is not technically how PCA works, but might help a layman understand].


How can I use Machine learning for inter-relationship between Features?

I use pair plots to study the inter-relationship between features. Pair plot gives the first level information of the features. Seaborn library has pairplot function and even matplotlib has the same.

Another thing you can use heatmap which gives the co-relation between features, by using heat map we can see features that are high co-related and may eliminate one of them. As a word of caution you must good reason or domain knowledge to drop a feature


How do GBM algorithms handle missing data?

LIGHTGBM will ignore missing values during a split, then allocate them to whichever side reduces the loss the most. Section 3.2 of this reference explains it.

There are some options you can set such as usemissing=false, which disables handling for missing values. You can also use the zeroas_missing option to change behavior. GitHub


How to identify possible false negatives in labels?

I think your problem is not with identifying the false negatives, it's with defining the classification problem properly.
If I try to phrase what you're trying to predict, it would be something like this: "Given the data about a drilling point, what's the probability of encountering a difficulty up to depth $x$".
So, if you have a hole which didn't reach that depth $x$, it's not a false negative - it's just a missing label (no test... yet).
If you want to make your prediction based on the information known before drilling (or, at least, before drilling up to a given depth $z$) - limit your model to that information.


Literature on selecting specific dimensions in a word embedding vector

There is not much theoretical work on the topic. The number of dimensions for an embedding is an empirical question depending on the corpus and the task.
The "GloVe: Global Vectors for Word Representation" paper by Pennington  showed that there are large gains in accuracy by adding dimensions up to couple of hundred then diminishing returns.

The most common number of dimensions is either 128 or 256. 128 and 256 are both powers of 2 which can speed up training time.


Has the Random Forest algorithm ever been used in Reinforcement Learning applications?

I expect someone somewhere has used a RF estimator inside RL to approximate action values, if only to assess it as a comparison to other function approximators. However, it does look like from a web search that this is not used widely, and I could not find an example either.

The main problem with a RL/RF hybrid with RF as the value estimator is that the random forest base algorithm is not an online one - it works across a finalised data set and processes the whole batch in order to do things like bagging. Even when used as an estimator using experience replay, support for online learning is a desirable feature, and values are typically fed into supervised learning part in small or medium batches. That is because the action values that are learned by the internal estimation function in reinforcement learning are non-stationary.

Once you have learned the action values for the current policy, and in most RL well before estimates for the values have converged, you change the policy. This changes the expected action values, so your estimator must be able to forget older data and bias towards most recent values. Algorithms that can be made to work online can do this, purely offline ones cannot.

However, there are online Random Forests which have the necessary alterations to work with non-stationary data. I don't see any reason why they could not work. Decision trees in general have been used successfully as estimators in RL - see Reinforcement Learning with Decision Trees. It looks like that in principle it could be done.


How to capture the detail of an attribute in a sentence?

You can try using dependency parsing for that, e.g., as implemented in Spacy or any other NLP toolkit.
The details should be adjective modifiers (dependency label amod) of what you call attribute that you detected using a NER-like approach.

Alternatively, they can form compound nouns as "airy cotton" in your example. In this case, it is a word that depends on the entity with the dependency label compound. But I guess this will be rarely the case because such compounds will be more often recognized as one entity.

Spacy has an online demo for dependency parsing.


Should I scale or normalise my dataset before clustering?

As often in Machine Learning, there is no clear answer. In fact, both are valid options [1, p. 116]. However, for k-means min-max-scaling is usually used in practice [2]. So min-max-scaling would be the default choice and it's what I'd recommend. But as so often you can simply try both and see which provides better results (i.e. better internal cluster validation measures, such as the Silhouette Index).
References:
[1] https://dbs.ifi.uni-heidelberg.de/files/Team/eschubert/lectures/KDDClusterAnalysis17-screen.pdf
[2] https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152173


The function dataframe.columns.difference() gives you complement of the values that you provide as argument. It can be used to create a new dataframe from an existing dataframe with exclusion of some columns. Let us look through an example:

In [2]: import pandas as pd

In [3]: import numpy as np

In [4]: df = pd.DataFrame(np.random.randn(5, 4), columns=list('ABCD'))

In [5]: df
Out[5]:
          A         B         C         D
0 -1.023134 -0.130241 -0.675639 -0.985182
1  0.270465 -1.099458 -1.114871  3.203371
2 -0.340572  0.913594 -0.387428  0.867702
3 -0.487784  0.465429 -1.344002  1.216967
4  1.433862 -0.172795 -1.656147  0.061359

In [6]: df_new = df[df.columns.difference(['B', 'D'])]

In [7]: df_new
Out[7]:
          A         C
0 -1.023134 -0.675639
1  0.270465 -1.114871
2 -0.340572 -0.387428
3 -0.487784 -1.344002
4  1.433862 -1.656147

The function returns as output a new list of columns from the existing columns excluding the ones given as arguments. You can also check it:

In [8]: df.columns.difference(['B', 'D'])
Out[8]: Index(['A', 'C'], dtype='object')

I suggest you to take a look at the official documentation here.


Siamese vs matching network for correct image category matching

A standard approach for this kind of problem is to take the representations of the images computed by some pre-trained neural network (e.g. imagenet). As "representation" of the image we usually take the vector after flattening the output of the last convolutional layer block.
If your images are not of general domain, but are of a very specific domain, and you have enough data, you may want to train an autoencoder on them instead of using a pre-trained model.
You now compute and store the representations of all the images you want to compare against. Once you have the representations, for each image you want to match, you simply compute the closes image in the representation space. There are libraries like Facebook's faiss that are very fast computing binary similarities over a large number of vectors.
This kind of approach is cheap in terms of compute, it is scalable and it is widely used.


How to add time as a feature into clustering algorithm?

I've decided to try LDA:

cvectorizer = CountVectorizer(min_df=4, max_features=10000, ngram_range(1,2))
cvz = cvectorizer.fit_transform(df['word'])
n_topics = 32
n_iter = 2000
lda_model = lda.LDA(n_topics=n_topics, n_iter=n_iter)
X_topics = lda_model.fit_transform(cvz)

Then I've concatenated time feature to the LDA vector:

 X_all  = np.hstack((X_topics, X_time))

Where X_time was my time feature column.


DeepMoji is a fun project that came out of MIT, which predicts emojis that are most related to an input sentence. Gimmicks aside, it seems perfectly adequate for your task of anger detection.

Paper
Blog post
Pretrained models: PyTorch or Keras.


Understanding the training phase of the tutorial "Using Keras and Deep Deterministic Policy Gradient to play TORCS" tutorial

Look at the documentation of tf.gradients and you will see that the third parameter is used to weight the gradient calculation of first param wrt to second param. This is phrased in the documentation as

  A user can provide their own initial grad_ys to compute the derivatives using a different initial gradient for each y (e.g., if one wanted to weight the gradient differently for each value in each y).

So it essentially uses the grad_ys parameter as an element-wise multiplication.

The parameter update gradient calculation in the project code is feeding in the already-calculated value of $\frac{\partial \mu(s|\theta)}{\partial \theta}$ as a placeholder, and using it for this third param.

So the multiplication of the two gradient factors occurs within tf.gradients.


Automatically generating images for an art show with deep learning software

You can try to generate morphed images and then scale any size. You can use this or similar libraries.


How to extract information from plot images?

In the general sense this is most definitely a non-trivial problem and any full featured solution will require a considerable degree of work.

Success will probably be dependent on how much you constrain your problem.
Monochrome will be easier than color.
Limited font, font size, line and marker styles will help.

The more constrained, the more chance of success.

For line detection, such as axes, you might want to consider the Hough and Radon transforms.

Here are some stackexchange questions covering some of the topics you might want to address, sadly no bargraphs, some have a manual component which you might think how to automate. They are a good starting point to review some of the difficulties involved in the problem.

Selecting a curve:
https://mathematica.stackexchange.com/questions/44355/how-to-make-a-curve-selectable-from-a-scaned-image-and-convert-it-to-a-list-of-c

Datapoints and Axes:
https://mathematica.stackexchange.com/questions/1524/recovering-data-points-from-an-image

Scatter:
https://mathematica.stackexchange.com/questions/26356/how-do-i-find-the-coordinates-of-points-in-this-image

Datapoints:
https://mathematica.stackexchange.com/questions/3831/how-can-i-extract-data-points-from-a-black-and-white-image

https://mathematica.stackexchange.com/questions/14444/obtain-data-points-from-a-graph-on-an-image-without-axes


In general, the further away the green line is from the red line, the more the model is overfitting, however eventually enough data will cure all overfitting (there will be so much data the model can't possibly memorize all of it), and that's why the lines converge to being together (stops memorizing, red line goes down, starts generalising, green line goes up). Some models need more data to learn than others however, and so as you can see, the LogisticRegression model reaches it's best performance much faster than, for example, the SVC.
An interesting case is the KNN, who's red line doesn't go down, but rather up. I'm pretty sure the reason for this is to do with how the KNN works, it compares instances it knows to classify new instances. Thus, the KNN doesn't really memorize... New instances it can compare with will never hinder it's performance on the training set (red line). However it too will also eventually converge, the lines together.


Should the output of regression models, like SVR, be normalized?

In regression problems it is customary to normalize the output too, because the scale of output and input features may differ. After getting the result of the SVR model, you have to add the mean to the result and multiply that by the standard deviation, if you have done that during normalizing.

  How can one explain the impact of output normalization for svm regression?

If you normalize your data, you will have a cost function which is well behaved. means that you can find local parts more easily. The reason is that you have to construct the output using the input features in regression problems. It is difficult to make large values with small normalized features but with small numbers, making a normalized output is easier and can be learned faster.


In which situation should we consider a dataset as imbalanced?

I think you can speak of imbalanced targets if (in case of a binary classification problem) the classes are not represented in a 50:50 manner. This is almost always the case.
With about 25/75 in your case, I would see this as „imbalanced“. There are some strategies to deal with this problem, such as (re)sampling so that you achieve a 50:50 balanced sample (essentially you will lose observations in the majority class here). Alternatively you can use synthetic oversampling (SMOTE) and related rechniques.
However, some packages come with built-in options to deal with unbalanced targets, e.g. sklearn‘s random forest (option class_weight). Check the docs.


You are using multiple frameworks (pandas, keras / tensorflow, nltk, reg, scikit-learn) and doing a lot of manually processing. It would be better if you used a single framework. A single framework would automatic many of the steps and appropriately handle training and prediction. The result would be creating the same one-hot encoding scheme at the token level.


Word based perplexity from char-rnn model

Actually, there is a formula which can easily convert character based PPL and word based PPL.

$PPL = 2^{(BPC*Nc/Nw)}$

where $BPC$ is character based $PPL$, $Nc$ and $Nw$ are the number of characters and words in a test set, respectively.

The formula is not completely fair, but it at least offers a way to comparing them. The following are some reference.

[1] Hwang K, Sung W. Character-Level Language Modeling with Hierarchical Recurrent Neural Networks[J]. 2016.

[2] Graves A. Generating Sequences With Recurrent Neural Networks[J]. Computer Science, 2013.

[3] T. Mikolov, I. Sutskever, A. Deoras, H. Le, S. Kombrink, and J. Cernocky.Subword language modeling with neural networks. Technical report, Un-published Manuscript, 2012.


Why is the input to an activation function a linear combination of the input features?

The main reason is that a linear combination of the input followed by a non-linearity stacked on top of eachother is a universal function approximator. Which means that no matter how complicated the true underlying function is, a neural network can approximate it to an arbitrarily small error.

There's also the efficiency factor since a linear combination of $n$ inputs each having $m$ dimensions can be represented using a single matrix multiplication $h=X \times W$ where $X$ is an $n \times m$ matrix (where each row is an example and each column is a feature of that example) and $W$ is an $ m \times d $ weight matrix. And computers are VERY efficient at doing matrix multiplications. Thus, the more you build your model to use matrix multiplications the better.


Time series binary classificaiton with labelling issues

From your description of the data, this is not a time-series problem. Time is not a factor here, for each user you have a set of variables after you choose a time threshold. Although you seek the "when", that's probably out of your reach, based on your explanation.

If I understand correctly you're doing classification but you're unsure about your "data collection process", at least for the "NO" labeled obs.

Why not fix a time stamp, and register all the variables you think are correlated with the outcome, and keep the outcome as you defined (never buys = "NO", buys at least once = "YES).

I'd treat this as a classification problem but instead of looking at the classic metrics (from confusion matrix), I'd be more interested at the probabilities you get from each obs, compared to the truth (that you know up to that point in time).

Imagine a user, with an estimated probability of 0.90 (90%), but the labels says "NO". He's someone who the model thinks that should have bought but didn't, he's highly likely to be interested in a sale.

Maybe you could cluster on this kind of information, creating groups of users where your model is wrong, but it's really confident in it's estimation. Basically you're more interested in false positive obs.

But also false negatives can give you insight, again looking at the probabilities you get.

I know this won't probabily be possible but if you could collect more data, I'd suggest a variable (or more), that measure the relationship of the user with sales.

Also, I agree about the decision of removing the fact that the user plays for long, that's probably "too much correlated" with your outcome.

Without knowing more about the data, that's my two cents.


I prefer not to add drop out in LSTM cells for one specific and clear reason. LSTMs are good for long terms but an important thing about them is that they are not very well at memorising multiple things simultaneously. The logic of drop out is for adding noise to the neurons in order not to be dependent on any specific neuron. By adding drop out for LSTM cells, there is a chance for forgetting something that should not be forgotten. Consequently, like CNNs I always prefer to use drop out in dense layers after the LSTM layers.


You could try some competitions from kaggle.

Data Science courses from Coursera, edX, etc also provide forums for discussion.

Linkedin or freelance sites could be other possibilities.


How do I represent a hidden markov model in data structure?

With 29 states and 841 possible transitions to track whilst reading a file with 2000 entries (word, tag), then you should not be experiencing a speed problem when using a dictionary of dictionaries.

Assuming your data structure as described called transition_counts, and receiving data in pairs, (this_pos, next_pos) then running 2000 times:

transition_counts[this_pos][next_pos] += 1

takes only a fraction of a second. This is similar for code that calculates $p(POS_{t+1}|POS_t)$:

total_from_pos_t =  sum(transition_counts[pos_t].values())
prob_pos_tplus_one = transition_counts[pos_t][pos_tplus_one] / total_from_pos_t

This is very fast. Your problem is not with the representation.


What is the role hidden vectors play in RBM?

From my point of view, both answers are correct - it depends on what you want to achieve.

One way to use an RBM is as a generative model. If you wanted to do this, you would train the network in order to obtain weights generating a distribution the network which is as close as possible to the real distribution of the input data (not exactly true as CD is known not to converge to the sample distribution, but this just as an aside). You could then, for instance, reconstruct images from partial or sample completely new images. In this context, the main purpose of the hidden units is to increase the power of the network - a Boltzmann machine without hidden units is not able to distinguish target distributions with the same second moments. However, the actual value of the hidden units during and after is less important, as your primary goal is to generate samples.

If you want to use the model to detect features, the hidden units have an additional role. To explain this, suppose you use contrastive divergence as a learning algorithm. In each iteration, you would then sample the hidden units from the visible units and the visible units from the new values of the hidden units. If the result is close to the original value of the visible units, i.e. if your reconstruction error is small, then, apparently, the hidden units and the weights have learned enough about the training data to reconstruct the value of the visible units from the value of the hidden units. As you say, this is in fact similar to a dimensional reduction or an autoencoder. You could then build a second RBM on top the hidden units, i.e. you could start to stack RBMs to build deep networks, where each new layer of hidden units learns more and more abstract features from the training data.


Anomaly detection on sparse categorical data

No need for machine learning here.
After you've transposed the dataframe, just count the number of unique combinations in the new column, and then rank them by frequency. Set a suitable threshold of "rareness" (like freq=2 below) and you will have your list of strange combinations.
There's a tool in Pandas for this called df.values_count()
e.g.

combination
freq

a,b
1

a,c
1

a,d
1

a,b,c
2

a,b,c,d
10

b,d
10

Then just compare you new combinations with your "bank of rare combinations", and update them if they are no longer rare.


For binary classification problem, naturally, classes are called positives and negatives. Positive is usually a class of detection something that we are interested in, for example, spam or disease detection. According to documentation for sklearn.metrics.roc_curve, it takes as argument probabilities for positive class. For most binary classifiers, output probability usually refers to the probability of positive class (which is marked as 1 in the data). So, use probabilities of the class marked as 1 in the data.


Running Jupyter notebook on remote server with local files

What sort of virtual machine are you using?
If you are using Google colab for example you can connect it to your drive and add data from your drive.


You may use ngrams to get the frequent sequences of you events.

Here a little example

 library(tau)
 seq <- "ababcdcde"
 textcnt(seq, method="ngram", n=3L, decreasing=TRUE)

   _   a  ab   b   c  cd   d  _a _ab aba abc  ba bab  bc bcd cdc cde  dc dcd  de de_   e  e_
   2   2   2   2   2   2   2   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1

Select the longest and most frequent ngram, here ab and cd.
You may also trade off the length and frequency to get maximal compression. The n parameter limits the length of the ngram.


Finding a kernel with feature transformation

Yes, it is valid.

Kernel is just an inner product in the feature space. In fact, here you have explicitly started from the feature transformation and we understand the feature transformation.


The data in our relational DBMS is getting big, is it the time to move to NoSQL?

A few gigabytes is not very "big". It's more like the normal size of an enterprise DB. As long as you go over PK when joining tables it should work out really well, even in the future (as long as you don't get TB's of data a day).

Most professionals working in a big data environment consider > ~5TB as the beginning of the term big data. But even then it's not always the best way to just install the next best nosql database. You should always think about the task that you want to archive with the data (aggregate,read,search,mine,..) to find the best tools for you problem.

i.e. if you do alot of searches in you database it would probably be better to run a solr instance/cluster and denormalize your data from a DBMS like Postgres or your SQL Server from time to time and put it into solr instead of just moving the data from sql to nosql in term of persistence and performance.


LSTM/RNN model fails on new test data - TFLearn

It looks like you are using the same training and test data for each iteration. This is causing your model to be overfit.

You need to use cross-validation. I would recommend using fewer training iterations and doing the following:

Split your training data into k chunks where k is your number of training iterations
At each training iteration make your training set by combining k-1 chunks and your validation data the remaining chunk. Note that this should be done so that your validation and training sets are never the same across two training iterations. Additionally the same chunk should never be in both your train and validation sets.

You can keep the the accuracy metrics on each training iteration if you want to see how your model's accuracy is converging, but the true accuracy of your model can only be measured on a holdout set which was not used in this validation scheme.

If you have enough data I would recommend doing an 80/20 train/validation initial split on your data, doing steps 1 and 2 on the 80% split and measuring the accuracy on the 20% split.

If you really want to be careful and have a lot of data you could do the following:

Perform a 60/20/20 train/validation/test split
Split both the train and validation sets by the number of iterations you are using.
At each iteration, use k-1 chunks of training data and validate against one chunk from the separate validation set.


How does TF-IDF classify a document based on "Score" alloted to each word

It's not a single TFIDF score on its own which makes classification possible, the TFIDF scores are used inside a vector to represent a full document: for every single word $w_i$ in the vocabulary, the $i$th value in the vector contains the corresponding TFIDF score. By using this representation for every document in a collection (the same index always corresponds to the same word), one obtains a big set of vectors (instances), each containing $N$ TFIDF scores (features).
Assuming we have some training data (labelled documents), we can use any supervised method to learn a model, for instance Naive Bayes, Decision Trees, SVM, etc. These algorithms differ from each other but they are all able to take into account all the features for a document in order to predict a label. So in the example you give maybe the word "mobile" only helps the algorithm eliminate the categories "sports" and "literature", but maybe some other words (or absence of other words) is going to help the algorithm decide between categories "Business" and "Tech".


The framework you cope with is semi supervised. You have mostly unlabelled data and you can have some labeled data by manual labelling.

Active learning is one method to cope with the situation, by focusing your labelling efforts in the most beneficial areas. You can read a survey on these techniques at Settles, Burr (2010), "Active Learning Literature Survey" (PDF), Computer Sciences Technical Report 1648. University of Wisconsin–Madison, retrieved 2014-11-18

Please note that even if you are focusing your labelling efforts using active learning, it is still a significant constraint.

There are other methods of copying with semi-supervised framework, like co training.The classical reference on co-training is "Blum, A., Mitchell, T. Combining labeled and unlabeled data with co-training. COLT: Proceedings of the Workshop on Computational Learning Theory, Morgan Kaufmann, 1998, p. 92-100"


How to import large datasets into Google Colab?

One option is that you can download the dataset into your system and save it in an easily accessible directory. Then, run the following codes:
from google.colab import files

data = files.upload()

After running the above line, you will get a Choose File button where you can directly browse your system and choose your file.
Added the screenshot for your reference:


Is it possible to change test and train data size when using crossvalind function with Kfold param?

K-fold means that the validation step will be performed k times, each of them using a fraction $\frac{k-1}{k}$ for training and $\frac{1}{k}$ for validation. If you want a fixed validation fraction, choose the number of folds that fits:

90%/10% : 10-fold
75%/25% : 4-fold
etc.

I don't know about Matlab libraries or functions, but if you want to do custom folds, you'll probably have to sample train and validations sets "by hand".


Sales Forecasting - Random Forest - Which features should I use for out of sample forecasting?

The question is poorly phrased, I've tried to edit it to the best of my abilities. However here are the problems you've stated,

Some of the features which are being used now can't be used later because they might not be known. Will it affect the model?
If they can be used, what type of algorithm can be chosen?

The answer to the first problem, you have to check the accuracy first before making choosing any new features if the rest of your features give a good enough accuracy then there is no need to choose new features.

The second problem, to predict those values of the features you are using now if they are discrete in nature try classification algorithms whichever fits the model best, else try something along the lines of regression if the input values are continuous. And then use these predicted values along with your existing model and check how the accuracy varies.


There seem to be a few options, but I found rasterfairy which is very easy to install and use. Has the added bonus of being able to fit to a rectangular grid, but also circular and other arbitrary shapes.

A very nice IronPython notebook example: https://github.com/Quasimondo/RasterFairy/blob/master/examples/Raster%20Fairy%20Demo%201.ipynb

And some example results:


Finding the usage percent to perform predictive analysis for new users

Create a new column containing the % of times each user used the search bar (i'll call it chance)

You could eliminate all users outside of 'n' standard deviations of the new column eliminate both high and low values or you could manually set a threshold
calculate the logarithm of each landed on home page ($landed_i$) count with the base as the max landed on home page value ($landed_{max}$) $log_{landed_{max}}(landed_i) = weight$, then multiply every weight by it's chance to create chanceWeighted. This weighted chance ensures that more frequent users have more weight attributed to them. You can now find whatever average you wish off this data
if you wanted to decay users depending on last visit you would need to record the users last visited date and them perform another function on them similar to the weighted average


Transformer architecture not working on toy problem

I am not sure but maybe the excess zeroes are the culprit. I guess the results you are getting are just the accumulation of the biases. I would suggest converting your one-hot vectors to dense vectors using an Embedding Layer.


Applying Sci-kit Learn's kNN algorithm to Fresh Data

KNN is not fitted to "the k-nearest neighbor points closest to the test data points". You specify the fit option, like:
neigh = KNeighborsClassifier(n_neighbors=3)
neigh.fit(X, y)

Usually this will be xtrain, ytrain, while you test the model performance using "new" (unseen) data and compare the true targets to the prediction.
neigh.predict(xtest)

or
neigh.predict_proba(xtest)

See docs: https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html


In the context of the link, a skewed data set is referring to a dataset with a class imbalance problem. They are trying to build a classifier, but they have many more negative examples than positive examples. It's not a very precise term, but I've heard to used in this context a few times.


plotting time series data using matplotlib python

This is happening because the dates in your column are not sorted. To fix this either sort the values by the date column or, even better, plot using the timestamps instead of their dates. Matplotlib inherently handles datetime format better than it does datest represented as strings!
The best option however, is to add the timestamps as the dataframe's index and use pd.Series.plot() instead of matplotlib.
data.index = data['time_stamp']
data['c_16_avg_a'].plot()


How to reverse ReLU activation in deconvolution

It's not reversing the ReLU, but applying normal ReLU activation again after max pooling and max unpooling.

It is used again

  "to obtain valid feature reconstructions at each layer (which also should be positive)"


I have a similar dataset atm and have been doing some research, not yet finished.

What needs to be kept in mind is the need for generalization given that there is such few samples and many dimensions.

The linear regression model is not robust enough given the dataset, it does not handle a high number of dimensions, compared to samples, well. Even if you could tune in order to get a good result, you are still probably overfitting.

Stepwise feature/model selectors such as Cp-statistics does not seem to do well with so many dimensions vs samples.

Ridge, Lasso, and Elastic net seems to be the way to go for feature selection. These penalized linear regression models handles the high number of dimensions vs samples much better. The linear regression is penalized in order to generalize better and not overfit, creating a less complex solution.
R: GLMNET - Penalized regression
Depending on alpha, you may receive a different feature set, coefficient that are not zero. E.g Lasso is biased for not returning features that are very correlated with each other while ridge might return a set of variables/dimensions/predictors/features that are correlated with each other.

K-Fold Cross-validation could be used in order to check the features selected are selected over and over again. This could be used in order to make sure that you don't overfit.

I hope this gives you a few ideas.


Predicting evolution of an unknown mathematical function with machine learning?

You don't need any ML to do this task. If you search on Wikipedia Radioactive Decay you will lear in the "decay rate" section that there is a known behaviour of the times of the decays. In particular the decay of a radioactive nuclide follows what is called a Poisson process, hence the number of events in a given time window follows a Poisson distribution and the delay between two consecutive events follows an exponential distribution with a mean value that is the inverse of the decay constant. As you have the timestamp of each decay event you can calculate the delays between them and fit with an exponential distribution to obtain the decay rate of Po210. Otherwise you can calculate the number of events in a given time window and using the Poisson hypothesis you can calculate the same value for the decay rate.
As the radioactive decay is a true random event that follows quantum physics rules you won't be able (in any possible way) to know when the process wi take place, you can only estimate the probability that it will take place in a given window of time.


Should we keep all channels when doing image classification?

Well, it depends on your images and how you want to classify them.

If we take a really basic example, if you want to classify images by their color, you obviously should keep all 3 channels.

In the other hand, if you want to classify images by a shape. For example squares and circle, you don't necessarily need to use all 3 channels.

But changing the number of channel isn't the only preprocessing you can do on your images before classifying them.

You can also change the color space for example (the colorspace LAB can be usefull in some case) , or even use Morphological Image Processing methods to improve your image by, for exemple, bringing out the contrasts of your images or cleaning impurities.

All of this depends of your images, and what you want to do with them.


Keras model.predict giving different shape from training label array

The 10 outputs came from the fact that you have 10 neurons in the final layer of your network.
If you change your model to
model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(1, activation='relu')
])

its output will have a shape of (200, 1).


Classification Dataset which is linearly non separable

In Python you can use the in-built make_moons and make_blobs functionalities of scikit-learn, changing the parameters accordingly.

The "titanic dataset" and "mpgcars" also contain non-linearly separable categories, in addition.


Best method for similarity searching on 10,000 data points with 8,000 features each in Python?

There are two main paths:

Load all vectors into memory. If you are able to load vectors into memory, then you might be able to search the space with "clever" brute force. One such method is found in this paper.
Keep vectors on disk. If you follow this path, then you have to index the vectors. You are basically building a search engine. Common open source search engines are: Apache Solr and Elasticsearch

It also depends on what kind of what kind of search is needed. A working definition of "close" vector is needed. The most common definitions of "close" vectors can be found here.

Fais is a new library for efficient similarity search of vectors. It is designed for many vectors (> 1,000,000) each being relatively small (10s to 100s  of dimensions). It may or may not scale to your problem. If it does not scale because of high dimensionality, you can reduce dimensionality with Principal component analysis (PCA) or t-SNE.


Question About Coming Up With Own Function for Distance Matrix (For Clustering)

It's trivial to see that you don't want to compute the distance matrix. Just compute the size of the entire matrix, even assuming zero memory overhead... 35 million x 35 million x 8 bytes per double /2 for optimizing symmetry. That is something like 5 Petabyte?

Forget anything that is not linear. Try to shrink your data first by weeding out the easy cases first: duplicates, single-time visitors where you have too little data to say much, etc.

Then begin experimenting with a manageable sample first. If you find something on a sample then it might be useable on the entire data, too. And if a method does not work in the samples it likely will not work on the entire data (but you'll have to wait much less to know it doesn't work).


How to properly rotate image and labels for semantic segmentation data augmentation in Tensorflow?

I have found the following solution.

Labels

Here is a snippet filling an empty space in label with ignore label.

IGNORE_LABEL = 255
label = tf.subtract(label, IGNORE_LABEL)
label = tf.contib.image.rotate(label, angle)
label = tf.add(label, IGNORE_LABEL)

This way, by subtracting IGNORE_LABEL, you shift the interval to [-255, 0]. When label (segmentation labels) is rotated, empty spaces are filled with zeros. By adding back IGNORE_LABEL, you shift interval back to [0, 255] meaning that empty space is set to IGNORE_LABEL.

Images

Similar trick can be applied to image rotation to fill empty space with preferred value.


The difference is caused by the fact that seaborn.distplot and matplotlib.pyplot.hist use different defaults for the number of bins. The bins are ranges of values for which the number of observations are counted before being plotted. For more information on what bins are check the Wikipedia page for histograms.

In your example, the standard matplotlib plot has bigger bins than the seaborn plot since it uses bins=10, whereas seaborn seems uses the Freedman-Diaconis rule to determine the number of bins, which in this case would give a bin width of about 0.5 and bins=18.

Setting the number of bins used equal for both the seaborn and matplotlib.pyplot plots gives the following histograms:

As you can see, using the same value for the number of bins gives the exact same plots. I used the following code to produce this plot, in which you can also change the number of bins used by both plots to compare them.

import seaborn as sns
import matplotlib.pyplot as plt

# Set number of bins
nbins = 10

# Load dataset
x = sns.load_dataset("tips")

# Set up subplots
fig, axs = plt.subplots(1, 2, figsize=(12, 5))

# Seaborn plot
sns.distplot(x.tip, ax=axs[0], bins=nbins, kde=False)
axs[0].set_title("Seaborn plot")

# Matplotlib.pyplot plot
axs[1].hist(x.tip, bins=nbins)
axs[1].set_title("Matplotlib.pyplot plot")

# Set title
fig.suptitle(f"Histograms using $bins=${nbins}")

fig.show()


I recommend using LSTM RNN or CNN algorithm to pick up the most popular product of on-going month based on the past purchasing history. I have created a working product ranking model for online shopping site lands'end in the United States.

In order to archieve the goal of this solution, you need to follow several steps.

collecting the dataset

I think you already gathered the dataset for this part.
Processing of the dataset for supervised learning.

Cleaning up the dataset - please remove rows that contains NaN columns
mapping the products into eyes - you need to map the categories and products into row matrix with the element of 1 and 0.
Normalization - you need to rescale all real numbers into [0, 1]
split the dataset into the training, valid and test data.

Build Deep learning model like LSTM RNN

select the deep learning framework. I recommend Tensorflow, Keras.
set the dimension of model. i.e. the number of layers, the number of neurons per layer.
optimizer, metrics

Training the model and testing
Get the predicted ranking of the category by monthly.

Protip: the most important thing to get the better result is which column would be set the output for ranking. You should sum up the total purchasing count per product or per category by month or week.


group similar subjects and train only using them

Clustering would indeed give you something like group1, group2, etc., i.e. it would assign every instance to a group meant to represent similar instances together. With the model you could also assign any new instance to one of the groups in the model, i.e. find the group that this instance is the most similar to. In general clustering doesn't give you which predictors are the most important for a group (there might be ways to calculate this from the model, but afaik it's not standard).
Note that with clustering alone you can't predict the target variable for a new subject. However you could obtain the proportion of positive/negative for every group, i.e. the chances of being positive/negative knowing that the instance belongs to a particular group. You could also do the opposite: chances of belonging to a particular group knowing whether the instance is positive/negative.
An important question is whether you want to include the target variable in the features used for clustering:

If it's included, the model uses it like any other feature to group instances by their similarity.
If it's not included, the model itself doesn't know it so the proportion of positive/negative in a cluster doesn't depend on it. This might be more insightful, depending on what you want to obtain. Another advantage of not including it is that you can still use the outcome of the clustering for training a binary classification model afterwards, typically by adding the cluster an instance belongs to as a feature.

As you can see, there are a few important design decisions to make depending on the goal.
[edit: answer to comment]
In order to use the clustering as a basis for classifying any new instance, here are a few general options:

Basic option: predict the cluster the instance belongs to, then just output the probability of positive of the cluster.
Clustering model + classification model: train a classifier with all the training instances, with one additional categorical feature corresponding to the cluster the instance belongs to. The classification might ge improved compared to only the original features.

Variant: add one feature for every possible cluster, where the value represents how similar the instance is to each cluster (most clustering methods can calculate a distance or similarity against any cluster).

Multiple independent classification models: based on the clustering, train a classification model for every cluster independently, using only the instances from this cluster. Advantage: this allows the classification model to take into account more fine-grained patterns within this cluster. Disadvantage: the model has less instances to be trained from, especially for small clusters.