How to explain the outcome of k-means clustering?

I made some similar studies on process descriptions on computers. The main difficulty was to discard words that were not meaningful. Which is hard because:

Some words appear in every clusters, but in different proportion. They should not be discarded, like "pilot". It is normal that "pilot" appears often, but depending on the number of occurrences it might signify that the pilot made a decisive action for the accident.
Some words appear very rarely, but they are the key words for the reason why the accident happened. Then it is very hard to find a way to give them more importance without adding direct prior knowledge to your clustering.

What I could suggest is:

First try different k value. Why did you chose 3 ?
Try a different algorithm for 2D visualization. For example t-sne is good for reducing high dimension in 2. Look at the examples here http://scikit-learn.org/stable/auto_examples/manifold/plot_lle_digits.html#example-manifold-plot-lle-digits-py
Try working directly on your word occurrences to check whether your word differ between clusters. For example, you can compute the KL-divergence of each word for each cluster given their global distribution. See here https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence
If you can't get good results, think of another way of vectorizing. For example word2vec is good at finding words that occurs often together and could be more suitable to your case.


Why models performs better If normalize test data and train data separately?

If apply normalization on training and testing in a separate way, I get really good results 85% (and sometimes more) and the further steps I try to do next work better as well.

The problem with applying normalization across instances on the test set separately is that the test set represents any new data. So in principle the model should be able to give a prediction for a single instance independently from any other instances, in which case there is no set of instances to obtain the mean/std dev from. More importantly, the prediction of the model for a given instance should always be the same. Normalizing on the test set breaches this principle, because it makes the prediction for a particular instance depends on the other instances in the test set.

I don't think "separate normalization" is unethical strictly speaking, because it doesn't imply using any of the test data at training stage (whereas normalizing before splitting the train/test sets would). However it's theoretically incorrect for the reasons I mentioned above.

The fact that you obtain such a big difference in performance by normalizing "separately" points to a very different distribution of the data between training and test set (or a bug somewhere along the process). I'd suggest investigating that, maybe there's some error in the data?


Is it possible to use Jaccard similarity instead of Cosine similarity in gensim document similarity?

If you have trained a gensim model, that object can act as a dictionary to give you the vector projection (via https://radimrehurek.com/gensim/models/word2vec.html)

$ model['computer']  # raw numpy vector of a word
array([-0.00449447, -0.00310097,  0.02421786, ...], dtype=float32)

So it is possible to manually implement any vector comparison that you choose.  Cosign similarity is typically chosen because it performs relatively well when compared to other methods of grouping high dimensional projections.

The way I could envision implementing Jaccard Similarity would be to identify a list of key words on a per document basis, and when comparing document, include words that are synonyms as intersections.

Based on reviewing the gensim document comparison text (https://radimrehurek.com/gensim/similarities/docsim.html) I don't believe there is a native implementation.


What is the difference between cache() vs prefetch() in tensorflow?

The tf.data.Dataset.cache transformation can cache a dataset, either in memory or on local storage. This will save some operations (like file opening and data reading) from being executed during each epoch. The next epochs will reuse the data cached by the cache transformation.
Prefetch overlaps the preprocessing and model execution of a training step. While the model is executing training step s, the input pipeline is reading the data for step s+1. Doing so reduces the step time to the maximum (as opposed to the sum) of the training and the time it takes to extract the data.
Source


Augmentation on test dataset and validation dataset

Your test and validation dataset should reflect the type of data you would expect when you deploy your model in the actual setting. So usually you do not apply augmentation to the validation and test dataset, since in the real setting you will not receive some strange augmented images.
Another way to think of it is if you apply augmentation to your validation dataset then you will actually measure how your model will perform on augmented data and not on 'real' data. You will use this to inform the best settings for your model so you will end up with a model tuned to perform well on augmented data.


How to extract location related terms from raw text in python

I fiund these two tools which can find geographical terms in a text file.
mordecai: Full text geoparsing as a Python library. Extract the place names from a piece of text, resolve them to the correct place, and return their coordinates and structured geographic information.

, and CLIFF-CLAVIN: CLIFF-CLAVIN parses news articles and pulls out people, organizations and places mentioned.


Difference between LASSO penalty in neural network and just LASSO regression

I think Lasso Regression, is a type of regression model which implements the lasso penalty (L1 regularization) within its algorithm (different from normal regression algorithms).

Lasso penalty is more of a generalized penalty that can be put on any learning algorithm.

The concept behind both ideas are the same, but their area of implementation is different.

Hope this helps !!


I did something like that when I was at university. Professor asked me to implement and compare a variant of k-nearest neighbor with the original one.

I've applied the 10 fold cross validation over several datasets with both the algorithms and then I've applied the Friedman test with Holm post hoc over the accuracy means. You can use also simple paired t-test or others, it depends on your needs.

My professor also suggested me to use this graphical tool.
Hope this helps

SPOILER

It turned out that my k-NN variant wasn't better!


NLP: To remove verb and find the match in a sentence

As indicated by @hssay, your way seems to be PoS tagging and then removing verbs. If you don't want to get your hands dirty, you might prefer to use the off-the-shelf Google natural language web API. If you try the UI, click on the Analyse button, and then under the Syntax tab, look for Part of Speech and VERB.

Another similar toolkit that you could use as an API is StanfordNLP.


How does combining neurons create non-linear boundaries?

I think what you are referring to is how neural nets work as universal function approximators. Check out this link for an intuitive explanation.


Why BERT tokenizers function differently?

Each BERT variant is trained with text that has been prepared differently, e.g. as the name implies, BERT uncased is trained with text where all letters are lowercase. This means that the vocabulary extraction process has also use lowercase text as input, and therefore gives as result a different vocabulary than the same vocabulary extraction process used with text in its original casing.
Note that, as the vocabularies are different, each model should be used with the tokenizer it was used to train it. Using a model with a different tokenizer may lead to bad results.
The choice of tokenizer, therefore, is tied to the choice of BERT model. The criteria to use one or the other (e.g. uncased vs. cased), depends on the case. For instance, for doing named entity recognition (NER) in English, it may be important to keep the original casing so that the model can more easily distinguish proper nouns.


Setting class-weights in a batch in where a certain class is not present

I don't think that the weight you use matters, whether you set it to $10^6$ or $10^8$. This is because, as examples of that class are not in the batch, the loss function for that batch will not have any of those examples contributing, so that weight won't appear. If something does not appear in the loss function, it is not used at all. For that reason, it doesn't really matter what weight to use.


1) Feature Selection should be done by AutoML on the other hand preprocessing is normally done by the user in order to make sense fo the data.

2) AutoML takes care of the hyper-parametrization.

3) The disadvantage that I mostly find is that is extremely computationally expensive. And from what I have seen in Kaggle most of the winning solutions use manual ML, not AutoML.

4) For me, one of the advantages is that sometimes it finds a good algorithm that I have not tried (or thought) and that it avoids me spending some coding time. Also, it happens to do some good ensembles of different models.

5) You can do Manual ML in your side and then doing an ensembling with your personal ML model and the AutoML. This doesn't guarantee improvement but it could boost your performance.

You can have a look at H20 AutoML, I quote the documentation, I believe it can be helpful in this case in order to have an intuition about it:

  Although H2O has made it easy for non-experts to experiment with machine learning, there is still a fair bit of knowledge and background in data science that is required to produce high-performing machine learning models. Deep Neural Networks in particular are notoriously difficult for a non-expert to tune properly. In order for machine learning software to truly be accessible to non-experts, we have designed an easy-to-use interface which automates the process of training a large selection of candidate models. H2O’s AutoML can also be a helpful tool for the advanced user, by providing a simple wrapper function that performs a large number of modeling-related tasks that would typically require many lines of code, and by freeing up their time to focus on other aspects of the data science pipeline tasks such as data-preprocessing, feature engineering and model deployment.

You can also have a look at this blog of Bojan Tunguz where he defines what are the phases of AutoML:

Level 0: No automation. You code your own ML algorithms. From scratch. In C++.
Level 1: Use of high-level algorithm APIs. Sklearn, Keras, Pandas, H2O, XGBoost, etc.
Level 2: Automatic hyperparameter tuning and ensembling. Basic model selection.
Level 3: Automatic (technical) feature engineering and feature selection, technical data augmentation, GUI.
Level 4: Automatic domain and problem specific feature engineering, data augmentation, and data integration.
Level 5: Full ML Automation. Ability to come up with super-human strategies for solving hard ML problems without any input or guidance. Fully conversational interaction with the human user.


Is there a way to visualise what/how my image classifier has identified an image?

What you are looking for is a model with an "Attention mechanism". Unfortunately, this means that you have to change your model and to retrain it.
It is more complicated that a simple image classifier but I can assure you that it is worth it to learn how it works. Or you can simply find an existing repository that implement it.

Alternatively, you can train your model on smaller images that contain exactly the melanoma. The preprocessing will be much heavier.
Then you will scan your bigger images with a bounding box this is known as sliding window. You will have probabilities for each small images.

Sliding Window (Formally):

The propose is:
Given a classifying model $F(J)$ where $J$ are images of size $w \times h$ you will apply this model to a image $I[m,n]$ by slicing it into many blocks $B[m,n]$ where
$$ B[m,n] = I\Biggl[\Biggl(m-\frac{w}{2}:m+\frac{w}{2}\Biggr),\Biggl(n-\frac{h}{2}:m+\frac{h}{2}\Biggl)\Biggr]$$

And you will apply the model as $F(B[m,n]), \forall m \in [w,W-w] and n \in [h,H-h]$ where $W$ and $H$ are the width and height of $I$.

Important to notice that for OpenCV images the height is actually the first coordinate of the variable (numpy array).

If you have a mean size of melanoma images you can run through  $m$ and $n$ with steps of half of the mean size.

So you might search for which parts of the image where classified as melanoma by looking into $$H[m,n] = F(B[m,n])$$ and using it as a binary mask to identify melanoma classified image parts.


How can I use different loss functions for images with different labels in tensorflow?

This is not possible, for 2 reasons:

It doesn't make mathematical sense. The Gradient must explore one loss function that is optimized to distinguish between more than one class. Using a loss function to classify a single class is equivalent to not learning anything.
In ML, the models you train are meant to generalize on data they have never seen before. Because of this, you can't use different loss functions because once you test its performance on test data nobody knows to what class they belong.


Which data science framework should I start learning?

Matplotlib is for visualization. Pandas is for handling data frames. Numpy is more matrix-oriented. I feel like a good starting point is numpy with the various casting/slicing functionalities and some libraries such as script/optimization. Matplotlib would help you “see” what you are doing. For data science specifically, data frames are easier to deal with and more affine to reality (they are tables), so pandas are more indicated for starting. For Machine Learning a good source is the book by Ageron on scikit learn and tensor flow. There you could see some applications of the various libraries. If instead, you want to see some hard theory, if I remember correctly  “statistical machine learning” is great as a book (don’t remember the author). I personally did a little bit of both and it ended up being useful. For basic models, it’s definitely better if you understand the theory first.


Is there any kaggle competition for finding the feature for affecting revenue?

Is there any kaggle competition out there doing EDA (Explotary data analysis) not prediction for finding the most significiant feature that affects the net_revenue or sales ?

Although it is hard to prove a negative, I would say "no" to this.

Kaggle competitions are based on continuous metrics that can be ranked, such as getting best log loss, mean average precision etc.

Identifying the "most significant feature", although a potentially useful question to answer from a data set, is not something that can be ranked automatically and objectively in a competition.

It is possible you will find competitors performing that kind of analysis within a prediction challenge. It is fairly common to rank features by relevance, especially when using ML techniques that support doing so, such as random forests or xgboost. I am pretty sure I have seen forum posts in Kaggle discussing these things and searches for a "golden feature" which is a similar concept - usually some processed combination of one or more original features.

In addition, there have been a few competitions where the entrants were expected to produce supporting documentation. Discussion of feature relevance could be an important part of such entries. These are much rarer competitions at Kaggle than the more direct finding best predictive model.

So you may find competitions where identifying a "best feature" is part of the winning strategy, but the goal of the competion will likely be getting the best mean log loss on the test set.


Is there an API to extract questions and labels from yahoo answers?

It seems as though there is a lot of advice on the Yahoo Developer Site.

Basic steps will be:

get yourself a standard Yahoo account
using that account, get yourself an Application ID (basically an API key)
(probably) get a unique password to go with your API key

This will all allow you to authenticate yourself when making requests for data.

There are daily rate limits for data requests, they say:

  Each service may have different limits on the number of results returned by each query, as well as other variations in supported parameters. The documentation for each service specifies the default number and maximum number you can receive.

I should also point out that the service is not intended for commercial use, so if you plan to make a product and sell it, based on Yahoo data, you need to probably discuss the API usage with them.

The results will be in a form that requires parsing, such as XML, as it seems to be a standard web requests API. This means you can look at tools such as BeautifulSoup4 or the xml ElementTree. You can then choose the specific parts of the received data from Yahoo's API.


I found the answer in https://stackoverflow.com/questions/55868975/keras-how-to-evaluate-model-accuracy-evaluate-generator-vs-predict-generator/57630536#57630536
I just had to add shuffle = False in the test_datagen.flow_from_dataframe, and the confusion matrix was good


Deciding on the number of components in PCA

The accuracy is varying and not linear, meaning that if the number of components increase, the accuracy will not necessarily increase.

This is not unexpected. Choosing the right number of components requires balancing the extra information given by the additional dimensions and the useless noise and redundancy present therein. Even though PCA is a linear transform, and even if you used a linear classifier, the dependence of the classifier performance on the number of components is generally very non-linear.

Two options (or alterations thereof) are common:

Pick a predefined level of variance to keep (usually 90% or 95%, at least for me) and just stick with it. Think of it as regularization, rather than as a hyper-parameter.
Treat it as a hyper-parameter and optimize over a validation set for the number of components to keep $k$. This is essentially what you are doing already. It's perfectly reasonable to simply choose the $k$ that gave the best performance (on a held-out set).


How do you effectively predict the top 20% most likely customers to churn from a dataset?

If you use k-fold cross validation, then every customer in your dataset will lie in the test set of one of the k folds. That might help alleviate the issue at hand. Finally you can train your model against your entire train set and run inferences against net new customers.


What does it mean when the model Learning Curve displays this behavior?

Yes, this is overfitting. As you can see in your loss curve the training loss is steadily decreasing like it should, but at the same time your validiation loss is increasing. On data the model hasn't seen before it's actually getting worse!
This means your model is starting to learn the training data, and does not instead keep the learning generalized. As soon as validation loss is increasing you should stop training.
I'm answering this question although overfitting is a general concept and has been answered before because this question might want to have an answer that is specific to LSTMs.


Named Entity Recognition: NLTK using Regular Expression

You have a great idea going, and it might work for your specific project. However there are a few considerations you should take into account:

In your first sentence, Obama in incorrectly classified as an organization, instead of a person. This is because the training model used my NLTK probably does not have enough data to recognize Obama as a PERSON. So, one way would be to update this model by training a new model with a lot of labeled training data. Generating labeled training data is one of the most expensive tasks in NLP - because of all the man hours it takes to tag sentences with the correct Part of Speech as well as semantic role.
In sentence 2, there are 2 concepts - "Former Vice President", and "Dick Cheney". You can use co-reference to identify the relation between the 2 NNPs. Both the NNP are refering to the same entity, and the same entity could be referenced as - "former vice president" as well as "Dick Cheney". Co-reference is often used to identify the Named entity that pronouns refer to. e.g. "Dick Cheney is the former vice president of USA. He is a Republican". Here the pronoun "he" refers to "Dick Cheney", and it should be identified by a co-reference identification tool.


How to categorize unlabelled promotional email data

So the question asks what model can you use to put texts into topic categories in a unsupervised way.
The main model that we use for this sort of task is Latent
Dirichelt Allocation (LDA) (https://programmerbackpack.com/latent-dirichlet-allocation-for-topic-modelling-explained-algorithm-and-python-scikit-learn-implementation/)
Also, there is another method you could try, which is to convert texts into document embedding using a Doc2Vec (https://medium.com/wisio/a-gentle-introduction-to-doc2vec-db3e8c0cce5e) model, which will convert the text into an n dimensional vector. From these document embedding you could use any form of clustering (KNN or Agglomerative Clustering) to cluster the texts into topics.


Unexpected Error Message in RStudio; while using 'twitterR' Package

It's because of the whitespace in the parameter for geocode.

It should be:

my_tweets <- searchTwitter('@jabhij', geocode="20.593684,78.96288,2000km")


Yes, Orange has a Dimensionality Reduction Widget under the Unsupervised Learning tab. Click PCA (Principal Component Analysis)


Is there a way to aggregate frequency distribution for many runs?

My question is is there a way to aggregate all the frequency distributions for all the fitness values among the 30 runs? so that I can only show a single plot for all the runs.

Yes absolutely, you can do that with boxplots or violin plots, one boxplot (or violin plot) for each fitness value. The height and shape shows the distribution of the values across runs for one fitness value, and allows visual comparison between the different fitness values.


How to deal with possible data leakage in time series data?

There is no need for sample tests.

A customer may have received many loans 1 to n - 1. To predict the default rate of nth request at time t(n), you are allowed to use any information up until t(n). When a user has no transaction history before t(1) system cannot predict the default rate for her; except maybe based on her age, income, etc. However, for the next loan request at t(2) system can use the transactions between t(1) and t(2), but still cannot use any transaction that happened after t(2). For any particular prediction at t(n), events happened after t(n) must never be used.

Regarding "it does not involve using information not available at prediction time", it does involve using information not available at prediction time t(n), since system is trying to utilize transactions that occur after t(n).


I need some guidance how to develop myself in deep learning

I am not an expert in the field, but I'd like to share with you my personal opinion. Taking online courses and reading books is great, but the problem is you don't get to practice much on your own. I know there might be assignments and projects included, but they're mostly guided and pretty straightforward, or related to a certain topic of the chapter/lecture.

However, real world problems are a bit different. They usually encompass several smaller sub-problems that might not be clear at the beginning, and there are a plenty of choices that you have to make on your own. Usually you'd have to try several solutions before you're confident what suits your problem best. I believe it's something you learn mostly by practice. So I think given that you have enough knowledge now, it's about time to use your knowledge and build interesting stuff, stuff that would speak for you. Kaggle is also a great way to move forward, since there are tons of competitions and data sets that could inspire you, and help you know where you stand among others. With every project you try to do, you discover something new or learn to appreciate something you already know more. Plus you can always learn from others who publish their kernels.

Whatever you choose to do, I wish you all the best :)


How to get predicted class labels in convolution neural network?

What you have are predicted class probabilities. Since you are doing binary classification, each output is the probability of the first class for that test example.

To convert these to class labels you can take a threshold:

import numpy as np

probas = np.array([[0.4],[0.7],[0.2]])
labels = (probas < 0.5).astype(np.int)
print(labels)
[[1]
 [0]
 [1]]

For multiclass classification  where you want to assign one class from multiple possibilities you can use argmax:

probas = np.array([[0.4, 0.1, 0.5],[0.7, 0.2, 0.1],[0.3, 0.4, 0.3]])
labels = np.argmax(probas, axis=-1)
print(labels)
[2 0 1]

And to get these as one-hot encoded arrays you can use LabelBinarizer:

from sklearn import preprocessing

lb = preprocessing.LabelBinarizer()
lb.fit_transform(labels)
array([[0, 0, 1],
       [1, 0, 0],
       [0, 1, 0]])

And for multilabel classification where you can have multiple output classes per example you can use thresholding again:

probas = np.array([[0.6, 0.1, 0.7],[0.7, 0.2, 0.1],[0.8, 0.9, 0.6]])
labels = (probas > 0.5).astype(np.int)
print(labels)
[[1 0 1]
 [1 0 0]
 [1 1 1]]

Some packages provide separate methods for getting probabilities and labels, so there is no need to do this manually, but it looks like you are using Keras which only gives you probabilities.

As a sidenote, this is not called "normalization" for neural networks. Normalization typically describes scaling your input data to fit in a nice range like [-1,1].


Semantic segmentation of an image with multiple labels per pixel

Three separate models (one per channel) will easily learn to predict the >channel category,  but it will never output r0g1b2 and r2g3b0 classes in 64 class model because it have never seen these classes. How to overcome this problem?

The only way to solve this problem is to use channels (one for each skin damage type) for each pixel, and treat it as a regression rather than a classification.
In other words, use a multi output regression.
For your output, use a convolutional layer that gives the same number of rows and columns as the input, but 4 channels (one for each skin damage type).
Your ground truth (y_true) should be an array the same width and height as your input, but with 4 channels (one for each skin damage type) with each holding the severity rating of that pixel for the corresponding skin damage type.
Your loss function could be something used for regression such as MAE (mean absolute error).
This is because a classification will output 0 for classes it has never seen samples of since that is what minimizes the loss. A regression, on the other hand, will treat the target variable as continuous, and even if it hasn't seen examples of all severity levels of Type A skin damage (for example), it can still output them.
You could then choose thresholds to classify the outputs into the corresponding labels.
Ex. for each output <0.5 is 0, >0.5 is 1, >1.5 is 2, >2.5 is 3, >3.5 is 4
so [0.1, 1.6, 0.7, 5] means {A: 0, B: 2, C: 1, D: 4}
This is also more useful and easier to interpret in a clinical context as the severity is ordinal rather than just categorical, so a doctor would be better off knowing that a particular mild case (for example) had a prediction of 1.4 rather than 0.6, because they both correspond to a mild prediction but a 1.4 is closer to moderate and may be treated differently than a 0.6
This is an interesting problem from a learning and research perspective, but I develop deep learning based prognosis and diagnosis models using medical images at a big pharma/life sciences company and can tell you that a dataset of 200 images for a task this complex will be insufficient for decent performance or reliable results. multi-fold training/validation/testing AND some slight image augmentation will be necessary, but probably still insufficient.


Is there an algorithm for sampling shortest paths?

I don't know whether this answers your question. But have a look in these algorithms:

The Shortest Path Faster Algorithm (SPFA) is an improvement of the Bellman–Ford algorithm which computes single-source shortest paths in a weighted directed graph. The algorithm is believed to work well on random sparse graphs and is particularly suitable for graphs that contain negative-weight edges.

Dijkstra's algorithm solves the shortest-path problem for any weighted, directed graph with non-negative weights. It can handle graphs consisting of cycles, but negative weights will cause this algorithm to produce incorrect results.

The Shortest Path algorithm calculates the shortest (weighted) path between a pair of nodes. In this category, Dijkstra's algorithm is the most well known. It is a real time graph algorithm, and can be used as part of the normal user flow in a web or mobile application.

More Info at:

https://www.sciencedirect.com/topics/computer-science/shortest-path-problem
https://medium.com/basecs/finding-the-shortest-path-with-a-little-help-from-dijkstra-613149fbdc8e
https://link.springer.com/chapter/10.1007/978-3-319-05813-9_9


Flipping the labels in a classification problem

To begin with, I don't quite get it why you want to flip them. In the binary case, you flip Negatives and Positive, so True Negative becomes True Positive and so do FP/FN. Hence you flip specificity/true negative and sensitivity/recall values, so overall accuracy and F1 stay the same.


R: Calculations based on frequencies / grouped / aggregate data

I think working with "collapsed" or "summarized" data as opposed to the data itself directly inside the SQL database is counter-intuitive. Also, trying to download the original table for analysis in R is a very bad route if the data is too big to fit in the memory, and given that you already have a tool like SQL, which is designed for that specific purpose. You can calculate the mean, median, quantiles, etc. in SQL; and then pull that into R (even into Excel) and draw box plots.

Even though overkill, here is what you can do:

Calculate the mean, median, quantiles from the frequency chart by coding basic math in R. For example mean of a series is the sum over frequency(i)*value(i)/sum(frequency(i)), where i=1,...,n, and n is the number of distinct values in the frequency chart. Median is the value which is in the mid point of series 1,...,sum(frequency(i)).
You can recover the original series from summary statistics, but memory is again of concern. Let's say your original series was this:

  mySeries <- c(5,9,4,4,2,5,8,9,9,5,5)

But obviously it sits in the database, and this is what you get from SQL as you described:

> fromSQL <- data.frame(table(mySeries))
> fromSQL
  mySeries Freq
1        2    1
2        4    2
3        5    4
4        8    1
5        9    3

You can just use the rep function to recover the series like so:

> recoveredSeries <- as.integer(as.character(rep(fromSQL$mySeries, fromSQL$Freq)))
> recoveredSeries
 [1] 2 4 4 5 5 5 5 8 9 9 9

Now define a function that returns the summary of this series and gives you what you want:

> multiSummary <- function(x) {
+   c(mean = mean(x), median = median(x))
+ }
> multiSummary(recoveredSeries)
    mean   median
5.909091 5.000000

As I said, in my opinion, all of this is overkill. Different tools are tailored for different needs, and trying to do something with a tool that is not really the best for your need will give you less-than-optimal results and cost you a lot of time. My recommendation would be performing SQL operations, or going for MapReduce framework for this kind of purposes, depending on the size of your data.


I do not believe that Keras returns the MNIST data shuffled. You can see that it is not the case below.

First I will define a function which we will use to plot the MNIST data nicely.

import matplotlib.pyplot as plt
%matplotlib inline

# utility function for showing images
def show_imgs(x_test, decoded_imgs=None, n=10):
    plt.figure(figsize=(20, 4))
    for i in range(n):
        ax = plt.subplot(2, n, i+1)
        plt.imshow(x_test[i].reshape(28,28))
        plt.gray()
        ax.get_xaxis().set_visible(False)
        ax.get_yaxis().set_visible(False)

        if decoded_imgs is not None:
            ax = plt.subplot(2, n, i+ 1 +n)
            plt.imshow(decoded_imgs[i].reshape(28,28))
            plt.gray()
            ax.get_xaxis().set_visible(False)
            ax.get_yaxis().set_visible(False)
    plt.show()

Then let's reload and plot the MNIST dataset 4 separate times. We will see that the order of these is unchanged

for i in range(4):
    (x_train, y_train), (x_test, y_test) = mnist.load_data()
    show_imgs(x_train, x_test)
    print('Training labels: ', y_train[0:10])
    print('Testing labels : ', y_test[0:10])

  Training labels:  [5 0 4 1 9 2 1 3 1 4]
  Testing labels :  [7 2 1 0 4 1 4 9 5 9]

  Training labels:  [5 0 4 1 9 2 1 3 1 4]
  Testing labels :  [7 2 1 0 4 1 4 9 5 9]

  Training labels:  [5 0 4 1 9 2 1 3 1 4]
  Testing labels :  [7 2 1 0 4 1 4 9 5 9]

  Training labels:  [5 0 4 1 9 2 1 3 1 4]
  Testing labels :  [7 2 1 0 4 1 4 9 5 9]


Input contains NaN, infinity or a value too large for dtype('float32'). Without ruining Dataset

If NaNs are not too frequent you can try to drop the rows containing it.

Otherwise if your algorithm can't handle NaNs (i.e. lightgbm that is also a tree based algorithm can) you have to fill those values somehow. Usually it's a good idea to use a value that is out of the range of normal values (like -99 for a variable with only positive values) so tree can learn to treat those values differently.

In your case with binary (0/1) data it's a little different. I would put 0.5 so model can learn with a single cut to put missing values to either group (0 or 1) depending on which one makes more sense. Don't worry if all 3 groups are very different model should be also able to learn it given that you have enough data for the tree to make another cut.


Mixture Density Network: determine the parameters of each Gaussian component

Actually, the upper alpha and the upper sigma are not free parameters to be set, they are just used to represent the output activations corresponding to the mixture coefficients and the variances, respectively. They are used to distinguish derivatives with respect to the alpha and sigma. I say it from page 275 of the book “pattern recognition and machine learning” by Christopher Bishop”:


LSTM with return_sequences - "Training a model on multiple timesteps simultaneously"

Setting return_sequences to True or False depends entirely on which one is appropriate with respect to how you are going to make predictions at inference time:

If you want your model to make a prediction only when it has consumed a whole sequence as input, then you use return_sequences=False, so that your loss is only based on the output after ingesting the whole sequence.
If you want your model to be able to make predictions after any number of timesteps, you set return_sequences=True and set the loss over all the time step outputs. As we are dealing with time series, the appropriate loss could be the average squared error over all time step outputs.

About the statement "With return_sequences=True the model can be trained on 24h of data at a time", I think it means that you are training the model for each of the 24h time steps, instead of the single prediction trained with return_sequences=false.


Extracting words belonging to a key from the text

You have two options:

Use regular expressions to extract features of interest based on your dictionary. You may have mixed results, since, for example, dual sim cards will be micro or nano, so you will get two types from single description. However, that just says, that the dictionary isn't well structured.
Train your own NER (named entity recognition) system. For that, first, you will need a data set which will consist of your descriptions and marked entities. This would require manual labour, either yours or paid one one some crowd sourcing platform. When you will have a marked-up dataset, you could use NLTK to train a NER. Markup may be done in any way familiar to you, but, generally speaking, xml-like markup is the best choice since there are many ways to train a NER, and most of frameworks know how to parse xml-like tags:

Example markup:

<screen_size>5.5"</screen_size> (13.97cm) full hd <resolution>1920 x 1080</resolution> pixels ips screen gorilla glass display etc.


Using scikit Learn - Neural network to produce ROC Curves

You could use sklearn.metrics.roc_curve.

Besides, Here is an example of what you want to do.

from sklearn.metrics import roc_curve, auc
fpr2, tpr2, threshold = roc_curve(y_test, clf.predict_proba(X_test)[:,1])


How to identify outliers from a small list of numbers?

An "outlier" is an observation that is so unexpected that we suspect it wasn't valid -- corrupted by noise or something. But what is unexpected? An observation that is highly improbable? But then how do we know what's probable?

Unless you're able or willing to make some assumptions about the distribution that generated these numbers, you can't really declare things outliers. For example, quartiles don't help, since it will only help you find the x% largest or smallest values. But every data set has these -- indeed every data set has a min and max, and being the min or max doesn't mean being an outlier.

In practice, I sense there's an assumption lurking here, that the numbers are probably normally distributed about some mean with roughly some standard deviation. If you know what the mean and/or stdev is supposed to be, you can use it directly to decide how unlikely an observation is and discard it as an outlier if it exceeds a threshold you choose.

You can take the mean and stdev of this sample as a surrogate for that, and for a large enough sample, the sample mean and stdev could be close enough to the real population mean and stdev to work. Here, with such small sets, the outliers influence stats like the mean so much that they throw off attempts to evaluate them in terms of the stats they influence. It's kind of circular.


Gridsearch XGBoost for ensemble. Do I include first-level prediction matrix of base learners in train set?

You should tune the meta-estimator using whatever data you want it to eventually predict with.  This should definitely include the base model predictions (else you aren't actually ensembling), and may or may not include (some of) the original features.

One important note though: you should not be training the meta-estimator using "predictions" of the base models on their own training data; those are more accurately called estimations rather than predictions, because the base models already had access to the truth.  A common approach is to train the meta-estimator on out-of-fold predictions from a cross-validation training of the base models.

If the base models are quite good, then it's reasonable that the xgboost model might only use one tree; it just has to tweak the already-good predictions from the base models.  But, consider dropping the learning rate or otherwise increasing regularization, to see if more trees can perform better.


RBF neural network python library/implementation

You can simply use gaussian as the nonlinearity and use it in Tensorflow. I guess it is the easiest solution.


In order to train some supervised learning algorithm to identify 'Problem' and 'Solution', you need to somehow generate some data that has labels of these things, which may be your best bet.  So you would need pieces of article text that you know are 'Problems' and 'Solutions'.  Then the trained algorithm could identify new articles based on the training set.  But I imagine this is a lot of work to generate if you don't have this training data to begin with.

You can also try some unsupervised method, for example you may split each article into multiple overlapping parts, and compute TFIDF vectors of each part.  Do this for many articles, then cluster the TFIDF vectors.  You can then look at your clusters and hope one of them is generally capturing the 'Problem', and another the 'Solution'.  It will probably be very messy, and you'll have to clean it up with some post-hoc heuristic, depending on what the result looks like.  I think this approach would be hard to actually get to work well, but it would avoid having to obtain (or create) a labeled data set.

Finally, if you look at articles you may be able to devise generally good heuristics (e.g. the solution might be the first paragraph in a section titled 'Discussion' 70% of the time) to bolster both of these approaches, and if the article patterns are repeatable enough you may be able to just use heuristics to do a generally decent job.  Since articles are often very similarly structured, this is where I would start to see how far I can get, and how much I can simplify the problem using heuristics before passing the simplified problem to the other methods.


Reducing search iteration over millions of data

You are on the right track. In an interview situation, this should be a good answer.  Another answer would be to pre-calculate list of 20 stores in ETL (or CRUD services or DB triggers) and store list of N closest stores in master-detail table.

I have implemented these two solutions in projects :

In database, create indices on lat and lon columns AND add a where clause with a bounding box.

SELECT distance_function(cur_lat,cur_lon,lat,lon) as distance

WHERE (lat between cur_lat -1 and cur_lat +1) AND (lon between cur_lon
      -1 and cur_lon +1)

(+Order by, top 20 rows clause depending on SQL dialect)

This solution cuts search space to a bounding box. This operation in log(n) in time complexity (due to B+ tree indices in DB)

For 100 million rows, results should be available in less than 1 ms.

Choice of +-1 four bounding box is arbitrary in this example. In reality, you will have to store this box with each row (1000s of stores in 1 mile in NYC, London etc; 1 store in 100 miles in other locations)

Implement QuadTree https://en.wikipedia.org/wiki/Quadtree . In this context it is similar to your solution but takes care of extreme cases where density of stores varies across geographical regions.


Clustering the data if some values in each vector is not applicable

First, are your categories independent of each other? If this is really four parallel problems, solve one at a time and it'll simplify things a bit.

Second, the heart of clustering is the distance function between any two observations. Because your data has missing values, you need to explicitly decide how to handle them. A common approach is to ignore the problem by throwing away any rows that have any null values, which won't work at all for your data.

A simple approach would be to ignore any NA columns, but keep the rest of the columns--for category C, the difference between rows 1 and 2 would be 4 (columns 11, 14, 15, and 16 are all off by one). There's an issue with this approach that I'll get to later, but let's implement it for now.

For each category, we need three rows of centroid positions (we might as well start off with randomly selected rows that have no missing values in them). We then calculate distance to each centroid using a bunch of summed ifs: if(matrix cell != NA, matrix cell - centroid cell, 0) across all the columns. (If you're doing this computation in a spreadsheet, I'd use intermediate cells to calculate the pieces instead of dealing with one huge formula.)

We then optimize the centroids with a standard iterative procedure: each row gets a group membership corresponding to the closest centroid, then each centroid becomes the mean of its group members (and it looks like your mean function automatically excludes NAs), then we repeat until satisfied.

Why did I specify that we had to seed with a row without NAs? If you allow NA measurements in the centroids, this will fail because a row of all NAs is as central as it gets! If our update step never lets us move to NAs, the NA column will be transient, but what happens when we take the mean of a missing column? (In your sample data, consider what happens if rows 4-12 are a group in the B category.)

So if we want to allow NAs in the centroids, we need to modify our distance function to something like the following:

If both cells are numbers, the distance is the difference
If both cells are NA, the distance is 2
If only one cell is NA, the distance is 3.

Those numbers are chosen such that for any full column of numbers, a numerical value in the centroid position will be closer, but if the column has sufficiently many NA rows, a NA value in the centroid will be closer. You might want to adjust the values so that the NA threshold looks like what you want it to.

In order to find the centroid of a set of rows using our distance function, we should use an optimization procedure, but we can cheat: if the number of NAs in the group is above the threshold, we set it to NA, and otherwise we set it to the mean.


To save a model's state, it is enough to save the model's parameters. If you are using Torch, then you can save it as follows torch.save(model.state_dict(), path_to_save).

When you want to resume training with a saved model, you would have to first create the model instance, and then you can use model.load_state_dict(torch.load(path_of_saved_model)) to update the new instance with the parameter values of the saved model.

You can find more reference here: https://pytorch.org/tutorials/beginner/saving_loading_models.html

There will be something similar for Tensorflow or any other framework that you use.

You can also have an evaluation function that evaluates how good your model is after a certain number of iterations, and save the one which performs the best.

Hope that helps!


Metrics to determine K in K-cross fold validation

The rule of thumb is the higher K, the better.

I think a better rule of thumb is: The larger your dataset, the less important is $k$.

However, it is useful to have a general understanding of the impact of $k$ on the performance estimator (leaving aside computational costs):

Increasing $k$ decreases the bias because the training set better represents the data
Increasing $k$ increases the variance of the estimator because the training data sets are becoming more similar

Also note that there is no unbiased estimator for the variance of the $k$-fold CV. Together this means that there is no metric that can tell you the best $k$ if you leave computational costs aside. Some empirical studies suggest that 10 is a reasonable default.

And to be clear, $k$ is not a hyper-parameter you want to tune to find the best accuracy. If you start performing $k_2$-fold CV to find the best $k_1$ something hopefully feels wrong.


Check out this blog post. There you find a summary of two fairly new papers from the field, which might start you off. One about semantic segmentation, the other about a classification problem.


Data science tools for easing the participation of a business into their scoring system

From a Machine Learning perspective, the answer would be to let the non tech users annotate a sample of products with scores and then train a model which predicts scores for all the products. Then the model could be refined and updated by regularly repeating this process, possibly using active learning in order to select the most useful instances and minimize the number of required manual annotations.
Many variants are possible: users could be asked to just say whether they think a score is too high/low, or compare two products and say which one should have a higher score, etc.


How does the naive Bayes classifier handle missing data in training?

In general, you have a choice when handling missing values hen training a naive Bayes classifier. You can either choose to either

Omit records with any missing values,
Omit only the missing attributes.

I'll use the example linked to above to demonstrate these two approaches.  Suppose we add one more training record to that example.

Outlook  Temperature  Humidity   Windy   Play
-------  -----------  --------   -----   ----
rainy    cool        normal    TRUE    no
rainy    mild        high      TRUE    no
sunny    hot         high      FALSE   no
sunny    hot         high      TRUE    no
sunny    mild        high      FALSE   no
overcast cool        normal    TRUE    yes
overcast hot         high      FALSE   yes
overcast hot         normal    FALSE   yes
overcast mild        high      TRUE    yes
rainy    cool        normal    FALSE   yes
rainy    mild        high      FALSE   yes
rainy    mild        normal    FALSE   yes
sunny    cool        normal    FALSE   yes
sunny    mild        normal    TRUE    yes
NA       hot         normal    FALSE   yes

If we decide to omit the last record due to the missing outlook value, we would have the exact same trained model as discussed in the link.
We could also choose to use all of the information available from this record.  We could choose to simply omit the attribute outlook from this record. This would yield the following updated table.

           Outlook            Temperature           Humidity
====================   =================   =================
          Yes    No            Yes   No            Yes    No
Sunny       2     3     Hot     3     2    High      3     4
Overcast    4     0     Mild    4     2    Normal    7     1
Rainy       3     2     Cool    3     1
          -----------         ---------            ----------
Sunny     2/9   3/5     Hot   3/10   2/5    High    3/10   4/5
Overcast  4/9   0/5     Mild  4/10   2/5    Normal  7/10   1/5
Rainy     3/9   2/5     Cool  3/10   1/5

            Windy        Play
=================    ========
      Yes     No     Yes   No
False 7      2       10     5
True  3      3
      ----------   ----------
False  7/10    2/5   10/15  5/15
True   3/10    3/5

Notice there are 15 observations for each attribute except Outlook, which has only 14. This is since that value was unavailable for the last record. All further development would continue as discussed in the linked article.

For example in the R package e1071 naiveBayes implementation has the option na.action which can be set to na.omit or na.pass.


Hierarchical clustering, with complete linkage will find clusters with a maximum pairwise distance i.e. diameter.

You need two parameters:

Distance function
Maximum distance = height where to cut the tree


Can I sum up feature vectors of a user‘s collection?

You can use total sum of boolean values. That will be fast and give a general notion of similarity.
A more useful metric might be Hamming distance, the sum of matching booleans between two vectors.


After plotting my predicted values against the true label values, I didnt quite get the answer I was looking for

The true quality is 4,5,6... the predicted quality deviates from the true one. This deviation is expressed by the variation of the dots on the x axis. The line in the plot shows, that there is positive correlation between your prediction and true states (which is good and expected).

Think about the magnitude of mae 0.5 in your case. Means „half a class wrong“. You likely have some concentration of predicted values around the true value (not good to spot in such a figure) and some outliers, viz relatively bad predictions, which are far away from the true class. You can also see that your prediction is bad for low and high classes.

I guess this is a linear regression? Try multinominal logit or the like. In you linear regression the prediction can be any rational number. What you would like to have as a prediction is a class, so 4,5,6 or whatever the class is called.


Does keras' model.fit() remember learning rate when called multiple times?

Provided that you are in the same scope, will remember not only the learning rate but the current state of all tensor, hyper parameters, gradients and so on.
In fact you can call fit many times instead of setting epochs and will work mostly the same.


Sklearn PCA with zero components example

Think of it this way: a PCA "transform" with $k$ components essentially approximates your $n$-dimensional data points by projecting them onto a $k$-dimensional linear subspace, trying not to loose too much data variance along the way.

More precisely, what you are doing is representing your original points $y \in R^n$ as:

$$y \approx \mu + Vx$$

where $x \in R^k$ are the lower-dimensional "new coordinates", $\mu \in R^n$ is the mean of your data and $V\in R^{n \times k}$ is the matrix of principal component vectors.

The "new coordinates" $x$ tell you how many steps you need to make along the $k$ principal components in order to reach the best possible linear approximation to $y$ by starting your trip from $\mu$.

Now, if $k=0$ the model becomes:

$$ x \approx \mu. $$

In other words, you are modeling all of your data as a single, fixed center point. Of course, you do not need to store any "new coordinates" here (because you do not need to move away from the mean), hence it does not make much sense as a "transform", but it is, none the less, a proper probabilistic model (a maximum-likelihood fit for a Gaussian distribution of errors, to be precise).

In particular, you can speak about the log-likelihood of data under this model (which is, up to an affine transform, equal to the sum of squared errors here, but is not as trivial as you might think in the general case) and we can compare various models, choosing the one with the best likelihood. This is exactly what is done in the example from the docs you mention in the question.


In Keras library, what is the meaning of "nb_steps_warmup" in the DQNAgent Object initialization?

Often times in reinforcement learning the error rate of the first few steps will be very large and may cause your parameters to oscillate. This is usually attributed to the lack of specificity of the deeper layers in your network. Thus we can come up with some schemes where the learning rate changes in a pre-determined way. For example we can use constant warm-up or gradual warm-up.

The convergence of stochastic gradient descent is a function of the learning rate and the batch size. When the batch size is increased too much then the needed increase in the learning rate can be such that it is beyond the possible curvature of the loss function. We thus introduce warm up as a means by which we can introduce large learning rates without the instability.


Yes, you can run a regression using the features you have, and predict the revenue but you will need more features to run an effective analysis. Maybe adding features for the genre and wether the music is trending or not. To turn it into a binary classification problem you will have to decide on a cut-off and label everything above that as 1 and below it as 0, the median might be a good cut-off but solely depends on your data. This entire problem will boil down to feature engineering, which in my view is one of the hardest thing to do. Also you will need a lot of data for this. You can create features wether the song was featuted on the billboard. See, try to think of it like a human and add such features to your model. Like as a person I think the music that will generate more revenue will be :

Popularity and Reputation of the artist.
Wether the genre is trending these days.
Has the song won any awards.

Stuff like this. So you wilk have to find features that quantify these things. Try to find some open source datatset for this kind of problem and work from there.


Random forest is a learning algorithm. It is an ensemble learning algorithm that uses decision trees as base learners. You wrote the steps for it correctly.

Rain forest is not a learning algorithm. It is an algorithm of constructing a decision tree (how to do splitting) when the dataset is so large that it does not fit the memory. In rain forest, the whole dataset is not required for making a splitting decision. Only some aggregated information (AVC-set for an attribute or AVC-group if you have more memory) is required.

If your dataset is large, and memory is small, you can use rain forest to build several different decision trees. Then use random forest algorithm with those trees as base learners.


What does RMSE points about performance of a model in machine learning?

There are multiple factors to consider, but the first thing to realize is that in regression, you don't want to think about whether an example is "correct" or "incorrect" but rather how close it was to the true target value.  Therefore you can ignore your original intuition about "80% of predictions are 'correct'."

Second remember that RMSE is in the same space as your target values.  So it is relative to the variance in your target values.  The benchmark of random guessing should get you an RMSE = standard_deviation.  So lower than this, your model is demonstrating some ability to learn; above that number, you haven't even learned to guess the mean correctly.

There isn't a cutoff for "my model is doing well" in RMSE space, just like with other metrics.  Everything is relative to a naive solution/benchmark or the state-of-the-art.


Best statistic books which have solutions

Some Statistical Books that I have used are:-

Introduction to Statistical Learning by Trevor Hastie and Rob Tibshirani - Good free book with R and Python examples

Elements of Statistical Learning

Statistical Computing with R


How to detect when the "bibliography" of a paper has began?

Here are a few heuristics that could help solve the problem without ML:

If all the papers are from the Arxiv, you can download the source tex files and regenerate the paper without the bibliography
You can use Arxiv-Vanity or their code to render the document into HTML (or some intermediate representation), then use that hierarchical structure to easily remove the part with a bibliography
Look for the word "references" followed by high word density of the words "in", "pages" and years in the 1900s and early 2000s

I guess what I am saying is "the input already has a fairly rigid structure, why don't you use that instead of statistics?"


Predicting probability of order arriving on or before customer Requested Delivery Date(RDD)

The most important thing here is to model your problem as time series, as you know, each example consists of one customer that has many orders over time. see Predicting Clinical Events via Recurrent Neural Networks


Can we select features by examining scatterplots?

Like one of the commenters, I also question the utility of these scatterplots in this situation.

What you've described is a standard multi-class classification problem. Your outcomes are labeled (clubs, spades, etc.), and you have seven features (f2-f8) with which to predict the outcomes.

So, simply try putting all of the features into a standard classification model (e.g., logistic regression), and see what comes out. If you believe that some features are unimportant, you can eliminate those features from the model, and examine the cross-validation score (accuracy, f1-score, etc.) to see if the feature was truly predictive or not. For a linear model, looking at the coefficients and their standard errors can also be useful in this regard. Another approach for feature selection for generalized linear models is to examine the deviance. Hill/Gelman write in their book (Data Analysis using Regression and Multilevel/Hierarchical Models) that "when an informative predictor is added to a model, we expect the deviance to decrease by more than 1. When $k$ predictions are added to a model, we expect the deviance to decrease by more than $k$".

What you should not do is to examine scatter plots to determine which features are important. The fundamental problem is that even though the data might be well-separated in seven dimensions (i.e., using all seven features), this separation might not show up well in a scatter plot since this plot is only two-dimensional.


You can see your affinity matrix as a weighted adjacency matrix of a graph and apply modularity-based community detection algorithms on that. Just note that modularity based algorithms have resolution problem i.e. finding very small communities is difficult in presence of large ones.


For multiclass classification problems there are multiple algorithms which are inherently built in a way to be able to solve them.
Some examples: kNN, naive bayes, decision trees...

For the performance to be accurate on all labels and for the classifier to show little bias, you can use other approaches: you can oversample minority classes or undersample majority classes, in a way that all the labels have the same number of points associated with them.

Here you can find some interesting answers about how to fight against class imbalances on decision tree classification: https://stats.stackexchange.com/questions/28029/training-a-decision-tree-against-unbalanced-data


Encoding Categorical Data Without Increasing the Dimension

I would strongly advise against doing anything like this: a features which is semantically discrete should be typed as such. There's nothing to be gained about casting categorical values into real numbers:

It obfuscates the meaning to a human analyzer
For categorical variables which are not ordinal (i.e. have no natural order), it introduces a serious bias for learning algorithms (note: most attributes in the dataset mentioned in the question are ordinal).
Real number binary representation can lead to approximations, thus possible errors when used as labels and compared for strict equality (something that a categorical variable should support)
It becomes impossible to fix errors in the data

Any conversion in the data should always rely on reasons which are specific to the problem/method to be used, not on some agnostic technical transformation. There's no universal recipe for encoding features (categorical or other), one has to understand what they represent and how an algorithm could use them in order to determine the best representation.


adding logic combinations of boolean features in classification

Try it and see what happens. Neural networks don't have enough representational power to learn an XOR operation without at least one hidden layer, so there are definitely some interesting features you can construct with logical operations. The AND operation is equivalent to multiplication, which corresponds to interaction terms linear models.

But yeah, it does depend on the model. For example, a decision tree can learn these kinds of features on its own (although it won't necessarily). For exame, an AND operation would correspond to two tests on the same branch.


Unbalanced class: class_weight for ML algorithms in Spark MLLib

Algorithms in MLLib are always used as baseline in production scenario , and they indeed can not handle some industrial problems , such as label imbalance . So if you want to use them , you have to balance your instances .

Besides , mechanism of BSP in Spark , you can simply see as data parallel ,  might be the main reason why Spark does not cover that problem . It might be hard for Spark to dispatch instances to all nodes in cluster , while the partial instances of each node share the same label distribution as the whole .

At last , you only have to weight the loss value for every minor labeled instance during your iteration process if you want to implement it .

Hopes this will help you , good luck -)


How to convert images (.jpg) to vectors for image classification

You can use imread function from imageio library with np.array from numpy:

from imageio import imread
import numpy as np
filename = '/path/to/dog.jpg'
vectorized_picture = np.array(imread(filename))

When entering this data into a ML engine, you'll probably need to reshape it to a column matrix (at least in Prof. Andrew Ng's Coursera Machine Learning courses that I've been doing). If so, you can do that by:

reshaped_vectorized_picture = vectorized_picture.reshape(1, -1).T

You need to install both libraries through apt, brew or pip (preferred one)

pip install numpy imageio


As for a complete machine learning package on GPU's, no such package exists.  However, there are actually a handful of R packages that can use GPU's.  You can see these packages on the CRAN High Performance Computing page.  You should note that most of these packages do require you to have a NVIDIA card.  Of the packages available, there are three packages you most likely would utilize unless you have a special case.

gputools - if interested in distance computations (only NVIDIA).
gmatrix - general numeric computations (only NVIDIA).
gpuR - general numeric computations (any GPU via OpenCL).*

* NOTE - At the risk of self promotion I am the author of the gpuR package.

You can likely use the latter two packages to reproduce existing machine learning algorithms.  I am actually using my gpuR package to create a GPU accelerated neuralnet package but this is in progress.

So in summary, if you are determined, the basic resources are available in R.  But if you need something in the immediate future, you will need to explore other resources/approaches as pointed out by @YCR.


data splitting into 3 sets based on years

Seems to me the best (or at least quickest) way to do this would be have all the data in a Pandas dataframe, then create masks based on year and create new dataframes for each group. Ex:

train_df = data[data['year'].isin(['2015', '2016', '2017'])
validate_df = data[data['year'] == '2018']
test_df = data[data['year'] == '2019']

Hope this is what you're looking for. If not, let me know and we can work out another solution.


Why Do Neural Networks Prefer Dense Features

A one-hot encoded feature will act like a lookup table, picking out from the first layer in the network a dense representation for that categorical value. The first layer transforms the one-hot into a dense representation via matrix multiplication.


How to use sklean pipeline to deal with data that read in line by line

Sklearn has some strategies on its website. Mainly, some of their estimators allow incremental learning, meaning that you can feed the data in streaming fashion during training.

To use it with a pipeline, I would refer you to this StackOverflow post. You essentially have to break down the pipeline during training, and then during inference, you can use it as is.


Which algorithm would be suitable for clustering a billion datapoints?

I don't know if there are other clustering methods which would work with this amount of data, but with K-means I would suggest this:

Run K-means with a varying number of instances picked randomly and study how much variation there is between the centroids depending on the data size (you can also study the variation across different random samples). I would expect the centroids to stabilize quite quickly with respect to data size: assuming the centroids become stable with data size N, there's no point running the full K-means process with more data.
Having obtained a model with N data points (the centroids), the model can be applied to all the remaining data points in order to find which cluster they belong to.

This is much more efficient than running the K-means process over the whole data since in the second stage the centroids are fixed, the algorithm doesn't have to iteratively update them.


Algorithm or formula to measure happiness?

You should be looking towards Natural Language Processing, specifically at Sentiment Analysis.

The link I provided is a good starting point for learning about sentiment analysis.  If this is what you are looking for, it is available as part of Stanford's Core NLP.


How to shuffle input data using stochastic gradient decent?

If I understand your question correctly, I would advise you not to spend time on this.

If you have data that has a structural order to it, for example time-series data where each sample follows from the previous, then shuffling would be throwing that information away.

If your data is not sequential, on the other hand, then computing all possible permutations and training the model on each of them is really just a waste of time. We shuffle the data e.g. to prevent a powerful model from trying to learn some sequence from the data, which doesn't exist.

Training a model on all permutations might be a way to uncover the correct order of the data, is you know it has an order, but it was already shuffled. Otherwise, this experiment would be as useful as training the same model $(n^2)!$ times, just using a different random seem each time.

Have a look here for some more points on shuffling data and why we might do it. Here is more discussion on shuffling input samples.


About what I have understood, you need to use time series forecasting algorithms. As your data is time series, you can look at these algorithms based on Deep Learning. You can also use classical algorithms for time series forecasting. You can use LSTM also which is good, details are given in link.
Look here


Layers' function in a convolutional neural networks

This is a basic property of convolutional networks. First layers identify simple features and as you go deeper in the CNN each layer uses the features of the previous layer to build more complex ones.

I would say the state-of-the-art article about this is Visualizing and understanding Convolutional Networks by Matthew D. Zeiler and Rob Fergus.
You can download it here


Total variance = within-class variance + between-class variance.

i.e. if you compute the total variance once, you can get the between class inertia simply by

between-class variance = total variance - within-class variance


Yes. You can add weights to the dataset's attributes.
To first add the weights:

Open the dataset in explorer
Perform any required filtering (if necessary)
Click the "Edit" button on the top panel
Right click on the respective attributes you wish to assign a weight to
Once you implemented your respective weights, click "OK"

You have now implemented weights to your attributes! However, one must keep in mind that these weights will only work for some boosting classification algorithms such as AdaBoostM1 and its variants (MultiBoostAB, etc.). Additionally, the sampling option must also be disabled and base classifier must also be a WeightedInstancesHandler. Otherwise, the algorithms likely won't use the weights correctly.


Sanity check: low PPV but high AUC scores?

AUC of ROC is not affected by PPV (precision), but by recall (true positive rate) and fall-out (false positive rate).

As per your results, model A gets a better PPV hence a lower recall and a lower ROC AUC. Inversely, model B gets a better recall hence a better ROC AUC and a lower PPV.

So this is entirely reasonable due to the definition of the two metrics.


May I use the same data in several time series intervals?

I dont see a problem in „reusing“ observations. Its just new/different info coming in. The question is more if the model will train/perform well with a short lookback or timelag. Why dont use a long(er) sequence of data to predict (eg) the movement in 3h steps. I wonder why you dont use all the data you have to train the LSTM and update the data stepwise. More data usually is better!


Binary text classification with TfidfVectorizer gives ValueError: setting an array element with a sequence

TfidfVectorizer returns a (sparse) 2-D array or a matrix. You can't set the column df['text'] to a matrix without messing up the dimensions. Instead, you need to concatenate the result from TfidfVectorizer with the remaining features in the dataframe.

df_text = pd.DataFrame(tfidf.fit_transform(df['text']).toarray())
X_train, X_test, y_train, y_test = train_test_split(pd.concat([df[['tid']],df_text],axis=1), df[['cat']])


context-based model and interaction terms

If you have gender data, use it. You're right that it will probably be a binary dummy feature (i.e., male = 0 or 1, and so on). You can use gender directly as a predictive feature, and/or interact it with other variables. You don't need to split your dataset by gender. Adding arbitrary information increases the noise in your set and decreases model quality. But you can learn whether gender is arbitrary or not! Add gender and compare to a model without gender to see if there's any change.


For sake of convenience, I'll go in ascending order of how the neural
network is working

The forward propagation of the LSTM cell is where the input is fed  ->  passed through the hidden states  ->  output is achieved (either at each time-step or at the end depending upon the type of problem
like classification, forecasting, etc.,)
Then after receiving the output the error between the actual value and the predicted value is calculated. This is where the loss
function comes into the picture
Classification problem - cross-entropy/log-likelihood
Regression Problem - Mean Squared Error, Mean Absolute Error functions are used.
So, after calculating the error only we backpropagate through time (BPTT) in case of RNN neural networks which updates weights of the
matrices and cell states of the LSTM cells. In practice, we refer to
these backpropagation algorithms as optimization algorithms like
Gradient Descent, etc.,
The optimization algorithms like RMSProp, Adam are much faster in practice than the standard gradient descent algorithm.

Overall, the training part of any neural network algorithm is that at
the early stages of training, it will perform very horribly and over
a period of time, with the correct loss function and backpropagation
algorithm it will try to reduce the chances of classifying error and
will converge fast depending upon the backpropagation algorithm used.


I wouldn't consider DTW to be outdated at all. In 2006 Xi et al. showed that

  [...] many algorithms have been proposed for the problem of time series
  classification. However, it is clear that one-nearest-neighbor with
  Dynamic Time Warping (DTW) distance is exceptionally difficult to
  beat.

The results of this paper are summarized in the book "Temporal Data Mining" by Theophano Mitsa as follows:

  In [Che05a], a static minimization–maximization approach yields a maximum error of 7.2%. With 1NN-DTW, the error is 0.33% with the same
  dataset as in the original article.
  In [Che05b], a multiscale histogram approach yields a maximum error of 6%. With 1NN-DTW, the error (on the same data set) is 0.33%.
  In [Ead05], a grammar-guided feature extraction algorithm yields a maximum error of 13.22%. With 1NN-DTW, the error was 9.09%.
  In [Hay05], time series are embedded in a lower dimensional space using a Laplacian eigenmap and DTW distances. The authors achieved an
  impressive 100% accuracy; however, the 1NN-DTW also achieved 100%
  accuracy.
  In [Kim04], Hidden Markov Models achieve 98% accuracy, while 1NN-DTW achieves 100% accuracy.
  In [Nan01], a multilayer perceptron neural network achieves the best performance of 1.9% error rate. On the same data set, 1NN-DTW’s rate
  was 0.33%.
  In [Rod00], first-order logic with boosting gives an error rate of 3.6%. On the same dataset, 1NN-DTW’s error rate was 0.33%.
  In [Rod04], a DTW-based decision tree gives an error rate of 4.9%. On the same dataset, 1NN-DTW gives 0.0% error. • In [Wu04], a
  super-kernel fusion set gives an error rate of 0.79%, while on the
  same data set, 1NN-DTW gives 0.33%.

Please see the original book for a list of the mentioned references.

An important thing to note here is the fact that Xi et al. even managed to beat the performance of an MLP back in 2006. Even though the situation might look a bit different these days (as we have better and faster Deep learning algorithms at hand), I would still consider DTW a valid option to look into when it comes to signal classifications.

Update

I would also like to add a link to a more recent paper called "The Great Time Series Classification Bake Off: An Experimental Evaluation of Recently Proposed Algorithms" from 2016. In this paper, the authors "have implemented 18 recently proposed algorithms in a common Java framework and compared them
against two standard benchmark classifiers (and each other)". The following quotes from the paper stress that DTW is (or at least was in 2016) indeed still relevant:

  Many of the algorithms are in fact no better than our two benchmark classifiers, 1-NN DTW and Rotation Forest.
  For those looking to build a predictive model for a new problem we would recommend starting with DTW, RandF and RotF as a basic sanity
  check and benchmark.
  Received wisdom is that DTW is hard to beat.


machine learning application in ticket pricing

This sounds a lot like a linear optimisation problem: given certain resources and constraints, maximise profit. If you are new to this, check out some of these resources:

walkthrough of an example
in depth, this book
practical example using Python #1 - and here #2

Machine learning is generally an approach to model and understand data, hopefully in such a way as to allow us to make predictions in the future, given some data that is representational of past observations. It more often than not considers a dynamic world and non-linear functions: given the state of the world, what should the ticket price be? Practically speaking, selling tickets doesn't work that way - it would be very confusing for customers if ticket prices always change! One more point that would concern me with your specific problem, is that your variables (ticket price, frequency of trains etc.) are not mutually exclusive from the number of tickets sold. If you just make each ticket cost $1000 dollars each, profit would be zero, because nobody would by a ticket. What I want to say is: we're not classifying cats and dogs here  :-)

With your problem and using machine learning, I would be more inclined to try answering questions such as:

How many passengers can we expect next Saturday?
Which are peak travel times in a certain region?
which factors should we improve to increase customer feedback scores?

Maybe you can try looking at one of these approached to help in the optimisation, i.e to help find your linear constraints.

One can of course attempt to frame it as a pure machine learning model, and there are many ways of doing this; it is optimisation, after all. I think the first step, whether using some machine learning algorithm or a linear optimisation construction, is to understand your data, the effects of each factor and their relationship to your target variable to be maximised/minimised (profit/loss, respectively).

So, as I was told back when learning about this myself: "The first step is to translate words into linear inequalities". So you could think about the factors you have and understand which ones effect price. If there are constraints/limitations, these should also be considered. For example, if you know that you are not allowed to sell more than 10,000 tickets on one day due to capacity constraints of your system.
There are some interesting points made in this thread over on Cross-Validated.

Now you understand the data a little better, you could think about which models to try out. Without knowing more about your data, I can't really offer more guidance as to which models might be worth trying, but hopefully this answer helps you along that path.


Dictionary based statistical NER learner

It sounds like you're describing using a gazetteer and training, which is not particularly unusual. See here for example:

  How do we go about identifying named entities? One option would be to look up each word in an appropriate list of names. For example, in the case of locations, we could use a gazetteer, or geographical dictionary, such as the Alexandria Gazetteer or the Getty Gazetteer.

Using a gazetteer has tradeoffs; see here for some discussion.

Here's an example of someone using it in CRF++ as a feature.


svm KKT condition violation that alpha choice in SMO

Equation $12$ of the paper states the $KKT$ condition:

\begin{align}
\alpha_i = 0 &\iff y_iu_i\geq1 \\
0<\alpha_i < C &\iff y_iu_i = 1 \\
\alpha_i = C &\iff y_iu_i \le 1
\end{align}

Hence if $\alpha_i < C$, we are suppose to have $y_iu_i \geq 1$.

To violate $y_iu_i \geq 1$ means $y_iu_i < 1$,

$$y_iu_i < 1 = y_i^2$$

is equivalent to

$$y_i(u_i-y_i ) < 0$$

Using the notation of the paper, it is $r_2=y_i E_i < 0$.

Numerically, we consider $y_iE_i$ to be negative if it is smaller than $-\epsilon$ for some small positive number $\epsilon$.


Question about "1x3 and 3x1 conv is equivalent to 3x3 conv"

A 3x3 convolution can do things a 1x3 followed by a 3x1 cannot do, for instance diagonal filtering.

As you said, it's all about the number of parameters, the first convolution has far more capabilities than the second solution.


How to build an overfitted network in order to increase performances

The reason of overfitting is generally because of the high(unnecessary) complexity model. So if you train a too complex(large) model, you will get a high training accuracy and low test set accuracy. Then you can start to fight with overfitting  with getting more data, dropout, regularization, early stopping, global average pooling, feature scale clipping or dropping some of layers from your network.


What does it mean to be “stable to deformation”?

Images are susceptible to deformations, i.e. afine or arbitrary deformations, such as "melting" effects.

In computer vision algorithms for non-constrained environments it may be desirable for a predictive model to be "stable", i.e. robust, ideally invariant, to arbitrary transformations that are common in that problem. E.g. a digit classifier would benefit from been stable to common paper deformations such as those caused by wrapping and unwrapping a long letter.

It is said that pooling layers insert certain stability to these deformations, this is analysed by Ruderman on his paper Learned Deformation Stability in Convolutional Neural Networks and in Pooling is neither necessary nor sufficient for
appropriate deformation stability in CNNs.

Some features, such as SIFT, BRISK and HoG try to deal with the most common deformations in image (Scale and Rotation). Most of convolution-dependent methods are already invariant to shifting as that is a feature of convolutional filtering itself.


Welcome to the community,

I do not know about other libraries, but gensim has a very good API to create word2vec models. In order to preprocess data, you have to decide first what things you are gonna keep in your vocab and whatnot. for ex:- Punctuations, numbers, alphanumeric words(ex - 42nd) etc.

In my knowledge, the most generic preprocessing pipeline is the following:-

1) Convert to lower
2) Remove punctuations/symbols/numbers (but it is your choice)
3) Normalize the words (lemmatize and stem the words)

Once this is done, now you can tokenize the sentence into uni/bi/tri-grams.

Have a look at this

The generic format to put data in gensim.models.word2vec()'s sentence parameter is : [[tokeneized sentence 1], [tokenized sentence 2].....and so on]

Hope it helps, thanks!!


xgboost: Is there a way to perform regression on rates/percentages data?

In principle: yes, you will have the same problem as with OLS. However, since xgboost is tree-based (and by that non-parametric), you may get relatively accurate estimates, meaning that values which are below zero or above one would be rare (at least the problem should be less severe than with OLS). In this case you could simply restrict results to $\hat{y} \in [0,1]$. An alternative would be to do a multiclass classification task where you have 100 classes $y=[1\%,2\%,...,100\%]$. Boosting usually performs well on classification tasks.

Just a hint: xgboost can be somewhat "heavy" in data handling. Boosting tools such as lightgbm are a good alternative. I prefer lightgbm over xgboost.


Deep Learning to estimate what is beyond the edge

I think the closest problem that has been addressed with deep learning is image inpainting, that is, filling a blacked out region in the image:

For instance, this paper: Semantic Image Inpainting with Perceptual and Contextual Losses.

So it is certainly possible to fill missing information from an image with deep learning.


Is it safe to use labels created from unsupervised model to train a supervised model using the same data?

Please take care of stratification while sampling data for training. You should mention stratify = y in train_test_split .

High ROC-AUC may be pointing to imbalance dataset - when-is-an-auc-score-misleadingly-high

Also check this out - k-fold-cross-validation-auc-score-vs-test-auc-score


How much text is enough to train a good embedding model?

It is not the number of many articles that matter but the total number of words.
Enough "meaningful/good" is an empirical question that depends on the dataset. One way to test the results of a newly trained model is the Google analogy test set which compares a new model's predicted word to established embedding benchmarks.
As far as minimum number examples needed for each unique token in vocabulary, the general consensus is there should be at least 40 examples per token. If there are fewer than 40 examples for a token, the vector estimates can be unstable and the token should be dropped from training.


ValueError: Error when checking input: expected conv2d_13_input to have shape (3, 150, 150) but got array with shape (150, 150, 3)

Change this:

model.add(Conv2D(32, (3, 3), input_shape=(3, 150, 150),padding='same'))

to this:

model.add(Conv2D(32, (3, 3), input_shape=(150, 150, 3),padding='same'))

And read the doc: https://keras.io/layers/convolutional/#conv2d

In particular the section about data_format:

  data_format: A string, one of "channels_last" or "channels_first". The ordering of the dimensions in the inputs.  "channels_last" corresponds to inputs with shape  (batch, height, width, channels) while "channels_first" corresponds to inputs with shape  (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be "channels_last".


Testing for gender composition between groups

For analysis of your clusters you can use the silhouette coefficient or silhouette width. These are available in cluster and factoextra package in R.

I will explain what basically in silhouette analysis:

The silhouette coefficient is calculated as follows:

1) For each observation i, it calculates the average dissimilarity between i and all the other points within the same cluster which i belongs. Let’s call this average dissimilarity “Di”.

2) Now we do the same dissimilarity calculation between i and all the other clusters and get the lowest value among them. That is, we find the dissimilarity between i and the cluster that is closest to i right after its own cluster. Let’s call that value “Ci”

3) The silhouette (Si) width is the difference between Ci and Di (Ci — Di) divided by the greatest of those two values (max(Di, Ci)).

Si = (Ci — Di) / max(Di, Ci)

So, the interpretation of the silhouette width is the following:

Si > 0 means that the observation is well clustered. The closest it is to 1, the best it is clustered.
Si < 0 means that the observation was placed in the wrong cluster.
Si = 0 means that the observation is between two clusters.

Basic code to calculate the above and visualise:

library(cluster)
library(factoextra)

sil <- silhouette(clustering$cluster, dist(input))
fviz_silhouette(sil)


No.
What you refer to (the difference between the means of group A and B) is actually the effect size, and it has absolutely nothing to do with the p-values.
The situation is nicely summarized in the (highly recommended) paper Using Effect Size—or Why the P Value Is Not Enough (emphasis mine):

Why Report Effect Sizes?
The effect size is the main finding of a quantitative study. While a
P value can inform the reader whether an effect exists, the P value will not reveal the size of the effect. In reporting and
interpreting studies, both the substantive significance (effect size)
and statistical significance (P value) are essential results to be
reported.
Why Isn't the P Value Enough?
Statistical significance is the probability that the observed
difference between two groups is due to chance. If the P value is
larger than the alpha level chosen (eg, .05), any observed difference
is assumed to be explained by sampling variability. With a
sufficiently large sample, a statistical test will almost always
demonstrate a significant difference, unless there is no effect
whatsoever, that is, when the effect size is exactly zero; yet very
small differences, even if significant, are often meaningless. Thus,
reporting only the significant P value for an analysis is not
adequate for readers to fully understand the results.

In other words, the p-value reflects our confidence that the effect indeed exists (and it's not due to chance), but it says absolutely nothing about its magnitude (size).
In fact, the practice of focusing on the p-values instead of the effect size has been the source of much controversy and the subject of fierce criticism lately; see the (again, highly recommended) book The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives.
The following threads at Cross Validated may also be useful:

P value as a measure of effect size?
Are effect sizes really superior to p-values?


Python TypeError: __init__() got an unexpected keyword argument 'decision_function_shape'

Your snippet is almost exactly the same as scikit-learn's example (except for kernel='rbf' and probability=False) and works fine under version 0.18, provided the needed imports are present.

Update: the version of scikit learn used is 0.16.1, and in that version, SVC did not have as many arguments as in 0.18, as per the docs. You should therefore use something like this:

model = SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0, degree=3,
            gamma=0.0, kernel='linear', max_iter=-1, probability=True,
            random_state=None, shrinking=True, tol=0.001, verbose=False)

For upgrading scikit-learn to 0.18, (assuming you use pip) do this:

pip install scikit-learn==0.18 --force-reinstall


What's the difference between probabilistic programming such as pyro and belief networks?

A probabilistic program and a Bayesian Network are both ways of specifying probabilistic models. Any model that can be specified as a Bayesian Network can also be specified by a probabilistic program, in fact by a probabilistic program that has no control flow. Roughly

Bayes Nets == Straight line Probabilistic Programs

For example consider the Bayes Net

This Bayes Net is equivalent to the probabilistic program (in Pyro)

def model():
    p_rain = pyro.param("p_rain", torch.tensor(0.2), constraint=unit_interval)
    p_sprinkler = pyro.param("p_sprinkler", torch.tensor([0.4, 0.01]),
                             constraint=unit_interval)
    p_wet = pyro.param("p_wet", torch.tensor([[0.0, 0.9], [0.8, 0.99]]),
                       constraint=unit_interval)

    rain = pyro.sample("rain", Bernoulli(p_rain))
    sprinkler = pyro.sample("sprinkler",
                            Bernoulli(p_sprinkler[rain.long()]))
    wet = pyro.sample("wet", Bernoulli(p_wet[rain.long(), sprinkler.long()]))

More generally, probabilistic programs can contain control flow (if, for, while) and recursion. Some of these extra features are expressed in extensions to Bayes nets, e.g. some for loops can be expressed as plates in Bayes Nets.


Classification algorithm with multiple output for a set of features

Your problem can be considered as multiclass classification problem. So, you have a dataset of features X and the predictor y. Where X contain Income, age, sex,etc. and y is an item that one customer will buy with higher probability.

To achieve your goal and predict the probability of a customer you can use any classifier from scikit-learn Library (if you use python) and call the method predict_proba after you fit the classifier, as shown in the following example:

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier

# Make a random multiclass classification problem
X, y = make_classification(n_samples=1000,
                           n_features=10,
                           n_informative=10,
                           n_redundant=0,
                           n_repeated=0,
                           n_classes=5,
                           random_state=0,
                           shuffle=False)

# Fit any classifier
rfc = RandomForestClassifier(n_jobs=-1,max_features= 'sqrt' ,n_estimators=50, oob_score = True)
rfc.fit(X,y)

# Prediction for one example
one_example = X[10,:].reshape(1,-1)
outcome     = rfc.predict_proba(one_example)

The outcome of this problem with 5 classes is

array([[0.76, 0.12, 0.02, 0.06, 0.04]])

Which represent the probability of each class for this example (or customer in your case)


You have not specified that what neural network you are using but as comments, you should try to fit your data first. You have to try to find a model that learns your training data. For this purpose you don't have to increase the number of data, at least not at this stage. You should try to find a good model which suits your data. For this purpose you have to change the hyper-parameters of your neural network, e.g. number of layers or number of neurons in layers. You can take a look at here and here which the former can help you and the latter helps you understand the features learned by CNNs in case you are using them.

For using F1 score in Keras I've not seen but you can implement it and pass it to compile method, take a look at here.


What conclusion can I get when the variable is influenced by other but there isn't any correlation?

First of all as your target value is continuous so it's a regression problem.
If the correlation doesn't give desired value it means there exist no linear relationship between variable and target variable you should look into mutual information.
Also, while interpreting a relationship, one should be careful to not confound correlation and causality, because although a correlation demonstrates that a relationship exists between two variables, it does not automatically imply that one causes the other.
So if you increase the values to get positive correlation of that variable with target it might not necessarily increase the same as there might exist a confounder.
Mutual information versus correlation


General programs/libraries for studying user search behavior?

I know of few though they do not model user behavior, they just try to present best search results and compare to user ideal. lemurproject.org has books based on it though it is old and in java. For example the user activity recorder works with firefox pre big numbers versioning. That is something like 10 years ago. Maybe has a modeler but almost certainly out of date. Apache Lucene is simpler though made for bigger data sets and regularly updated. Terrier IR platform is similar.


Do some measurements to identify your bottlenecks.

Here, I suggest to not use Silhouette. Because it is much slower than k-means. Silhouette need O(n²) distance computations every run!

The obvious was to speed this up would this be to compute and store a distance matrix just once. This won't help k-means, but it will make multiple Silhouette runs a little bit faster (it will still be O(n²), but only array lookups instead of distance computations now). Nevertheless, this will not be scalable to large data.

The better approach is to use a cheaper heuristic to guess the "best" value of k. None such measure is perfect anyway.


implementing forward and backward of a Linear model

This is because the derivative wrt $b$ is $1$: $\frac{\partial E}{\partial b} = 1$
dout is the derivative of loss function wrt prediction. Using chain rule,
$$
\frac{dE}{dw} = \frac{dE}{dy}\frac{dy}{ds}\frac{ds}{dw}
$$
The last term is the vector of input features $x$. In your case dout is the combination of the first two terms. For example, for MSE loss and sigmoid activation  dout $= (y-L)y(1-y)$
This is often used in optimizers for momentum calculation
For MLPs, you need to compute gradients for coarse layers using gradients of deep layers. For example, for MLP with one hidden layer with features $\mathbf{z}$ (hence 3 in total) vector of gradients wrt weights in the input layer $\mathbf{w}^0$ would be
$$
y= \sigma(\sum_kw^1_k \cdot\sigma(\sum_jw^0_jx_j))\\
\frac{\partial E}{\partial \mathbf{w^0}} = \frac{\partial L}{\partial y} \frac{\partial y}{\partial s} \frac{\partial s}{\partial \mathbf{z}}\frac{\partial \mathbf{z}}{\partial \mathbf{w}^0}  = \frac{\partial E}{\partial \mathbf{z}}\frac{\partial \mathbf{z}}{\partial \mathbf{w}^{0}}\\
\frac{\partial E}{\partial \mathbf{w^0}} = (y-L) y(1-y) \sum_j\frac{\partial s}{\partial z_j}\frac{\partial z_j}{\partial \mathbf{w^0}} = (y-L) y(1-y) \sum_j\frac{\partial s}{\partial z_j}\frac{\partial z_j}{\partial s_j}\sum_i \frac{\partial s_j}{\partial w_{ij}}\\
\frac{\partial E}{\partial \mathbf{z}} = (y-L)y(1-y)\frac{\partial s}{\partial \mathbf{z}} = (y-L)y(1-y)\mathbf{w}^1
$$
So, in other words, in order to compute gradients for weights in the input layer, you need gradients wrt neurons in the hidden layer


You can try converting your word representation into a document representation by simply taking the average over the word vectors for a document. For example, if a document has 9 words with (9, 200) dimensions, by taking an average over the words you can have a document representation with a dimension (1,200).
After you have your document-representation, you can use T-SNE to find similar documents. The documents with a similar topic or theme will cluster near each other. You can always improve the document representation by improving your word vectors.
Check this