PCA, covariance, eigenvector matrix and rotation

Having 2 uncorrelated normal variables $x_1 \sim N(0, 1)$ and $x_2 \sim N(0, 1)$, one can correlate them through a linear transformation:
$$A = \begin{bmatrix}a & b \\ c & d\end{bmatrix}$$
$$X' = AX$$
The covariance matrix of the transformed correlated variables $X'$ is given by:
$$\Sigma' = A A^T$$
For a rotation by an angle $\theta$, and scaling for adding individual standard deviations, the transform is  (sign is same as that given):
$$A =  \begin{bmatrix}cos(\theta) & sin(\theta) \\ -sin(\theta) & cos(\theta)\end{bmatrix} \begin{bmatrix}\sigma_1 & 0 \\ 0 & \sigma_2\end{bmatrix}$$
This produces the covariance matrix $\Sigma'$ as:
$$\Sigma' = \begin{bmatrix}\sigma_1cos(\theta) & \sigma_2sin(\theta) \\ -\sigma_1sin(\theta) & \sigma_2cos(\theta)\end{bmatrix} \begin{bmatrix}\sigma_1cos(\theta) & -\sigma_1sin(\theta) \\ \sigma_2sin(\theta) & \sigma_2cos(\theta)\end{bmatrix}$$
The cross-variance component is given by:
$$\Sigma'_{12} = \sigma_1\sigma_2(cos^2(\theta)-sin^2(\theta))+(\sigma_2^2-\sigma_1^2)cos(\theta)sin(\theta)$$
This is the relation of cross-variance on angle $\theta$. Setting $\theta = 45^\circ$ we get:
$$\Sigma'_{12} = \frac{\sigma_2^2-\sigma_1^2}{2}$$
Setting $\sigma_1 = 0.333$ and $\sigma_2 = 1$ then $\Sigma'_{12} = 0.445$
Note $n = \frac{1}{tan(\theta)} \ne \Sigma'_{12}$ does not represent the covariance component as shown by analysis above.
PCA decomposes the covariance matrix into un-correlated components. In other words it undoes the rotation (the scaling remains), since the rotation correlated the components in the first place.
Explicitly PCA decomposes the covariance matrix into:
$$\Sigma' = U S U^T$$
where $U$ is an orthogonal matrix of eigen-vectors and $S$ is a diagonal matrix which represents individual variances.
Going back to $\Sigma' = A A^T$ and representing the rotation by $R$ we have:
$$\Sigma' = R \begin{bmatrix}\sigma_1^2 & 0 \\ 0 & \sigma_2^2\end{bmatrix} R^T$$
Thus the orthogonal matrix of eigen-vectors $U$ corresponds to the (orthogonal) rotation matrix $R$.
Thus the eigen-vectors should (precisely in theory, approximately in practise) correspond to the rotation matrix (up to some sign factor and permutation which remains arbitrary).
For further info see:

Bivariate normal distribution
Understanding the Covariance Matrix
Interesting Properties of the Covariance Matrix


How to know the probability of correctness of a test data in a Binary Classifier

Accuracy is a statistic that you can compute on a dataset if you know the true labels. For a single image, the accuracy is either 0% or 100% based on if you get it right or wrong.

In the newer versions of Keras, the predict method returns the probabilities of the classes, what you want to print is (if I guess correctly), the probability score for the best scoring class.

Not that this is not an unbiased estimate of the class probability. Neural classifiers tend to very confident even if their prediction is incorrect.


How to get the probability/closeness of a sample belonging to a specific cluster?

Try Gaussian Mixture Model (GMM) as it is similar to KMeans, but differs in a few ways. In a nutshell think of KMeans as a hard clustering model where each sample is assigned to only one cluster, whereas GMM is a soft clustering technique that calculates the density (probability) of each Gaussian mixture (which can be considered as clusters) containing the data point in question. You can get both labels and probability scores from the model. Try it and see if it helps in your case. It is available from the SciKit Learn library.
Another approach in case if you have to stick to KMeans could be:

Take the cluster centers from the KMeans model.
Take your test sample vector and pass these as parameters to the softmax function to get the probability score for all the cluster centers per sample.


Vector space model cosine tf-idf for finding similar documents

Unfortunately, the math simplifies to show that you can't rigorously justify restricting the cosine similarity comparison of the vectors based on their lengths.

The key point is that the cosine similarity metric normalizes based on length, so that only the unit vectors are considered.  I know this isn't necessarily the answer that you wanted, but the math clearly shows that the cosine similarity metrics is agnostic to vector length.

Lets look at the math in more detail:

You are applying a cosine similarity metric and requiring that that metric be larger than 0.6:

$$similarity=\cos{(\theta)}=\frac{\mathbf{A}\cdot\mathbf{B}}{||A|| ||B||}\geq0.6$$.

But the scalar lengths on the bottom can be distributed into the cross products above (distributive property):

$$\frac{\mathbf{A}\cdot\mathbf{B}}{||A|| ||B||} = \frac{\mathbf{A}}{||A||}\cdot\frac{\mathbf{B}}{||B||}=\hat{\mathbf{A}}\cdot\hat{\mathbf{B}}$$.

Now $\hat{\mathbf{A}}$ and $\hat{\mathbf{B}}$ are vectors that point in the same direction as $\mathbf{A}$ and $\mathbf{B}$ but they have been normalized to length one.  So the definition of the cosine similarity metric is the take the original vectors, normalize them to length one, and then measure the dot product of the unit vectors.

Therefor:

$$similarity=\cos{(\theta)}=\frac{\mathbf{d1}\cdot\mathbf{d2}}{||d1|| ||d2||}=\hat{\mathbf{d1}}\cdot\hat{\mathbf{d2}}\geq0.6$$

depends only on the orientation of the vectors and not on their magnitude (i.e. length).

Reconciling this with what you are doing:

Despite what the results of the linear algebra show, you might still be seeing a statistically significant result.  Practically speaking you may be finding that the statistics show that length restrictions are valid for your data.  For instance, you might be finding that tweets never share a cosine similarity $\geq0.6$ when compared with Tolstoy's "War and Peace".  If your statistics look good for using $||d2|| > .8 ||d1||$ and $||d2|| < ||d1|| / .8$ then I suggest you go with it as these type of canopy restrictions are very useful in saving computing time.

You can perhaps reconcile what you have been doing with distance metrics by also considering the Euclidean distance.  Where as the cosine similarity only returns a value between -1 and 1 based on the angle between the two vectors, the Euclidean distances will return values that depend on the lengths of the two vectors.  In some sense, you are combining aspects of the Euclidean distance with cosine similarity.

It makes fairly good sense to require that the relative lengths be within 25% of one another in the sense that this combines an aspect of Euclidean distance to create group-by canopies, which cuts computation time, then length agnostic cosine similarity can be used as the final determinant.

Note that 1/.8 =1.25, so d2>=.8d1 is a tighter restriction than d2<=d1/.8.  I suggest using d2>=.75d1 and d2<=1.25d1 as this is symmetric.

Hope this helps!


Comparing AUC, logloss and accuracy scores between models

I will start with your last question as the answer explains the other questions. In general, this is a hard question. That is exactly why there are different metrics available. Depending on your problem and what you want to achieve, you should choose the metric that measures this best.

The differences in metrics can be found all over the internet. Others have explained it much better than I can.


GPU vs TPU for convolutional neural networks (NLP)

I don't think it makes a lot of sense to compare a generic TPU to a generic GPU. There is probably a factor of 10 or greater between a low-end GPU and the best GPUs on the market in terms of compute capability. Google has itself developed 3 generations of TPUs, each more powerful than the last.
I haven't kept up with the latest developments, but last I was aware, TPUs don't necessarily dominate over GPUs in all aspects. They may provide more compute per watt, but that's not the same as compute per unit or compute per dollar.
More specific to the architecture details, google has it's own documentation on exactly what sorts of computation TPUs are good at, versus what is more suitable for a GPU.


Forecasting time series outside the training/test set

Let's say you trained a forecasting model using the following base data:

time       | index_1 | index_2 | label
01/01/2018 | 80      | 70      | 1
01/02/2018 | 60      | 30      | 0
01/03/2018 | 75      | 90      | 1

You used time, index_1 and index_2 to predict label. Then you would simply need a dataset like this to predict 01/04/2018:

time       | index_1 | index_2
01/04/2018 | 60      | 75

Using your model on this data set should predict the label-value.

Now in a time series this can be more complicated, let's say what you actually want to predict is the label-value of time X from the indicie values of time X-4 months. In this case your data to build the model should look like this:

time       | index_1_lag_4_months | index_2_lag_4_months | label
01/04/2018 | 80                   | 70                   | 1
01/05/2018 | 60                   | 30                   | 0
01/06/2018 | 75                   | 90                   | 1

This model would predict the label-value for 01/04/2018 based on the indice values of 01/01/2018. To actually get a prediction we again give a data set like this:

time       | index_1 | index_2
01/04/2018 | 60      | 75

Only the output would not be the label-value for 01/04/2018 but instead for 01/08/2018.


How to run XGBoostregressor using reg: tweedie as objective?

As I understood, it is a versioning issue. However, it should be noted that before installing the latest version, the old version should be manually removed. Then, try to install the new version.


Kohen Kappa Coefficient of Naive Bayes with 62% overall accuracy is better than Logistic Regression with 98% accuracy?

Logistic Regression is only predicting one class (in this case the negative class)! Because of the high imbalance in the data, this model gives a high accuracy score. This metric, however, isn't reliable for imbalanced datasets. A more proper metric like Cohen's Kappa penalizes this behavior.

Naive Bayes, on the other hand, tries to predict both classes. It misses a lot more predictions this way, but it's Kappa is higher.


How to get an aggregate confusion matrix from n different classifications

There are a few ways to achieve your "master confusion matrix".

Sum all the confusion matrices together:  Like you suggested, summing this results in a confusion matrix.  The problem with this is you can not interpret totals.
Average the entries.  This method is the same as number one, but you divide each entry by the number of trials (~400 in your case).  This would be my preferred method because then you can actually translate each category to a (mean) +- (an error measurement) and actually see which categories are the most volatile or stable.  Careful with interpreting this 'error measurement' though.
Report a problem specific measurement of the confusion numbers.  For example, if your numbers have outliers, medians would preferred over means.

There are other statistics that are possible to report as well.  You can redo the method to keep track of individual classifications.  Then we can say other important stats like '% of classifications that stay the same and are accurate', etc...


Is the prediction algorithm absolutely the same for all linear classifiers?

Is it true that linear classifiers differ only in the Learning
  algorithm, but do they do the same during Prediction y = w1*x1 + w2*x2
  + ... + c?

Yes, all parametric linear classifiers try to predict the weights for the same equation $y = w1*x1 + w2*x2 + ... + c$ and differ in how they try to optimize the cost function.

  If I used one method for Training (for example SVM with linear kernel
  function), then can I use other method for prediction (for example
  Perceptron) with the same output result?

You will not be able to achieve this using standard libraries. But, if you code your models from scratch and extract weights from one model after training and use those weights to make a prediction with a different model, you'll get a sufficiently close answer. This is due to the fact that once you have learned the weights $(w1, w2,...., wn)$, you know the nature of data and can predict the target variable using the equation $y = w1*x1 + w2*x2 + ... + c$. Note that, this will be true only in the case of parametric linear models as only then the above equation would be applicable.

  And can I do the same, if I used SVR (Support Vector Regression with
  linear kernel function) for Training, then can I use Perceptor as
  linear regression method for Prediction value?
  Well, yes again with the same restrictions as above. This is because you will get a decision boundary as the output which depends on the data. Note that, there will be minor changes in the output as the error of different models is different.

Also, any other differences in the output will be due to the difference in the learning algorithm and how it operates on the data. You can verify this by training two models on the same data and extracting the weights after training. If the weights are close enough (which means you have learned the correct underlying pattern), you can use any of the two models on test data to predict and you will get similar values.


Regression vs Classification to rank by profit

This actually isn't too surprising given the context in my opinion. Classifying an observation as "1" = spent money in past and "0" = did not spend money in past causes you to lose information that in your case is important; that is, the amount of money that each person gave.

Imagine these two scenarios, for example. In one scenario, you send your letter to ten people and all ten people send back 100 dollars each each. Using a classification scheme, these ten people are all "1"'s. In the second scenario, you send your letter to ten people again, but now only one of ten respond. However, this one person sends back a cheque of 10,000 dollars. From a pure profit point of view, clearly the 2nd option is superior.

Ultimately, both your classification and regression models answer two separate questions. Your classification model simply predicts how likely someone is to spend any amount of money, (by how you have phrased the question, anyway) whether that is 10 dollars or 10,000,000 dollars there exists no distinguishing between the two with a classification model (as you phrased it). The regression model predicts the exact amount that each user is expected to spend which if I am reading your question correctly, is what you care about.

If you want to make the problems closer in scope, perhaps try defining in your classification model as "1" if person spends over some profitable amount of money, and "0" if they don't. While I don't agree very much with doing this (you lose information this way by "bucketing" a continuous variable, and also, why make the problem harder?) you might get "closer" results.


Apparently this functionality is left out intentionally, see here. I'm afraid you have to use SVD, but that should be fairly straightforward:

def pca(X):
    mean = X.mean(axis=0)
    center = X - mean
    _, stds, pcs = np.linalg.svd(center/np.sqrt(X.shape[0]))

    return stds**2, pcs


How and When features are attached to target label

I assume you're training a CRF model, or some other variant of sequence labeling, right?

I'm not especially familiar with Mallet but it's very unlikely that any model would let you add features at testing stage: the trained model and the test data must have exactly the same features, no more no less.

Apparently the features you want to add consist of a similarity score with a few particular categories, I assume this score is calculated independently during pre-processing? If yes this would usually be represented like this in the data:

<token>   <PLACE_LIKE> <CITY_LIKE> <COUNTRY_LIKE> <label>
KFC       1.0          0.0         0.0            PLACE
Bangalore 0.0          1.0         0.0            CITY
INDIA     0.0          0.0         1.0            COUNTRY

Note that this format with token + N features must be followed in both the training and test data. Therefore you must apply the same pre-processing steps for both.


recommender system: how to compare different scores when calculated individually?

You can continue to use machine learning for this model-combination step too. You're describing a form of ensembling. How about:

set aside some purchases as your validation set V, and keep the remaining as training set TRAIN
for TRAIN, use k-fold cross-validation to create your separate models
use TRAIN again with k-fold cross-validation to ensemble the models together
report performance of the ensembled models on V

Ensemble techniques will take into account the predictive quality of each individual model.


Ignore unseen columns with OneHotEncoder

It is not possible. And this has nothing to do with OHE, if you choose any other encoding methods, still you are facing the same problem! Not having similar feature sets between train and test contradicts with fundamentals of developing ML models using train/test. This is already discussed in length in a stats.stackexchange question, or here concisely.

My two cents into the matter:

I would first recommend figuring our the nature of missing column in the test data. Do they miss sometimes or they are never present? If they do miss sometimes, how do they look like when they are present? If they are never present in the test, you have no choice than dropping them in the train set. However, if they are present every now and then in the test, and they contain similar content to the train, maybe imputing them is kind of possible, and having a NaN in test when not present in the test is a valid approach to try. Stil I would recommend doing so, as you may add artifacts (information) that may not represent the reality or even distort the reality so to say.
OHE: Please note that OHE is quite sensitive to its members. As it was mentioned in one of the answers and I am sure you are aware too, OHE create new column for each of its members and fill them with 0 or 1. For example, for your column a, you will have a_one, a_two, and a_three columns after OHE. And if even one of these members (one, two, three) are not present in column a in your test set, your pipeline raises an error. OHE is good for cases that you have always a known set of members for each features, otherwise you are better good choosing other categorical encoding methods like Hashing, Binary, or even recently introduced Entity Encoding.

Good luck!


When you hit the limits of an application like Apple Numbers, or Excel, you need to start using a programming language like Python, R or C. Using a programming language, you write your own application, which is not constrained by arbitrary limits like 220 rows by 214 columns. (Of course, you are still constrained by the physical memory and the way that a particular language addresses it: R and Python tend to be more resource-intensive than C).

Let me give you an example using Python. You can install Python for your system from the official site. If you install one of the latest versions, you will already get pip as an installer of further Python packages. Install Pandas by starting a terminal (on OS X, there is a 'Terminal' application, on Windows this used to be a MS-DOS window ...), and typing into that terminal:

pip install pandas

You can then start Python in an interactive session by typing python in a terminal. In the Python shell, you can read the data as mentioned in @tasos 's answer:

>>> import pandas as pd
>>> df = pd.read_csv('name_of_your_file.csv')

Here you can use the describe method of the df DataFrame to get some statistics about the read data. These numbers are for a randomly generated data frame:

>>> df.describe()
          0         1         2
count  3.000000  3.000000  3.000000
mean  -0.150869  0.444380 -0.117066
std    0.751421  0.697880  0.565328
min   -1.017424 -0.356764 -0.506030
25%   -0.386514  0.206466 -0.441313
50%    0.244396  0.769696 -0.376595
75%    0.282409  0.844952  0.077416
max    0.320422  0.920208  0.531427

Then you can generate plots for different columns, one against the other, using the Pandas plotting functions. Check the many examples in the linked documentation. Here is a small example from there:

import matplotlib.pyplot as plt
import pandas as pd
import bumpy as np

df = pd.DataFrame({'a': np.random.randn(1000) + 1, 'b': np.random.randn(1000),
                   'c': np.random.randn(1000) - 1}, columns=['a', 'b', 'c'])
plt.figure()
df.plot(kind='hist', alpha=0.5)

Usually, you will need to use another call more in order to save it to a file:

plt.save('output.png')  ## or *.svg, *.pdf

In the file you should find something similar to this figure.


Imbalanced dataset in MLP classifier in python

You can try using data re-sampling techniques. They can be divided in four categories: undersampling the majority class, oversampling the minority class, combining over and under sampling, and creating an ensemble of balanced datasets.

The above methods and more are implemented in the imbalanced-learn library in Python that interfaces with scikit-learn. See ipython notebook for an example.


Getting unexpected result while using CountVectorizer()

The problem lies in the line:

keys = 'keys_' + str(i+1)

Here, keys becomes a string variable while I guess you would expect it to take the value of the list you defined in the first lines..

Try instead with a dictionary:

my_keys= {
"keys_1" : ['funny', 'amusing', 'humorous', 'hilarious', 'jolly'],
"keys_2" : ['horror', 'fear', 'shock', 'panic', 'scream'],
"keys_3" : ['romantic', 'intimate', 'passionate', 'love', 'fond']
}
text = ('funny amusing fear passionate')

for i in range(3):
    keys = my_keys['keys_' + str(i+1)]
    cv = CountVectorizer(vocabulary = keys)
    data = cv.fit_transform([text]).toarray()
    print(data)


Is there any activation function that can make batch normalization obsolete?

What you describe sounds a lot like Scaled-Exponential Linear Units (SELUs), that are the core of Self-Normalizing Neural Networks, which where presented at NIPS 2017.

A short summary from here is that:

  If the mean and variance of the input is in certain range, then the mean and variance of the output should (1) also in that range and (2) converge to a fixed point after iteratively applying the activation function.

You might want to have a look at the reddit post comments. If you want to fully understand them, you can go ahead with the 90 page-long appendix of the arxiv preprint.

They got a lot of attention when they were presented, but I think they have not delivered up to the expectations, as no one seems to be talking about them lately on the internet.


Can we apply to GridSearchCV to Logistic regression .?

I believe that the first thing you should do is to convert your data to a numeric type.

Maybe this starter tutorial can help you
https://towardsdatascience.com/introduction-to-natural-language-processing-nlp-323cc007df3d


Pearson Correlation is a technique for investigating the relationship between two quantitative, continuous variables, for example, age and blood pressure. Pearson's correlation coefficient (r) is a measure of the strength of the association between the two variables, a number between -1 and 1. [reference]

This following scatter plots from SPSS tutorial explains different pearson correlations visually:

From your scatter plots, however, it looks like there is no linrar correlation between between sensor 6 vs. class label. I do not know why you expect a correlation??!! That is why using pandas.DataFrame.corr, which is by default based on Pearson correlation, should in principle should lead to a near-zero correlation. What strikes me though is that how this gives you high correlations for some sensory data namely sensor0, sensor4 and sensor8!! Maybe post part of your actual dataset rather just the statistics. I am curious why we see those correlations.

Two questions:

What are the data types of the class_label and sensory data columns? Having a wrong data types could lead to wrong correlation values in pandas. See this post
Shouldn't you be looking at the correlation between a continuous value vs. categorical varaibles? See this post as a good reference.


How can I create a new column of binary values from my TfidfVectorizer sparse matrix?

var = (0,833)
new_var = np.zeros(1000)
for i in var:
    new_var[i] = 1

I would need your sparse matrix datatype to give you a more precise answer.


Connection between cross entropy and likelihood for multi-class soft label classification

If I understand it correctly, you are asking how log likelihood in a multi-class classification problem relates to the cross entropy loss. So here is my try:

Assuming we have a multi class classification problem ($C$ different classes) where we estimated the conditional probabilities for each class given the data $x$ (e.g., using a neural network) and where the classes are one-hot encoded:
$$
\hat{p}_i^{(k)} \in [0, 1] \quad \text{denotes the estimated probability of the } i\text{th sample for the } k\text{th class}\\
p_i^{(k)}\in \{0, 1\}\quad \text{denotes the true probability of the } i\text{th sample for the } k\text{th class}\\
y_i^{(k)} = p_i^{(k)} = 1_{y_{i}^{(k)}==1} \quad \text{hard labels, i.e.,} \begin{cases} 1 & \text{if }i \text{th sample belongs to } k\text{th class},\\ 0 &\text{else.}\end{cases}
$$
Note that $\hat{p}_i^{(k)} = \hat{p}_i^{(k)} (\textbf{x}_i, \theta)$ is actually a function that depends on parameters $\theta$ (e.g., network weights) and the input data $\textbf{x}_i$. In the end, we want to optimize the parameters $\theta$. For the sake of simplicity, we just write $\hat{p}_i^{(k)}$ (silently knowing that these estimated probabilities come from some kind of function approximator).

The idea of maximum likelihood is to maximize the likelihood of our estimated conditional probabilities given the data, i.e.,
$$
\large{
\max P(Y|X) = \max \left( \prod_{i=1}^N P(y_i| x_i) \right) =\max \left( \prod_{i=1}^N \prod_{k=1}^C  \hspace{0.2cm} \left(\hat{p}_i^{(k)} \right)^{1_{y_i^{(k)} == 1}} \right)}
$$
Let's take a moment to digest this formula. It's actually pretty easy what we are doing here, we want that the estimated conditional probabilities $\hat{p}_i^{(k)} \in [0, 1]$ correspond to the true labels $p_i^{(k)}\in \{0, 1\}$. Note that the true labels are hard labels, e.g., $\textbf{p}_0 = \begin{bmatrix}0 &1&0 \end{bmatrix}^{\text{T}}$, but the predictions are (probably) soft labels, e.g., $\hat{\textbf{p}}_0 = \begin{bmatrix}0.2 &0.7&0.1 \end{bmatrix}^{\text{T}}$. We optimize our likelihood by maximizing the predicted probability of the true label, i.e., where the label $y_i$ corresponds to the class $k$, this is denoted by the indicator function $1_{y_i^{(k)} == 1}$

In the extreme case where our conditional probabilities always predict the true class with probability 1, this yields the maximum possible value for our likelihood:
$$
\max P(Y|X) = 1
$$

By definition of the hard labels, we can rewrite the formula of the likelihood into
$$
P(Y|X) = \prod_{i=1}^N \prod _{k=1}^C {\left(\hat{p}_i^{(k)}\right)}^{p_i^{(k)}}
$$

Now we take the negative log-likelihood (hence the maximization problem becomes a minimization problem):
$$
\max P(Y|X)  \equiv \min \left(- \log P(Y|X)\right) = \min \left(- \sum_{i=1}^N \sum_{k=1}^C p_i^{(k)} \cdot \log \left(\hat{p}_i^{(k)}\right) \right)
$$
(This formula is the same as in your question).

Let's suppose, there is only one datapoint $N=1$, then we get
$$
- \log P(Y|X) = - \sum_{k=1}^C p^{(k)} \cdot \log \left(\hat{p}^{(k)}\right) = H(p, \hat{p})
$$
This is known as the cross-entropy loss, i.e., minimization of the cross-entropy loss corresponds to maximum likelihood when hard labels are provided.


As such we don't have much ways to handle under-fitting issue.
Generally I follow following ways to handle it:

Add more features to training data. That includes deriving new features front the existing one.
Make a complex model.
Increase iterations.

Adding new training data only, I believe will not solve this problem.


Does MNIST generalise to european handwriting?

for sure this is too late for you but as i used mnist some days ago and could not get good results for my own handwriting... maybe someone else is in search for a solution:

  MNIST does represent american handwriting and will not achive good
  results in recognition of european writing.


Error using corpus widget in Orange v 3.25.0 text mining and no import document option?

I believe your Text version is outdated. Please update Text add-on. For issues with Orange, please always head over to the issue tracker and see if your issue already exists. If not, open a new issue.


Read news articles on schools for inspiration.

Student achievement compared by demographic
Student advancement (or lack therefore) across grade
Class popularity by enrollment
Average cost per student


Can i build an image classification model where each image has multiple labels?

One model will work in this case.
Lets put it this way, let's say we are trying to solve multi-class(lets say 5) classification problem in neural networks we will have 5 neurons in the final dense layer and ideally use softmax as activation function and categorical_crossentrophy as loss function.
Now coming back to multi-label classification lets take an example:
Your car1 has labels like Sedan,Blue,and Type(Good).
Your car2 has labels like SUV,Black, and type(Bad).
You will these many neurons in your final layer lets say here we have 6 different types of labels for 2 cars, and we do one hot encoding for the target variables.

The difference is we use sigmoid and binary_crossentrophy as activation and loss functions. This will give probabilities of the labels for which each image belong to.
Any questions, please post.


How do you calculate how dense or sparse a dataset is?

It's actually defined on the first page:

  ... sparsity level (ratio of observed to total ratings) ...

In other words, the fraction of the user/item rating matrix that is not empty. Remember that the problem is that most user-item pairs have no rating, and we wish to estimate them.

Example:

Let there be three users and four products. The number of possible ratings is $3\times4 = 12$. If every user rates only one product each (regardless of which product), the density is 3/12 = 25%.


A simple way to start with word-level:

Given a phrase credit cart, you'd like to correct this to credit card. Take all the correctly spelled words (assume for now that anything found in your dictionary is correctly spelled even if grammatically incorrect) and embed them into vectors using your favorite embedding, or better yet, fine-tuning that embedding on a corpus related to yours.

Next, for all the misspelled words, generate a small number of candidates based on edit distance. Then choose the winner from this set based on the distance from the mean of the embeddings from the correctly spelled set.

This certainly isn't optimal, but would be a framework to get started.

A way to start with word-level (sequence modeling):

Get your favorite word-level sequence model, fine tune it on corpus of your choosing.

Then, you process the document to be checked one word at time, and if word is misspelled, predict the next word based on the context before it. You should peel back the last layer to see the probabilities, then select the top N of those words most likely.

Then score the predicted next words based on edit distance to the actual next word, and choose that.

Final notes

For both these schemes you'd certainly have to do some engineering work and may need to weight by relevance or likelihood of the word, and so on.

Finally, I suspect the truly optimal way to do this is just to have a giant corpus of spelling mistakes & subsequent corrections (something a MS or a Google would have from their document software usage databases). With that much data, you might just be able to look up the most common misspelling and replace it, no ML needed. Or at very least narrow the candidate set of "correct" replacements for another simple scoring technique.

Otherwise, you could try with a character-level model to do something similar to my second example and move forward one character at a time, and for those misspelled words, generating a distribution of probabilities per character. Based on this you could cleverly try to figure out the best "path" that moves through your likely character distributions and again score the resultant candidates with some type of embedding distance to the words in the vicinity. Thus getting you the most likely character combination to replace the misspelled word with. It's not a great solution since you're not taking into account the words after, but perhaps with a bidirectional model you might be able to do this.

Anyway, hope this long ramble was somewhat helpful! Remember, a lot of deep learning for real use cases is just engineering!


Converting ML Azure random forest to Python - what is "Number of random splits per node"?

From the Notes section in your link above: "The features are always randomly permuted at each split."

So using the max_features parameter should be analogous to "number of random splits per node".


AUC ROC Threshold Setting in heavy imbalance

-->ROC curves should be used when there are roughly equal numbers of observations for each class.

-->Precision-Recall curves should be used when there is a moderate to large class imbalance.

enter code here
 from sklearn.metrics import precision_recall_curve
 precision, recall, thresholds = precision_recall_curve(y_test, probs)
 auc = auc(recall, precision)


is it ok to get 100% accuracy in random forest classifier algorithm?

There could be many reasons why you achieved 100% accuracy.One of them could be:Duplicates in your data which are repetitive in both train and test data.I would suggest you to try the following steps:
1.Check if there are any duplicates in the original data
2. Try a different Train-Test split like 80-20
3.Try k-Fold cross validation
4.Checkout for Precision and Recall apart from accuracy.
Hope this helps!


You may want to read about the task of Relation Extraction. The most common scenario is to extract triplets (entity1, relation, entity2). Usually entity1 and entity2 are given and the list of possible relations is known. Common methods use a classifier, others can be rule-based.

With your example from the sentence "Bob is married to Lisa since 1983." the system can extract (Bob, is_married, Lisa)

A variant is Open Information Extraction which tries to extract relations but with no information specified.

There exist multiple projects on the subject such as DeepDive, IEPY or openie-standalone.

This this presentation or this review give an overview of the task.


There is no fundamental difference between using RNNs for text vs. for numeric values. In fact, text is more difficult because you have to preprocess the text to convert into numeric values first.  In any case you might have a look at this tutorial:

https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/

I find this site to be pretty useful in general for examples of working with Keras.


is it better to correlate and encode or encode and correlate?

Correlation is for continuous variables, example Pearson co-relation. Using Pearson correlation for categorical or ordinal variables are not recommended. If you are encoding the data, i can imply that it is a categorical variable.
For categorical variables you can find association between the variables (instead of correlation) using Chi-squared test of independence. This test is quite lenient , in the sense, even for a weak association it will show a low p-value. To further validate the result of Chi-squared test of independence , you can calculate Crammers V-test which will give a specific value between 0 and 1 . The analysis gets easy with Crammers V-test. Crammers V-test uses Chi distribution.
Whether you calculate association before or after the encoding (for categorical variable) it will not make any difference because it works on the frequency table rather than the actual values of the column


Passing tuples (key, value) into parameterized SQL query in Python

Generally speaking, you need a string to be executed that contains placeholders to accept the values contained within the tuple. Also, what you are calling your "tuple" is actually a Python dictionary, but that isn't a problem.

From the sqlite3 documentation, we can see there are two ways to correctly use the execute() method:

an example command is given as:

# This is the qmark style:
cur.execute("insert into people values (?, ?)", (who, age))

# And this is the named style:
cur.execute("select * from people where name_last=:who and age=:age", {"who": who, "age": age})

And the second example, the named style is exactly what would work with your situation. You can try the following, using the keys of the dictionary as the named placeholders:

cmd = """insert into YOUR_TABLE values Marke=:Marke, Kilometerstand=:Kilometerstand, Erstzulassung=:Erstzulassung, Kraftstoffart=:Kraftstoffart, "Leistung (PS)"=:"Leistung (PS)", Getriebe=:Getriebe"""
cur.execute(cmd, your_dictionary)

So each of the keys of your dictionary matches the name of the corresponding column in the target table.

Notice that I put the entire cmd in triple " to make a string literal that can itself contain the " character. I think it will be required, because you have a column name in your table that contains a space: Leistung (PS).


Is it possible to change pandas column data type within a sklearn pipeline?

You sure can.

One solution off the bat is to extend your estimator that
 takes object type variables. So what does that mean.

Library that you said are all estimators in the sklearn form fit, predict methodology. So all you have to do is something as follows:

> class modifiedTraf(oldTraf):
>        def __init__(self):
>           super(modifiedTraf, self).__init__():
>
> bla bla
>
>       def fit(self, X):
>           X[col] = X[col].astype(object)
>           super(modifiedTraf,self).fit(X)
>           return self
>       def transform(self, X):
>           X[col] = X[col].astype(object)
>           X=super(modifiedTraf,self).transform(X)
>           return X

Learn more about overriding sklearn classes and super argument for example here


How to transform raw data to fixed-frequency time series?


Why are Chunking and IOB tags necessary?

BIO(L) tagging is important (but as you correctly noted, not necessary) part of a NER pipeline. Main idea behind such split is to facilitate learning in following manner.

Take English as an example, some words will (most likely) never end a Named Entity, like adjectives, so the model will never tag them as the L(ast) part of a named entity. The same applies to the L-tag.

What is crucial, is that many models, like Conditional Random Fields, learn not only tags themselves, but also the transition probability, so, if you will get some chunk of text that is tagged as B_ O_ L_, that sequence is incorrect, but when you learn the transitions as well, model will figure out, that if you get a strong Beginning and End, the inside part should also be a entity part.


Can Shapley/Lime values be used for unsupervised learning?

I don't think so, not directly. SHAP is trying to explain each feature's effect on the prediction, but you have no label here. It might be better to ask therefore, what are you trying to explain?
In the case of an isolation forest, you can find the short path through the trees to any anomaly. That path tells you why the trees separated it, based on what features. It may not be super interpretable, but can be read off directly.


The main aim of a regularization is to reduce the parameters values which in turn will reduce the effect of the feature associated with that parameter on the models prediction. For example, Since the intercept Theta0 basically doesn't have a feature associated with it, We don't penalize it.


Getting low accuracy on keras pretrained word embeddings example

The code has been changed to remove headers.  See comment on github:

"Newsgroups message contains header like 'Newsgroups: alt.atheism', which inflates the accuracy to 0.95 (2 epochs).
After removing the header, the val accuracy is 0.47 (2 epochs) and 0.71 (10 epochs)."

https://github.com/fchollet/keras/pull/5585

This confused me for days!


Training of Region Proposal Network (RPN)

The first answer in your commented link answers one point about how region proposals are selected. It is the Intersection Over Union (more formally the Jaccard Index) metric. So how much of your anchor overlaps the label. There is usually a lower limit set for this metric to then filter out all the useless proposals, and the remaining matches can be sorted, choosing the best.

I recommend reading through this excellently explained version of a proposal network - Mask-R-CNN (Masked Region-based CNN).
If you prefer looking at code, there is the full repo here, implemented in Keras/Tensorflow (there is also a PyTorch implementation linked somewhere).

There is even an explanatory Jupyter notebook, which might help make things click for you.


How to transform specific type feature to yield better prediction?

ML has sometimes problems to find relations obvious for people, so I would help a little bit by creating a new feature:

relX = (maxX-minX)/minX

because some diseases are more correlated with jumps of some parameters than with their level, however, such relX may be too much correlated with target and may cause overfitting.
Usually, after creating such a feature, it is better to remove one of the original features, sometimes both.


Reliable way to verify Pyspark data frame column type

If you don't have business knowledge, there is no way you can tell the correct type, and no way you can 'confirm' it. You can at most make assumptions about your dataset and your dataset only, and you for sure have to inspect every value.

In your example, you created a new column label that is a conversion of column id to double. You could count all rows that are null in label but not null in id. If this count is zero you can assume that for this dataset you can work with id as a double. That doesn't necessarily mean that in a new dataset the same will be true for column id.


Problem with calculating error rate for KNN

K-NN is a measure of distance, thus the result of your equation will depend on the scale of your data. If the ratings are in a scale from 0 to 100. Then if you always predict very poorly you are evidently going to have values much larger than 1.

For example, for a very bad predictor

import numpy as np

Y = [100, 90, 100, 90]
Y_p = [10, 10, 10, 10]

np.sqrt(np.sum(np.subtract(Y, Y_p)**2)/len(Y))

  85.14693182963201


Your first example is basically not a sequential model. You have an input and an output and that's it. You don't need a recurrent layer for that...

What I would suggest you do is:

    e               f               g
    ^               ^               ^
 LSTM_t1  ---->  LSTM_t2  -----> LSTM_t3  ----->  ...
    ^               ^               ^
  abcd            bcde            cdef

Obviously, you will lose the ability of predict the first characters but hopefully your dataset will include a lot more examples using those specific characters so it will learn how to use them eventually. I would say this is the most general way of producing char to char text generation models. A couple of examples including implementations: here and here.


This issue is caused because the following environment variables are not set correctly
PKG_CONFIG_PATH and LD_LIBRARY_PATH
for my configuration, i should execute this:

export PKG_CONFIG_PATH = /usr/local/lib
export LD_LIBRARY_PATH = /usr/local/lib

also set this commands on R console:

Sys.setenv(HADOOP_HOME="/usr/local/hadoop/")
Sys.setenv(HADOOP_BIN="/usr/local/hadoop/bin")
Sys.setenv(HADOOP_CONF_DIR="/usr/local/hadoop/conf")


Does image's background matter for detector training (CNN)?

Of course, it matters.

Which one is best completely depends on your problem. The golden rule for machine learning problems is that you want the data you train on to be as representative as the data the model will encounter once live.

Typically, faces cropped dataset will be an easier problem because the CNN can directly focus on the face, but if your use case will use images of faces that are not cropped, your model might not perform well.

So use whatever data you expect to encounter later on. The only thing you could do is add a face-cropping algorithm before your network, and then you could train on the faces cropped dataset right away.


I can´t give you an perfect answer because there is no code, dataset and the target what you want to achieve.
Because the feature importances from random forest, is calculated based on the training data given to the model, not on predictions on a test dataset. That means, that is not the true prediction power. You should check, if there are difference on training and test results, when you run a random forest model. Another oportunity is the permutation feature importance.
But I made a similar experience and solved this on another way.

check correlation (shows linear dependency)
check with the pps (shows polynomial dependency)
https://datascienceplus.com/correlation-vs-pps-in-python/
check feature and permutation importance

With these 4 options, I got a better view into my dataset.
Hope I could help you a bit.


Predicting Intent to do X with a confidence score or intent percentage score?

You could use the probabilities output by LogisticRegressions predict_proba method.

Almost all classifiers give you a probability in sklearn. One exception is the support vector classifier which will give you a points distance to the decision hyperplane, which can be interpreted as a confidence (you can get probabilities for the support vector classifier, but it is through a computationally costly cross validation process).


Building Stacking machine learning model using three base classifiers

No, in generally speaking, even minor changes should affect your performance. Changing your meta-model should normally have a visible impact in your model's performance.

Two things you can try:

Check for any problems in your code.
Maybe your test set size is really small. For example if you have 5 test samples, it isn't difficult for all models to get 4/5 (i.e. 80% accuracy). As your test size increases so should the variance of your models' performance.


scikit-learn n_jobs parameter on CPU usage & memory

I can imagine a value of -1 consumes all available resources as and when they become available. Depending on which function you are talking about, it seems data is copied for each of the jobs, which can lead to memory problems if the dataset is large enough. Here is a snippet of information from the docstring of GridSearchCV:

If `n_jobs` was set to a value higher than one, the data is copied for each
point in the grid (and not `n_jobs` times). This is done for efficiency
reasons if individual jobs take very little time, but may raise errors if
the dataset is large and not enough memory is available.  A workaround in
this case is to set `pre_dispatch`. Then, the memory is copied only
`pre_dispatch` many times. A reasonable value for `pre_dispatch` is `2 *
n_jobs`.

So it might be a good idea to use pre_dispatch to put an upper limit on your memory consumption.

Otherwise, why are you setting it to -1? You should just set it to the number of physical cores on your machine, or maybe 2 times that number, if the task can be multi-threaded.

EDIT:

It seems setting n_jobs=-1 does indeed just select all physical cores and maximises their usage. Have a look at the comments in this answer on StackOverflow.

If you have not set pre_dispatch, it will of course try to copy a lot. This is why you run out of memory. If you have 4 cores there will be, by default, 8 copies of the dataset made (as described above in the quote).

Here is another thread, which looks more at the performance side


About 1M rows of data. Should I restrict myself to few columns as well?

Did you try it?

In general, I believe the curse of dimensionality is way overrated. As a rule of thumb, the curse of dimensionality refers to having more columns than rows. I doubt that you have 1M columns...

A robust algorithm, such as Logistic Regression, can often deal with sparse data. I sometimes get the feeling that this algorithm doesn't get the credit it deserves. Especially in production (due to high bias and low variance), I prefer it whenever possible. Nothing worse than an algorithm that fails in production...


Unhashable type 'list' while looping through dataframe in Python

TypeError: unhashable type: 'list'.

You will get this error when you are trying to put list as key in dictionary or set because list is unhashable object.
Example you trying to input code such as
dict1 ={ 1:'one', [2]:'two'}
print(dict1)

O/p:
TypeError Traceback (most recent call last)
<ipython-input-73-c4e2fd1e6bf0> in <module>
----> 1 dict1 ={ 1:'one', [2]:'two'}
      2 print(dict1)

TypeError: unhashable type: 'list'

solution :
Try to convert the list into tuple
Hashable object in Python:

int
float
decimal
bool
string
tuple
complex
range
frozenset
bytes

UnHashable object in Python

list
set
dict
bytearray
customclasses


How to Standardize/Normalize three sets of test data of different lengths

Here is one idea.

First plot the time vs distance. These lines would have different slopes between the measurement periods and you would essentially estimate the speed between measurements. Here is a graph with one measurement window:

Since they're racing on the same track, the total distance driven should be approximately equal. You might need to stretch or shrink the height of lines to make sure they travel the same distance. (This might not be a good assumption for your case.)
Now, go from the y-axis and for regular distance intervals, look up the slope of the graph.
Then plot the observed estimated speed for each distance period. You'll end up with speed measurements at regular distances which will now have the same length of samples.

This way you'll have a fairly good approximation of the speed at which racers traveled at specific parts of the track. I.e. you can compare cornering speed etc.


When is centering and scaling needed before doing hierarchical clustering?

My question is related to the centering and scaling, Is it absolutely necessary to both scale and center the data?, even in scenarios where all the data is coming from the same platform and with same units of measurement.

It depends on the type of data you have. For some types of well defined data, there may be no need to scale and center. A good example is geolocation data (longitudes and latitudes). If you were seeking to cluster towns, you wouldn't need to scale and center their locations.

For data that is of different physical measurements or units, its probably a good idea to scale and center. For example, when clustering vehicles, the data may contain attributes such as number of wheels, number of doors, miles per gallon, horsepower etc. In this case it may be a better idea to scale and center since you are unsure of the relationship between each attribute.

The intuition behind that is that since many clustering algorithms require some definition of distance, if you do not scale and center your data, you may give attributes which have larger magnitudes more importance.

In the context of your problem, I would scale and center the data if it contains attributes like patient height, weight, age etc.

This answer on a similar question has more.


Consider multi-label classification, instead of a binary sentiment your dataset would have a level/degree/probability target for each emotional label of whatever dimension size you want.

In the context of neural networks, it's just a matter of replacing softmax with sigmoid loss layer.

If getting such large dataset with desired supervised labels is hard or not possible, one idea may be to use models trained on any sentiment datasets out there (there are even some multi-dimensional too) and apply additional un(semi)-supervised learning techniques to extract what you want.


Difference between sklearn make_pipeline and imblearn make_pipeline

The imblearn package contains a lot of different samplers for easy over- or under-sampling of data.
These samplers can not be placed in a standard sklearn pipeline.

To allow for using a pipeline with these samplers, the imblearn package also implements an extended pipeline. This pipeline is very similar to the sklearn one with the addition of allowing samplers.

If you want to include samplers in the pipeline, use the imblearn pipeline. Otherwise, use the sklearn one.

The code for the imblearn pipeline can be seen here and the sklearn pipeline code here. Note that make_pipeline is just a convenient method to create a new pipeline and the differnece here is actually with the pipelines themselves.


Reasoning behind using Deep Learning on non-local data

It appear that I forgot one point of the ANN, at least forgot one of its effects : the activation function.

It is true that for linear activation, multi-layer can be reduce to a single-one, but with a non-linear function,

  a two-layer neural network can be proven to be a universal function approximator.

Sources

However, it is true that I dont understand now why to use more than two hidden layers...


Linear Regression in R Mapreduce(RHadoop)

I suggest using machine learning libraries with already functional linear regression
Spark MLlib or hivemall.


Why is an activation function notated as "g"?

The addition of the activation layer creates a composition of two functions.

"A general function, to be defined for a particular context, is usually denoted by a single letter, most often the lower-case letters f, g, h."

So it comes down to the reason that he uses the hypothesis representation h(x)=wX+b which is a function, and that is wrapped by an activation function denoted as g. The choice of g seems to be purely alphabetical.


Data produced as an output to Dumbo API of Python not getting distributed to all the nodes of cluster

In hadoop etc folder,hdfs-site.xml was having number of replication units as 1.That is why,all the files were getting saved on one single node.I changed it and the problem resolved.


Decreasing n_estimators is increasing accuracy in AdaBoost?

In short, AdaBoost works in that way that it trains in subsequent iterations and then measures the error of all available weak classifiers. In each subsequent iteration, the "validity" of incorrectly qualified observations is increased, so that classifiers pay more attention to them. So confusion matrix could be shown after each iteration(after 13).
In case n_estimators=19 its look like you have the perfect fit so for bigger values of n_estimators model starts to overfit, which gives worse performance. In your case please read about early stopping. That technique help you find you best value of n_estimators.

https://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_early_stopping.html


How to calculate Accuracy, Precision, Recall and F1 score based on predict_proba matrix?

To compute performance metrics like precision, recall and F1 score you need to compare two things with each other:

the predictions of your model for your evaluation set (in what follows, I'll call them y_pred)
the true classes of your evaluation set (in what follows, y_true).

From what you write, you have obtained just the predictions of your model, and that's what you have in y_pred. You have constructed y_pred so that each of its components is equal to the class that is assigned the maximum probability by your model. All fine here!

The key ingredient that you are missing is the array of true classes (in your question, you called them "tags") associated to your evaluation examples. You totally need this information to understand if the predictions of your model are correct (or not). You should be able to construct an array y_true containing the true classes/tags of your examples from...knowing the actual ones.
For example, if your 1st text belongs to class 3, your 2nd text belongs to class 1, your third text belongs to class 2, your y_true will be an array like

y_true = np.array([3, 1, 2, # ... the rest of components ])

Now, to compute accuracy, precision, and recall, you need to compare y_true and y_pred. If they coincide, congratulations: that means that your algorithm works perfectly on your evaluation set!
In general though not all the components of y_pred will coincide with y_true. To quantify agreement/discrepancies you can use metrics like accuracy, precision, etc. You can code them yourself, but the scikit-learn library comes with functions for the purpose.

For instance you can easily compute accuracy, precision, recall and F1 score, even the confusion matrix for your problem with the code

from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

# print accuracy
print("Accuracy: ", accuracy_score(y_true, y_pred))

# print precision, recall, F1-score per each class/tag
print(classification_report(y_true, y_pred))

# print confusion matrix, check documentation for sorting rows/columns
print(confusion_matrix(y_true, y_pred))

P.S. Note that in what I did I didn't use your y_pred_one_hot (which is not a confusion matrix!), and that the precision is not zero (it may be, but you have to compute it using y_true)!

P.S. Beware using predict_proba with Naive Bayes as output probabilities may not be calibrated.


Can a classifier be trained with reinforcement learning without access to single classification results?

If every data point has a ground-truth label (i.e., one of the six labels), than any supervised learning technique can work, including Random Forest. If the labels come in batches, then the parameters of the model can be updated with each new batch. Either completely retrain the model with data up to the current time point or incrementally update the model parameters with the new data.

There is no reason to apply reinforcement learning unless it is necessary.

If every data point does not have a ground-truth label, then reinforcement learning might be appropriate. Then you can not use Random Forest or any other supervised learning technique. You'll need to pick a Reinforcement Learning technique that can handle sparsely labeled data.


How to transition between offline and online learning?

Although RL algorithms can be run online, in practice this is not stable when learning off policy (as in Q-learning) and with a function approximator. To avoid this, new experience can be added to history and the agent learn from the history (called experience replay). You could think of this as a semi-online approach, since new data is immediately available to learn from, but depending on batch sizes and history size, it might not be used to alter parameters for a few time steps.

Typically in RL systems like DQN, you would train for some randomly sampled batch between each action, out of some window of historical data (maybe all historical data, maybe last N steps). The amount of training you perform between actions is a hyper-parameter of your model, as is any sampling bias towards newer data.

For example in the Atari game-playing paper by Deep Mind team, the agent sampled a mini-batch with 32 observations (state, action, reward, next state) to train the neural network on, in between each action, whilst playing the game online.

The concept of an epoch does not occur in online learning. If you are using each epoch to report performance metrics, and want to continue using comparable numbers, then you can pick some number of training iterations instead. There are no fixed rules for this, but you might want to consider reporting the same statistics on similar number of iterations as you trained on historical data - e.g. if you had 10,000 training samples in your history and are now training online with a mini-batch size of 50 per time step, then report progress every 10,000/50 = 200 time steps.


Are there any algorithms for solid polygon clustering?

Finally I just implemented solution as I planned and described it in Medium post.
This is "K-Means for solid polygons". I made a playground on my github.io.

A bit more details of solid clustering approach.

K-Means complexity is O(nkdi) or O(nk) per iteration without d,

n is the number of d-dimensional vectors (to be clustered)
k the number of clusters
i the number of iterations needed until convergence.

Complexity estimation of solid clustering

Suppose that clustered polygon has no self-intersections.

k - number of clusters
m - number of points in clustered polygon (to not mix with K-Means n).

Step complexity estimations

Building Voronoi diagram for k centers with Fortune's algorithm: O(k log(k)). Expected that you have centers and cells as result.
Intersect each Voronoi cell with clustered polygon.
I use algorithm from polygon-clipping package.

From description:

  "The Martinez-Rueda-Feito polygon clipping algorithm is used to
  compute the result in O((n+k) log(n)) time, where n is the total
  number of edges in all polygons involved and k is the number of
  intersections between edges."

Total number of edges for us is m and edges in Voronoi cell Tmp. In complex polygon case m >> Tmp. I don't know how to estimate number of intersections but I think with big m it can be ignored.

So full estimation of this step O(m log(m) k).

Complexities of polygon area and center of weight are O(m).

What is the number of polygon points estimation?

In worst case one Voronoi cell polygon has (k-1) edges.
Clustered polygon has m edges.
As I can imagine in the worst case intersection of the worst Voronoi cell and the worst polygon will produce 3n/2 edges polygon.
As a result the upper boundary of this step is O(nk).

Centers of weight are new centers - move to step 1.

Final complexity estimation is O(k log(k) + m log(m) k + m k) => O(m log(m) k) per iteration.

What does it mean?

If we want to solve the same problem of solid polygon clustering with K-Means then we need to discretize original polygon to K-Means points.

K-Means result will be less accurate.
If n >> m log(m) then K-Means will be slower than solid
clustering.

Interesting points

K-Means includes density as part of clustering process. Some points may be heavier than other.
Solid clustering ignores density so weights of all clustered points are equal.
We can see bottleneck effect in solid clustering. Clustering tries to spread cluster centers to have equal density. But cluster center can't jump out of its polygon so it slowly moves from dense area to spatial. In certain conditions we can see the same effect in K-Means.

I think this bottleneck problem may be significantly reduced with center jumps - jump center of the smallest cluster to the biggest one.


Neural Turing Machine - how the shifting works?

Ok, it makes sense if we don't think about the formula

Look at the bottom half of the following slide, taken from Page 20, "Neural Turing Machine" side-presentation by Kiho Suh; June 19, 2017


Computational aspects are typically ignored by statisticians

I think what the author is speaking about is the time/memory complexity of algorithms that statisticians may don't care about. Make a model which is mathematically well proven may be more important to statistician eyes than making approximation to render a model feasible in real life.

I encourage you to look at the complexity of frequent mathematical operations.

Often operation like Singular Value Decomposition, Matrix Inversion, Matrix transpose are used and their cost is way higher than the "upper bound scalable time complexity limit" $O(n.log(n))$ which prevent any utilization on massive datasets.

As an example, you can easily imagine than we are rapidly limited with time complexity because we are not able to increase computational power -- or wait -- more than approximately linearly with the increase of problem size. Take a very common $O(n^2)$ complexity, you can't afford to wait $1000000$ more longer -- or multiply your computational power by $1000000$ to keep same app duration -- when you multiply your dataset size by $1000$ (except if your operations are really fast...)


Is there any NLP library or package which can help in adding comma, punctuation, newlines appropriately to text?

This can be solved with "text segmentation". NLP libraries have code for breaking given text into :

Sentences
Phrases
Words

With this, you can break text into sentences and insert . or ? for each sentence. Similarly, dependency tree will help with inserting some punctuation marks (not all).

Example (breaking text into sentences):

import spacy
nlp = spacy.load('en_core_web_sm')
text = "I was expecting a surplus of cute close-ups but Burton does surprisingly little to win us over He's never been big on treacle but a bit more warmth in this chilly movie which barely follows the outline of the 1941 original would have gone a long way"
text_sentences = nlp(text)
for sentence in text_sentences.sents:
    print(sentence.text)

Output is :

  I was expecting a surplus of cute close-ups but Burton does
  surprisingly little to win us over

and

  He's never been big on treacle but a bit more warmth in this chilly
  movie which barely follows the outline of the 1941 original would have
  gone a long way

More details : https://spacy.io/usage/linguistic-features


You are going in the right direction. Dates can contain a lot of information depending on the task you want to learn. A problem with your suggestion of using the day of the year straight up is that the last day of the year and the first day of the following year are very close to each other, while in your representation they are the furthest away. An alternative to this could be using the fact that this is cyclical and map the day_of_year feature onto a circle and use the coordinates on this circle as features to represent this. A lot of the important features regarding dates are cyclical, like the day of the week, the day of the month, the hour of the day etcetera. An alternative which allows for easier non-linear relationships would be to one-hot encode them as classes. This has the advantage of a more free representation, but the downside that it will cost more features and it cannot generalize based on the fact that two options might be next to each other.

With regards to additional domain knowledge, like knowing that being close to christmas, or close to salary day might be relevant, it's almost always beneficial to add these features to your data. This could be done by calculating absolute distance to christmas, or days before pay day etcetera. Some of these features can be work intensive to create, for example localized holiday features, but they can significantly influence people's decisions and if that is what you are trying to predict, adding these will benefit your performance.


Cannot create new percent columns from CSV file

Try using groupby with a custom function and apply.

df = pd.DataFrame()
df['RunNumber'] = ['Run1','Run1','Run2','Run2']
df['A']=[500,60,20,30]
df['B']=[5,5,2,2]
df['C']=[5,10,2,6]
df['D']=[3,4,5,4]
df['E']=[65,56,56,44]

def calculate_sum_percent(input_df):
    output_df = pd.Series()
    output_df['AB_sum']=input_df['A'].sum() + input_df['B'].sum()
    output_df['CD_sum']=input_df['C'].sum() + input_df['D'].sum()
    output_df['AB_perc']=100*(input_df['A'].sum() + input_df['B'].sum())/input_df.sum().sum()
    output_df['CD_perc']=100*(input_df['C'].sum() + input_df['D'].sum())/input_df.sum().sum()
    return output_df

stats_df = df.groupby('RunNumber').apply(calculate_sum_percent)
display(stats_df)


Multiclass classification of timeseries data using NN

How to approach the solution using a NN ?

Given your data, it doesn't look like an RNN/LSTM is the most sensible choice.

It looks like you have two features and a binary output. RNN/LSTM are useful with variable-length input, otherwise, they are a bit overkill.

Also, make sure that you have a reasonable amount of data before using a deep learning approach. They usually have many parameters to train, and if you don't have enough data points, this might be difficult to do.

If you want to stick to NN, I'd say a regular multilayer perceptron might do the job. If you are open to other options, I would try decision trees for this particular problem.

How do I convert the categorical value of OUI to numerical entities

You could use one-hot encoding. There are plenty of implementations out there (i.e. scikit-learn), or you could do it yourself.


Clustering a very large number of very small clusters with most data unrelated

At this data size, you can still use hierarchical clustering.

You can stop the clustering early when the similarity is too low.

But all of these approaches are pretty inefficient. The proper approach is to not use clustering at all. Instead, use similarity search or, e.g., minhash.


How would you teach multiplications to a neural network?

You need to limit the network model to one which naturally would generalise your problem.

For a simple multiplication, this would be a single linear neuron. You can also train bias so that the network represents y = Ax + B.

If you take that very simple network give it just 2 or 3 examples to learn from, and train it using least squares error, it will quickly converge to the function you want. A single layer linear network like this can generalise to y = Ax + B where x, y and B are vectors, and A is a matrix - i.e. it can solve simultaneous linear equations provided you supply enough examples to do so (usually number of examples equals number of dimensions of the vectors).

This is actually really trivial, and if you train such a simple network using any modern NN library, it will be very fast to learn your linear function. However, in the general case of having a specific non-linear function and wanting the network to learn it in a general sense, it is not possible. The network can be made to learn to generalise reasonably well near where you have provided example inputs and outputs, but will output incorrect results when you give it an x value far away from the training examples.

It is important to note that the network does not learn the math formula for your function. It finds the best approximation given its own internal model. But by picking a network where the internal model is a good match to the function you want to learn, it is going to generalise well to it, and may be able to get it exactly if there is no noise in the examples and enough of them.


Why we add a constant value column in our DataFrame sometimes?

In linear regression you need that column to have lines which are not constrained to pass through origin. Think of linear model $y = b_1 x_1 + b_2 x_2 + ...$. Iff all $x_i$ are 0, y must be 0, you need an additional parameter to pass that constraint.


Relating two data sets at the row level where table one may correlate to zero or more rows in another

I would like to predict the monetary impact that an outage will have on the electric grid.

It may help to think of your problem as two problems:

Given an outage (or set of outages), predict the number of constraints that will result
Given a constraint, predict the monetary impact

You would need to build two regressive models - one for each of the problems.  It's difficult to give advice about which specific techniques or algorithms to use without a better understanding of the domain and of your dataset.

It sounds like you have a set of timestamped historical outages and a set of timestamped historical constraints.  To me, your description implied that there was no authoritative mapping between outages and constraints, so the first wrinkle may be assigning blame for each outage to one or more constraints.  This likely requires domain knowledge that most SO contributors won't have.

Another wrinkle is that you only have date and time information about outages and constraints.  If the temporal information explains a lot of the variance in number of outages and cost of those outages, then no problem.  Otherwise, it's going to be difficult to build accurate models.

If you go with the approach I suggested above, then you have a timeseries forecasting problem on your hands.  Traditional algorithms for this type of problem include auto-regressive models like ARMA, ARIMA, SARIMA, etc.  You can read up on those to see if they seem suitable for your problem.

Alternatively, Facebook recently released an open-source forecasting library Prophet designed to be easy-to-use and to work well out-of-the-box.  It's super easy to try out, so you could give it a shot.


The screenshot looks like output of Keras, so I assume you're using a deep neural network. The training (epoch) is organized with batches of data, so that optimization function is calculated within subset of whole dataset. The console output shows the accuracy of the full dataset, so the optimization of a single batch can decrease the accuracy of the other part of the dataset and decrease the global result.

The other possible reason is that you provide test set to Keras API and the console output is result of the test set that is not used for training of the model. If it's true you can see that your model is overfitting.


As far as I understand you simply don't have to worry about that.

So, you have a sample split randomly in control and treatment. You measure something before treatment, and after, for the same individuals. Because you measure on the same individuals than you have paired measures: delta = before - after. You are interested to measure if the mean of delta for control sample is significantly different than the mean of delta for control sample. This is done with paired sample test known also as dependent test.

If you assume a normal distribution you can use paired t test. If you can't reasonably assume the normal distribution than you can use Wilcoxon signed rank test.

The idea of pairings in test is to eliminate the effect of confounders. What you computed can be the effect of a confounder, but if the sample was split randomly in control and treatment you should not worry about that, unless you have strong reasons to doubt the randomization procedure.  In the later case you perhaps should include confounders in the equation and take the route of random effects modeling.


ValueError: Tensor Tensor("activation_5/Softmax:0", shape=(?, 2), dtype=float32) is not an element of this graph

The solution to this issue is predict from the keras model when running a tensorflow graph as default.

import tensorflow as tf
graph = tf.get_default_graph()

global graph
with graph.as_default():
    result = loaded_model.predict(x)


If you use the test set for model selection (it does not matter whether you use some random state, or do hyperparameter tuning) then it is dirty and can no longer be used to predict the prediction performance.

The general workflow you should follow is this:

Split into train and test. Lock "test" in a safe.
Use cross-validation on the train set only to tune parameters.
Fix the hyperparameter to the best setting found. Write them in boldface as 'final parameters' into a stone plate on your wall.
Train a new classifier on the entire training set using exactly these parameters
Get the test set from the safe, and run the classifier on the test set exactly once to get a prediction of how well your classifier is going to work in the future
If the results are substantially worse than those from 2.: panic. It's not going to work.
Optional: If you are confident that hyperparameters do not depend on the data size but that you need every little bit of training data, you could train a classifier on train+test with the in-stone parameters. This is risky though, a you don't have a test set anymore.
Deploy, but monitor that it really performs as good as predicted.

The golden rule is: if you used the test set, you must not change the model anymore. You've got one shot to score.

That is why you split the train set again in CV.


Feature extraction - wavelet transformation + autoregression

The SVM is used after the wavelet transform. The wavelet transform is a way to detect correlations between the parts/segments of the signal. Once you have these 'features' you can train more or less any classifier on that data.


Algorithm for detecting parity of integers

A fundamentally linear model like logistic regression will never work well, because its assumptions are not at all true for your data set. It presumes that probability (OK, really, log odds) of being positive or negative changes linearly in each input, but, it alternates with each integer value in your input. KNN's assumption likewise does not match. For each integer, its neighbors have an opposite classification. These will never work well as applied to your input, because your input doesn't match their usage.

If however you include the parity of the input (X mod 2) as a feature, they should all trivially learn this model.

This is a pretty good simple lesson in feature engineering and understanding the assumptions you take on when applying an algorithm.


Finding frequencies in a noisy, "uneven" dataset

Just to point you in one possible direction: you could treat this problem as one of probabilistic mixture modeling.

Imagine that each person's drink ordering is governed by a probability distribution. That distribution may be characterized by the time since their last drink order.  As time passes, the probability that they will order another drink increases until eventually they order another drink and the time resets.

One possible model for the time between drink orders for a single person is the exponential distribution.  If you consider many people ordering, the resultant times would likely be a mixture of exponentials. The problem then comes down to fitting a mixture of exponentials to the table's drink data.  You would likely have to supply some additional prior data as well to get a meaningful model (otherwise it's difficult to tell the difference between 5 people ordering 2 drinks a piece or 1 person ordering 10 drinks).


python pandas optimization: filtering on text index values

I've found a solution: instead of filtering through all the values, I compute a hash table (in fact a DataFrame) of the possible values and filter on those values:

Step 0: 0:00:00.014251
Step 1: 0:00:00.010097
Init fast filter: 0:00:00.353645
Step 2: 0:00:00.027462
Step 3: 0:00:00.002550
Step 4: 0:00:00.027452
Globally: 0:00:00.435839

It runs faster because there are lots of duplicated values.

Here's a class which is doing that stuff. It works for indexes and normal columns:

import pandas as pd
import numpy as np

class DataFrameFastFilterText(object):
    """This class allows fast filtering of text column or index of a data frame
    Instead of filtering directy the values, it computes a hash of the values
    and applies the filtering to it.
    Then, it filters the values based on the hash mapping.
    This is faster than direct filtering if there are lots of duplicates values.
    Please note that missing values will be replaced by "" and every value will
    be converted to string.
    Input data won't be changed (working on a copy).
    """

    def __init__(self, data, column, isIndex):
        """Constructor
        - data: pandas Data Frame
        - column: name of the column - the column can be an index
        - isIndex: True if the column is an index
        """
        self._data = data.copy()  # working with a copy because we will make some changes
        self._column = column
        self._isIndex = isIndex
        self._hashedValues = None
        self._initHashAndData()

    def _initHashAndData(self):
        """Initialize the hash and transforms the data
        1. data: a. adds an order column
                 b. creates the column from the index if needed,
                 c. rename the column as 'value' and
                 d. drops all the other columns
                 e. fill na with ""
                 f. convert to strings
        2. hash: data frame of 'hashId', 'hashValue'
        3. data: a. sort by initial order
                 b. delete order
        """
        self._data['order'] = np.arange(self._data.shape[0])
        if self._isIndex:
            self._data[self._column] = self._data.index.get_level_values(self._column)
        self._data = self._data.loc[:, [self._column, 'order']]
        self._data.rename(columns={self._column: 'value'}, inplace=True)
        self._data['value'] = self._data['value'].fillna("", inplace=False)
        self._data['value'] = self._data['value'].astype(str)
        self._hash = pd.DataFrame({'value': pd.Series(self._data['value'].unique())})
        self._hash['hashId'] = self._hash.index
        self._data = pd.merge(
            left=self._data,
            right=self._hash,
            on='value',
            how='inner',
        )
        self._data.sort_values(by='order', inplace=True)
        del self._data['order']

    def getFilter(self, *args, **kwargs):
        """Returns the filter as a boolean array. It doesn't applies it.
        - args: positional arguments to pass to pd.Series.str.contains
        - kwargs: named arguments to pass to pd.Series.str.contains
        """
        assert "na" not kwargs.keys()  # useless in this context because all the NaN values are already replaced by empty strings
        h = self._hash.copy()
        h = h.loc[h['value'].str.contains(*args, **kwargs), :].copy()
        d = self._data.copy()
        d['keep'] = d['hashId'].isin(h['hashId'])
        return np.array(
            d['keep'],
            dtype=np.bool
        ).copy()

And the example code is adapted as follows:

...
tic = datetime.datetime.now()
fastFilter = DataFrameFastFilterText(data, 'identifier')
print("- Init fast filter: %s" % str(datetime.datetime.now() - tic))

tic = datetime.datetime.now()
filt = filt | fastFilter.getFilter(rexpPlus)
print("- Step 2: %s" % str(datetime.datetime.now() - tic))

tic = datetime.datetime.now()
filt = filt & (~fastFilter.getFilter(rexpMinus))
print("- Step 3: %s" % str(datetime.datetime.now() - tic))
...


Can LSTM have a confidence score for each word predicted?

Couldn't you require your softmax output to exceed a threshold for the prediction or you call it "not confident"?

I have not tried word prediction, but the softmax gives you a value for each output, where all of those values sum to 1. So pick the column with the highest value as your predicted word and use that value to determine your confidence. (It may be fairly low, since there might be a small chance of lots of words and then larger chances of your most-common three.)


Running a query in R after establishing dbconnect

I understand your question as: I try to send this query to the database but it doesn't work.
As far as I can see the trouble is here:

"select id as ID,
       date_c."Professional_Status" as

You first start with double quotes and R thinks that the string ends at the next double quotes, so before Professional_Status. Use either single quotes like you do in date = '2018-01-19'
Or use single quotes at the ends and double in between. Don't mix and match, it will not work.
Since Profesional status is not defined variable nor is it a valid R statement this will fail.

Some small tips:
If you want to substitute professional_status with a variable you have defined in your R session, use something like paste("text", variable " text") or use the excellent glue package that allows you to write glue("text {variable} text")
A final tip, if you use an IDE, like for instance RStudio, you can actually see that the text ends at the first second quote, because the highlighting is different.


How can I split an image into rectangles?

AWS Rekognition works very well for detecting typewritten text but not so well for detecting hand written symbols. You can see this is true in your example. What I would do is detect all the boxes associated with the words Problem. Then you can extract a box relative to the position of this word. Then you will have extracted the hand written numbers in each box.


Imputation missing values other than using Mean, Median in python

So if you want to impute some missing values, based on the group that they belong to (in your case A, B, ...), you can use the groupby method of a Pandas DataFrame. So make sure your data is in one of those first.

import pandas as pd
df = pd.DataFrame(your_data)              # read documentation to achieve this

Then, it is just a case of chaining a few steps together:

df["Value"] = df.groupby("Brand")["Value"].transform(lambda x: x.fillna(x.mean()))

df.groupby simply groups the dataframe into sub-dataframes (groups), such that each group only contains one Brand
transform() will apply a function to a dataframe - so to each of the individual groups created in groupby
the nameless function (a lambda function) calls the DataFrame's fillna() method on each dataframe, using just the mean() to fill the gaps

You can simply substitute the mean() method for anything you like. You could also create a more complicated function, ifyou need it, and replace that lambda function. It would simply need to take a dataframe as input and return a dataframe with a comparable index.


The $2^{|S|}$ number of POSSIBLE questions is defined by the number of different combinations of subsets of $S$ that could be part of the first partitions, then the second and so on until the tree is finished. That being the worst case.

Imagine that: $loc \in{} N$ is the first separation, so it's divided into Yes/No. Then $loc \in{} S$ is divided into Yes/No.

Then $loc \in{} E$ is the third partition and the answer is the same (Yes/No), you have 8 (2^3) questions asked. There are many combinations of use/no use/which to combine in which step, etc, but ultimately you will be asking the tree to consider all possibilities, which are $2^{card(S)}$.


Finding lookalike customers for Digital Media targeting

This seems to be a pretty common scenario in digital marketing, and a few companies have published their approach to lookalike modeling.

Here are a few links:

Lookalike at LinkedIn
Lookalike at Pinterest
Lookalike at Yahoo
Another lookalike from Yahoo
Academic paper on lookalike (not sure where the authors work)
Google's lookalike patent (this one is a lot harder to read)

Maybe you can draw inspiration from what these companies have done.  Most of the lookalike systems that I've read about fall into one of the following categories:
segment-approximation models, regression models, and similarity-based models.

Segment-approximation is a similar idea to collaborative filtering.  The idea is to find which interests are shared among your seed audience (that is, your past converters).  Then you expand the audience by finding new individuals who have a similar interest profile.  Obviously this can only be done if you have some information about your potential customers' interests.

Regression models try to compute a membership function which ranges from 0 (not similar to the seed audience at all) to 1 (a member of the seed audience).  You use this model to score each potential customer, and those which receive a sufficiently high score are included in the lookalike audience.  One challenging aspect of this approach is that you have to use semi-supervised training.  You have a set of positive examples (past converters), and you have a much larger pool of mixed-label examples (people who haven't converted).

Similarity-based models generally attempt to learn a useful representation for customers in an unsupervised way.  You can then run clustering in the representation space, and your lookalike audience will be those clusters which have high overlap with your seed audience.
Alternatively, you can do similarity comparisons in the representation space, and construct a lookalike audience by thresholding the similarity you find between converters and other potential customers.

The specific learning algorithm that you use will probably vary based on the data you have available.  The links above may spark some ideas.

Hope that helps!


Scikit MLPClassifier vs. Tensorflow DNNClassifier

Maybe you want to use a more complicated activation function - i.e. leakyReLU or you want to add batch normalization. From a brief look at the API there is no option for batchnorm in SKlearn. Also it appears that you can't use dropout with MLPClassifier.

You might also feel more comfortable with tensorflow features - i.e. tensorboard, or you might feel more comfortable with SKlearn. To each his own.


Significant drop from validation accuracy to test accuracy

I don't think this is ususual, I experiance it quite frequently with regression problems. Generally I think it means the model is underspecified so instead of learning the actual relationship between X and y it overfits. Generally I find that I have to do a bit of feature engineering, and ensure I'm only using features that are really necesary - too many features or features with spurious relationships to the target can really hurt a RF model in particular in my experiance. Also make sure as part of your cross validation that you're cross validating parameters which directly influence model complexity i.e. tree depth, number of trees etc.


The ultimate end goal of your modeling is going to affect the way you want to format your data. It's a good practice whenever you start a machine learning project to ask yourself, what is the precise question you want to answer, because whatever model you generate, it's only going to make sense if used in the context of the question asked.

If in your case you want to predict loan approvals, then first you need to check whether that information is even present in the data you have. The JSON you have just shows historical data, but do you know the outcome for each data point? Is that another feature, maybe recorded in the "someValue", "title" area of the file? Without it, you really can't do anything.

If you can get that information, then it's perfectly fine to generate a variable for each year: assets_year1, assets_year2, ... assets_year8, investments_year1, ... etc. True, year8 might have a lot of Null values, but that's not necessarily bad. For example, most models working with text data consist of really sparse training matrices, yet they do very well in practice. Depending on the algorithm, it might weed out those variables anyways. With Null values, you just have to try imputing the Null values differently, and you can also create an indicator variable for whether the column has missing values, and see how different settings change performance.

To take it a step further, you can also create additional features to encapsulate the nuances that are occurring over the years. Create features like the historical average, average change per year, total number of years of history, etc. Creative feature engineering is the hardest part, but can lead to big changes in performance.


Character recognition neural net topology/design

Your design makes some sense, but there is no need to limit connections even if you expect to represent probabilities of upper/lower case separately, because they will interact usefully. E.g if the character could most likely be one of o, O, Q, G then this might be useful information to choose the correct one.

If you went ahead, you would need to train this network without the final layer (so that it learns the representations you expect, not some other group of 52 features), then add the final layer later, with no need for special connection rules, just use existing ones. Initially you would training the new layer separately from the full output of the 52-class net i.e. probability values, not selected class. Then you would combine with the existing net and fine-tune the result by running a few more epochs with a low learning rate on the final model.

That all seems quite complex, and IMO unlikely to gain you much accuracy (although I am guessing, it could be great - so if you have time to explore ideas, you could still try). Personally I would not take your hidden layer idea further. The full 52-class version with simple logic to combine results is I think simpler. This is also not necessary, the neural net can learn to have two different-looking images be in the same class quite easily, provided you supply examples of them in training. However, it may give you useful insights into categorisation failures in training or testing.

It is not clear from the question, but if you are not already using convolutional neural network for lower layers, then you should do so. This will make the largest impact on your results by far.


How to determine the most active time of a user from active directory logs

As always, start by looking at the data.  Pick a user, and look at their typical login times.  Maybe plot a histogram to show the distribution of time of day of their login.  Pick another user, and look at them.  Pick a few more.  Do you see any patterns?  Do they have a common shape?  Do users fall into a few clusters?  Try to understand the data well.

Then, once you have a feeling for what normal behavior tends to look like, and how it varies from user to user, you can start to think about how to build a model that captures this random behavior and helps you distinguish normal vs abnormal behavior (perhaps by some kind of anomaly detection).

But as always: start by visualizing the data.  Asking how to approach the problem or what machine learning method to use is premature, until you have a better understanding of the data.


Calibration using predict_proba vs class_weight

There are several options for you:

class_weight should boost the loss function towards the preferred class. This option is supported by various estimators, including sklearn.linear_model.LogisticRegression,

sklearn.svm.SVC, sklearn.ensemble.RandomForestClassifier, and others. Note there's no theoretical limit to the weight ratio, so even if 1 to 100 isn't strong enough for you, you can go on with 1 to 500, etc.

You can also select the decision threshold very low during the cross-validation to pick the model that gives highest recall (though possibly low precision). The recall close to 1.0 effectively means false_negatives close to 0.0, which is what to want. For that, use sklearn.model_selection.cross_val_predict and sklearn.metrics.precision_recall_curve functions:
  y_scores = cross_val_predict(classifier, x_train, y_train, cv=3,

                               method="decision_function")

  precisions, recalls, thresholds = precision_recall_curve(y_train, y_scores)

If you plot the precisions  and recalls against the thresholds, you should see the picture like this:

After picking the best threshold, you can use the raw scores from classifier.decision_function() method for your final classification.

Finally, try not to over-optimize your classifier, because you can easily end up with a trivial const classifier (which is obviously never wrong, but is useless).
As said there are 2 stages to make this Tuning: in the model training stage (like custom weights) and the prediction stage (like lowering the decision threshold).
Another tuning for the model-training stage is using a recall scorer. you can use it in your grid-search cross-validation (GridSearchCV) for tuning your classifier with the best hyper-param towards high recall.
GridSearchCV scoring parameter can either accepts the 'recall' string or the function recall_score.
Since you're using a binary classification, both options should work out of the box, and call recall_score with its default values that suits a binary classification:

average: 'binary' (i.e. one simple recall value)

pos_label: 1 (like numpy's True value)

Should you need to custom it, you can wrap an existing scorer, or a custom one, with make_scorer, and pass it to the scoring parameter.


(Assuming this is a similar question to what was posted on Cross Validated, but was closed): You can merge the data from both matrices while adding a variable that specifies the origin (group), then plot them together in ggplot2:

library( ggplot2 )

N1 = 19
N2 = 17
M  = 10

m1 = matrix( rnorm(N1*M,mean=0,sd=1), N1, M)
m2 = matrix( rnorm(N2*M,mean=0,sd=1), N2, M)

y     = c( as.vector( t( m1 ) ),  as.vector( t( m2 ) )  )
x     = c( rep(1:10, each = N1 ), rep(1:10, each = N2 ) )
group = c( rep( '1', N1 * M ),    rep( '2', N2 * M )    )

df = data.frame( state = x, value = y, group = group )

ggplot( df, aes( x = state, y = value, colour = group ) ) +
    geom_point() +
    ggtitle( "State values in group 1 and 2" ) +
    labs( x = "State", y = "Value" ) +
    scale_x_continuous( breaks = seq(10) )

This is the result:


Is there any machine learning algorithm that can solve this problem?

You want a supervised approach. Clustering will not care about your target variable and perform arbitrary splits that don't help.

Likely a decision tree can be helpful here, if you use a good implementation that can split the same feature multiple times (to break it into intervals). There are other approaches you could try, such as piecewise regression etc.


Why do my target labels need to begin at 0 for sparse categorical cross entropy to work?

Have a look at this stackoverflow answer, it seems to be caused by that the fact that your labels need to be zero indexed as the argmax function also return the index based on a zero indexed array.


Trainer Component Error in Tensorflow Extension : AttributeError: module 'user_module' has no attribute 'trainer_fn'

The reason is tfx doesn't support Keras sequential model, but function API. After migrating to use function API, it works in fine in the trainer pipeline.

def get_model(show_summary=True):

#one-hot categorical features
num_A = 4,
num_B = 3,
num_C = 2,
num_D = 8,
num_E = 12,
num_F = 4,
num_G = 16,
num_H = 26

input_A = tf.keras.Input(shape=(num_A,), name="A_xf")
input_B = tf.keras.Input(shape=(num_B,), name="B_xf")
input_C = tf.keras.Input(shape=(num_C,), name="C_xf")
input_D = tf.keras.Input(shape=(num_D,), name="D_xf")
input_E = tf.keras.Input(shape=(num_E,), name="E_xf")
input_F = tf.keras.Input(shape=(num_F,), name="F_xf")
input_G = tf.keras.Input(shape=(num_G,), name="G_xf")
input_H = tf.keras.Input(shape=(num_H,), name="H_xf")

inputs_con = tf.keras.layers.concatenate([
input_A,
input_B,
input_C,
input_D,
input_E,
input_F,
input_G,
input_H])

dense_1 = tf.keras.layers.Dense(50, activation = 'relu')(inputs_con)
dense_2 = tf keras.layers.Dense(25, activation = "rely") (dense_1)
output = tf.keras.laters.Dense(1, activation = "sigmoid") (dense_2)
model = keras.Model(inputs=inputs, outputs=outputs)

_inputs = [
input_A,
input_B,
input_C,
input_D,
input_E,
input_F,
input_G,
input_H]

model = tf.keras.models.Model(_inputs, output)

model.compile(optimizer='rmsprop',
          loss='binary_crossentropy',
          metrics=['accuracy'])

if show_summary:
    model.summary()

return model