Measuring distance preservation in dimensionality reduction

The confusion

The point of confusion is probably the use of variable $n$ to denote the dimension in Euclidean distance, which is later re-used for the number of points. Euclidean distance is better to be written as
$$d_X(\boldsymbol{x}_1, \boldsymbol{x}_2)=\sqrt{\sum_{i=1}^{\color{red}{d}}(x_1^i - x_2^i)^2}$$
in the original space, and
$$d_Z(\boldsymbol{z}_1, \boldsymbol{z}_2)=\sqrt{\sum_{i=1}^{\color{red}{k}}(z_1^i - z_2^i)^2}$$
in the reduced space. Where $\boldsymbol{z}_a=(z_a^1,\cdots,z_a^k)$ is the transformation of point $\boldsymbol{x}_a=(x_a^1,\cdots,x_a^d)$ by $W$. EDIT: in your notation, $Z[:, a]$ is the transformation of $X[:, a]$.

Distance preservation measure

A measure of distance preservation for transformation $W$ should be defined in such a way that compares the distance of two points in the original space with the distance of the same two points in the reduced space.

Accordingly, a measure that compares all $\binom{n}{2}$ pairwise distances could be defined as:
$$DP(W) = \frac{2}{n(n-1)}\sum_{a=1}^{n} \sum_{b=a+1}^{n} |d_X(\boldsymbol{x}_a, \boldsymbol{x}_b) - d_Z(\boldsymbol{z}_a, \boldsymbol{z}_b)|$$

where for a constant $k$, smaller $DP(W)$ implies better distance preservation by $W$.

Note that various measures of distance preservation can be defined based on this foundation. For example,

Restricting the comparison to only $K$ nearest neighbors of a point in the original space, leading to $nK$ comparisons instead of $\binom{n}{2}$, or
Using the Pearson correlation of distances rather than their difference, since we generally want close (distant) points in the original space to be also close (distant) in the reduced space, or
Measuring the distance to point's tangent space instead of the point itself, etc.


Change raw count to a proportion based on a filter in Tableau

Yes, you can do this without reworking your, and as always, there are few ways to do it. Without seeing exact data, I would try setting table calculation to percent of total and then alter compute with to Table Down / Table Across depending on the data layout


Target and output shape/type for binary classification using PyTorch

According to your questions:

Labels should be long and advised.
[num_samples, ]
It should have two outputs.
If your batch_size=200 then target somehow similar to this: [0, 1, 0, 1, 1, 0, ....1]
3rd ^


How to check if audio samples have only noise or are silent?

First you should apply speech enhancement algorithms to remove noise from speech. Then, you should use VAD (Voice Activity Detector) to remove silence from speech.


Measuring the success of text summarization

You will first need a human written summary as your correct output. You can then compare the original and generated ones using Rouge scores. They compare the similarity between two giving paras by comparing different combinations of sub-phrases. Search for Rouge and you'll get different types of rouge scores, you'll probably need to use more than one. Also, there are libraries available that calculate rouge scores so you don't have to implement it yourself!


How to create clusters based on sentence similarity?

For this problem, I would start with a simple bag of words model and use that as a baseline.

This is an example : https://medium.com/@MSalnikov/text-clustering-with-k-means-and-tf-idf-f099bcf95183 , http://brandonrose.org/clustering

For logs, this might be sufficient since it will easily filter out common words like date etc.

If this does not work well for a given usecase, next step is to try Doc2Vec and similar methods that project sentences to a vector space.


Pandas change value of a column based another column condition

What I want to achieve: Condition: where column2 == 2 leave to be 2 if column1 < 30 elsif change to 3 if column1 > 90

This can be simplified into where (column2 == 2 and column1 > 90) set column2 to 3. The column1 < 30 part is redundant, since the value of column2 is only going to change from 2 to 3 if column1 > 90.

In the code that you provide, you are using pandas function replace, which operates on the entire Series, as stated in the reference:

  Values of the Series are replaced with other values dynamically. This differs from updating with .loc or .iloc, which require you to specify a location to update with some value.

This means that for each iteration of for x in filter1 your code performs global replacement, which is not what you want to do - you want to update the specific row of column2 that corresponds to x from column1 (which you are iterating over).

  the problem is 2 does not change to 3 where column1 > 90

This is truly strange. I would expect the code you provided to have changed every instance of 2 in column2 to 3 as soon as it encountered an x >= 30, as dictated by your code conditional statement (the execution of the else branch). This discrepancy may stem from the fact that you are assigning to column2 the result of global replacement performed on the column Output (the contents of which are unknown). In any case, if you want your program to do something under a specific condition, such as x > 90, it should be explicitly stated in the code. You should also note that the statement data['column2'] = data['column2'].replace([2], [2]) achieves nothing, since 2 is being replaced with 2 and the same column is both the source and the destination.

What you could use to solve this particular task is a boolean mask (or the query method). Both are explained in an excellent manner in this question.

Using a boolean mask would be the easiest approach in your case:

mask = (data['column2'] == 2) & (data['column1'] > 90)
data['column2'][mask] = 3

The first line builds a Series of booleans (True/False) that indicate whether the supplied condition is satisfied.
The second line assigns the value 3 to those rows of column2 where the mask is True.


Train on multi-domains, then fine-tune on specific domain

Generally speaking, training with a broad distribution of data and then fine tuning on a specific domain generally leads to better performance and faster training time.
There are techniques such as meta-learning, domain invariance using techniques (ex. Gradient Reversal) or minimizing distance between the mean domain feature space using Maximum Mean Discrepency metrics. You could also familiarize yourself with empirical risk minimization and develop an intuition grounded on experiments. You could also check out 'A theory of learning from different domains' by Ben-David et al.


How to plot multiple columns with ggplot in R?

If I understand right your question, you are looking to plot selected numerical columns against a selected categorical column of your dataset, am I right ?

If so, you can have the use of dplyr, tidyr and ggplot2 packages to achieve this.

Starting with this dataframe:

  id        num1      num2      num3 cat cat2
1  C -0.48892284  1.417909 2.8884577   a    f
2  C -0.62795166  1.472390 1.6625688   c    f
3  B -0.04691673  2.731553 0.9692889   c    e
4  B  0.16261812 -1.152528 2.4308332   a    d
5  C  1.29230591 -1.609465 2.2089074   a    f
6  E -0.46355650 -1.070132 0.4517597   b    f

Basically, you are selecting first your columns of interest (here num1, num2 and cat), then, you reshape data into a longer format using pivot_longer function to obtain something like that:

library(tidyr)
library(dplyr)
df %>% select(id, num1, num2, cat) %>%
  pivot_longer(., cols = c(num1,num2), names_to = "Var", values_to = "Val")

# A tibble: 200 x 4
   id    cat   Var       Val
   <fct> <fct> <chr>   <dbl>
 1 C     a     num1  -0.489
 2 C     a     num2   1.42
 3 C     c     num1  -0.628
 4 C     c     num2   1.47
 5 B     c     num1  -0.0469
 6 B     c     num2   2.73
 7 B     a     num1   0.163
 8 B     a     num2  -1.15
 9 C     a     num1   1.29
10 C     a     num2  -1.61
# … with 190 more rows

Finally, you can add to this pipe sequence the plotting part by calling ggplot and geom_boxplot :

library(tidyr)
library(dplyr)
library(ggplot2)
df %>% select(id, num1, num2, cat) %>%
  pivot_longer(., cols = c(num1,num2), names_to = "Var", values_to = "Val") %>%
  ggplot(aes(x = Var, y = Val, fill = cat)) +
  geom_boxplot()

Is it what you are looking for ?

Data

set.seed(123)
id <- sample(LETTERS[1:5],100, replace = TRUE)
num1 <- rnorm(100)
num2 <- rnorm(100)*2
num3 <- rnorm(100)+2
cat <- sample(letters[1:3],100, replace = TRUE)
cat2 <- sample(letters[4:6],100, replace = TRUE)
df <- data.frame(id, num1, num2,num3, cat,cat2)


Initializing a recurrent neural network (RNN) in Keras

In line 809, you can find that SimpleRNNCell takes a bias_initializer as an argument to the constructor function and it defaults to zeros.


Neural Network Model using Transfer Learning not learning

Just take 2 images from your training data. One from class 'crack', another one from class 'not crack'. Now, check if your model can get a training accuracy of 100% which means it can overfit on the training dataset or not. If it is not able to do that, something is extremely wrong about the model.


Best tool for analyzing inventory that has binary sales outcome of yes or no

Logistic regression would be an ideal candidate for assessing the probability of a sale, but it would be wise to take Emre's comments on 'how many' into account.  In this case even a basic aggregation of your top 50000 with a 'yes' dependent variable would get you there-- but why stop there?

You may want to isolate whether or not seasonal (or other cyclical) factors are influencing purchasing behavior (e.g. if you were attempting to optimize inventory levels for the summer purchasing habits).  In this case, linear regression with dummy variables accounting for the season or month may be a good way to delve deeper (this touches on the 'classification' point made by Emre).  The more historical data you have here, the better.

Back to your point on 'fast'-- you mention Excel so I am assuming Python or R are being ruled out.

RapidMiner or Knime would fit your requirement for 'fast' in any of these modeling contexts.  Otherwise, if you're curious, scikit-learn (python) has about 3 dozen other general linear models freely available (R has a vast library as well).

Attached is a link on how to do this kind of modeling within RapidMiner--

Logistic Regression Example (RapidMiner)

https://www.youtube.com/watch?v=0npgGPK0zR8

Happy modeling!


How does Max Pooling (of size 2x2) change the size of receptive field in CNN?

Before anything, let's try to understand what Max-pooling actually does. Intuitively, Max-Pooling takes the maximum of the value inside the kernel as the maximum value is something that causes a larger impact from the picture. When the size of the kernel is 2x2, half of the values denote the actual value so the receptive field increases. A visualization could be found in this video. Also you can look at this video too


If you're not modifying anything you can call it validation. If you are, you can call it preprocessing.


If you use any forecasting model, then it will do over fitting on your data set, as you have just 19 observations. At least 1000 observations are the good point to start applying any machine learning model.

I need to predict Response Rate, % Promoters, % Detractors and %
Neutrals all of which are numeric variables.

You can do data exploration where you start using bar charts, line charts, and try to find the pattern for each of those variables. Using this, if not accurate, but you would be able to understand lump sum forecasted value for upcoming months.
Even built in functions are there in excel. Check this out: https://www.excel-easy.com/examples/forecast.html


I think that what is influenced by the type of your independent variables is not the structure of your neural network but the encoding of your data. If your independent variables are ordinal, then you can map them to integer numbers or other ordered sets (and maybe normalize them to make them between 0 and 1).

If your target variable is a label, it is a classification problem. Therefore, the neurons in the last layer in your neural network should have an activation function optimized for classification problems, like sigmoid.

You also need a specific loss function, like binary crossentropy. There was a case study, however, that showed that for ordinal classification (when your target labels are ordinal), mean squared error works well.

You can take any activation functions for all hidden layers in your network as long as they are non-linear. The choice of a particular activation function will influence the rate of convergence of your learning algorithm but I don't see how a particular activation function in hidden layers may be preferable for a certain type of input variables, like if they are ordinal or not.


Cross validation when training neural network?

I couldn't say what the authors refer to by

  best by cross-validation

but I'll mention a simple and general procedure that's out there:

You are correct that analyzing one estimate of the generalization performance using one training and one test set is quite simplistic. Cross-validation can help us understand how this performance varies across datasets, instead of wonder whether we got lucky/unlucky with our choice of train/test datasets.

Split the whole dataset into k folds (or partitions), and train/test the model k times using different folds. When you're done, you can compute the mean performance and a variance that will be of utmost importance assessing confidence in the generalization performance estimate.


Extract or subset hundreds of columns from a data frame

Hopefully I've understood your question correctly.

Assuming your text file looks like this?

GeneName1
GeneName2

You can read that in using the readLines() function:

cols <- readLines("name_of_text_file")

Which returns cols as a vector of those names:

> cols
[1] "GeneName1" "GeneName2"

Which can then be used to subset the data frame as per your example:

mydata[ , cols]


How does MITIE perform named entity recognition?

After having used MITIE for a few weeks, I feel like I at least have enough to answer my basic questions:

(and 3.) All models need to be trained from scratch - there is no online method to add new samples to the model as they come in. This is unfortunate because MITIE takes at least 45 minutes to an hour to train on a ~20k-sized dataset.
The datasets I used were ATIS, CoNLL 2003, and DBpedia

I've found MITIE to be quite good as far as classification accuracy goes, although it takes a bit of work to prepare datasets for it.


Is this random forest logical correct and correct implemented with R and gbm?

Is the code correct?
Yes, it looks correct to me

How to calculate Errors:
Calculate accuracy for train data
Calculate accuracy for test data

If train error is high (RMSE in your case) - high bias, retrain model with more trees, less learning rate, more data (if possible)
If train error low, but test error high - high variance or overfit: regularization, increase k-fold
If tran and test error low - your model is good to go.
Hope this answeres your question.


How do I interpret loss in a neural network?

Thinking in terms of optimisation, loss is convergence indicator

NN can be expressed as an optimization problem.

In the context of an optimization algorithm, the function used to evaluate a candidate solution (i.e. a set of weights) is referred to as the objective function. When we minimise this function we also refer to it as loss function

Cross-entropy and mean squared error are the two main types of loss functions to use when training neural networks.

Graph that you plotted indicates just convergance behavior on two different sets of our optimisation problem. If they diverge too much, we are overfitting. If they stop, we stopped learning.


Multiclass Discriminative Training of i-vector Language Recognition,
this article contains update formulas for mean and covariance (diagonal).

First you should obtain parameter estimations using ML (so calculate mean and covariance from class data) and then iteratively update them according to formulas (16-20). I'm not sure about the meaning of C0.


After we have

$$w^Tx + b = \pm \delta$$

We can always divide everything by $\delta$,

$$\left( \frac{w}{\delta}\right)^Tx + \left( \frac{b}{\delta}\right)=\pm1$$

Now, we can set $\tilde{w}=\frac{w}{\delta}$ and $\tilde{b}=\frac{b}{\delta}$.

$$\tilde{w}^Tx+\tilde{b}=\pm1$$

This is as if we have set $\delta=1$ from the beginning.

The derivation of the distance formula has been given in equation $(19)$ in the article that you linked to and you might like to be more specific if you can't understand it. The distance should be $\frac{2\delta}{\|w\|}$ if $\delta$ is not set to be $1$.


empty fields to be filled with previous values

You can use fillna function with ffill method:

df.fillna(method='ffill', inplace=True)


which open-source frameworks exist for fact-extraction

I don't know many examples but I'm aware of at least one such tool, specialized for the medical domain: SemRep

  SemRep is a UMLS-based program that extracts three-part propositions,
  called semantic predications, from sentences in biomedical text.
  Predications consist of a subject argument, an object argument, and
  the relation that binds them. For example, from the sentence in (1),
  SemRep extracts the predications in (2).

  1.We used hemofiltration to treat a patient with digoxin overdose that was complicated by refractory hyperkalemia.

  Hemofiltration-TREATS-Patients,
  Digoxin overdose-PROCESS_OF-Patients,
  hyperkalemia-COMPLICATES-Digoxin overdose,
  Hemofiltration-TREATS(INFER)-Digoxin overdose

In general, this is closely related to the problem of semantic role labeling:

  semantic role labeling (also called shallow semantic parsing) is the process that assigns labels to words or phrases in a sentence that indicate their semantic role in the sentence, such as that of an agent, goal, or result.

Apparently there are some implementations available: https://framenet.icsi.berkeley.edu/fndrupal/ASRL


Implementing the Dependency Sensitive CNN (DSCNN ) in Keras

Intermediately I finished the network and it is working great :-) Anyone who is also looking for this network can download it on my Github page.
To refer directly to my questions:
Question 1:
The proposed code actually makes partially sense:

One should use the shared-embedding layer to save memory usage, so this part makes sense
In the initially proposed code I used TimeDistributed layer, this is the wrong way for the purpose of the network. We actually have to use a simple shared LSTM layer. Time distributed layers are needed for many-to-many networks. For more information about this, I refer to karpathy's blog and Brownlees blog.
I applied the second LSTM layer before the concatenation, with this I applied two LSTMs to capture the joint meaning of the sentences. That is not the intention of the DSCNN, so we have to apply the one LSTM layer after the concatenation

So far the code looks like:
sentence_inputs = [Input(shape=(max_sentence_len, embedding_dim,), name="input_" + str(i))
                   for i in range(max_sentences_per_doc)]

# LSTMs and Average Pooling (sentence-level)
shared_sentence_lstm = LSTM(units=embedding_dim, return_sequences=True, activation='tanh')
shared_average_pooling = AveragePooling1D(pool_size=max_sentence_len)
sentence_modeling = [shared_sentence_lstm(sentence_inputs[i]) for i in range(max_sentences_per_doc)]
sentence_modeling = [shared_average_pooling(sentence_modeling[i]) for i in range(max_sentences_per_doc)]

doc_modeling = Concatenate(axis=1)(sentence_modeling)
doc_modeling = LSTM(units=embedding_dim, activation='tanh', return_sequences=True)(doc_modeling)

The convolutional layer with multiple filter sizes has to be applied "manually", therefore we run the convolutions in a loop and concatenate the resulting layers after we flattened them (you could also apply GlobalMaxPooling, then you would not have to flatten the layer, but with this you would lose many information of your feature vectors).
conv_blocks = []
for sz in kernel_sizes:
    conv = Convolution1D(filters=filters,
                         kernel_size=sz,
                         padding="valid",
                         activation="relu",
                         strides=1)(doc_modeling)
    conv = MaxPooling1D(pool_size=2)(conv)
    conv = Flatten()(conv)
    conv_blocks.append(conv)
doc_modeling = Concatenate()(conv_blocks) if len(conv_blocks) > 1 else conv_blocks[0]

After this we can normally apply an activation layer (e.g. softmax or sigmoid, with or without dropout) and compile the model

Question 2 and 3: We need either an embedding directly in the batch-data or the embedding layer.
If you implement your network to have the embedding already in the batch data, you have to generate it only once. It works with small datasets and small embedding sizes (small depends on your memory size etc.), but with large datasets (e.g. 2000 words per document and an embedding dimension of 300) you rapidly exceed like 30gb of memory. Therefore it definitely makes sense to use the embedding layer, since it stores each word vector only once. The downside with this embedding layer is, that it has to perform the word index to word vector replacement after each batch, this costs a little bit of runtime, but since it is only a lookup operation this is not very expensive. So be encouraged to use the embedding layer.
Question 4:
It would not. Firstly, because of some parameter issues (e.g. Conv1D accepts only one integer for the kernel_size), secondly, it implements another architecture, that makes no sense (cf. question 1, TimeDistributed layer).
Question 5: No :-), see question 4


What can i do after a PCA with the results?

Well PCA, as suggested above by @CarltonBanks, does help you remove features with the least correlation and use mash the features together such that they have the highest correlation.

To answer your question, how to visualize higher dimensions using PCA

Transform the feature matrix with the number of components of your data set to 2 or 3
This ensures you can represent your dataset in 2 or 3 dimensions. To simply see your answer just plot this transformed matrix into a 2d or 3d plot respectively.
This helps you visualize a higher dimensionality data as a 2d or 3d entity so while using regression or some predictive modeling technique you can assess the trend of data.

Should we use PCA in machine learning algorithms more often?

Well, that strictly depends, using PCA reduces the accuracy of your data set so unless you need to save up some space caused due to a lot of features with bad correlation and the overall accuracy doesn't matter. If your machine learning model scenario is similar to this then it is ok to proceed.

However, most use of PCA is as you asked before for visualizing higher dimensionality data to determine the data trend and to check which model fits best.


stratify parameter will preserve the proportion of target as in original dataset, in the train and test datasets as well.

So if your original dataset df has target/label as [0,1,2] in the ratio say, 40:30:30. That is, for every 100 datasets, you can find 40, 30 and 30 observations of target 0,1 and 2 respectively.

Now when you split this original using the train_test_split(x,y,test_size=0.1,stratify=y), the  methods returns train and test datasets in the ratio of 90:10. Now in each of these datasets, the target/label data proportion is preserved as 40:30:30 for the classes [0,1,2].

Often, we want to preserve the dataset proportions for better prediction and reproduceability of results


How to fit a model to the V-shaped data?

After plotting out the density of the dependent varaible, flux, I would recommend exploring a Generalized Linear Model.

flux appears to be the output of a gamma distribution or perhaps a log-normal distribution. I would recommend exploring a GLM, especially with the lme4 package in R. GLMs are convenient transformations of the prototypical linear regression, and thus are easily interpreted. They are flexible and well-founded in theory as well.

Is there any work which might suggest a data generating process for your problem? You might ask yourself things you know about the process: is it true that x values are always greater than 0? Is there an upper bound?


Do precision-recall curves have a constant shape/pattern?

I wonder if PR curve(Precision-Recall curve) has a constant shape(pattern)?

For a model that is better than random (having a ROC above tpr = fpr line), which is mostly the case, precision and recall roughly have an inverse relationship. Therefore, we should expect that a (precision, recall) curve generally decreases (checkout this post on the relationship between precision and recall).

  Can I compute AUC of PR curve to compare which model is better?

Yes. It is even preferred over AUC (of ROC) when classes are imbalanced in the test set (for example, check out this blog, this Kaggle notebook, and this 2015 paper). Here is an image from the referenced blog

which shows that PRC (right), unlike ROC (left), is robust to imbalanced classes.

And here is a quote from the referenced Kaggle notebook (texts in brackets are added by me):

  ROC curve is not a good visual illustration for highly imbalanced
  data, because the False Positive Rate ( False Positives / Total Real
  Negatives ) does not drop drastically [i.e., does not produce a lower curve with smaller AUC] when the Total Real Negatives is
  huge [i.e., negative class is much larger than positive class].

  Whereas Precision ( True Positives / (True Positives + False
  Positives) ) is highly sensitive to False Positives and is not
  impacted by a large total real negative denominator.

which is illustrated as below

emphasizing on the black model (black curve) that is shown to be powerful in ROC  but not that much in PRC; given that negative to positive ratio is around 600 (the ratio should be 1 when classes are balanced).


Pandas loads a csv file incorrectly, but without throwing an error

You need to read the file with the index_col=False argument:
# wrong way:
df = pd.read_csv("IntPayments26102020.csv")
print(df.head())
# result :
                  Date     Travels  ...  Debt Services And Payments  Total
10/12/2003   234234.00     23424.0  ...                3.574570e+05    NaN
1/12/2005        12.00         2.0  ...                1.160000e+02    NaN
4/28/2006   7119901.48   9950000.0  ...                3.887479e+08    NaN
5/2/2006        346.00    345435.0  ...                4.632060e+05    NaN
5/12/2006    152204.22  11500000.0  ...                4.743224e+08    NaN

[5 rows x 11 columns]

# correct way:
df = pd.read_csv("IntPayments26102020.csv", index_col=False)
print(df.head())
# result :

         Date     Travels  ...  Debt Services And Payments         Total
0  10/12/2003   234234.00  ...                    23423.00  3.574570e+05
1   1/12/2005       12.00  ...                       21.00  1.160000e+02
2   4/28/2006  7119901.48  ...                 45602245.83  3.887479e+08
3    5/2/2006      346.00  ...                    34534.00  4.632060e+05
4   5/12/2006   152204.22  ...                 71560790.20  4.743224e+08

[5 rows x 11 columns]

From the docs:

index_col : int, str, sequence of int / str, or False, default None
Column(s) to use as the row labels of the DataFrame, either given as string name or column index. If a sequence of int / str is given, a
MultiIndex is used.
Note: index_col=False can be used to force pandas to not use the first column as the index, e.g. when you have a malformed file with
delimiters at the end of each line.

From where you can arguably guess why the default setting index_col=None will not work with your data - your rows are also separated by comma ,:
"Date","Travels","Inward Money Transfers","Cash Sales to BDC And Banks","Letters of Credit","Total Direct Remittances","Remittances to Travelex","Remittances to Amex","WDAS","Debt Services And Payments","Total"
"10/12/2003","234234","23424","23423","2342","342","23423","3423","23423","23423","357457",
"1/12/2005","12","2","12","12","21","12","12","12","21","116",


XGBoost (R Interface) throwing "label set cannot be empty" error

In python I needed to set the label param in  dtrain = xgb.DMatrix(X_train,label=y_train,feature_names=cfg_col_X)

PS I was going to comment but missing rep points.


Efficient algorithm to compute the ROC curve for a classifier consisting of an ensemble of disjoint classifiers

If I understood the question correctly, you have trained an algorithm that splits your data into $N$ disjoint clusters. Now you want to assign prediction $1$ to some subset of the clusters, and $0$ to the rest of them. And amont those subsets, you want to find the pareto-optimal ones, i.e. those who maximize true positive rate given fixed number of positive predictions (this is equivalent to fixing PPV). Is it correct?

This sounds very much like knapsack problem! Cluster sizes are "weights" and number of positive samples in a cluster are "values", and you want to fill your knapsack of fixed capacity with as much value as possible.

The knapsack problem has several algorihms for finding exact solutions (e.g. by dynamic programming). But a useful greedy solution is to sort your clusters in decreasing order of $\frac{value}{weight}$ (that is, share of positive samples), and take the first $k$. If you take $k$ from $0$ to $N$, you can very cheaply sketch your ROC curve.

And if you assign $1$ to the first $k-1$ clusters and to the random fraction $p\in[0,1]$ of samples in the $k$th cluster, you get the upper bound to the knapsack problem. With this, you can draw the upper bound for your ROC curve.

Here goes a python example:

import numpy as np
from itertools import combinations, chain
import matplotlib.pyplot as plt
np.random.seed(1)
n_obs = 1000
n = 10

# generate clusters as indices of tree leaves
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_predict
X, target = make_classification(n_samples=n_obs)
raw_clusters = DecisionTreeClassifier(max_leaf_nodes=n).fit(X, target).apply(X)
recoding = {x:i for i, x in enumerate(np.unique(raw_clusters))}
clusters = np.array([recoding[x] for x in raw_clusters])

def powerset(xs):
    """ Get set of all subsets """
    return chain.from_iterable(combinations(xs,n) for n in range(len(xs)+1))

def subset_to_metrics(subset, clusters, target):
    """ Calculate TPR and FPR for a subset of clusters """
    prediction = np.zeros(n_obs)
    prediction[np.isin(clusters, subset)] = 1
    tpr = sum(target*prediction) / sum(target) if sum(target) > 0 else 1
    fpr = sum((1-target)*prediction) / sum(1-target) if sum(1-target) > 0 else 1
    return fpr, tpr

# evaluate all subsets
all_tpr = []
all_fpr = []
for subset in powerset(range(n)):
    tpr, fpr = subset_to_metrics(subset, clusters, target)
    all_tpr.append(tpr)
    all_fpr.append(fpr)

# evaluate only the upper bound, using knapsack greedy solution
ratios = [target[clusters==i].mean() for i in range(n)]
order = np.argsort(ratios)[::-1]
new_tpr = []
new_fpr = []
for i in range(n):
    subset = order[0:(i+1)]
    tpr, fpr = subset_to_metrics(subset, clusters, target)
    new_tpr.append(tpr)
    new_fpr.append(fpr)

plt.figure(figsize=(5,5))
plt.scatter(all_tpr, all_fpr, s=3)
plt.plot(new_tpr, new_fpr, c='red', lw=1)
plt.xlabel('TPR')
plt.ylabel('FPR')
plt.title('All and Pareto-optimal subsets')
plt.show();

This code will draw a nice picture for you:

The blue dots are (FPR, TPR) tuples for all $2^{10}$ subsets, and the red line connects (FPR, TPR) for the pareto-optimal subsets.

And now the bit of salt: you did not have to bother about subsets at all! What I did is sorted tree leaves by the fraction of positive samples in each. But what I got is exactly the ROC curve for the probabilistic prediction of the tree. This means, you cannot outperform the tree by hand-picking its leaves based on the target frequencies in the training set.

You can relax and keep using ordinary probabilistic prediction :)


What algorithms are available to identify similarity scores between two records?

2 Approaches.

a) Siamese networks, let the net tell you whats similiar and what lies together.

b) just perform multiclassification if the data permits, I dont know all the details but it looks like it.


How can a NN guess the velocity of an object from a video source?

That is not possible without external labeled data (i.e., each video would have to be labeled with the object speed). Without labels, the neural network would be unable to learn the object's velocity because the object might be moving in any direction relative to the camera. For example, an object directly away would be getting smaller without motion blur.


Why would 2 sets of similar training samples take significantly longer to train?

There could be multiple factors that could cause this to happen:

If the data types are different for the two images - np.float32 vs np.float64
Your computer might be performing some compression and decompression to optimize memory usage. If most of the pixels are very different from each other in the Arabic dataset, then the compressed size may be larger and require more time to decompress.
Your computer happened to have other competing processes running at the same time and distorted the execution results for one of your runs.

I would suggest you to use methods like timeit that do multiple runs to verify that this behaviour is consistent.


Handling NA Values in the Chicago Crime Rate data set

Assuming Im reading the format of you table correctly, the NA's are exclusively for geo-location attributes. With "District" being the most complete field with only 47 missing values. Are the locations of certain crimes censored for data privacy reasons?

Of the 6 M crimes in your data set:

10 % or ~600,000 are missing or censored for Ward, Community area or name
1 % or ~60,000 are missing spatial coordinates.

You could determine most of the missing Ward and Community data by analysing the coordinate pairs in a GIS.

How you handle missing data will largely be determined by the hypothesis and analysis you intend to do. You mention Time Series, it should be fine to include all crimes for straight time series at the Chicago level. For higher spatial resolutions, you can do district (dropping 47 Missing points), and so on. How much effort you devote to handling the missing data, such as by using steps outlined above, will depend on how useful that data may be for your analysis. I suspect that most people working with this data tend to drop the NA's as they are mapping the crime rates in different areas.


Architectures that take inputs of mixed sampling rates

With a search online, I couldn't find a satisfying answer, so I came up with a way. I tested and it gave pretty nice accuracy. This method is inspired by this paper.
Similar to the approach in the paper, I first select the number of convolution kernels I want to use, and depending on the sampling rate of a data point, I apply convolution to it using the convex hull of the set of convolution kernels.
So how exactly do I come up with the weight? There are a lot of hyperparameters, but what I did is assign a sampling rate to each kernel by dividing up the expected range of sampling frequency. I then subtract the sampling rate of the input datapoint to see which kernel is the closest to it. Then I take a softmin of the absolute difference, which makes sure the resulting weights add up to 1.
There are many things to play around here: Do we uniformly assign the rates? How many kernels to use in a set? How do you calculate the weight?...


Gaussian Mixture Models as a classifier?

Some unsupervised models can make predictions, but not ones that necessarily match the original class labels. Once a GaussianMixture model has been fitted, it can predict which of the clusters a new example belongs to. This is exactly what the predict and predict_proba functions do in this case, and given that the number of clusters is set to 3, the number of classes, the predict function will predict a label from $\{0, 1, 2\}$.

However, this still raises the question of how does the GaussianMixture assign particular labels to the clusters? In general, it is arbitrarily chosen, but in the example from sklearn that you linked, they cheat when initialising the cluster centers:

# Since we have class labels for the training data, we can
# initialize the GMM parameters in a supervised manner.
estimator.means_init = np.array([X_train[y_train == i].mean(axis=0)
                                for i in range(n_classes)])

The initial position for each cluster center is at the center of each class, which also has the additional consequence of correctly ordering the cluster labelling to match the original class labels. This means that the GMM predicting which cluster a new instance belongs to is equivalent to predicting which class it might belong to in this case. I believe this has been done for easy visualisation of the different covariance matrix options.


There has/is been a lot of hype around the word "Big Data". So, as someone who has been setting up an analytics infrastructure at a fast growing company, I would like to share my experiences.

You would like to check this out, which I have written for an almost similar question on CV.

As the tools like H2O, R, Julia, Spark, etc; are being evolving for dealing and playing with Big Data, the size of the data is becoming a less of a problem these days.

As there is a lots of data available, and a wonderful set of tools at disposal, the art of analytics and data science is becoming more of a pleasure for data scientists rather than pain.

But, the real challenge is the availability of people who know what they are doing, who know how to use these tools for mining gold from the data.
Just anyone who has run a Linear Regression on a data set can't call himself a data scientist. One has to explain what the data is, what can be pulled out of it, and why is the algorithm applied on it, and why not something else.

How do they differ from a small amount of data?

In my experience, there isn't any huge differences. The only differences which I have found are algorithms which help find the needle in the hay stack, like MinHash, HyperLogLog, etc; which had to evolve for dealing with the size.


some algorithms to identify of a pattern in text data

Since you don't seem to have any annotated data, the best you can do is probably this:

Optional first step: remove stop words (there are many such lists available, for example https://pythonspot.com/nltk-stop-words/)
Calculate the Inverse Document Frequency for every word in the vocabulary. This is intended to measure the importance of a word based on how often it's used, in the sense that a word used in fewer sentences makes it more important for these sentences.
Calculate TF-IDF for every word in a sentence, then rank the words by their TF-IDF weights. In general the top ones are supposed to be the most relevant. Optionally you can decide to select the top N words as keywords.


Word Embedding for Pos tags can easily be trained using pos tags sequence. There are lot of ways that you can get the trained model. I did it by gensim's word2vec api. Here is the link to it:
https://radimrehurek.com/gensim/models/word2vec.html

Also if you want memory efficient solution, radim(creater of gensim) provides a great tutorial: https://rare-technologies.com/word2vec-tutorial/

You just need to pass pos sequence of your training data, resulting vector size, min frequency count etc. You can view api's documentation for more detail.


Sentiment analysis of tweets (Train model on a labelled dataset and use on some other unlabelled data)

The natural approach is to use a labelled dataset and a supervised learning technique. You can start with something simple, like using tf-idf for feature generation and train a simple logistic regression model.
I think this is the first thing you should try, I see it more likely to succeed than the unsupervised techniques, and it is simple enough.


Do non-parametric models always overfit without regularization?

No - non-parametric methods only means that the method does not assume a function form of the data. There are non-parametric methods such as Random Forest that do not always overfit. In fact nonparametric methods could underfit, it could lack the ability to fit the training data. An example of this would be a decision stump.


Forecasting Foreign Exchange with Neural Network - Lag in Prediction

A couple of ways to improve your design:

Consider a different normalization: The sigmoid function will attenuate large moves.  It is likely precisely these large non-linear moves that attracted you to using neural networks in the first place.  Why remove them?  A simple whitening of the data may be better
As pointed out by Nima, your model can only predict what is knowable from the data you give it.  If you are only fitting on data using historical prices, it will only give you what is predictable from them.  Things such as news events / earnings surprises / holidays / option market flows will not be fed into your model.  Consider adding these series as well.
Lots more historical data.  Neural networks often require very large sample sizes because they are trying to estimate a very large parameter space.  Although, more data doesn't always imply more information, this will still likely help.
Experiment with different network architectures.  The number of layers / size of the layers / different gradient decent algorithms / different activation functions / dropout etc..


Continuous variable not supported in confusion matrix

The confusion matrix is used to tell you how many predictions were classified correctly or incorrectly. You are looking at a regression model, which gives you a continous output (not classification).

So when you run confusion_matrix(y_test, y_pred) it will throw the ValueError because it expected class predictions, not floating point numbers.

Are you trying to predict classes, or really just a number output? If not, then you should not be using the confusion matrix.

If you want to predict e.g. 1 or 0 for your y values, then you would have to convert your linear regression predictions to either of these classes. You could say any value in y_pred above 0.7 is a 1 and anything below is 0.

cutoff = 0.7                              # decide on a cutoff limit
y_pred_classes = np.zeros_like(y_pred)    # initialise a matrix full with zeros
y_pred_classes[y_pred > cutoff] = 1       # add a 1 if the cutoff was breached

you have to do the same for the actual values too:

y_test_classes = np.zeros_like(y_pred)
y_test_classes[y_test > cutoff] = 1

Now run the confusion matrix as before:

confusion_matrix(y_test_classes, y_pred_classes)

which gives output:

array([[2, 3],
       [0, 0]])


Why does MAE differ after prediction (Neural Network)?

Your MAE_val is:

MAE_val= np.mean(np.absolute(y_val - pred_val ))

On the other side, you fit your model on:

history = model.fit(x_train, y_train, ...

So you are calculating them on different objects. Training data on one side, and validation set for a final evaluation.


how to make moving moving graphs that vary in time like a video?

You can find a list of great library here.
I would recommend this one but you have to write some css to have that same effect.


What is generative and discriminative model? How are they used in Natural Language Processing?

Let's say you are predicting the topic of a document given its words.

A generative model describes how likely each topic is, and how likely words are given the topic. This is how it says documents are actually "generated" by the world -- a topic arises according to some distribution, words arise because of the topic, you have a document. Classifying documents of words W into topic T is a matter of maximizing the joint likelihood: P(T,W) = P(W|T)P(T)

A discriminative model operates by only describing how likely a topic is given the words. It says nothing about how likely the words or topic are by themselves. The task is to model P(T|W) directly and find the T that maximizes this. These approaches do not care about P(T) or P(W) directly.


Independence of Features assumption in Naive Bayes

Statistical independence is a pretty straightforward thing. If $$p(A\cap B) = p(A) p(B)$$ then $A$ and $B$ are independent (in other words if marginal distributions are equal to conditional). If you want, you could even check that on your data. Though it would be easier to check:
$$p(A|B) = p(A) \ \text{and} \  p(B|A) = p(B)$$ instead of constructing a joint distribution. The latter is easy, if your features are categorical then you could estimate  $p(A)$, $p(B)$ , $p(A|B)$, $P(B|A)$ as sample frequencies. If one of A or B is categorical computations are also simple. If both A and B are numeric, you need to fit a KDE (kernel density estimation) model to all probability distributions.
However, on practice it is simpler and faster just to fit a Naive Bayes and check its performance on a test set.


Reason for better performance of variants of SGD when local minimas of Neural Nets are equivalent?

I believe reason for using these variants of SGD is to solve "bad" local minima problem

That is not accurate. The variants are mostly about accelerating steps of gradient descent when faced with shallow or rapidly changing gradients, or with gradients that need adaptive learning rates because some parts of the network get stronger gradient signals than others. They do this by weighting or adjusting the step sizes in each dimension, based on additional knowledge to the current gradients, such as the history of previous gradients.

Shapes like saddle points or curving gullies, in the cost function "landscape" can cause difficulties for basic SGD. Take a saddle point as an example - there is no "bad" minima, a saddle point can be quite high up in a cost function. But the gradient values can be very low, even if they pick up again if you can take steps away from the saddle point. The trouble for SGD is that using just the gradient is likely to make updates oscillate up and down the steep parts of the saddle, and not move in the shallow direction away from the saddle point. Other difficult shapes can cause similar problems.

For a visualisation of the difference in behaviours of some of the optimisers at a saddle point, take a look at this animation (I'd like to find a creative-commons variant of this and include in the answer), and a blog that references it.

In addition, for deep learning network, there is a problem that gradients can "explode" or "vanish" as you work back through the layers. Using ReLU activation or similar can help with that, but is not always possible (think of RNNs which need sigmoid activations inside LSTM modules). An optimiser like RMSprop deals with this by normalising gradients used by the weight update steps based on recent history. Whilst SGD could more easily get stuck and fail to update the lower layer weights (weights closer to the input) - either not updating them much at all, or taking too large steps.


In fact, you use 1D convolution.
Given that the dimension of the output of embedding layer is 100, that the kernel size is 5, and that the number of filters is 128,
You have 100x5x128 = 64000 weights. Add to this 128 biases and you get 64128 parameters.

Note that parameter sharing is used, so that there is only one set of weights and biases per filter, in depth.


How to visualize categorical data over time?

I'll give it a go, but a lot depends on how you want to communicate your results.

try to keep the buckets, and add counts to them, so you end up with something like:

date bucket count
2015-09-11 2-6 6
2015-09-12 2-6 9
2015-09-13 2-6 2
2015-09-11 10-16 9
2015-09-12 10-16 15
2015-09-13 10-16 4

This you can communicate either via a line chart, but I would go with side by side bar charts per period, maybe aggregated to week or month depending your timeline. If you have many ranges, splitting them in different graphs.

Don't index the ordinal values. The reader of the graph should not look somewhere else to figure out what the numbers mean. Just use counts with the ordinals and use line charts or bar charts, summary tables or a cross(table)
Count the occurences and use side-by-side or stacked bar charts and color the bars according to the colors you have. But human perception will always give a higher bar / line / area more weight than a smaller / lower one. Or just represent it as a table. That way you don't give any opinion except that some colors might occur more frequently than other.


The following code worked, thanks for the assistance.

chocolateData$Cocoa.Percent = as.numeric(gsub("[\\%,]", "", chocolateData$Cocoa.Percent))


Imagine you have 20 coefficients to test for and also have target accuracy (or whatever metric you're interested in) that you aim to beat. It acts as a threshold.

One tradeoff could be that performing "backwards regression" means you would in theory start with you maximum accuracy and be decreasing each time you remove a parameter. You might cut your threshold after removing only 3 parameters! So don't necessarily need to continue.

Conversely, starting with a single parameter and working your way up, you would have to try 17 models 5o reach your threshold.

This scenario could of course be reversed, so it is important to understand your data and make an informed decision about what suits your situation best.


How can you build a model that extracts data out from receipts?

The simplest pipeline would be to do the following:

OCR
Named Entity Extraction
Entity Disambiguation

OCR

This is basically transforming your receipts into plain text.

If you have scans (pictures) of the receipts, then you need a method that can deal with images. For example, you could use tesseract.

Named Entity Extraction

This is detecting the parts of interest in the text. This includes detecting dates, prices, currency, locations, names, and so on. Once you detect a named entity, for example: price=2.59, you still don't know if that is the unit cost, or total cost after tax, or total cost before tax, but that will come in the next step. All you know that this is an area of interest.

There are tools that are able to detect most named entities that you might want to use. For example: spaCy. The cool thing about spaCy is that you can train it to recognize more named entities that you might want.

Entity disambiguation

This is identifying the purpose of the named entity that you extracted. For example, given price=2.59, is it the unit cost, or total cost after tax, or total cost before tax.

This is where rules might come in handy, because it will be difficult to teach a machine learning model to identify them. You can have rules like:

highest value = total cost after tax
unity quantity × unit price = total cost before tax
total cost before tax × 1.21 = total cost after tax
and so on...


Parameters optimization algorithms in Weka

Supposing that you refer to the performance of tuning algorithms as the improvement in algorithm performance (accuracy, error, etc.). The effectiveness of parameter optimization (or tuning) is different for different algorithms. Many papers have discussed it, and Olson et al. have shown how much performance may vary for several algorithms. SVM and LR (those that you mentioned in your question) were not improved so much after tuning.

If you have bad performance in several algorithms you tried, the problem may not be in the tuning algorithms, but in your features-target set. I suggest you take a look at the Domingos paper, it's a nice read to build a successful model.


What is an efficient clustering approach for grouping multivariate timeseries data into optimally persistent states?

You can trivially modify DBSCAN (see GDBSCAN) to take time into account rather than dropping the precious time information.

I.e. neighbors are those points that are within a time difference of dt, and that have a minimum similarity.

Apart from that, classic "change in mean" analysis techniques for time series come to my mind. Maybe also changepoint detection.


What's the difference between Sklearn F1 score 'micro' and 'weighted' for a multi class classification problem?

F1Score is a metric to evaluate predictors performance using the formula

  F1 = 2 * (precision * recall) / (precision + recall)

where

  recall = TP/(TP+FN)
  and
  precision = TP/(TP+FP)

and remember:

When you have a multiclass setting, the average parameter in the f1_score function needs to be one of these:

'weighted'
'micro'
'macro'

The first one, 'weighted' calculates de F1 score for each class independently but when it adds them together uses a weight that depends on the number of true labels of each class:

$$F1_{class1}*W_1+F1_{class2}*W_2+\cdot\cdot\cdot+F1_{classN}*W_N$$

therefore favouring the majority class.

'micro' uses the global number of TP, FN, FP and calculates the F1 directly:

$$F1_{class1+class2+class3}$$

no favouring any class in particular.

Finally, 'macro' calculates the F1 separated by class but not using weights for the aggregation:

$$F1_{class1}+F1_{class2}+\cdot\cdot\cdot+F1_{classN}$$

which resuls in a bigger penalisation when your model does not perform well with the minority classes.

The one to use depends on what you want to achieve. If you are worried with class imbalance I would suggest using 'macro'. However, it might be also worthwile implementing some of the techniques available to taclke imbalance problems such as downsampling the majority class, upsampling the minority, SMOTE, etc.

Hope this helps!


What Non-linearities are best in Denoising RNN Autoencoders and where should the go?

I have some familiarity with Recurrent encoder-decoder archtectures, especially with seq2seq models. This is a standard implementation:

from tensorflow.keras import Sequential

from tensorflow.keras.layers import Bidirectional, LSTM
from tensorflow.keras.layers import RepeatVector, TimeDistributed
from tensorflow.keras.layers import Dense

from tensorflow.keras.activations import elu, relu

seq2seq = Sequential([

    Bidirectional(LSTM(len_input), input_shape = (len_input, no_variables)),

    RepeatVector(len_input),

    Bidirectional(LSTM(len_input, return_sequences = True)),

    TimeDistributed(Dense(100, activation = elu)),

    TimeDistributed(Dense(1, activation = relu))

])

where len_input is the length of the input trends, and np_vars the number of variables that compose each observations (if it's a univariate series, set it to 1). The choice of Bidirectional() wrappers is optional.

The model works like this: The input LSTM layer encodes the sequential information to a vector containing a latent representation. This vector is replicated by the RepeatVector() layer, its signal is then calculated using Dense() layers, and distributed for each timestep prediction sequentially (by the TimeDistributed() wrapper). You don't worry about non-linearities at this point, since this is learned by internal LSTM states and by Dense() vectors. Fancier implementation would require stacked Recurrent layers, or deeper feed-forward blocks.

If your goal is to build a Denoising Autoencoder you should pair each input series to its "clean" counterpart, so that the Network learns how to reduce noise.


How to save/load a Model (Pickle) with a specific path/directory

The "\" escapes the following sign when parsed, so the path cannot be read.

Use "/" instead and it should work. Also this question was probably more of a SO question ;)


You might want to have a look in XLNet.
In XLNet you can input a sentence, then take the state and inject it into a new run if the XLNet with a second input sentence, inject it into a third one with a third sentence and so on.
This, in essence, allows you to process text of virtually unlimited length.
Please see also this.


How to grid search feature selection and neural network hyperparameters in the same grid?

I see two solutions:

either you pass a list of dictionnaries to param_grid avoiding irrelevant combinations
or you use a single variable in your pipeline for feature_selector__feature__selector_k and classifier__input_shape

First solution:  you can generate the right list of combinations using something close to this:
param_grid = [
    {
        'feature_selector__feature_selector__score_func' : [f_classif],
        'feature_selector__feature_selector__k' : [k],
        'classifier__input_shape' : [k],
        'classifier__dropout_rate' : [0.0, 0.5]
    }
    for k in [7, 9, 15]
             ]

Second solution, you can use a specific class that create your model when fitting based on the shape of X. Here is a code sample:
class MyKerasClf():
    def predict(self, X):
        y_pred_nn = self.clf.predict(X)
        return np.array(y_pred_nn).flatten()

    def create_model(self, learn_rate = 0.01, weight_constraint = 0 ):
        model = Sequential ()
        model.add (Dense (units = 64, activation = 'relu',
                       input_shape = (self.input_shape, )))
        model.add (Dropout (self.dropout_rate))
        model.add (Dense (32, activation = 'relu'))
        model.add (Dense (1, activation = 'sigmoid'))
        model.compile (loss = 'binary_crossentropy',
                    optimizer = Adam (lr = learn_rate),
                    metrics = ['accuracy'])
        return model

    def fit(self, X, y, **kwargs):
        self.input_shape = X.shape[1]
        self.clf = KerasClassifier(build_fn = self.create_model, verbose = 2)
        self.clf.fit(X, y, **kwargs)

    def set_params(self, **params):
        if 'dropout_rate' in params:
            self.dropout_rate = params['dropout_rate']
        else:
            self.dropout_rate = 0.0

Then you can use the class in your pipeline
X, y = make_classification(n_features=50, n_redundant=0, n_informative=2,
                           random_state=42, n_clusters_per_class=1)

my_scaler = StandardScaler ()
steps = list ()
steps.append (('scaler', my_scaler))
standard_scaler_transformer = Pipeline (steps)

my_feature_selector = SelectKBest ()
steps = list ()
steps.append (('feature_selector', my_feature_selector))
feature_selector_transformer = Pipeline (steps)

# Create a specific clf
my_clf = MyKerasClf( )
pip_clf = Pipeline (steps = [('scaler', my_scaler),
                         ('feature_selector', feature_selector_transformer),
                         ('classifier', my_clf)],
                verbose = True)

param_grid = {'feature_selector__feature_selector__score_func' : [f_classif],
              'feature_selector__feature_selector__k' : [7, 15],
              'classifier__dropout_rate' : [0.0, 0.5]
             }
cv = RepeatedStratifiedKFold (n_splits = 5, n_repeats = 1, random_state = 42)
grid = GridSearchCV (estimator = pip_clf, param_grid = param_grid, scoring = 'f1',
                     verbose = 1, n_jobs = 1, cv = cv)
grid_result = grid.fit(X, y)

Note: I although added the dropout to be tested in the gridsearch as an example.


TS-SS and Cosine similarity among text documents using TF-IDF in Python

Normalised vectors have magnitude 1, so it doesn't matter if you explicitly divide by the magnitudes or not. It's mathematically equivalent either way.

I see no reason that you couldn't use normalised vectors in TS-SS, but it seems that the main motivation for using TS-SS in the first place is that it makes more sense for vectors that may have different magnitudes. I would try both cosine similarity and TS-SS for your problem, and see if there is a noticeable performance difference.


So, I understand from this question that you want to understand the grammatical structure of these messages, as well as the sentiment.

In terms of grammatical structure, this depends on how far you want to parse the grammatical structure, whether that is as simple as Part of Speech (POS) tagging (Noun, Verb, etc.) or if you need to parse the sentences as effectively parse trees, which show how each word depends on others.

For POS tagging, you can use nltk's POS tagging methods (link: https://www.nltk.org/book/ch05.html)

To get an idea of how each words relates to each other, one way to do it would be dependency parsing (principles are shown here: https://medium.com/@5hirish/dependency-parsing-in-nlp-d7ade014186)

In terms of sentiment analysis, there are many ways you can approach this. One approach I can think of from the top of my head would be using a encoder-decoder model architecture.
In this type of architecture, we firstly encode the input text as a sequence of tokens (using something like an RNN / LSTM) into a 'hidden representation'. Then we decode this 'hidden representation' using for example a standard neural network where we have a final softmax output layer, which gives a probability distribution over sentiment classes. The sentiment is then chosen as the index with the highest probability.
Here is a really good paper, which talks about sentiment analysis using an encoder-decoder architecture, which could be of some use: https://www.aclweb.org/anthology/Q18-1002/


Best MIMO prediction algorithm for categorical variables

You are trying to predict random numbers here. By design the lottery numbers are unrelated and one round has no impact on the next round. For machine learning you should use input that has some kind of "predictive power". I mean data that as some information or correlation with the target variable. You can of course train a prediction algorithm on that data like that, but it does not make sense to me. In case you find something that is a solid prediction, that might mean the lottery does not draw random numbers.
...this actually should be a comment, but StackExchange does not let me write a comment.


When in doubt, bootstrap.

Make a list with 970 0s (the non-sales at \$10) and 30 10s (the sales at \$10) - customers_10
Make a list with 975 0s (the non-sales at \$20) and 25 20s (the sales at \$20) - customers_20
Repeat lots of times (maybe N = 100,000 times - increase this until the the result settles down)

Randomly take 1000 samples (with replacement) from both lists - samples_10, samples_20
Sum the samples for each list - sum_s10, sum_s20
Record which one generated more revenue (i.e. whether sum_20 > sum_10)

The percentage of times sum_s20 was greater than sum_s10 is your X

(Another approach, which would be equivalent in the limit of N large, would be to model this as a binomial process. But processing is cheap, bootstrapping is robust and easy to implement - getting the binomial model right would be easier to mess up).


how to predict an image using saved model

Having problems with dimensions is very common. As others have mentioned, the method predict expects to get a batch of images. You should run model.summary() to see what the expected dimensions of the input are. The batch size itself might have been designed to be flexible during training, by specifying None in the first dimension on the model input_shape parameter.

Without knowing more of the dimensions, I cannot say exactly where your problems lies. You could include that information by editing your question.

It will likely help to have a look at the documentation for predict function of the model objects.

There is an argument: batch_size, which defaults to 32 if not fixed by the model itself, which you can see from the model.summary(). If you set this equal to 1, perhaps you will get a prediction.

Below is a modified version of your code that I would expect to return a prediction.

from keras.models import load_model
from keras.preprocessing import image
import numpy as np

# dimensions of our images    -----   are these then grayscale (black and white)?
img_width, img_height = 313, 220

# load the model we saved
model = load_model('model.h5')

# Get test image ready
test_image = image.load_img('/Images/1.jpg', target_size=(img_width, img_height))
test_image = image.img_to_array(test_image)
test_image = np.expand_dims(test_image, axis=0)

test_image = test_image.reshape(img_width, img_height*3)    # Ambiguity!
# Should this instead be: test_image.reshape(img_width, img_height, 3) ??

result = model.predict(test_image, batch_size=1)
print result

Note: there is ambiguity in the dimensions where I highlight it. A colour image will have three dimensions: (height, width, 3). Black and white will have only two dimensions: (height, width).


forecasting - likelihood of customers participating in next month sales

Forecasting is about predicting a variable as a function of time, such as the number of sales during coming months). Your problem is a regression problem, where the output is between 0 (no chance) and 1 (certain).

Any method to solve a regression problem is valid, but I would recommend starting with the simplest thing you think may work, to establish a baseline performance that you can try to improve.

This problem is often phrased as churn prediction, which is just the opposite chance from what you need (although definitions of churn vary), so you can google and see how people do that.

One easy approach that allows you to calculate this is to use the RFM model, where R, F and M stand for recency (time since last purchase), frequency (of purchases) and monetary (total money spent). You could use these features and a logistic regression as your baseline model.


Reducing MAE or RMSE of linear regression

You can try things like outlier removal, reducing data skewness, or stacking different models. For an advanced regression guide, checkout this kernel from Kaggle.


Create new column in Orange using Python Scripting Widget

You'd think it would be basic. So far I'm not finding it to be so. Here's one example that worked for me. I'm sure it can be much simpler.:
from Orange.data import ContinuousVariable, DiscreteVariable, Domain

out_data = in_data.copy()
var1 = ContinuousVariable("NewCol")
domain = out_data.domain
new_domain = Domain(attributes=domain.attributes + (var1,), metas=domain.metas, class_vars=domain.class_vars)
out_data = out_data.transform(new_domain)

i2 = out_data.domain.index(out_data.domain["NewCol"])
i1 = out_data.domain.index(out_data.domain["ExistingColumn"])
out_data.X[:, i2] = out_data.X[:,i1] * 2


Is multi-device trajectory prediction possible using only classical machine learning?

Any machine learning algorithm could possibly learn a solution once the problem properly defined.
One technique that could aid in learning is to bin the locations. Instead of predicting a specific x and y (and being precisely wrong), predict a more general region. Based on the topological structure of the location, bin similar location points together.
One class of classic machine learning techniques to try would be tree-based, such as decision tree, random forest, or boosted trees.


Getting parameters of the best model with crossvalidation in with SparkMLLib

It's not the most elegant solution but you can use the following to at least zip together evaluation metrics and hyper parameters

zip(cvModel.avgMetrics, paramGrid)


Do grid search on the same Level of "imbalancedeness" that you plan/are able to do your Training and Evaluation on.
So that means that if you saw that imbalanced data set does not skew your model predictions or results in other unwanted Outcomes, done use the maximal dataset possible. But on the other Hand if your model is strongly overfitting because of imbalanced dataset then optimisation with grid search will make him overfit more in that direction.


Transfer Learning for CNNs and Batch Norm Layers

but what was being trained when training=True?

Let's try to understand BatchNormalization(BN) layer first as it has more elements.
TL;DR -
γ, β are learned. These are initialized just like normal weights and learned in Backpropagation.
May read this crisp and spot-on answer on these parm Stat.SE

Formally, BN transforms the activations at a given layer  x  according to the following expression:
 BN(x)= γ⊙(x−μ)/σ + β
coordinate-wise scaling coefficients  γ  and offsets  β.
[Quoted - http://d2l.ai/]

each BN layer adds four parameters per input: γ, β, μ, and σ (for example, the first BN layer adds 3,136 parameters, which is 4 × 784). The last two parameters, μ and σ, are the moving averages; they are not affected by backpropagation, so Keras calls them “non-trainable”. However, they are estimated during training, based on the training data, so arguably they are trainable. In Keras, “non-trainable” really means “untouched by backpropagation.”
[Quoted - Hands-on machine learning with scikit-learn keras and tensorflow, Aurélien Géron

training=True: The layer will normalize its inputs using the mean and variance of the current batch of inputs.
training=False: The layer will normalize its inputs using the mean and variance of its moving statistics, learned during training. [Quoted - Keras doc for BN]

So, if you will not set it False it will continue updating μ and σ with every batch of test data example and normalize the output accordingly. We want it to use the values from the Training phase.
By default, it is False and fit method set it to True.

Dropout
Dropout is simpler of the two. We need this Flag here so that we can compensate(during testing) the loss to the Output value(on an average basis) due to the switched-off(during Training) Neurons.

Suppose p = 50%, in which case during testing a neuron would be connected to twice as many input neurons as it would be (on average) during training. To compensate for this fact, we need to multiply each neuron’s input connection weights by 0.5 after training. If we don’t, each neuron will get a total input signal roughly twice as large as what the network was trained on and will be unlikely to perform well. More generally, we need to multiply each input connection weight by the keep probability (1 – p) after training. Alternatively, we can divide each neuron’s output by the keep probability during training (these alternatives are not perfectly equivalent, but they work equally well)
[Quoted - Hands-on machine learning with scikit-learn keras and tensorflow, Aurélien Géron

On the example of Models difference
Though, these are subjects to try and check.
 But I believe generally we start to fine-tune when we believe that the upper layer is smoothened to match with the initial layers, to avoid a large flow in forward and backdrop. So the logic stated to keep it Flase in 2019 example might not hold too strong every time.


Are there libraries similar to cleverhans for different types of classifiers?

Try foolbox

  Foolbox is a Python toolbox to create adversarial examples that fool neural networks. It requires Python, NumPy and SciPy.

Also, join the google group of cleverhans, it's really good. Plus Ian Goodfellow is there along with Nicolas Papernot.


How to build confusion matrix , when predicted value and actual value is in sentence?

You could create 2 new columns (1 for the truth, 1 for the prediction) which have a more generalised value like PROCESS_RESUMED. You can then use those columns to create the confusion matrix.


Polynomial Regression Not Overfitting as Expected

Good news is that your code seems to be theoretically correct.
Bad news is that the numpy.linalg.pinv has numerical stability problem when fitting polynomials of high order. The problem is driven by decreasing determinant of the AT*A matrix.
Alternative approach would be to use numpy.linalg.lstsq which tend to provide slightly better results.
However, according to my tests with your sample data (shown below), even numpy.linalg.lstsq cannot consistently fit the points exactly as soon as number of points exceeds 10.
For example for only 15 points (and polynomial of degree 14) the determinant of AT*A decreases to ca 10E-150 making the matrix almost singular.
In the figure shown below I plot the log10 of determinant as a function of average distance between x-points across 1000 experiments: the pattern is very clear, the smaller the average distance the smaller the determinant. It appears plausible: the closer the x-points the more linearly dependent become the equations.
However, in practice, we are often interested in cases when the order of polynomials is significantly lower than the number of observations. In such cases the results are more numerically stable.

import numpy as np
import matplotlib.pyplot as plt
import math
import random
plt.rcParams['figure.figsize'] = [10, 5]
#*************************************************************
# added random seed for reproducibility
np.random.seed(123456)
#*************************************************************
##############################################################
def sample(n_samples):
    x = np.random.uniform(0, 1, size=n_samples)
    y = np.sin(2*np.pi*x) + np.random.normal(0, 0.3, n_samples)

    return [x,y]
##############################################################
def poly_regress(data, n_param):
    #coeff = np.zeros(n_param) # Coefficient vector initialization
    # Calculate the vandermonde matrix of input x
    vand_x = np.vander(data[0],N = n_param + 1, increasing=True)
    # Computes the coefficient vector via the normal equation
    coeff_pinv = np.linalg.pinv(vand_x.T @ vand_x) @ (vand_x.T @ data[1])
    #*************************************************************
    # added alternative more precise approach
    coeff_lstsq = np.linalg.lstsq(vand_x, data[1])[0]
    #*************************************************************
    pred_pinv = 0
    pred_lstsq = 0
    for i in range(n_param + 1):
       pred_pinv += coeff_pinv[i]*np.power(data[0], i)
       pred_lstsq += coeff_lstsq[i]*np.power(data[0], i)
    return np.array([
        np.average(np.diff(np.sort(data[0]))), np.linalg.det(vand_x.T @ vand_x),
        np.average(abs(pred_pinv - data[1])), np.average(abs(pred_lstsq - data[1]))])
##############################################################
# evaluate results
for num_pts in [3, 5, 15]:
    iter = 1000
    res = np.zeros(iter*4)
    res.shape = (iter,4)
    poly_order = num_pts - 1
    for ii in range(iter):
        # this function returns:
        # [0] average distance between x points: np.average(np.diff(np.sort(data[0])))
        # [1] determinant of the A^T * A matrix: np.linalg.det(vand_x.T @ vand_x)
        # [2] MAE of linalg.pinv-based approach
        # [3] MAE of linalg.lstq-based approach
        res[ii] = poly_regress(sample(num_pts), poly_order)
    plt.plot(res.T[0], np.log10(abs(res.T[1])), 'x', color='tab:grey', label= "log10(abs(Det))")
    plt.xlabel("avg(diff(sort(data[0])))")
    plt.legend()
    plt.title("Num data points: " + str(num_pts) + ", poly degree: " + str(poly_order) +
              ". Calculations repeated " + str(iter) + " times.")
    plt.show()
    plt.plot(res.T[0], (res.T[2]), 'rd', color='tab:red', label= "MAE linalg.pinv")
    plt.plot(res.T[0], (res.T[3]), 'rP', color='tab:blue', label= "MAE linalg.lstsq")
    plt.xlabel("avg(diff(sort(data[0])))")
    plt.legend()
    plt.title("Num data points: " + str(num_pts) + ", poly degree: " + str(poly_order) +
              ". Calculations repeated " + str(iter) + " times.")
    plt.show()


You can use the below type code.
import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, LSTM, Concatenate
from tensorflow.keras.models import Model

# input of first NN
input_l1 = Input(shape=(2,))
out_l1 = Dense(1)(input_l1)

# input is 2nd NN
input_l2 = Input(shape=(2,))
out_l2 = Dense(1)(input_l2)

# concat layer output shape will be (None, 2) becuase we concatinated 2 dense layer outputs/
concat_vec = Concatenate()([out_l1, out_l2])

# we need 3d input to LSTM i.e. ( Batch_Size, no of time steps, feature space)
# We have 2 inputs so expanded dim to (None, 2, 1)
expanded_concat = tf.expand_dims(concat_vec, axis=2)

# LSTM
lstm_out = LSTM(15)(expanded_concat)

model = Model(inputs=[input_l1, input_l2], outputs=lstm_out)


The hierarchy of methods for thresholds are:

Hand-coded thresholds
Learned thresholds with a decision tree
Learned thresholds with an ensemble of trees (e.g., Random Forest or XGBoost)
Learned thresholds with neural networks


How to measure the correlation between categorical variables and a continuous variable

The Chi-squared test measures the relationship between two categorical variables.

To measure the relationship between a categorical feature and a continuous feature, you can use an ANOVA test.

As an aside, you don't need to use a for loop for your encoding - get_dummies has the parameter columns which allows the user to specify which columns are encoded.


sklearn CountVectorizer token_pattern -- skip token if pattern match

Found this SO post which says to use the following regex:

\b[^\d\W]+\b/g

yielding the following:

>>> vec = CountVectorizer(token_pattern=r'\b[^\d\W]+\b')
>>> X = vec.fit_transform(docs)
>>> pd.DataFrame(X.toarray(), columns=vec.get_feature_names())
   aaa  is  more  some  text  this
0    0   1     0     1     1     1
1    0   0     0     0     0     0
2    1   0     1     0     0     0
3    0   0     0     0     0     0

What I needed in my regex were the \b word boundary characters of which I was not aware of. That does make this a misplaced question as it has nothing to do with data science or that discipline's tools (sklearn).


As the algorithm should not change the order of the lists you could just add the clusters list

 cities["cluster"] = cluster

If you are really paranoid you can add your input parameters a second time to the dataframe in the same way and compare the diff in values (should be 0).


Is Vector in Cosine Similarity the same as vector in Physics?

As you ask specifically for the Cosine Similarity technique, it has magnitude and direction, and similar to a vector which is used in Physics, as Cosine Similarity deals with vectors in an inner product space.

So, the magnitude of vectors is exactly the same as the formula in Physics (summating over the squares of the vector elements.)


How to prevent model from recognizing false Classes

I haven't actually played around with this problem, but here are a few resources:

AI.se question - Why not teach to a NN not only what is true, but
also what is not true?
How to Make a Neural Network say "Don't Know"? (maybe try
requesting fulltext through ResearchGate )
blog post - Making Your Neural Network Say “I Don’t Know” — Bayesian
NNs using Pyro and PyTorch

Edit: I just found the following questions here on DS.se:

Multi Class + Negative Class Image Classification Strategies
Fine-tuning a CNN for recognizing two classes, but also being able to tell if none of them is present in an image


How to classify a new email as spam/not spam?

I modified your code so the code runs as a block and is setup to predict new data:
import string

from nltk.corpus import stopwords
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report,confusion_matrix, accuracy_score
from sklearn.naive_bayes import MultinomialNB

#  Define training data
df = pd.DataFrame(data={'Email': [
"Hi, I am Andrew and I want too buy VIAGRA",
"Dear subscriber, your account will be closed",
"Please click below to verify and access email restore",
"Hi Anne, I miss you so much! Can’t wait to see you",
"Dear Professor Johnson, I was unable to attend class today",
"I am pleased to inform you that you have won our grand prize.",
"I can’t help you with that cuz it’s too hard.",
"I’m sorry to tell you but im sick and will not be able to come to class.",
"Can I see an example before all are shipped or will that cost extra?",
"I appreciate your assistance and look forward to hearing back from you.",],
'Spam': [1, 1, 1, 0, 0, 1, 0, 0, 0, 0]})

def fun(text):
    # Removing Punctuations
    remove_punc = [c for c in text if c not in string.punctuation]
    remove_punc = ''.join(remove_punc)

    # Removing StopWords
    cleaned = [w for w in remove_punc.split() if w.lower() not in stopwords.words('english')]

    return cleaned

# Create a vectorizer object to enable both fit_transform and just transform
vectorizer = CountVectorizer(analyzer=fun)
X = vectorizer.fit_transform(df['Email'])

X_train, X_test, y_train, y_test = train_test_split(X, df['Spam'], test_size = 0.25, random_state = 0)

classifier = MultinomialNB()
classifier.fit(X_train, y_train)

pred = classifier.predict(X_test)

print(classification_report(y_test ,pred ))
print('Confusion Matrix: \n', confusion_matrix(y_test,pred))
print()
print('Accuracy: ', accuracy_score(y_test,pred))

Here is how to predict on new data:
# Given a new email
new_email = "Hi, my name is Christopher and I like VIAGRA"

# Apply the same preprocessing steps and transformation
X_new = vectorizer.transform([fun(new_email)])

# Predict new email with already trained classifier
classifier.predict(X_new)


Sure, here is the github and here is the explanantion.


is there big difference between data Science , big Data and database?

Well, they are absolutely different things but that are somehow linked. I gonna go through each of them.

Data Base

Think of a data base (DB from now) like a computer which only purpose is to store data accesable to be read. By data, and focusing only in SQL-like DB, I mean basically tables of information like and excel file with columns and rows. You can think of a SQL DB like an ecosystem of excel tables which share some common field. So basically a DB is a hardware infrastructure which allows to write and read a given amount of information within it (in the very beggining they were plain computers, of course with the rise of internet specialized hardware appeared). You can build your own DB in your personal computer.

Big Data

" An intellect which at a certain moment would know all forces that set nature in motion, and all positions of all items of which nature is composed ... for such an intellect nothing would be uncertain and the future just like the past would be present before its eyes." P.S.Laplace

Laplace did not think over it very deeply before formulate his sentence, obviusly if god had come to him and give what he wanted soon or later he would have realized all that information was indeed useless for him. Where could him store all that? From where should him start to read? What can he do with such amount of information he could never end to compute? In case he could read everything what should he calculate first?

These all are questions Big Data tries to answer and find a solution to. Big data appeared together with the huge websites internet gave birth, such as Amazon or Google. At some point they need to store so many information that it was imposible to store it in a single computer, not even in a big one, so they need to use a set of computers for which previous standard technologies for DB didnt work anymore. This fact was also the seed for No-SQL DB.

More about Big Data and non-sql here:
http://www.kdnuggets.com/2016/07/seven-steps-understanding-nosql-databases.html

Data Science

Finally Data Science is a statistical science, which aim is to extract order out of chaos as any other science, however meanwhile the rest of the sciences are focused in a single "narrow" piece of knowledge, such as biology, chemistry, etc; data science is on the other hand multidisciplinar science that could face problems from a broaden origin. Examples would be marketing or business oriented, cosmology, etc.

So data science uses mathematical and computer science algorithms to provide some useful information out of a disordered set of data. And here is where the link with Big Data comes, actually is in the question is set before: What can he do with such amount of information he could never end to compute? So Data Science and Big data are a very usual marriage in most IT companies nowadays and in more specific fields day after day. But, data science is apply a set of mathematical algorithms to data (like apply a calculation in an excel file to create a new row) and big data is the technology to have a huge amount of excel files (I use the word "excel files" here is just to make it easier to grasp).


For some literature on the problem you may want to look at https://medium.com/sap-machine-learning-research/chargrid-77aa75e6d605 and https://arxiv.org/abs/1809.08799

For a commercially available solution Textract and Tika come to mind.


Any suitable way to describe the distributions of 2 Pandas Dataframes visually/graphically?

You can use the Kolmogorov-Smirnov Test. From Wikipedia

  In statistics, the Kolmogorov–Smirnov test (K–S test or KS test) is a
  nonparametric test of the equality of continuous, one-dimensional
  probability distributions that can be used to compare a sample with a
  reference probability distribution (one-sample K–S test), or to
  compare two samples (two-sample K–S test). It is named after Andrey
  Kolmogorov and Nikolai Smirnov.

Good for us, there is already an implementation for this in Scipy

You can use this dummy code to test it

from scipy import stats
p_value = 0.05
rejected = 0
for col in range(103):
    test = stats.ks_2samp(df1.ix[col,], df2.ix[col,])
    if test[1] < p_value:
         rejected += 1
print("We rejected",rejected,"columns in total")

I made the assumption that the columns that you want to compare are on the same index in both dataframes. If this is not true, you need to find another way. Maybe if they have the same name? You can do it like that.

from scipy import stats
p_value = 0.05
rejected = 0
for col in df1:
    test = stats.ks_2samp(df1[col], df2[col])
    if test[1] < p_value:
         rejected += 1
print("We rejected",rejected,"columns in total")

If the K-S statistic is small or the p-value is high, then we cannot reject the hypothesis that the distributions of the two samples are the same.

Edit: I haven't tried it before, but maybe you can do something like that

Boxplot

from scipy import stats
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#create 2 dataframes with random integers. I don't have data to simulate your case.

df1 = pd.DataFrame(np.random.randint(0,100,size=(10000, 102)), columns=range(1,103))
df2 = pd.DataFrame(np.random.randint(0,100,size=(10000, 102)), columns=range(1,103))

#apply the Kolmogorov-Smirnov Test

p_value = 0.05
p_values = []
for col in range(103):
    test = stats.ks_2samp(df1.iloc[col,], df2.iloc[col,])
    p_values.append(test[1])

#create the box plot

plt.boxplot(p_values)
plt.title('Boxplot of p-values')
plt.ylabel("p_values")
plt.show()

Heatmap

Another way is the heatmap.

import matplotlib.patches as mpatches

plt.rcParams["figure.figsize"] = 5,2
x = range(1,104)
y = np.array(p_values)
fig, (ax,ax2) = plt.subplots(nrows=2, sharex=True)

extent = [x[0]-(x[1]-x[0])/2., x[-1]+(x[1]-x[0])/2.,0,1]
im = ax.imshow(y[np.newaxis,:], cmap="plasma", aspect="auto", extent=extent)
ax.set_yticks([])
ax.set_xlim(extent[0], extent[1])

values = np.unique(np.round(y.ravel(),2))

colors = [ im.cmap(im.norm(value)) for value in values]
# create a patch (proxy artist) for every color
patches = [ mpatches.Patch(color=colors[i], label="Level {l}".format(l=values[i]) ) for i in range(len(values)) ]
# put those patched as legend-handles into the legend
plt.legend(handles=patches, bbox_to_anchor=(1.05, 2.2), loc=2, borderaxespad=0. )

ax2.plot(x,p_values)

plt.tight_layout()
plt.show()

Code for heatmap: stackoverflow


"DeepTree" network with binary classifiers at each node

You can try the SVM with your decision function :

https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC

it can handle muilt classfication problem with default ovr decision function

if you stick with tensorflow you can only try linver svc : https://www.tensorflow.org/api_docs/python/tf/contrib/learn/SVM


In theory, categories themselves can be handled in decision trees. However, most python implementations will only work with numbers, so you need to convert them. It looks like you already have. OneHot Encoding is one way to do that, but now that you've done that, it isn't necessary. You may try other schemes to turn them into 1 2 3, if there isn't a logical reason to how you have them this way. This is called Categorical Encoding. One popular way is to order them alphabetically and number them. Another way, in order of their frequency. At the end of the day, it won't matter a whole lot because the random forest will bin the number according to its algorithm. It might make minor differences though, so maybe try different ways. But no, you won't need to OneHot encode your categories because they are already numbers.


How to compute the Jaccard Similarity in this example? (Jaccard vs. Cosine)

Cosine similarity is for comparing two real-valued vectors, but Jaccard similarity is for comparing two binary vectors (sets). So you cannot compute the standard Jaccard similarity index between your two vectors, but there is a generalized version of the Jaccard index for real valued vectors which you can use in this case:

$J_g(\Bbb{a}, \Bbb{b}) =\frac{\sum_i min(\Bbb{a}_i, \Bbb{b}_i)}{\sum_i
max(\Bbb{a}_i, \Bbb{b}_i)}$

So for your examples of $t_1 = (1, 1, 0, 1), t_2 = (2, 0, 1, 1)$, the generalized Jaccard similarity index can be computed as follows:

$J(t_1, t_2) = \frac{1+0+0+1}{2+1+1+1} = 0.4$

Alternatively you can treat your bag-of-words vector as a binary vector, where a value $1$ indicates a words presence and $0$ indicates a words absence i.e. $t_1 = (1, 1, 0, 1), t_2 = (1, 0, 1, 1)$. From there, you can compute the original Jaccard similarity index:

$J(t_1, t_2) = \frac{2}{2+1+1} = 0.5$


Hyperparameter tuning for stacked models

Do not mix dividing the data into k-fold with cross validation.

You can use the 4 folds ( training data) to optimize the base classifiers. You can also find the best hyper parameters by applying cross validation on your training data, re-train using all training data ( the 4 folds) and then test using the last fold to generate the meta data.

After you finish generating the meta data you can now use all the data the 5 folds to train the base learners. The higher the convergence between the final set of base classifiers and the ones you used to generate the meta data the better. This is why the less the data you have the larger the k you need to go with and vice versa.

Finally, you use the meta data to train the meta classifier

Here is one last note to think about: using cross validation to optimize base learners my not be very beneficial, but why?
If you have access to N different training algorithms and you can use cross validation to optimize them and choose the best one ( the one that leads to a very low bias error), then there is a high probability that you don't need to use stacking. But stacking is very beneficial when all these N algorithms can not lead to 0 bias error (if you have variance error use bagging), it is a case that may confuse the cross validation algorithm. The other thing is using cross validation to optimize algorithms may lead to different algorithms that have similar biases, which reduces the benefits from the stacking technique.

you need to know that stacking, is one of the most tricky ensembles and this is why it is not well studied in literature in compare to bagging and boosting. In the other hand it is proven that it is very important, especially for practical purposes.


What is the difference between a (dynamic) Bayes network and a HMM?

From a similar Cross Validation question follows @jerad answer:

HMMs are not equivalent to DBNs, rather they are a special case of DBNs in which the entire state of the world is represented by a single hidden state variable. Other models within the DBN framework generalize the basic HMM, allowing for more hidden state variables (see the second paper above for the many varieties).
Finally, no, DBNs are not always discrete. For example, linear Gaussian state models (Kalman Filters) can be conceived of as continuous valued HMMs, often used to track objects in space.
I'd recommend looking through these two excellent review papers:

An Introduction to Hidden Markov Models and Bayesian Networks by Zoubin Gharamani
Dynamic Bayesian Networks by Kevin Murphy


It seems to be an error in the documentation as the target is validated to be a classification target.

 check_classification_targets(y)

The default label propagation would however just propagate the existing 'labels' i.e. regression values, so not much would be gained in this case.


There is GoogleNews package for Python. It allows queries by data range, keyword, and language.


Error using twitter R package's userTimlien

It seems to be working well on my configuration:

Ubuntu Vivid and R:

> sessionInfo()
R version 3.1.2 (2014-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] twitteR_1.1.8

loaded via a namespace (and not attached):
 [1] bit_1.1-12     bit64_0.9-4    bitops_1.0-6   DBI_0.3.1      httr_0.6.1
 [6] magrittr_1.5   RCurl_1.95-4.6 rjson_0.2.15   stringi_0.4-1  stringr_1.0.0
[11] tools_3.1.2

Maybe you should update packages versions?


What approaches are there to not display a search result that a user has no permission to see?

Since you aren't sure which software you would like to use, I can explain this concept only in general, by example. The clearest example is a content management system like Drupal or Wordpress (not necessarily a data science tool but the idea is the same).

In a content management system, there are different types of content which can be accessed. In your case, there would be two types - document type A and document type B. There can be an unlimited number of documents, each assigned one or more types.

The software has a permission system that gives access to each document type. This is built with roles. In your example, there would be a system administrator role, and a general user role.

Each role has permissions associated with it. You would configure system administrator to be allowed to view document types A and B, but you would configure general users to only be able to see document type A.

Each user registers an account in the software system. A site administrator assigns each account one or more roles. So a user with the system administrator role would be able to see all documents of type A and B. However, if a user only has the general user role, they would only be able to see documents of type A. Also noteworthy is the fact that each user can have multiple roles and will get all permissions associated with every role they have.

USER -has-> ROLE(s)

ROLE -has-> PERMISSION(s)

CONTENT TYPE -requires-> PERMISSION(s)

Every time the content is requested, the software system ensures that the user making the request has the permission to access the content.

When a search is run, the software system has to validate every potential result individually before displaying the search results to the user. The user must have the permission to see each piece of content that is be delivered back from the search.


Image classification if rotated version same

Just encode your images first and do the classification on the encoded data (Wavelet transformation). Parts of similar image regions will result in very similar encoded matrix. Further denoising the encoded matrix by zeroing the absolute values that are less or equal some small constant should reduce overfitting and improve your accuracy.

Here an example for a simple Haar-Wavelet transform encoder:

First we take the R values of the RGB-image and divide it into 8x8 matrices such as this one:
   $\begin{pmatrix}{97,95,99,94,97,88,90,95}\\{88,86,89,90,95,96,99,98}\\{95,99,94,91,88,85,90,94}\\{78,75,84,86,80,74,77,99}\\{71,75,78,70,71,84,91,90}\\{67,74,69,70,65,68,74,70}\\{59,55,64,68,70,84,86,90}\\{79,82,88,90,85,82,88,90}\end{pmatrix}$

Now we begin with the first row: $\begin{pmatrix}{97,95,99,94,97,88,90,95}\end{pmatrix}$

Grouping two adjacent values we obtain the 4 groups:

{97,95},{99,94},{97,88},{90,95}

Next step is to calculate the average of each group:

(96, 96.5, 92.5, 92.5)

And the difference of the second value of each group and the average:

95-96 = -1, 94-96.5 = -2.5, 88-92.5 = -4.5, and 95-92.5 = 2.5

Putting them together: (-1, -2.5, -4.5, 2.5)

Attaching them to the list of our average values we get:

(96, 96.5, 92.5, 92.5, -1, -2.5, -4.5, 2.5)

Now we repeat that process, but only using the first 4 values:

Grouping: {96, 96.5}, {92.5, 92.5}

Averages and differences: (96.25, 92.5, 0.25, 0)

One last time the same process but now only using the first 2 values:

Grouping: {96.25, 92.5}

Average and difference: (94.375, -1.875)

Putting it all together we obtain the new values of the first row, where we can see, that one value was already zeroed out:

(94.375, -1.875, 0.25, 0, -1, -2.5, -4.5, 2.5)

The next steps would be just applying the above process on the other 7 rows and then doing the same for every column of the resulting 8x8 matrix. Then doing the same on every other 8x8 part of our RGB-image's R-matrix and then doing the same for the G-matrix and B-matrix. The resulting RGB-matrices will be very sparse because of all the zeroed values. We got rid of all low entropy pixels what is left are the core features of the objects from the image. Using some simple similarity metrics will result in a high accuracy classifier.

The whole process can be very easy implemented using a simple recursive function, or by using linear algebra and matrix multiplications.
This is a lossless compression, meaning the resulted matrix still have all needed information to decode it and getting the original image.
However it is possible to apply some denoising routines to get more zeroes, but doing it, information will be lost.


Why is my LSTM, stateful network getting 0 accuracy

Your code is absolutely fine. Also, the model is working as you can see the loss is continuously decreasing.

The thing is, this is a regression problem and accuracy is used for classification tasks. It just calculates how often predictions match labels by comparing if the classes are the same. See here for more details on how it works. So, just remove accuracy from your metrics and you are good to go.


In model validation (log regression), can AUC for the test sample be higher than that for the train sample?

This is just not true. Let's say that your future data is way easier to classify, then it can happen than temporal splits imply higher AUC. It often happens that future data is harder to classify because of shifts in the distributions of the variables, but if distributions are stable I don't see why not.