InterquartileRange takes up most instances in data set

The InterQuartileRangeFilter from weka library uses an IQR formula to designate some values as outliers/extreme values. Any value outside this range $[Q_1 - k(Q_3-Q_1), Q_3 + k(Q_3-Q_1)]$ is considered some sort of an outlier, where $k$ is some constant, and $ IQR = Q_3 - Q_1$.

By default weka uses $k=3$ to define something as outlier, and $k=3*2$ to define something as extreme value (extreme outlier).

The formula guarantees that at least 50% value are considered non-outliers. Having a single variable (univariate sample of values), it's practically impossible to reproduce your result.

Note however that this filter can be applied to a data frame. When applied like this, it will consider as an outlier any instance of the data frame which has at least one value of the instance considered as outlier for that variable.

Now, supposing that you have a data frame with 2 variables, totally uncorrelated (independent). Considering again that only 10% of the values from each variable are considered outliers, due to independence, one can expect that $(1-0.9)^2$ values will not be outliers. If you have $p$ variables like that in your data frame, you might expect to have only $(1-0.9)^k$ normal values, and is not very hard to arrive in that situation.

There are two things which you will have to consider. One is to increase the factors for outliers if in general too many values are considered outliers (ideally you would like to take a look at each variable graphically and if possible to get some idea about the distribution beneath). The second one is to check if you have many values which are totally independent. The second hint does not solve your problem but might give you a reason why it happens.


To compare two LDA topics, you're really trying to compute the distance between two probability distributions.

One such measure that's commonly used in these circumstances is the Hellinger Distance. To find the closest match for $x_1$ in the topics for $y$, you would calulate the Hellinger Distance between $x_1$ and each $y$ topic, then take the lowest one.

Keep in mind that there's no guarantee whatsoever that the "most similar" topic in this sense would be remotely, subjectively similar.


LSTM forecasting on multivariate time series

LSTM should be a good fit for this problem(as you mentioned in question).  It can identify patterns in sequence of data (For example : high memory usage is a leading indicator of  higher disk usage [due to higher usage of swap partition]).
This is a tutorial for similar use-case.


It's not really possible to adress concept drift in general. But I can bring two similar answers for drift of houses prices :

As other prices the drift is usually well measured and studied. As one would correct price for inflation, one can correct past house prices with a housing index (typically this index for the US). It will help your model having prices that are comparable over years.

Another way to tackle drift is to consider a ratio with a relevant variable that has a similar drift. For housing price, that might be median income of the neighborhood. This will give you a variable that is less sensitive to the overall drift.

As you can see those two methods are pretty much equivalent here in practice, as it mainly consist in correcting features and eventually, targets. The main difference is that in the first case you talk about dollars directly which is often more business oriented. Application of those methods can get a bit difficult if you try to use your model to predict the future and need to project housing index or median wage.


How to calculate the different metrics for multi class classification

As you can see this MCC formula is for binary classification, so you can only calculate its results by considering the problem as binary.
[edited to clarify OP's confusion] What is a confusion matrix? It shows for every true class $X$ as a row and every predicted class $Y$ as a column how many instances have true class $X$ and are predicted as $Y$. If there are only two classes (binary classification), the only possibilities are

$X$ positive and $Y$ positive -> TP
$X$ positive and $Y$ negative -> FP
$X$ negative and $Y$ positive -> FN
$X$ negative and $Y$ negative -> TN

However when there are more than two classes (multiclass classification) it's impossible to use this distinction positive/negative directly, so there are no general TP,FP,FN,TN cases.
With multiple classes one can calculate binary classification metrics for every class. This is done by considering the target class as positive and all the other classes as negative (as if they are merged into one big negative class).
Example: suppose we have classes A, B, C. If we focus on class A, the confusion matrix is like this:
    A   B   C
A   TP  FN  FN
B   FP  TN  TN
C   FP  TN  TN

to present it another way:
         A    B or C
A        TP    FN
B or C   FP    TN

Now if we focus on class B the confusion matrix becomes:
    A   B   C
A   TN  FP  TN
B   FN  TP  FN
C   TN  FP  TN

In your code the TP and TN categories are swapped:
TP(i)=C(i,i);
...
TN(i)=sum(C(:))-TP(i)-FP(i)-FN(i);


Sentence embeddings with LSTM to classify the sentences is not working

The problem is that you are encoding the pieces of text as vectors and feeding those vectors to the model, but then the first layer of the model is again an embedding layer.
You should only use the embedding once: either you embed the text out of the model (train_embed=encoder(messages)) or you pass integer inputs to the model and them inside the model (X=embedding_layer(input)), but not both.
The problem is masked by the fact that the embedding layer does not raise an error when it receives an input of type float. Instead, it silently casts the input to integer.
Apart from the problem itself, there are some extra things that you should note:

depending on what you do, you may need to adjust the input to the LSTM to make it have the 3 dimensions it expects.
Here, the universal sentence encoder creates a fixed size vector for each sequence, and then you are feeding such fixed-size vector to an LSTM. It doesn't make much sense to have LSTMs process non-sequential data. I think you could just apply an MLP (i.e. a couple of dense layers with ReLU activations plus a final dense with the softmax activation) to the vector.
You are applying too much dropout for this case where there is almost no data.


Logistic Regression is rather a hard algorithm to digest immediately as details often are abstracted away for the sake of simplicity for practitioners. To explain the idea behind logistic regression as a probabilistic model, we need to introduce the odds ratio, i.e. the odds in favor of a particular event. The odds ratio can be written $\frac{p}{1-p}$, where p stands for the probability of the positive event. We can then further define the logit function, which is simply the logarithm of the odds ratio (log-odds). The logit function takes input values in the range 0 to 1 and transforms them to values over the entire real number range, which we can use to express a linear relationship between feature values and the log-odds:

$logit \frac{p}{1-p} = w_{0}x_{0} + w_{1}x_{1}...w_{n}x_{n}= w^{T}x $ (1)

In Logistic Regression we are interested in predicting the probability that a certain sample belongs to a particular class, which is the inverse form of the logit function. It is also called the logistic function, sometimes simply abbreviated as sigmoid function:

$p(Y=1|x) = \phi(z=W^Tx) = \frac{1}{1+e^{-z}}$ (2)

$ p(Y=0|x) = 1 - p(Y=1|x) $ (3)

Here it comes the tricky part, the parameter estimations i.e. w (see eq. 1). It basically states that a solution to a Logistic Regression problem is the set of parameters w that maximizes the likelihood which is defined via maximum likelihood estimator (recall that we are going to define a loss function going from here). As logistic regression predicts probabilities, rather than just classes, we can fit it using likelihood:

$ L(w) = P(Y_{i}|w;x) = \prod_{i=1}^{n} \phi(z)^{Y_{i}}(1-\phi(z))^{1-Y_{i}}$ (4)

In practice, instead of maximizing this likelihood, it is easier to maximize the (natural) log of this equation, which is called the log-likelihood function:

$log(L(w)) = \sum\limits_{i=1}^n [Y_{i}log\phi(z)+(1-Y_{i})log(1-\phi(z))]$ (5)

Or to make it mathematically more convenient, we take the negative of this function, so that it can be minimized using gradient descent:

$ cost = - log(L(w)) = l_{i}(w) = \sum\limits_{i=1}^n [- Y_{i}log\phi(z)-(1-Y_{i})log(1-\phi(z))]$ (6)

The cost would look like this (taken from here):

Here then we take derivative with respect to parameters w (see eq. 1), rather than z (in your photo it is with respect to $\alpha$). Those derivations would be like what you showed. Still I recommend checking out this blogspot that goes through these steps in more details, or parts of implementation in python here, if needed.


My small script on value alteration in columns of a data not working


Could you generate search queries to poison data analysis by a search engine?

I am not sure how many queries you'd need to perform to drown out your actual search queries, but there is already is an actual browser addon which does this. This addon is called TrackMeNot and is available to install for both Google Chrome and Firefox. More in-depth information on how this addon works can be found on their website and the whitepaper (section 3), but in short it create a dynamic list of queries based on popular search terms.


Please list some well tested api's for arima model

Statsmodels:

Statsmodels is your best bet for a python library that includes ARIMA.  I have used it fairly extensively and am quite happy with it. But, its certainly not as well tested as R based ARIMA models.

R:

If you want something "well-tested" then your best bet is likely to use Rpy2 to call an R based ARIMA library from python. Rpy2 can be a bit tricky based on version reconciliation between python, R and Rpy2.

Here's a turorial on calling R from python using Rpy2

Hope this helps!


Use sklearn.preprocessing.OneHotEncoder for example and transfer the one-hot encoding to your web-service ( i'm guessing that's how you're using the model for inference ) via sklearn.pipeline.Pipeline. The pipeline will save the state of your fit on your training data and apply the same function on your production data.

Example :

pipeline1 = Pipeline([
                ('OneHotEncoder', OneHotEncoder())
            ])
pipeline1.fit(trainingdata.column1.values.reshape(-1,1))

This is how you create a pipeline containing the onehotencoder , fit your data on the pipeline. All is left is dumping your pipeline in a file, loading it later in your production environment, and call the transform method on your loaded pipeline :

joblib.dump(pipeline1,"pipeline1.joblib")
# Production environment
pipeline1 = joblib.load('pipeline1.joblib')
momo = pipeline1.transform(productiondata.column1.values.reshape(-1,1)).toarray()

And here , the variable momo contains your production data with the pipeline ( containing the one-hot encoding operation ) applied to it.


Why gradient boosting uses sampling without replacement?

Why is Gradient Boosting sampling done without replacement?

Your first question seems to suggest that the base classifier will always have a subsampling mechanism but this is not necessarily true.

Notice that in the Catboost documentation it is mentioned "Stochastic Gradient Boosting" not "Gradient Boosting". Stochastic Gradient Boosting is a variation of Gradient Boosting that is precisely based on building the boosting process using a subset of the data at each iteration. Therefore each base model does not see the whole train data, it sees only a subset (or minibatch) of the data.

You might want to do this for 2 main reasons: Faster train time, Regularization effect.

Why would it be worst to sample with replacement?

As long as you do sampling in your base classifier the  speed benefit should be quite similar. If your base classifier does not do any sampling then the Stochastich Gradient Boosting algorithm will converge much faster. Just like a Mini Batch version of a Neural network converges much faster than a Full Batch version.

Are there any sampling techniques used in GB that are with replacement?

Standard Gradient Boosting uses all the data to build the gradient at each step of the boosting process.

If you do full batch learning but your base classifier has a "subsample" parameter then you are essentially doing sampling with replacement at every step of the boosting process.


Audio Analysis : Segment audio based on speaker recognition

Initially I did using silence detection but later moved to pyAudioAnalsis which is better.

Check "Speaker Diarization" section in Segmentation in pyAudioAnalysis


How do I explain the relationship between Health and Weight?

Suggestion

You could bin the 'Health' vector to make a categorical variable (ideally consisting of similar number of observations, for example : 2-bin would be median-split of 'high' and 'low', 3-bin would be tertiles of 'high', 'medium' and 'low', and so on), and then do box plots of 'Weight' per bin. You might see that the bin of 'low' is different from the bin of 'high'.
The number of bins you choose depends on the distribution of the 'Health' variable and you can play around with that.

Implementation

library(dplyr) # for modifying datasets
library(ggplot2) # for plotting
library(magrittr) # for piping
stackodato <- data.frame("Health" = sample(0:10, 10), "Weight" = sample(0:200, 10)) # creating a pseudo dataset

stackodato %>%
mutate(binnedHealth = factor(dplyr::ntile(Health, 2), labels=c("low", "high"))) %>% # add "binnedHealth" column which has the "Health" variable categorized into two factors : "high" and "low"
ggplot()+geom_boxplot(aes(x=binnedHealth, y=Weight)) # boxplot showing the distribution of "Weight" split by the "binnedHealth" factor

You can also try this :

stackodato %>% mutate(binnedHealth = factor(dplyr::ntile(Health, 2), labels=c("low", "high"))) %>% ggplot()+geom_boxplot(aes(x=Health, y=Weight, group = binnedHealth))


This can be easily worked out using a standard Seq2Seq word-level model without any modifications in the training process. During inference, in the decoder, just mask the vocabulary based on the partial word input which has been provided.

By masking, I mean to assign a very high score (inf) to words which don't start with the partial word input, thus constraining the decoder to predict words which only start with the partial word input.


Same predictors in test set but I want different outputs

I hope I did not misunderstand your question, as otherwise, this may sound too trivial to you.

I don't think its per se a problem that the same X (predictors) values lead to different Y (end time). After all, if every combination of X would lead to a unique Y, what is the point of estimating a model? Think, of a simple algorithm may predict the majority, e.g. for the same 10 X, you may get

5 * Y1
3 * Y2
2 * Y3

so that your prediction is Y1 for this X.

That being said, I don't think your target variable needs to be categorical, I'd rather configure it continuous, the same with start time. And then, in the simple case, run a regression.

Further, I would not recommend grouping by the same X and add a categorical user type. This leads to a deterministic relationship (mapping), i.e.

User type 1 --> Y1
User type 2 --> Y2

because the way you derive user type is by finding the combination of X that leads to an unique Y.

I'd rather put this as a separate analysis, i.e. inferring user types from TV watching, where you basically cluster your data and say each cluster is a user type.

Edit

Based on your edited problem statement, recall that $Y$ is modelled conditionally on $X$, i.e.

$Y_i = E[Y_i|X_i] + \epsilon_i$.

So your estimated end time $\hat{Y}$ for a certain combination of $X$, e.g. $X_i = \{BBC, news, 20:00\}$, is the conditional expected value, i.e. mean estimate, but your predicted $\hat{Y_i}$ has a distribution (and if you would be more conservative, you could give a confidence interval for your prediction, which graphically speaking is a snippet from that distribution). This distribution is different for every $X_i$, see e.g. the figure in this doc

Now, to your request: what you could do is to sample from the distribution of $Y_i$ (the respective end time). Say you got 10 times the same $X_i = \{BBC, news, 20:00\}$, then sample 10 times from $Y_i$ and output that.

Yet, I don't think this to be the best solution. Graphically speaking again, I would provide the error bounds, i.e. as in the last graph here. In words, don't output 10 different predictions for the same input, but rather output that you predict the end time $Y_i$ for $X_i = \{BBC, news, 20:00\}$ is with 90% confidence between 20:55 and 21:10.


Data set of vectors of SVG paths for digits

I managed to fine some stuff

Dataset of vector digits on Kaggle
Microsoft now offers an API for the OneNote handwriting recognition (using InkML)
Another data set on Kaggle using InkML

I then found out about 'CROHME' (Competition on Recognition of Online Handwritten Mathematical Expressions) this also uses InkML.


You're an online retailer. Like Amazon.
You keep your purchase data for different categories of items in different tables, but all website users have one account with one ID.

Inner Join:
You have two datasets, one with User IDs and purchases of clothing data, the second dataset has User IDs and purchases of books data. You want to find out who purchases both clothes and books from your site.

You inner join to find the User IDs and purchases for people who bought clothing and books. Any users who didn't buy both of these items will be 'dropped' from the final table.

Left Join
You have one dataset with User IDs and account info(e.g. age, name) of all users. You want to build out this table to include some sales data, without dropping people from your complete list of users. So you would left join your purchase dataset to your account dataset (with User ID as the key). Any user who has made no purchases won't be dropped from your final table.

Right Join
The opposite of left join. E.g. you want to keep all purchase data, even if for some reason your Account Info table does not contain the ID of the user who made that purchase.

Full (Outer) Join
You want to Join account info and purchases tables, but you don't want to lose any entries in either table. So your final table will include Users who have made no purchases, and (perhaps mysteriously) purchases who have no users associated with them.


getting the weights of intermediate layer in keras

The easiest way to create a truncated output from a network is create a sub-network of it and apply weights of your trained network. The following example is a modification of what you have shown up there, but it will guide you out

Network you want to train originally

model = Sequential()
model.add(Conv2D(10, (3, 3), padding='same', input_shape=(60, 60, 3),
                 activation="relu"))
model.add(Conv2D(10, (3, 3), activation="relu"))
model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(5, activation="softmax"))
model.compile(
    loss='categorical_crossentropy',
    optimizer='adam',
    metrics=['accuracy'])

model.fit(
    train_data,
    train_label)

Now create a subnetwork from which you want the outputs, like from above example

model_new = Sequential()
model_new.add(Conv2D(10, (3, 3), padding='same', input_shape=(60, 60, 3),
                     activation="relu"))
model_new.add(Conv2D(10, (3, 3), activation="relu"))
model_new.add(MaxPooling2D(pool_size=(3, 3)))
model_new.add(Dropout(0.25))
model_new.add(Flatten())

model_new.compile(
    loss='categorical_crossentropy',
    optimizer='adam',
    metrics=['mse'])

# You need to apply fit on random array's created, just so as to initialise
# weights. Anyways you will replacing them with original ones from above.
model_new.fit(train_data, y=np.random.rand(40, 3610))

Now take weights from the first trained network and replace the weights of the second network like

model_new.set_weights(weights=model.get_weights())

You can check whether the weights are changed in the above step by actually adding these check statements like

print("Are arrays equal before fit - ",
      any([np.array_equal(a1, a2) for a1, a2 in zip(model_new.get_weights(), model.get_weights()[:4])]))

model_new.set_weights(weights=model.get_weights())
print("Are arrays equal after applying weights - ",
      all([np.array_equal(a1, a2) for a1, a2 in zip(model_new.get_weights(), model.get_weights()[:4])]))

This should yeild

Are arrays equal before fit -  False
Are arrays equal after applying weights -  True

Hope this helps.


Strategies for continuously assessing and improving model performance

I'm glad to see this question because this site gets such few questions on models that are actually in a production state.

If I was in your position, I would start to think about how I'm going to use this algorithm on a go-forward basis and start to log everything. Every new prediction that your algorithm makes is also a new data point for training.

So you need to take the time to create a feedback loop and (1) take new datapoints, put them into training, (2) re-train your algorithm, (3) deploy a new version and then (4) use that revised algorithm. You repeat this entire process on a continual basis throughout the life of your project. From the technical side, it's an exercise in re-inforcement learning since your algorithm won't start from a base of zero knowledge. Also, most of the steps I described will most likely require you to write new code, most of which will be related to process and workflow, not necessarily data science or algorithmic.

Finally, as an FYI, it is said that there are multiple algorithms out there that carry out these types of cycles on a continual basis, literally updating on a minute-by-minute (or faster!) basis, always providing answers on the latest available data points.


Percentage of missing values so that we can't perform imputation

In my opinion that depends on the context, like the question you want to answer, and also on the domain.

If the remaining absolute data set is still big and diverse enough to allow conclusions about that feature, the percentange of records that can not add to that imputation is not a problem, in my understanding.

It might also be that features sometimes are not just missing, but  do not apply to the particular kind of record.


Count the frequncy of words in a cell of a column in a series

Why shouldn't you use the Counter class; it's exactly what you need?

from pandas import Series
from collections import Counter
text="barack hussein obama ii brk husen bm born august 4 1961 is the 44th and current president of the united states and the first african american to hold the office born in honolulu hawaii obama is a graduate of columbia university and harvard law school where he served as president of the harvard law review he was a community organizer in chicago before earning his law degree he worked as a civil rights attorney and taught constitutional law at the university of chicago law school from 1992 to 2004 he served three terms representing the 13th district in the illinois senate from 1997 to 2004 running unsuccessfully for the united states house of representatives in 2000in 2004 obama received national attention during his campaign to represent illinois in the united states senate with his victory in the march democratic party primary his keynote address at the democratic national convention in july and his election to the senate in november he began his presidential campaign in 2007 and after a close primary campaign against hillary rodham clinton in 2008 he won sufficient delegates in the democratic party primaries to receive the presidential nomination he then defeated republican nominee john mccain in the general election and was inaugurated as president on january 20 2009 nine months after his election obama was named the 2009 nobel peace prize laureate during his first two years in office obama signed into law economic stimulus legislation in response to the great recession in the form of the american recovery and reinvestment act of 2009 and the tax relief"
df = Series(text).to_frame()
newdf = df.assign(word_count = lambda x: x[0].str.split(' ').apply(Counter)[0])
newdf['word_count']

  0    {'44th': 1, 'born': 2, 'november': 1, 'running...


Keras Conv1D model Input_shape value error

After running your code & the major issue found out is the lack of proper mentioning of input shape.

In your case insisted of passing shape=(6,) try to pass shape=(6,1) and make sure your data should in the following format.

x = [[695746319], [5], [1], [0], [0], [0], [0]]


Best method to deal with too many zeroes in regression problem?

Option (b) is a good approach. If you use a classifier that outputs probabilities, you can even multiply the two model outputs together to calculate a risk expectation that's interpretable as the joint likelihood.

If cpc is discrete valued and non negative, you could interpret that as a count and use a zero-inflated Poisson model. If it's not, you could convert it by rounding to the nearest cent or something like that.


Need suggestions on customer segmentation

So the question is about how to before customer segmentation on this data.
When I do any customer segmentation, I firstly think to myself, do I know how many segments prior to the analysis or not.
If I do,
Then I would use a clustering method like K-means clustering (https://towardsdatascience.com/understanding-k-means-clustering-in-machine-learning-6a6e67336aa1), where k refers to number of customer segments.
If I do not,
Then I would use something like agglomerative clustering (https://www.datanovia.com/en/lessons/agglomerative-hierarchical-clustering/).
When it comes to data representation, you would represent a customer and their (purchasing) behaviours as a vector of values (I will refer to as customer vector).
If the variables are numerical (e.g. number of items purchases), then you can put the numerical values in the customer vector. If the variables are categorical (e.g. products purchased), then we concatenate a one-hot encoded vector (https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/) of that variable to the customer vector.


Interpreting random seed effect on correlation in regression with small sample

This means that your model is not consistent. Your problem is hard because of the small sample size. I would recommend reducing the number of predictor variables.

7 predictors is quite a few for linear regression, even with a large sample size. Try and get the best results you can with 2 or 3 predictors.

If you can predict with 30% accurately all of the time it is better than 70% accuracy 10% of the time.


Data structure design for supporting arbitrary number of columns in table or database

I've done something like you're describing using mongoDB--I think you'll best use your time using some sort of NoSQL approach, rather than creating a specialized one-off solution. If you're using Python, I've had excellent experiences using PyMongo to handle reads and writes from within my code.

I would strongly caution you against adopting your approach #1. This could break very easily in the future, and there are databases designed to handle your exact problem!


Error when checking : expected dense_1_input to have shape (None, 5) but got array with shape (200, 1)

I had a very similar issue when executing predict/predict_classes;

  ...
    classes = model.predict(XpredictInputData)
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 1006, in predict
    return self.model.predict(x, batch_size=batch_size, verbose=verbose)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1772, in predict
    check_batch_axis=False)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 153, in _standardize_input_data
    str(array.shape))
ValueError: Error when checking : expected dense_1_input to have shape (None, 100) but got array with shape (100, 1)

The problem was occurring because the input array to keras predict was 1 dimensional (caused by a combination of a) numpy.genfromtxt producing a 1d array from a 1 row data file and b) a broadcasting failure). Keras predict appears to require a two dimensional array, even if there is only 1 prediction to be made.

I solved the issue by adding the following check before executing predict;

if (XpredictInputData.ndim == 1):
    XpredictInputData = numpy.array([XpredictInputData])


It seems likely that the live data is different somehow from your other data.  Cross-validation shows a 84% accuracy, and accuracy on the held-out set is 86%, which is pretty consistent and does not indicate overfitting.  Accuracy on the live data is 70%, which is significantly different.  That suggests that live data is somehow different in ways that are important to the classifier.  Perhaps concept drift has occurred.


Adjusting imbalance in classification problem reduce precision, accuracy but increase recall

Intuitively the problem of imbalanced data can be understood like this: if a classifier is not really sure how to classify an instance but it knows that most instances belong to class X, then whenever there's a doubt predicting class X is always the best decision. As a consequence, the classifier unavoidably assigns class X too often since all the "unsure" cases end up being labeled as the majority class X.
So first it's important to understand that resampling is not automatically the "cure" for imbalanced data:

Resampling doesn't provide the classifier with more information, it just presents it in a different way in order to force the classifier to pay more attention to the minority class.
In case the data is easy to separate, the classifier can do a perfectly good job without resampling. This means that whenever possible it's better to improve the features rather than using resampling, because that's what can actually help the classifier doing its job.

This being said, resampling is a useful technique in some cases. Assuming the standard choice of the minority class as the positive class, resampling will only increase recall at the expense of precision: as said above, the difference happens when the classifier cannot easily predict the class for an instance. In such a case it has two choices:

Assigning the negative majority class: more likely to be correct (True Negative), a small risk of False Negative error.
Assigning the positive minority class: more likely to be incorrect (False Positive), a small chance to be correct (True Positive)

Without resampling the classifier favors the first option, so it has few FP errors but quite a lot of FN errors. Therefore it can have quite high precision but low recall.
With resampling the classes are equal, so the classifier stops favoring the first option. Therefore it makes less FN errors but more FP errrors, which means that it increases recall at the expense of precision.


How can I identify and remove outliers in R

Cook's distance and the alternative method DFFITS are not strictly speaking methods to detect 'outliers' in the sense of purely anomalous values, rather they detect 'influential' points, where leaving the value in or out of the analysis noticeably changes the result. Hence, Cook's distance measures how much the beta values change when a specific observation is omitted, which is a different idea to detecting an outlier as such, although they will naturally often be the same points.

From the description of your problem, it is possible that detecting influential points is what you are actually trying to do.

To directly answer your question on the Cook's distance threshold, the F-statistic with p and n-p (where p is number of regressors, and n is observations) degrees of freedom is often used.

Outliers per se in regression are more often identified via residual analysis e.g. if the data point's corresponding residual is large in the context of the data set. This is often taken to be when the value lies more than three standard deviations away from the mean of the residuals (though note that in a sufficiently large data set, a well fitted model will still have some observations at this level)


Time series classification, without the time dimension

Honestly it seems you are quite far from what would need a supervised vision approach. I suggest you to try a simple non-ML approach first : extract text with a standard library then just label what would count as a 'bullet' then check if there is more than one in a row. This might just work and if it doesn't it will help you understand why.
Going the whole OCR way would imply a lot of pdf annotation for a gain that is not really clear without trying a simple benchmark.
Ps: if you want to deal with text, the relevant field is Natural Langage Processing not time-series.


Without looking at the actual data, all we can really do is second guess and suggest best practices. Here are some pointers you can pursue -

Gather more data - If it can be done, nothing like it.
Improve data quality - The algorithm will always be as good as the data. Try extensive cleaning methods like -
Lowercase (basic) so all your data is standardized
Stemming / Lemmatization - These techniques reduces a word to its root form
Try parts of speech tagging (POS) and retain important parts of speech like nouns and look at their overall importance
Stopword removal - Just like you tried doing word frequencies, there are popular word sets like the word the that don't help much. They are filtered out
Correcting Spelling mistakes - If you are expecting there will be spelling mistakes, might be a good idea to correct them
Converting the unstructured data - I see that you have used bigrams. Try using unigrams and trigrams as well, or in combinations, run your algorithm and see which one works better. Try CountVectorizer, TfidfVectorizer and other techniques like word embeddings as well
Algorithms - Finally focus on the algorithm itself. For Naive Bayes, focus on MultinomialNB. Try RandomForestClassifierand other ensemble family algorithms. Try Deep Learning techniques with keras. Fine tune hyperparameters based on the validation results

Of course, there are other best practices like splitting your data into train, test and cross validation sets. Evaluating the right metrics such as accuracy, precision, recall and the confusion matrix.

Hope this helps!


Always drop the first column after performing One Hot Encoding?

This question, in a slightly different form, was discussed herein earlier.
You are kind of right, but the best and safest way is to do One-Hot-Encoding and drop at the end because which column you want to drop at the very beginning?
In fact in pandas.get_dummies there is a parameter i.e. drop_first allows you whether to keep or remove the reference (whether to keep k or k-1 dummies out of k categorical levels). Please note drop_first = False meaning that the reference is not dropped and k dummies created out of k categorical levels! You set drop_first = True, then it will drop the reference column after encoding.


Performance metrics changing significantly based on batch size

Batch size is very related to the learning rate, especially in non-adaptive optimizers like the vanilla SGD that you are using.
I would suggest two alternatives:

Tune (reduce) the learning rate. You can check the answers to this SO question for some heuristics on how to choose the value. Apart from that, I recommend the article Don't Decay the Learning Rate, Increase the Batch Size.

Use an adaptive optimizer, like Adam.


best similarity measure for images with different angles

A brute-force method is simply to try all rotation angles and decide if 2 images are a rotation of one another.
However, there are features (eg fourier coefficients) which are rotation-invariant. So comparing these rotation-invariant features is a similarity metric for determining if 2 images are a rotation of one another.
References:

Rotation Invariance in Images
Rotation invariant indexing of shapes and line drawings
Which Transformation, Or Similarity Metric, Is Rotation, Shift and Scale Invariant?


Neural network regression problem, integer output neuron constraint

You are correct to approach this as a regression problem, mostly because you are interested in the order of your outputs. For example if there are 1000 people present and you predict 1005, it's a better prediction than 7005. If you were treating this as a classification problem, both of these would be interpreted as missclassifications.

The most practical way to approach your constraint is to round the output of your neural network, i.e. if it outputs 3456.78, round it up to 3457. You need to have a linear activation function to be able to use the NN for regression.


What is a good interpretation of this 'learning curve' plot?

The X axis is the number of instances in the training set, so this plot is a data ablation study: it shows what happens for different amount of training data.
The Y axis is an error score, so lower value means better performance.
In the leftmost part of the graph, the fact that the error is zero on the training set until around 6000 instances points to overfitting, and the very large difference of the error between the training and validation confirms this.
In the right half of the graph the difference in performance starts to decrease and the performance on the validation set seems to be come stable. The fact that the training error becomes higher than zero is good: it means that the model starts generalizing instead of just recording every detail of the data. Yet the difference is still important, so there is still a high amount of overfitting.


Text representation using TFIDF .toarray() freezes my computer. Too many features to handle?

If your downstream algorithm can work with sparse matrices, you can proceed with sparse matrix rather than using toarray.
Alternatively, try to minimize the number of features which are created. Check documentation of TfidfVectorizer (parameters like min_df, max_df, stopwords) to filter out un-important words.


How to properly store a list of pandas dataframes, so that I can import them in a new python script?

Use pickle package, it's meant to export and/or load python objects.

You can find information of how is done in this topic of stack overflow: https://stackoverflow.com/questions/4529815/saving-an-object-data-persistence


Google TPU: when/how will it be available to me?

Google has apparently said it would not be available for sale outside of Google.

http://www.forbes.com/sites/tiriasresearch/2016/05/26/google-builds-its-first-chip-just-for-machine-learning/#58ab2bcb7a92

Google also did mention it would be available to developers for Tensorflow and Cloud machine learning but otherwise it has been light on details or when this will become available.


How do I provide input and output for such a network structure in keras

You can pass in a list to the input parameter when you create the model.

model = Model(input=[conv2d_1_1, conv2d_1_2, conv2d_1_3, conv2d_1_4], output=dense3)

Make the input x a list of four numpy arrays of shape (3,6,3)

Make the output y an array of one-hot vectors corresponding to your 145 classes.

Then call

model.fit(x, y)

You probably want to change your last layer to have 143 dimensions in the output

num_classes =143
Dense(num_classes, activation='softmax', activation = 'softmax')


How can I improve my model on a very very small dataset?

From what you say, I think you should start with checking three options:

I) Ordinary least squares (OLS): Just run a „normal“ linear regression. This will not yield great predictions, but you could view the model as a causal one, if you can assume a linear relation between $y$ and $x$. When you have five predictors and 35 observations, you have a total of 29 degrees of freedom which is „okay“. When you estimate the model in „levels“, so just values as they are, you can directly interprete the estimated coefficients as marginal effects. E.g. a model $y=\beta_0+\beta_1 x + u$, tells you that when $x$ increases by one unit, $y$ changes by $\beta_1$ units, just like a linear function.

II) You can use Lasso/Ridge/Elastic Net: All of them are linear-like models with a penalty term to „shrink“ $x$ variables if they are „not useful“. This works like automatic feature selection if you like to say so. There is a great package by Hastie et al. for R. You can find it here. It is also available for Python.

III) Maybe (!) boosting could be an option as well: You would (likely) need to do some feature selection/engineering on your own. But Boosting is able to work with a small number of observations, with highly correlated features, and it often works well with highly non-linear problems. There are LightGBM or Catboost as possible Python packages. Find some minimal examples here.

With II) and III) you will find that you are not really able to „set aside“ a number of observations to check if your models work (because you don’t have much data). You could use cross validation (Ch. 5 in ISL, link below), but you need to see how it works. Instead of going for a predictive model, I tend to say that you might be better off starting with a „causal-like“ OLS model. With OLS you do not really need a „test-set“. OLS is very robust.

Since you seem to be new to statistical modeling, you might benefit from having a look at „Introduction to Statistical Learning“ (Chapters 3 and 6 in particular). The PDF is online and there is code for the Labs in Python and R. The advanced book would be „Elements of Statistical Learning“.

Good luck with your project!


Reinforcement Deep Learning for object detection

To my extent of my knowledge, RL is used as a model for attention mechanism in object detection field (particularly the REINFORCE algorithm which is a 'flavor' of policy gradient methods). You can take a look at one of the papers that first proposed the method: RAM


What are the inputs to a logistic regression? Probability or trial result?

From Wiki: The graph shows the probability of passing the exam versus the number of hours studying, with the logistic regression curve fitted to the data.

The probability curve is the output of the LR, not the input. Typically during training, the output class (or target class) will be discrete class labels with 1 or 0. During inferencing, the output will be a continuous value between 0 and 1. To generate the probability curve, just feed in different values of "hours studying" into the trained model.


df.groupby('device_id')['category'].apply(list).tolist()

There's your transactions LOL.

If you aren't limited to Python 2.7, I'd suggest Orange3-Associate which contains a frequent_itemsets() function based on FP-growth algorithm, which is orders of magnitude faster than Apriori.


Faster-RCNN how anchor work with slider in RPN layer?

For each anchor you find an IoU with the object in the picture and set 1 if IoUexceeds the threshold and 0 if it is below a lower tjreshold(e.g. 0.3). If it's a hit a bbox offset is calculated, distance between prediction and true bbox. Hence there are two loss fumctions: object/bg and bbox regression


How to find out if two datasets are close to each other?

You could take an Information Theory approach by finding the lowest Kullback–Leibler divergence between the distributions. There is a KL divergence option within SciPy's entropy function.

>>> from scipy.stats import entropy

>>> p = [0.21,0.24,0.36,0.56,0.67,0.72,0.74,0.83,0.84,0.87] # Data removed to make equal sizes: [0.91,0.94,0.97]
>>> q_1 = [0.13,0.21,0.27,0.34,0.36,0.45,0.49,0.65,0.66,0.90]
>>> print(entropy(p, q_1))
0.019822015024454846

>>> q_2 =[0.14,0.18,0.19,0.33,0.45,0.47,0.55,0.75,0.78,0.82]
>>> print(entropy(p, q_2))
0.01737229446663193

The second simulated distribution is closer than the first simulated distribution to the real distribution.

If you are interested in inference, you could run many simulations and compute p-values. That process is a variation of permutation testing.


Try a totally different approach, using Generative Adversarial Networks.

For this purpose you need:

A Generator
A Discriminator

See the scheme (credit O'Reilly):

The "Real Images" block in the scheme should be your training dataset (or ground truth). The Generator should generate the distorted words and the Discriminator should verify if the word is "adequately" distorted, based on a criterion of your choice, which can be any similarity measure between known words (database) and the generated one. Both the Generator and the Discriminator get trained on-the-go while in the training phase and in the end you will have two trained networks, of which the Generator would be very useful for your purpose.

Helpful source: https://deeplearning4j.org/generative-adversarial-network


Confusion regarding classification accuracy calculation and result

You might want to take a look at the confusion matrix  wiki page.

Accuracy = (TP + TN) / (P + N) = (26 + 1473)/1500 = 99.9%

I guess, if you want the break down by the class, it would have been:
1473/1473
and
26/27, but you have 1500 as the denominator in both classes.

Even though I think conventionally most people report on the entire model rather than just on a particular class.


Is GLM a statistical or machine learning model?

A GLM is absolutely a statistical model, but statistical models and machine learning techniques are not mutually exclusive. In general, statistics is more concerned with inferring parameters, whereas in machine learning, prediction is the ultimate goal.


I also found confusing the definition of likelihood the first time I encountered it. The only thing you need to remember is that likelihood refers to the probability of your data given a parameter. That is,

$$L(\theta| X) = P(X|\theta)$$

As you can see, the likelihood focuses on the parameter, so you're asking "What is the likelihood of this parameter given some observations?" In this case, the phrase "given some observation" assumes the observations $X$ are fixed and the parameter $\theta$ is not fixed. On the other hand, if you describe the same in terms of probability, you're trying to measure the probability of some observations $X$ given a fixed parameter $\theta$. It is only a matter of usage.

In the case of logistic regression, the likelihood function is described as:

$$L(\beta| y) = \prod_{i}^{N} \pi_{i}^{y_{i}}(1-\pi_{i})^{n_{i} - y_{i}}$$

where $\beta$ is the vector of parameters $\{\pi_{i}| i=1,...,N \}$and $y$ is the vector of Bernoulli variables. As you can see, this is the same (by definition) as $P(y|\beta)$ and that is what you have in the equation above: The product of the probabilities for each observation $y_{i}$ given its parameter $\pi_{i}$. As usual, each probability $\pi_{i}$ is equal to the inverse of the logit function.

In Naive Bayes, you have the same basic idea. The likelihood is defined as $L(p_{k}|x) = \prod_{i}^{N}p_{ki}^{x_{i}}(1-p_{ki})^{(1-x_{i})}$ where $p_{k}$ is a vector of parameters describing the probabilities of a class $k$ generating a word $x_{i}$).

In the EM algorithm, the likelihood is again $L(\theta|X,Z)$, where $\theta$ are your parameters, $X$ your observations and $Z$ are latent variables. The fact that you have a latent variables shouldn't confuse you, because this variable will be "integrated out"  in the expectation step of the algorithm.

In the previous equations, the calculation you need to do is self-evident if you think about it starting from your observations: First, I have some observations. Then I want to calculate the probability of obtaining these observation using some model. Most models require parameters to be fitted. So I want to calculate the probability of these observations given those parameters. That is, $p(X|\theta)$. Now, if you need to talk about likeliihood, then you simply write $L(\theta|X)$.


Self-designed objective for linear regression learning

This question seems really old, it may be helpful to new visitors.

Yes, we can define any objective function and use gradient descent to minimize it. For that the objective function need to have some properties like

It should be convex function else we'll stuck at local minima or may never find the minimum value
It must be differentiable on every point and must have non zero derivative. Else gradient descent will not make progress and get stuck

If cost function have these properties then we can use tensorflow (or other library) to create graph to calculate the cost and then to calculate its gradient using auto-diff algorithm and use gradient descent to minimize it.

Auto-diff basically breaks calculation of cost function into elementary arithmetic computation like addition, multiplication, division etc and use chain rule to find derivative.


Copying from the abstract I found here - read the paper for more insights:

In the year 2011, the CMS collaboration introduced the novel concept
of data scouting, allowing to take data that otherwise would be
rejected by the usual trigger filters. This special data flow, based
on event-size reduction, was created to maintain sensitivity to new
light resonances decaying to jets or muons, with very small online and
offline resources allocated to it. The challenges implied by this new
workflow and the solutions developed within the CMS experiment are
shown. This technique is now a standard ingredient for CMS data-taking
strategy. The present status of data scouting in CMS is presented.

I do not come from a Physics background, and hence I could not completely understand the context while reading the paper in the larger scheme of things. But atleast this gives you a heads up about what it is.
Coming to Data Mining, there are plenty of definitions you can find. My definition would be using raw data to find patterns or insights from large datasets. This could be done with the use of many algorithms from math, computer science or statistics.
Some data mining algorithms are k-nearestneighbor, k-means, naive bayes, and so on.
So what is the difference between data scouting and mining? In mining, you consider raw datasets and perform some operations. In scouting, if I am not wrong, you take the data discarded by some filters (do not know what these filters are) while performing some operation, and then considering that data to perform additional operations.


Structure of a multilayered LSTM neural network?

For layer 1, After all 5 Xi's are entered into the network, the 10 outputs from X5 from the first layer are then used as the inputs for the second layer. The other outputs from X1 to X4 are not used.

No, the outputs from all time steps 1,2,...,5 are used as the input of the second layer.

For layer 2, After all these are done, only the final 10 outputs from the layer 2 will be used?

It depends on you. Some people choose to use only the final size=10 output as an encoding of all the input. Some people choose to average or concat all the outputs of second layers. When using attention, all the outputs are used.

Here is a illustration of a 3-layer LSTM. Notice that all the outputs from the lower layer, not only the final output, are fed into the higher layer.

.


The answer depends on what you want to do with the hashtags/words and also on what tokenizer you are using.

Consider this example tweet:

  Hi, we need you!

  #Hi #Weneedyou

If you use TreeBank or WordPunct tokenizers the output will be:

 ['Hi', 'we', 'need', 'you', '!', '#', 'Hi', '#', 'Weneedyou']

However if you use Whitespace Tokenizer, the result is:

 ['Hi', 'we', 'need', 'you!', '#Hi', '#Weneedyou']

Similar discrepancies can be found in terms such as can't or pre-order for instance.

Additionally, you need to consider what is a token for your task at hand. In my previous example Hi make sense either with or without #. However Weneedyou without the hash is just a poorly written word. Maybe you want to keep the hashtags in your corpus or maybe not, but this depends on what you want to do with it.

So first you need to know how your tokenizer handles these cases and decide if you want to remove them beforehand or later to keep hashtags in your corpus or only (sometimes weird) words.

In the case of the @ is exactly the same, you can keep it, remove it or maybe delete the whole instance @user as you don't want to keep user names in your corpus. As I said, it all depends on your task.

PS: In case you want to play around with different tokenizers, try this.


Importance of variable to implement in Logistic Regression model in Python sklearn?

The problem you are describing is feature selection. Welcome to a particularly sticky problem. Your method of Pearson correlation certainly can help - simply drop those features with low and or insignificant correlation to your dependant variable. This is univariate selection.  Since you are using a logistic regression, you can also inspect the model coefficients and odds ratios to determine the effect size and significance of each feature.

But what if some combination of your independent variables produces a significant effect within your model? Say the combination of: gender, age, income, and house value for a binary response related to likelihood to purchase. This is multivariate feature selection.

For logistic regression, and other models, you can use "interaction terms" to investigate the combined effect of input variables. In Data Science and Machine Learning literature this would fall under the practice of feature engineering. There are a number of other procedural and rule of thumb methods to select inputs for multivariate models. For instance, one can use a decision tree method to evaluate important features across repeat random samples of the data, or one can simply use domain and conceptual knowledge of the process being modeled. In most cases you should use a variety of techniques and choose the inputs that make both mathematical and theoretical sense.


The derivative is a linear transform, and you're using a linear model. As you've demonstrated, nothing is gained by adding linear combination of other features (like the derivative) as a new feature in a linear model.

You may want a derivative coefficient to help interpret the model. You could use the normal model $x_{n+1} = a * x_n + b * n_{n-1}$ and then report the value $a-b$ to the client. A more general approach is to examine the frequency response of the model (the derivative is a high-pass filter, your model is a FIR filter).

Alternately you can rotate the basis of the input such that the derivative is a feature and the dimensions are still independent as follows:

You can think of the independent variables $x_n$ and $x_{n-1}$ as forming a standard basis for the two dimensions of your model's input, that is $\langle 1, 0\rangle$ and $\langle 0, 1\rangle$.
Neither $a\langle 1, 0 \rangle$ or $b \langle 0, 1 \rangle$ are orthogonal to  c$\langle1, -1\rangle$.

Therefore, if you want to use $c(x_n - x_{n-1})$ as a feature, I would also use the orthogonal feature $d(x_n + x_{n-1})$. That is,

$$ x_{n+1} = c(x_n - x_{n-1}) + d(x_n + x_{n-1}) $$.

You could think of this model as the derivative plus the integral.

The same logic applies to the choice of scaling function for the Haar wavelet.


How to deal with class imbalance problem in natural language processing?

Disclaimer: this answer might be disappointing ;)
In general my advice would be to carefully analyze the errors that the model makes and try to make the model deal with these cases better. This can involve many different strategies depending on the task and the data. Here are a few general directions to consider:

Most of the time the imbalance is not the real problem, the real problem is why the model can't differentiate between the classes. Even in case of extreme imbalance if the classes are easy to discriminate a model can perform very well. The imbalance only causes the model to assign the majority class when it doesn't have enough indication to decide, so it resorts to the conservative choice.
If the minority class is really small in absolute terms it's likely that there's not enough language diversity in the positive instances (data sparsity). This will usually cause a kind of overfitting which can be hidden by the fact that the model almost always assigns the majority class. In this case the goal should be to treat the overfitting, so the first direction is to simplify the model and/or the data representation.
Sometimes it can make sense to consider alternative ML designs: in a regular classification problem a model relies on the distribution of the classes, by principle. Some alternative approaches might not as influenced by the distribution, for example one-class classification. Of course it's not suited for every problem.

Overall my old-school advice is not to rely too much on technical answers such as resampling methods. It can make sense sometimes, but it shouldn't be used as some kind if magical answer instead of careful analysis.


How to i get word embeddings for out of vocabulary words using a transformer model?

I'm not sure there is a need for aggregation, or in other words you may have a pipeline mismatch. BERT sentencepiece tokenization is specifically meant to be passed to some set downstream pipelines, with the aim of the sentencepiece thing being to be able to cater to OOV words. By aggregating the sentencepiece tokens, you might be doing away with the benefit of being able to cater to OOV in your later pipeline.
If you are looking for whole word vector tokens, and want to work with OOV words, I would recommend looking at FastText instead. This algorithm more or less uses subwords, and it will also build tokens for OOV words by pretty much aggregating the subword information for that OOV word during a custom training step. The benefit here is that the aggregation step need not be part of your pipeline, and you can use these new vectors in any downstream task (except, of course, the pipelines that accept BERT sentencepiece tokens)


What if the votes for 2 classes are equal in an ensemble learning technique?

Depending on the implementation, this problem never occurs. Most of the implementations build an odd number of trees or models to ensure one class's dominance.
However, some implementations allow an even number of trees and models. In such a case, some algorithms simply just throw an error by mentioning the tie has occurred.
Especially in R, some of the libraries solved this issue by randomly picking one of the classes.
The last and best (I think) solution is choosing the class that is more frequent than another. In other words, the class that is more frequent in the training data is chosen arbitrarily by the algorithm to break the tie.


My question is — why does this token exist as input in all the
transformer blocks and is treated the same as the word / patches
tokens?

The CLASS token exists as input with a learnable embedding, prepended with the input patch embeddings and all of these are given as input to the first transformer layer. The CLASS token gathers information from all the patches using Multihead Self Attention (MSA). It is basically treated the same as patch tokens but at the end when doing classification, only the hidden output from the CLASS token are used as input to the classification layer.

Treating the class token like the rest of the tokens means other
tokens can attend to it. I'd expect that the class token will be able
to attend other tokens while they could not attend it.

I think that would help training all the weights during back-prop, including the weights of the embedding layer of the input patches.

Also, specifically in ViT, why does the class token receive positional
encodings? It represents the entire class and thus doesn't have any
specific location.

The positional encoding would tell that this is the first element of the sequence. Would help if you have a bi-directional transformer model like BERT.


Data visualization of frequencies of state transitions (possibly in R?)

How about a Sankey diagram with time on the x-axis and flow width representing state transition frequency. Here is a SO discussion on implementing Sankey diagrams in R.

One possible R package is {riverplot}... here is code showing the first transition in your data:

library(riverplot)
nodes <- as.character(sapply(1:2, FUN = function(n){paste0(LETTERS[1:3],n)}))
edges <- list(A1=list(C2=4), B1=list(A2=1,C2=1,B2=2), C1=list(C2=2))
r <- makeRiver( nodes, edges, node_xpos= c( 1,1,1 ,2,2,2),
                node_labels= c( A1= "A", B1= "B", C1= "C", A2="A",B2="B",C2="C" ))
plot( r )

Will produce this:


Converting non-numeric data values into equivalent rank scores

To me the approach looks overcomplicated. If you're not limited to that one algorithm use one-hot encoding and try out various classifiers. Many of them can predict probability which you can use to calculate the ranks.


The latent vector $z$ is just random noise.

The most frequent distributions for that noise are uniform: $z \sim U[-1,+1]$ or Gaussian: $z \sim \mathcal{N}(0, 1)$ . I am not aware of any theoretical study about the properties derived from different priors, so I think it's a practical choice: choose the one that works best in your case.

The dimensionality of the noise depends on the architecture of the generator, but most of the GANs I've seen use a unidimensional vector of length between 100 and 256.

In PyTorch, torch.Tensor.random_ and torch.randn can respectively be used to generate uniform and Gaussian noise.


Logic behind the Statement on Non-Parametric models

Non-parametric machine learning algorithms try to make assumptions about the data given the patterns observed from similar instances.By not making assumptions, they are free to learn any functional form from the training data.

Unlike parametric approach, where the number of parameters are fixed,
in non-parametric approaches the number of parameters grow with
training data.

If your data set is too small or otherwise is a set that is not representative of the entire population, then your result will be biased in more ways than possible with parametric methods.
So,to get better results in Non-parametric machine learning algorithms we need large amount of data where relationship between features are not known.
While non-parametric machine learning algorithms are often slower and require large amounts of data, they are rather flexible as they minimize the assumptions they make about the data.


Correlation between products based on purchases placed around the same date

Association rule mining is nearly always done with the apriori or eclat algorithms. There are a few papers which have utilized these in the time series context. This paper details a thorough implementation, though this is done with relation to identifying gene patterns, rather than customer purchases. The idea is still the same though, only there is slightly less complexity in your case. Unfortunately, this does not give a detailed Python implementation.

Depending on your situation (this is unlikely to be applicable from your description, but I list this just in case...), it may be useful to break from association rules and cluster your data instead, provided you have enough of it to treat it as continuous. A Python implementation of this is featured in this article.

Good luck!


If I understand correctly, in your problem the mineral-element vector $m1$ and $m2$ is known.

If that is the case, I would suggest LASSO regression. Suppose you have $n$ minerals $m_1$, $m_2$, ..., $m_n\in\mathbb{R}^p$ arranged into a matrix $M=[m_1;m_2,...,m_n]\in\mathbb{R}^{n\times p}$, and an unknown mixing vector $x\in\mathbb{R}^n$ (in your case, $x=[0.25; 0.75]$). Your sensor measures $y=M^\top x\in\mathbb{R}^n$ and you want to recover $x$ from $y$.

The LASSO regression aims to solve $\hat{x}=\arg\min_x ||y-M^\top x||_2 + \lambda||x||_1$. The $\ell_1$ penalty promotes sparsity in $x$ and could be beneficial when you have a lot of candidate minerals.

I don't think you need any special handling due to the compositional nature of the data, other than normalizing the final $\hat{x}$ if needed.


Postitive event but unsure when it occurred in time

This is referred to as "interval censoring": that is to say, you know the event occurred within an interval, but not exactly where in that interval.

I don't think there has been a lot of attention in the ML community for interval censored data (or any, really). However, there has been a reasonable amount in the statistical community, myself included. As such, I've written the R package icenReg for interval censored regression models. Regression models certainly can be used as ML tool, although these don't yet have all the bells and whistles that are more typical for ML problems (i.e. no penalized regression a la elastic net etc.).

However, icenReg at least contains a tool for generalized cross validation, although it is hidden from the public. It can be extracted by icenReg:::icenReg_cv.


Selecting Transforms with sklearn pipelines

The purpose of the pipeline is to assemble several steps that can be
cross-validated together while setting different parameters. For this,
it enables setting parameters of the various steps using their names

Check the docs
For your requirement, you will have to create multiple pipelines, you cannot do it via single pipeline.


How to scatter plot with each dimension having its own color

In scatter in Matplotlib, the first two arguments must be the $x$ and $y$ values that you want to combine. In your code, the first value ($x$) is len(inp_samples) (or len(out_samples)) which is the number 300, so all your points line on the $x=300$ vertical line.

In order to combine the inp_samples with the out_samples, I too would recommend using zip, as in your second sample of code. Just replace

plt.scatter(len(inp_samples), d_in, c=c)
plt.scatter(len(out_samples), d_out, c=c)

with

plt.scatter(d_in, d_out, c=c)

(and see if it works).

Edit

  So I thought to plot each point with a different color, but the color for a dimension to be same, so that i would know, which dimensional points are lying where.

In a scatter plot, there is no need for any special colouring -- by two values forming one pair of $(x_i,y_i)$, it is already given which point belongs to which.

You could also try to plot $(i,x_i)$ and $(i,y_i)$ in the same plot. In this case, points above each other belong together and you still don't need any special colouring.

colors = itertools.cycle(["r", "g", "b"])
for i in range(300):
    clr = next(colors)
    plt.scatter(i, d_in[i], c=clr)
    plt.scatter(i, d_out[i], c=clr)


If I understand you question correctly, there are two reasons why genetic algorithm might not a good idea for optimizing website features:

1) Feedback data is coming in too slow, say once a day, genetic algorithm might take a while to converge.

2) In the process of testing genetic algorithm will probably come up with combinations that are 'strange' and that might not be the risk the company wants to take.


What could be the reason of having a lower RMSE than MAE?

Yes, you are correct that in general $RMSE(x) \geq MAE(x)$ holds (see this answer for a good explanation of the different error measures and this paper for an interesting comparison of the two measures).

Therefore, your case must stem from other sources:

Randomness in models: the models you have applied are non-deterministic, e.g. your random forest randomly sub-samples features. This could either be solved by fixing all random parameters or measure RMSE and MAE on the same trained model:

model.fit(X_train)
y_pred = model.predict(X_test)
rmse = RMSE(y_test, y_pred)
mae = MAE(y_test, y_pred)

instead of

model.fit(X_train)
y_pred = model.predict(X_test)
rmse = RMSE(y_test, y_pred)

model.fit(X_train)
y_pred = model.predict(X_test)
mae = MAE(y_test, y_pred)

This answer shows a way to fix random seeds.

Randomness in data splits: Data splits are another source of randomness. So you either either make the split deterministic or measure both errors on the same split:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.7)
y_pred = model.predict(X_test)
rmse = RMSE(y_test, y_pred)
mae = MAE(y_test, y_pred)

instead of

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.7)
y_pred = model.predict(X_test)
rmse = RMSE(y_test, y_pred)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.7)
y_pred = model.predict(X_test)
mae = MAE(y_test, y_pred)


Using SVM as a binary classifier, is the label for a data point chosen by consensus?

The term consensus, as far as I'm concerned, is used rather for cases when you have more a than one source of metric/measure/choice from which to make a decision. And, in order to choose a possible result, you perform some average evaluation/consensus over the values available.

This is not the case for SVM. The algorithm is based on a quadratic optimization, that maximizes the distance from the closest documents of two different classes, using a hyperplane to make the split.

So, the only consensus here is the resulting hyperplane, computed from the closest documents of each class. In other words, the classes are attributed to each point by calculating the distance from the point to the hyperplane derived. If the distance is positive, it belongs to a certain class, otherwise, it belongs to the other one.


CV Pipeline for Time Series with Differing shape for X and y

I've done a bit of searching and have actually found a solution to this using tsfresh.

You can find the sklearn transformers here:
https://tsfresh.readthedocs.io/en/latest/text/sklearn_transformers.html

Here is the code snippet from the example.

pipeline = Pipeline(
    [
        ('augmenter', RelevantFeatureAugmenter(column_id='id', column_sort='time')),
        ('classifier', RandomForestClassifier())
    ]
)

df_ts, y = load_robot_execution_failures()
X = pd.DataFrame(index=y.index)

pipeline.set_params(augmenter__timeseries_container=df_ts)
pipeline.fit(X, y)

We pass an empty dataframe to pipeline.fit and specify the input data as a parameter to the tsfresh transformer. df_ts and y have a totally different shape!

:-)


Does the mean/median of a set sentence embedded vectors represent anything?

Well, the mean is pretty average for all the words.

These tend to all be quite similar, cluster in the center of the data, and have nearest neighbors to pretty bland, generic words.

The average word vector is not a good representation of what a text is about.


Solving a system of equations with sparse data

If I understand you correctly, this is the case of multiple linear regression with sparse data (sparse regression). Assuming that, I hope you will find the following resources useful.

1) NCSU lecture slides on sparse regression with overview of algorithms, notes, formulas, graphics and references to literature: http://www.stat.ncsu.edu/people/zhou/courses/st810/notes/lect23sparse.pdf

2) R ecosystem offers many packages, useful for sparse regression analysis, including:

Matrix (http://cran.r-project.org/web/packages/Matrix)
SparseM (http://cran.r-project.org/web/packages/SparseM)
MatrixModels (http://cran.r-project.org/web/packages/MatrixModels)
glmnet (http://cran.r-project.org/web/packages/glmnet)
flare (http://cran.r-project.org/web/packages/flare)

3) A blog post with an example of sparse regression solution, based on SparseM: http://aleph-nought.blogspot.com/2012/03/multiple-linear-regression-with-sparse.html

4) A blog post on using sparse matrices in R, which includes a primer on using glmnet: http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r

5) More examples and some discussion on the topic can be found on StackOverflow: https://stackoverflow.com/questions/3169371/large-scale-regression-in-r-with-a-sparse-feature-matrix

UPDATE (based on your comment):

If you're trying to solve an LP problem with constraints, you may find this theoretical paper useful: http://web.stanford.edu/group/SOL/papers/gmsw84.pdf.

Also, check R package limSolve: http://cran.r-project.org/web/packages/limSolve. And, in general, check packages in CRAN Task View "Optimization and Mathematical Programming": http://cran.r-project.org/web/views/Optimization.html.

Finally, check the book "Using R for Numerical Analysis in Science and Engineering" (by Victor A. Bloomfield). It has a section on solving systems of equations, represented by sparse matrices (section 5.7, pages 99-104), which includes examples, based on some of the above-mentioned packages: http://books.google.com/books?id=9ph_AwAAQBAJ&pg=PA99&lpg=PA99&dq=r+limsolve+sparse+matrix&source=bl&ots=PHDE8nXljQ&sig=sPi4n5Wk0M02ywkubq7R7KD_b04&hl=en&sa=X&ei=FZjiU-ioIcjmsATGkYDAAg&ved=0CDUQ6AEwAw#v=onepage&q=r%20limsolve%20sparse%20matrix&f=false.


Classifier and Technique to use for large number of categories

This sounds like a pretty gnarly problem, so a lot of finesse will be needed to solve it.  @Jérémie Clos has some good points, but I wanted to add some more general thoughts...

With the size of your problem, you might want to think about a scalable framework like Mahout or H2O rather than scikit-learn, which is an awesome shared memory library, but does have scalability limitations.

With a categorical feature with cardinality of 5000 and 80 million training cases, a uniform distribution will only give you 16,000 training cases for each positive result.  However, the distribution probably isn't uniform.  I suggest that you eliminate the least common categories until you have thrown out 5% of the data and assess the cardinality of the remaining 95%.  This may give you a better clue about how to proceed i.e. what percent of the cardinality remains?

Most multiclass classification algorithms are just lots of binary classifiers that are combined to produce a composite result, so will essentially do the work for you of splitting up the problem, binarizing the data (aka one-hot encoding), and training the classifiers, so you may not need to explicitly perform this splitting step.  Two exceptions to this, i.e. classifiers that can handle the multi-class problem intrinsically are decision trees and random forests (Mahout version). I would try both of these and hold them as benchmarks moving forward.

In terms of handling the addition of categories as data is added to the system, you will have to retrain sometimes.  This is sometimes referred to as the stability-plasticity dilemma.  The best way to handle this is probably only to make predictions on categories that comprise some threshold of the data and throw out the rest as outlier categories that are not powered well enough to predict on e.g. similar to the sanity check I suggest above.

Hope this helps!


Structured and unstructured pruning for deep learning models

You can do structured pruning for MLPs. You would remove things like whole neurons or layers (not specific weights).


Collaborative Filtering would definitely be a good start. This would work as follows. For each user, you can fill in rankings for restaurants they have not yet ranked by averaging the ranks (or some other aggregation) of similar users who have rated that restaurant.

Alternatively, you could take a content based approach. Here, each restaurant has a collection of keywords, and using or rating a given restaurant updates a user's profile for those keywords. Then, the user's profile can be compared (via cosine similarity or something) to all other restaurants profiles. Then you order accordingly.

It may be more useful to map into a latent space first, but up to you.


Data pre-processing before feeding into a deep learning model

It depends on the context of your problem.
If it is an sequence of images with not great quality then we need to perform some morphological operation on it e.g.grayscale,binary image etc.
Also if the digits are distorted,then we could translate it up-to certain extent for model to learn easily.


Should I fit my parameters with brute force

If you don't know which of the 10,000 sensors are good and which are bad, the data from those 10,000 sensors is useless for training a regression line / classifier.  You need labelled data, where you know both the value of the features and the health of the sensor.

Moreover, to be effective, you probably need your training set to contain both healthy sensors and bad sensors (where you know which ones are healthy and which ones are bad); it's not enough to just have data from bad sensors, because that doesn't tell you what healthy sensors look like.  In a pinch you could use one-class classification, but I don't recommend it -- your results will probably be poor, and you'll probably be better off obtaining labelled data from both healthy sensors and bad sensors.


Preparing Dataset Minority Class vs Majority Class

My question is: Does it matter if the minority class is set as the positive or negative label in relation to performance of training a model or affecting a loss function such as cross entropy?

No it doesn't.
However in binary classification it's customary to call "positive" the main class of interest, so be careful to be clear about which one is positive/negative when/if you present your results to other people.
Also be careful that precision and recall are usually calculated for whatever is called the positive class, so don't inadvertently use the results of the majority class instead of the one you're interested in.


Why a Random Reward in One-step Dynamics MDP?

In general, $R_{t+1}$ is is a random variable with conditional probability distribution $Pr(R_{t+1}=r|S_t=s,A_t=a)$. So it can potentially take on a different value each time action $a$ is taken in state $s$.

Some problems don't require any randomness in their reward function. Using the expected reward $r(s,a,s')$ is simpler in this case, since we don't have to worry about the reward's distribution. However, some problems do require randomness in their reward function. Consider the classic multi-armed bandit problem, for example. The payoff from a machine isn't generally deterministic.

As the basis for RL, we want the MDP to be as general as possible. We model reward in MDPs as a random variable because it gives us that generality. And because it is useful to do so.


From Labels to Graph: what machine learning approaches to use?

Your graph is characterized by a set of nodes and edges. It means given a start point x, show me the next node z1, then from z1 show me z2 and so on. This looks like the definition of graph-searching algorithm.
However, maybe by definining a 'cost function' that you want to optimize (minimize or maximize) you can convert it to a supervised learning task. Basically, given a graph composed of weighted nodes and edges, you want to choose a subset (E,V) that minimzes the cost function.
I am not sure this problem fits a generative problem (as you emphasize on the word generate) whose goal is to learn a distribution.


Lightweight execution of Spark MLLib models

You can export the model (Weights + Hyper parameters) to a common format from Spark and then execute the same in another Language/Framework.

https://docs.databricks.com/spark/latest/mllib/model-export.html

Above link has examples for Python / Spark, should work for Java as well.


reprocessing steps for images before training classification models

I assume you talking about preprocessing instead reprocessing.

Divided the image with 255.0 value is a normalization technic called min/max normalization. Like the other normalization methods, min/max normalization used to improve performance of CNN's.

Image process methods like gaussian blur, average filter etc. are using to remove the noise as you said. You might want to denoise your input image to avoid performance losses.

Using purpose of morphology methods are depends on what is your data and what do you want to do your dataset with these methods. You can refer this link to see the methods that cv2 shared with code snippets.

Greetings


For your first question, I would try to use Regex to identify the reference numbers since they seem unique comparing to normal words. I assume they are:

All capital
Start with one or more letters
After follow numbers
Any other pattern may follow (i.e. letters, numbers, hyphens)
Do not include spaces

Could be something like this: \b[A-Z]+\d+([A-Z]|\d|-)*\b

I can imagine you could do it better...


What is the difference between tensorflow saved_model.pb and frozen_inference_graph.pb?

saved_model.pb may represent multiple graph definitions as MetaGraphDef protocol buffers. Weights and other variables usually aren't stored inside the file during training. Instead, they're held in separate checkpoint files

.

frozen_inference_graph.pb has its variables converted into inline constants so everything’s in one file and ready for serving on any platform including mobile. Freezing process includes loading the GraphDef, pull in the values for all the variables from the latest checkpoint file, and then replace each Variable op with a Const that has the numerical data for the weights stored in its attributes It then strips away all the extraneous nodes that aren't used for forward inference. Reference: Tensorflow Model Files


DecisionTree split quality using chi-squared statistic

Base on another post about significance test for stopping decision tree in stats stackexchange website. CHAID, QUEST, GUIDE are three options. The QUEST was provide in binary format, but not available now.

The usage for an implementation of CHAID in R is expained here.
To install it on R

install.packages("partykit")
install.packages("CHAID", repos="http://R-Forge.R-project.org")


How to quantify ‘compute cost’ of training of xgboost model?

If you have the time on hand, you could simply measure the time taken for all combinations of hyper parameter values in a Grid Search, preferably with repetition. It's unlikely that any theoretical analytical expression will provide adequate accuracy for predicting the compute cost, as there as so many factors that contribute noise to the compute time.
You could even build a regression model to predict the computing cost of new hyper parameter combinations if you wanted to.


R: lattice equivalent of density2d in ggplot?

Use a custom panel function in which the density is estimated (e.g. using MASS::kde2d()):

library("lattice")
library("MASS")

set.seed(1);
dat <- data.frame(x=rnorm(100), y=rt(100, 5))

xyplot(y~x, data=dat,
       panel=function(x, y, ...) {
         dens <- kde2d(x=dat$x, y=dat$y, n=50)
         tmp <- data.frame(x=dens$x,
                       y=rep(dens$y, each=length(dens$x)),
                       z=as.vector(dens$z))
         panel.levelplot(tmp$x, tmp$y, tmp$z, contour=TRUE,
                         subscripts=1:nrow(tmp), region=FALSE, ...)
         panel.xyplot(x, y, ...)
       })


How to evaluate multi label image retrieval model

First consider analyzing the results per class (with normal accuracy/F-score/whatever) - this way you can have a good idea of where the algorithm is doing well and where it may be improved.
Consider only the exact match. It's a "hit" if the algorithm got the exact labels that it should have. You can then use the hit count to calculate precision, recall and F-score.
Hamming Loss: this is a favorite of researchers in my experience, it represents the how many of the total labels were misclassified

The relevant Wikipedia article seems to agree with me and adds the Jaccard index to the mix.

The paper that presents the famous Classifier Chains method (READ J. et al, Classifier Chains for Multi-label Classification, 2009) uses four different evaluation methods: a accuracy variation that closely resembles Jaccard distance, a similarly changed F1-score, and a log-loss function. The fourth method they use to evaluate is the area under the Precision-Recall curve, but that's one I'd argue should not be used (see the work on Precision-Recall-Gain curves by Peter Flach).


Why do neural networks use cost minimization of loss function and not profit maximization of profit function?

There is not necessarily a difference between the two, but the most important reason we minimize a cost function instead of maximizing a profit function is because of the optimization method. Neural networks are optimized using gradient descent, in which we use the derivative of the cost function to calculate the gradients and use the gradient to adjust the parameters. Assuming that you would use the accuracy as the profit function, it is not differentiable, meaning that you cannot calculate the derivate and therefore cannot use gradient descent (or in this case gradient ascent since we want to maximize a function). We then specify a cost function that is differentiable and minimize this as the cost function generally goes down as the profit function increases. You can use gradient ascent to maximize any arbitrary profit function as long as it differentiable.


How to restrict the values of predicted variable to be positive?

Use a generalized linear model which allows you to account for this. I do not recommend discretizing your continuous variable, you will lose a ton of information doing this and the size of the bins are arbitrary.

A standard linear model can be rephrased as a generalized linear model under a normal distribution and identity link function. Simply switch the link function to log, and derive your predictions = exp(XB), where XB is the linear predictor. Alternatively, change the distribution to a distribution with support on [0, infinity] like the gamma or inverse gaussian and use a log link. If you have count data (1 order, 2 orders, 3 orders, ...) rather than a fully continuous variable (550.567 dollars of orders, 300.234 dollars of orders, ...) then use a generalized linear model with a Poisson/log link instead.

As another alternative, log your predictor and assume Y = ln(target) follows a normal distribution. Then exp(Y) follows a lognormal distribution. That is, log your target variable, predict the logged target variable (using a standard linear model, normal/identity link usually) and then use exponentiation to get back your predictions. Note, however, that this might introduce some bias when you convert back to the standard unit. You can correct this by calculating the sample variance of your logged target variable in your training set and then letting your final prediction = exp(XB + variance^2/2). This comes from the expected value of a lognormal distribution.


In Naive Bayes classifier how is P(sneezing,builder|flu) = P(sneezing|flu)P(builder|flu)?

This is just a basic property of conditional independence. If two events A and B are conditionally independent, given event C then:

$$Pr(A \ and\ B \ |\  C) = Pr(A \ |\ C) * Pr(B \ | \ C)$$

or equivalently,

$$Pr(A \ | \ B \ and\  C) = Pr(A \ |\  C).$$

That is,

$$ Pr(A  \ and \  B | \ C) $$
$$ =\frac{Pr(A \ and \  B \ and \  C)}{Pr(C)} $$

(definition of conditional probability)

$$= \frac{Pr(A \ | B \ and\ C) * Pr (B \ and \ C)}{Pr(C)}$$
(by Bayes rule)
$$= Pr(A \ | \ C) * Pr(B\ | \ C) $$

(independence and definition of conditional probability again)

The independence assumption is what makes Naive Bayes "naive" because it is a bold assumption in general to think your variables are all conditionally independent, given a class. In Naive Bayes, we seek to find an expression for:

$$Pr(class \ r \ | x_1, x_2, x_3..., x_k)$$ where $x_1, x_2, x_3, ... x_k$ are the features/variables/predictors that we have. Applying Bayes rule;

$$Pr(class \ r\ | x_1, x_2,...,x_k) = \frac{Pr(x_1, x_2, ...x_k\ | class \ r) * Pr(class  \ r)}{\sum_{i=1}^m Pr(x_1, x_2, ...x_k\ |\ class \ i) Pr(class \ i )}$$

where m = number of classes.

The expression $Pr(x_1, x_2, ...x_k\ | class \ r)$, the conditional joint distribution of the predictors, is essentially impossible to estimate directly. So we make the assumption of conditional independence in the predictors, given a class (and that the distribution of $(x \ | \ class \  i)$ follows Gaussian, typically, but not always)

$$Pr(x_1, x_2, ...x_k\ | class \ r) = \prod_{j = 1}^{k}Pr(x_j \ | \ class \ r)$$

Hence:

$$Pr(class \ r\ | x_1, x_2,...,x_k) = \frac{\prod_{j = 1}^{k}Pr( \ x_j | \ class \ r) * Pr(class  \ r)}{\sum_{i=1}^m \left[\prod_{j = 1}^{k}Pr(x_j | class \ i) \right]Pr(class \ i )}$$.

We typically estimate the prior probabilities, $Pr(class \ i )$, using the MLE = sample proportion; # of training examples in class i / # of total training samples.


Problem when exporting scored labels to excel

As indicated by TitoOrt's comment the solution is simply changing the comma to the dot character as decimal point in Excel.

The reason was:

I am using the Spanish version of Excel. where is used the character dot (.) for decimals and not comma (,). AZURE uses dot for decimals and Office products in Spanish use comma by default.


Cluster documents based on topic similarity

As @Emre suggested, if you already have the distribution of topics in each document, you can represent each document as a vector $x_d \in \mathbb{R}^N$, where $N$ is the number of the unique topics in your collection. For documents, not exhibiting specific topics just fill the specific cells in each feature vector with zeros. Then, you can use some clustering algorithm such as nearest neigbors, using those feature vectors.

Example usage code in python below:

import pandas as pd
import numpy as np
from sklearn.metrics import pairwise_distances

# Initialize some documents
doc1 = {'Science':0.7, 'History':0.05, 'Politics':0.15, 'Sports':0.1}
doc2 = {'News':0.3, 'Art':0.5, 'Politics':0.1, 'Sports':0.1}
doc3 = {'Science':0.8, 'History':0.1, 'Politics':0.05, 'News':0.1}
doc4 = {'Science':0.2, 'Weather':0.2, 'Art':0.6, 'Sports':0.1}
collection = [doc1, doc2, doc3, doc4]
df = pd.DataFrame(collection)
# Fill missing values with zeros
df.fillna(0, inplace=True)
# Get Feature Vectors
feature_matrix = df.as_matrix()

# Get cosine similarity (i.e. 1 - cosine_distance) between pairs
sims = 1-pairwise_distances(feature_matrix, metric='cosine')

# Get the ranking of the documents given document 0
# from most similar to least similar. Don't take into account
# the first document, because it will be the same that the query was about
ranking_1 = np.argsort(sims[0,:])[::-1][1:2]
print ranking_1

ranking_2 = np.argsort(sims[1,:])[::-1][1:2]
print ranking_2

This takes 4 documents with $N=7$ unique topics, fills missing values with zeros and creates a similarity matrix between all documents. Then querying for documents 1(Science=0.7) and 2(Art:0.5) the most similar other document in the collection, we surely get documents 3(Science:0.8) and 4(Art:0.6) correspondingly.

You can try more sophisticated approaches regarding clustering and other distance metrics.


Can ridge regression be used for feature selection?

Regularization of Ridge causes its weight to become very close to zero, but not zero. In contrast lasso can make weights equal to zero because of the types of regularization they use.

I would recommend to use lasso if you have lots of features, and you think just few of them are important. Otherwise, even though your model becomes simpler you will have a poor accuracy.

There is ElasticNet which combines both, but it is more expensive since it uses two regularization.

I didn't really understand what you ask by

  Can the number of zeros in a weights vector change from zero to something else?

If you determined number of parameters beforehand their weights are unknown yet, not zero. That's what you are trying to find.

One last thing, for feature selection there are other methods. These(ridge, lasso) are just linear models for regression. If you want to identify which features work best etc. I suggest you to do a research on methods for this.


Clustering algorithm which does not require to tell the number of clusters

I would try DBSCAN algorithm first: fairly easy to tune (with, in particular, a notion of distance as you requested), and does not need to know the number of clusters.

There are a few other algorithms that can help you decide the number of clusters: Bayesian Gaussian Mixtures (see sklearn implementation) for instance, but it requires a bit more knowledge and work. There is also spectral clustering, but for this one, sklearn does not automatically find the number of clusters, so you will have to do create your own implementation and determine the number yourself, manually, by plotting the eigenvalues.


Understanding the Gini/AUC metric as out-of-development performance metric

The advantage which train/test/validation dataset separation has is that you separate your dataset into:

The individuals which you know the exogenous variables and the output: Training
The individuals which you know the exogenous variables and the output (but you suppose you don't know which the output is): Test
The individuals you know the exogenous variables but not the output: Validation

Every DS or ML model is made so it is prepared to receive a validation dataset in the future and try to get every metric just almost as good as if it was the train dataset.

The test dataset has the objective of simulating the situation of having data but not output, and then you have the output to measure the behaviour and comparing the modelled vs real output.

So, for a concrete answer:
The behaviour you should expect from the validation (or newly collected sample) is the same as the test dataset.

Given that the underlying phenomenon and sampling technique remains the same.

For more information:
https://towardsdatascience.com/train-validation-and-test-sets-72cb40cba9e7


Steps on how to use autoencoders to reduce dimensions

I will repost my answer on Quora here, about the "should I train the autoencoder on the whole dataset" part of the questions.

TL;DR: NO, you always use ONLY the training set for training, even for unsupervised learning with autoencoders!

What is the goal of using a test set ? It is meant to represent data that you model (including both the autoencoder and the random forest) have NEVER seen before.

Autoencoders are unsupervised learning models, but it does not mean that they cannot overfit. They can learn latent codes that do not generalize. Imagine an AE that encodes every data point x1, x2, … as a real number (one-dimensional code):

x1 -> 0.0001, x2 -> 0.0002, etc.

Do you think this autoencoder will generalize well ? No, it just remembered the entire training set and did not learn useful representations. That is why we use regularization in autoencoders, in very different forms : undercompleteness, sparsity, denoising, variational autoencoders (VAE)…

Conclusion, if you include the test set to train your autoencoder, it may in certain cases overfit, your performance estimates will be biased and you lose all the purpose of a test set.

(don’t hesitate to ask if you want me to detail some aspects of my answer or have more questions on this matter)

Concerning the architecture of you AE:

using a sigmoid output activation only makes sense if you data takes values between 0 and 1, as a sigmoid function squashes the output into the [0, 1] range. So either you normalize your data to the [0, 1] range, or you use a linear output activation.
With a sigmoid output activation, you should use a binary cross-entropy loss, which is designed for binary variables following Bernoulli distributions. With a linear output, use the mean squared error loss, which is designed for real-valued output (following Gaussian distributions).
Maybe Adam optimizer will converge faster that Adadelta. Try for yourself.


How to best handle imbalanced text classification with Keras?

First of all, to reassure you, SMOTE should work on text data. SMOTE will work on any data type as long as there is a way to compute the distance between data points.

Based on the error message you receive, it seems that it's an implementation issue (adding part of your code or how much data you have would greatly help).

As the error states, you have only 3 samples but the method requires at least 6. My guess is that something went wrong and you should have much more than 3 samples.


How to choose the best parameter values for TfidfVectorizer in sklearn library?

I think these parameters are mostly used when you combine the vectorizer and a machine learning model in a pipeline. Therefore, you should tune these parameters based on the outcome of your model training.

For example, if your task is to classify input texts, you may want to tune the max_features parameter such that the number of features is not too large, but your model can still perform reasonably well. In general, these parameters are used to control the size of the vector to be fed into your machine learning model, so as to avoid overfitting and improve the efficiency of model training.