I believe your problem boils down to clustering time-series of different lengths. According to your question, you want the longer time-series of a power user to be considered similar to time-series of similar pattern but much shorter.
Therefore you should look into clustering techniques and distance metrics which allow for these properties. I don't know your language of choice but here are some of the many packages in R that you might find interesting :
- Fréchet distance - one of the packages offering this is kmlShape
- Dynamic Time Warping included in base R
- Permutation Distribution Clustering - package pdc
This would also solve your data formatting problem as to setting values to -1 or NULL would not be needed anymore. hth.


In the Keras Tokenizer class, what exactly does word_index signify?

Are the words all words in texts, or are they maxed at SOME_NUMBER?

Yes, it will be maxed at the most common SOME_NUMBER-1 words.

  And are the dict values for word_index the frequency of each word, or just the order of the word?

It is just the index of the word in the dictionary.

You can read more here in the documentation.


It is a good practice to freeze the early layers when you finetune the model. There are two reasons why you want to do this :

Your new layers is initialized randomly and will always start with very big loss. Big loss means big gradient and if the weights are not frozen this will be propagated . Your model will become less stable and might fail to converge. To put it simply you don't want to put the blame of false prediction on the earlier layers.
You are saving a lot of computation resources. Training a neural network, is not a small issue, and training only a small section of the model will give you decent enough accuracy without expending too much computational resource.

Unfreezing the earlier layer is up to you on the later stage of training (Some people do some people don't) but if you feel you might need to do this to push for better performance, then just try it.


Probably it's problem with Pandas.Series, just change type to numpy array.

X_train[:,8] = impC.fit_transform(np.array(X_train[:,8]).reshape(-1,1))


What does the "randomly shuffle training samples" in stochastic gradient descent attain?

If you are not training with minibatches, but just one batch per epoch, then random shuffle does nothing. However, if you are training with minibatches, and the data in the first minibatch is related and vastly different from the last minibatch, the error that is back propagated from the first could be totally the opposite of the error in the final.

By shuffling the records, the minibatches as presumably more representative of the entire training data and so the back-propagation of the errors will be, hopefully, making slow but stead progress towards the global minima, not whipsawing back and forth chasing what would essentially be a class based minima.


Applying machine learning to cache algorithm

You need to validate your assumptions based on data driven analysis

Types of requests received on a day
Types of requests received on a holiday
Pattens for different times (Morning / Evening / Night)
Seasonality / Monthly / Weather based patterns

Post Analysis

Do you see correlation from Data & patterns
Do you see correlation from Data & Location
Do you see correlation from Data & Age / Gender / Locality

ML Models

Do you see correlation from Data & patterns
With the variables we can identity and model a forecast algo

ML Model is not the start. It is the outcome of detailed data and domain driven analysis


Dynamic pricing with aggregate constraints

Hi Asher11 and welcome to the community. I have worked in pricing in the past and I will try to give you some directions, although your question is quite general. It all comes down to how you want to formulate your problem. I suggest you to not look for models first but to figure out the problem you are facing and translate into math.

For example:

Do you want to know if a customer will buy a specific product according to some characteristics (features) of them? Then you have a prediction problem. You could possibly represent each customer with a set of features and try to predict if they will buy a product by training a Supervised Learning model. Start with linear regression in order to have a performance baseline.
Do you want to optimize profit/revenue? then you have an optimization problem and potentially you can use Reinforcement Learning methods to solve it. If the sequence of the events (offering and buying) does not play an important role in the decision then you can try algorithms from Bandit theory.

Another question you need to answer is why you want to use ML methods? You might be surprised but you will have a way easier solution by looking at Pricing Optimization with Linear Programming. You will need to make some assumptions about the supply and demand curve of each product (or a group of products), add appropriate constraints and optimize in order to get the best price. The product grouping might be done with ML methods or according to your intuition. By doing this exercise you will start getting an idea of the best way to formulate your problem.

Be aware in order to train an RL agent for optimizing profit/revenue it means that the agent will try random prices LIVE (RL training is based on trial and error). For this reason you need to look up for techniques on how you can pre-train a network to suggest prices within a specific range and then use RL live without the fear that the algorithm will output unreasonable values. Another way is to build a "market" simulator with the data that you have and train the RL agent directly there.

By writing the above, I tried to give you few keywords that you could look up in order to specify your problem better and possibly find a solution.


Anomaly detection in Time Series Data - Help Required

Welcome to DS forum. There are many methods available for anomaly detection of uni-variate, low dimension and high dimension data sets.

Supervised algorithms can be used if the past anomalies are recorded with flag variables and there is significant examples of flagged anomalies.

In unsupervised algorithms the key aspect of anomaly is which subset of the larger data set is considered when looking for anomolies i.e., global vs local and point vs contextual

Here are some useful links to get you started, anomaly detection in machine generated data and introductory overview of time-series based anomaly detection algorithms.

Hope this helps.


Sklearn - Override random_state=None by default

For scikit-learn can set np.random.seed(1), for example, and as long as nothing in your script is modifying the seed nondeterministically then you should get reproducible results.

This is described in the scikit-learn FAQ under How do I set a random_state for an entire execution?

However, I don't believe it is possible to do the same thing for pandas. See here for discussion.


What kind of data analysis problem is this?

You should start with exploratory data analysis to understand the data in as much detail as possible. Once you have a good understanding of the obvious patterns in the dataset,  you should then turn your focus towards answering the less obvious questions using modelling techniques.
The final item on the list that hints at building a recommender is an example of data modelling whereas all other items appear to be exploratory data analysis.
It appears this is an open ended problem. It is for you to look at the data, gain an understanding out of it and then figure out answers to interesting and less obvious questions that might interest the Business group.


RL's policy gradient (REINFORCE) pipeline clarification

For notation and visualizations please take a look at this excellent tutorial Policy Gradients.

For your questions:

The second is correct. In PGs we try to maximize the expected reward. In order to do this we approximate the expectation with the mean reward over samples of trajectories under a parametrized policy. In other words sample actions and get their respective rewards over a period of time within an episode. Compute the discounted return from last step back to the first step (this is your $V_t$ in your notation which usually you will find it as $R_t$ and is the discounted return). Multiply returns with logits and sum. Take a look at slides 8 and 9 of the (1) to see how the REINFORCE is being implemented along with this code examples (lines 59-75).
As you may have already realized, no. It's a return over an episode (multiple timesteps) and is calculated as a discounted sum of all the rewards that you collected. Even if you get 1 at the end and 0 everywhere else the reward is propagated back to the first step (doing it by hand helps a lot!) so all rewards at every timestep are converted into returns from that state and timestep.
Look at slide 13 of (1). Your intuition is correct. The maximum likelihood equation for a classification problem (cross-entropy) multiplied by the return equates the policy gradient loss. If you use a simple Neural Network with REINFORCE with two actions to perform a task you will notice that the gradients propagated back are the same as in a classification task but here are multiplied with their respective return (line 69 2) instead of the class label (0/1).

A bit of intuition for the 3: This is with accordance with PG methods considered as Model-free methods and map states to actions. If you use a clustering technique to cluster the hidden representations of the last layer of your network (after dimensionality reduction) and color the data points according to which action your network chose, you will find out that the representations are naturally clustered into separate groups depending the action. At a high level, you can state that the REINFORCE performs a type of classification having as label the reward signal.

Hope it helps!


XGBoost probability distribution tending towards the extreme

Since you're seeing probabilities concentrated near 0 and 1 (as is expected in gradient boosting), make sure you're not using Platt calibration; isotonic calibration is a better choice, or if you're willing to depart sklearn, beta calibration sounds promising.  Keep in mind that all these calibration methods should use a separate training set.  Also, have a look at calibration plots (which might show that Platt is unsuitable, and would also indicate whether the next paragraph is a possibility).

There's also a (slight) possibility that these scores are actually well-calibrated: e.g. if the data was generated in two clusters, and each cluster's points choose classes uniformly with probabilities close to 0 in one cluster and close to 1 in the other cluster, then "the right" answers are these probabilities, with the tree finding those clusters.

One approach to obtain "more regularly distributed probabilities" without actually calibrating them would be to make your model more conservative, using fewer trees and/or lower learning rate, more regularization, etc. but this risks hurting the model's performance.


As most of statistical models (regression, neural networks etc.) use probabilistic frameworks in their optimazation process (E.g. softmaxfrom a random distribution). You need a seedto reproduce your results. A seed is a fixed "value set" drawn from a random distribution.

If you would run your model several times you will get slightly different estimates as the starting values from your optimazation will slightly differ every time.

EDIT

Some additional information on the optimazation process when dealing with Neural Networks.

What you are optimizing in neural networks are the weights (connections) between the Neurons. Connections that have strong predictable character get somewhat stronger (more weight) relative to other connections with less predictive power. This process is done when you "train" your data. However to train your network you need some "inital weights" which are normally assigned by random (there are different approaches, but this is the most common one).

Like previously said - The inital weights will have impact on your optimazation and the results. That other users can check your model / reproduce your results you need to save the inital weights. - Therefore you use a seed


How can Clustering (Unsupervised Learning) be used to improve the accuracy of Linear Regression model (Supervised Learning)

Just an idea:

You may be able to cluster a continuous variable and add the clusters as indicator in a linear regression jointly with an interaction term.

Say your linear model is income (inc) explained by years of job experience (exp)...

inc = b0 + b1*exp + u.

Now say you have two groups (low/high skilled workers). If you cluster inc in two groups, and add this as a dummy with interaction term, you may be able to increase the fit of the model.

inc = b0 + b1*exp + b2*indicator + b3*exp*indicator + u.

The idea is that low/high skilled workers will increase income contingent on experience in a different way (different slope). However, this approach requires that you can identify clusters well.

Edit: same strategy may work with clustering the X.


There are several different methods.
One method to use Numba which is a just-in-time complier. Numba can compile Python to target CUDA. CUDA is a API for GPU computation.


How to build a model which can predict the probability of an event based on a set of timeseries data?

Since you have features that would be handled best with a recurrent neural net, AND some features that would be handled best with a feedforward net, what you can actually do is both and feed them into a main Dense layer which has a softmax output to give you the probability distribution.

This would be rather hard to do by hand, but luckily you are using Keras, which allows for this kind of modeling rather easily!

In the Keras functional API guide https://keras.io/getting-started/functional-api-guide/, there is a model actually very similar to what you are looking for, where the "Main" information is an LSTM layer (which you'd do for the stock prices), and the "Auxiliary" information would be (Name of entity, Type Of entity , Size , Country, Specific Attributes) etc...

The model looks like this:

The example model actually uses 2 loss functions (2 outputs), but you can easily build it to only have the one output. The code is all there so will be easy to replicate.

I basically use this kind of model for almost everything now and get great results, vs just LSTM alone.


Why activation functions used in neural networks generally have limited range?

The main goal of an Activation function is to add non-linearity but at the same time

It must not explode the large inputs else time to reach minima will become very long
It should have a smooth gradient, this will help the training keep going towards minima without being stuck

With your example function -

We will achieve non-linearity but gradient will be very large for big value and it will slow down the training

I believe you guessed this, so you asked the next question. What if I normalize the data -

Normalization helps in bringing two different features at the same level. But the larger value of the same feature will still be larger compared to the smaller one

But such function stops learning at both the extreme due to flattening gradient. Also, they require high computation
We have RELU without an upper limit. It's a good trade-off on all the mentioned points


How to determine a sufficient number of training examples for a linear regression classifier?

For every 15-25 samples, you can have 1 additional feature. In rarest cases it might be low as 10 samples for each feature, however, it is not recommended unless you have good knowledge of your data and can safely be sure or test that your model is not showing spurious results. For more, please read the source.
Using a few samples or breaking above mentioned rule of using too many features for a few samples (e.g. 80 samples 15 features) may cause overfitting. In this case, it will not be able to generalize your data instead will learn your input data in detail. However, those details will probably be redundant information or randomness of your data so, your model will be useless in production. Another possibility is that it may give a rise to spurious results. In other words, since your samples are few, your model can somehow find a way to imitate your target variable with no logical relationship but by chance.
Those, if you have few samples try to preserve the above-mentioned rule. If you have many features do a bivariate analysis to find how good are these features to explain your target variable alone. One other way is to use algorithms that are more robust against the overfitting and can handle many features of a few samples. For example, SVM is accepted as better than Linear Regression in such cases.


Convert a pandas column of int to timestamp datatype

You can specify the unit of a pandas to_datetime call.

Stolen from here:

# assuming `df` is your data frame and `date` is your column of timestamps

df['date'] = pandas.to_datetime(df['date'], unit='s')

Should work with integer datatypes, which makes sense if the unit is seconds since the epoch.


Label distribution over training, validation and test

yes of course, the model will learn to recognize the label 0.0 very well and maybe have an idea about the others, unbalanced dataset is common problem in ML, look at Oversampling and Undersampling


Answer

  Are there any heuristics for deciding on where to start with the number of layers for a neural network?

No there isn't one in general. The architecture of the network (including depth) is considered to the new "feature engineering". It's more of an art than science at the moment. Anything above 3 layers worth considering.

  I want to add a bunch of polynomial features

You don't need to do this kind of feature engineering like you would in traditional machine learning. If you have a network that is sufficiently wide and deep, your network should be able to "engineer" those features itself. Just think about how the first hidden layer is a linear combination of input features. And the second hidden layers is a linear combination of the first hidden layer. It should be obvious to see how a neural network is capable of structuring any polynomial features itself.

Suggestion

Try using the original 60 features directly and start with 3 layers. Given the size of the network, the training should take less than 10 minutes on a GPU-enabled machine. Compare your result with your benchmark model (you should always have a benchmark model). Play around with the hidden size and layers, rinse and repeat.


Applying and Visualizing k means clustering on a data set that has 9 features

You may use a dimensionality reduction algorithm like PCA method after performing clustering, to reduce the dimension of your clustered data into two dimensions and then visualize the clusters.


In general, you randomly splits the available data into following 3 sets. Train, validation and test. [Different ratios could be used depending on total amount of data at hand and the difficulty of the problem.] You can start with a simple 80%/10%/10% split.
You use the first two sets to build your model. So you would use the 80% split of your data to lets say build a logistic regression model. Now you use the 10% validation set to see how good your model is. You can iterate on this process until you are satisfied with the validation data set performance of the model.

Now, you use the last set (the "test" split) to see how your model would generalize on an "unseen" data set. [Here unseen means, your model never had access to "see" this data while in the learning phase]. You never use use this 3rd set to glean knowledge about how your model could be improved. Think of this set as a set your customer has, and you don't have access to, and to whom you are going to deliver your model.

Once you get a hang of this concept also learn more about how you can do cross-validation if you have limited data and to improve confidence on your model.


Correctly evaluate model with oversampling and cross-validation

I believe, the sequence for combination of CV and SMOTE should be as below.

1. Perform the 5 fold-CV ( Loop through for each fold )
2. Training Sample and Testing Sample ( for each fold )
3. Smote Training Samples
4. Train the model on the "training"
5. Prediction ( test samples )
6. Evaluate performances on test
Repeat for next fold


What kind of learning is needed for anomaly detection? Supervised learning, semi-supervised learning or unsupervised learning?

Supervised learning is when your model learns from a well-labeled dataset. By well-labeled dataset, I mean every row is categorized into a class. The model learns patterns from this dataset and makes a prediction. One flaw of supervised learning is that it only makes a judgement based on the patterns it has seen in the training dataset.

Unsupervised learning is the opposite of supervised. You have a dataset but you don't know anything about the classes of the dataset. In this case, you try to identify the patterns among the dataset and try to cluster the same patterns together and a cluster forms a class. The number of clusters depends on the data scientist and the data.

Semi-supervised learning algorithms are trained on a combination of labeled and unlabeled data. This is useful for a few reasons. First, the process of labeling massive amounts of data for supervised learning is often prohibitively time-consuming and expensive. Also, if the data is labelled by humans, it can contain bias from our judgment.

Anomaly detection falls under the bucket of unsupervised and semi-supervised because it is impossible to have all the anomalies labeled in your training dataset. There are several methods to achieve this, ranging from statistics to machine learning to deep learning. To have a detailed idea on these things, refer to the following link  https://www.datascience.com/blog/python-anomaly-detection


User-to-item and item-to-user recommendations

You might try several techniques to achieve your goal, I have researched and used these before, so it might help you.
Semi-Supervised Learning
Basically you need some historical data linking the products the users, therefore you can establish a primary knowledge of the relationship between some products and the users after that you cluster the same products/users and you label the rest of unlabeled products based on that.
Association rule learning
is a rule-based machine learning method for discovering interesting relations between variables in large databases. It is intended to identify strong rules discovered in databases using some measures of interestingness.
Collaborative learning/Federated learning
It's better to use it when you have informations about the buyers/sellers, therefore you can study the relationship between the product/(buyers/sellers), then products/shops, then you can supervise the learning between the products/shops.You need a high content quality data.
Cooperative Learning
You run diffrent test on diffrent combination of users/items, and you derive diffrent results from them, then you compare them against each other. For this you need some pre-determined rules to evaluate your own decision making results. You need some historical data of the products/users.
Combine collaborative learning and Cooperative Learning
This is what Google search engine is doing, he know what other people are looking for and build patterns for which is called collaborative learning, but if they give the results of it, it will not be specific for you as a user, so they add Cooperative Learning to emphasize your own search results experience. As interesting as it might sound it not that easy to code.


Multilingual Bert sentence vector captures language used more than meaning - working as interned?

Sadly, I don't think that Multilingual BERT is the magic bullet that you hoped for.
As you can see in Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (Wu and Dredze, 2019), the mBERT was not trained with any explicit cross-lingual task (for example, predicting a sentence from one language given a sentence from another language).
Rather, it was trained using sentences from Wikipedia in multiple languages, forcing the network to account for multiple languages but not to make the connections between them.
In other words, the model is trained with predicting a masked instance of 'cat' as 'cat' given the rest of the (unmasked) sentence, and predicting a foreign word meaning 'cat' in a masked space given a sentence in that language. This setup is does not push the model towards making the connection.
You might want to have a look in Facebook's LASER, which was explicitly trained to match sentences from different languages.
P.S
The fact that the sentences do not have the same representation does not mean that the mBERT cannot be used for zero-shot transfer learning across languages. Again, please see Wu and Dredze


What more does TensorFlow offer to keras?

Deep Learning frameworks operate at 2 levels of abstraction:

Lower Level: This is where frameworks like Tensorflow, MXNet, Theano, and PyTorch sit. This is the level where mathematical operations like Generalized Matrix-Matrix multiplication and Neural Network primitives like Convolutional operations are implemented.
Higher Level: This is where frameworks like Keras sit. At this Level, the lower level primitives are used to implement Neural Network abstraction like Layers and models. Generally, at this level other helpful APIs like model saving and model training are also implemented.

You cannot compare Keras and TensorFlow because they sit on different levels of abstraction. I also want to take this opportunity to share my experience of using Keras:

I do not agree that Keras is only useful for basic Deep Learning work. Keras is a beautifully written API. The functional nature of the API helps you completely and gets out of your way for more exotic applications. Keras does not block access to lower level frameworks.
Keras results in much more readable and succinct code.
Keras model Serialization/Deserialization APIs, callbacks, and data streaming using Python generators is very mature.
Keras has been declared the official high level abstraction for TensorFlow.


Data Interpretation with a double line graph

That looks a double axis or dual axis graph but isn't well done or I don't understand clouds well enough. It is almost like a Sankey diagram. Problems:

What does the X axis convey, if anything?
It appears that the cloud types are relatively stratified but I'm not sure what causes the waviness over the X axis. Is it related to the two lines for temperature and pressure?

Compare that to a phase diagram like below. I'm not a chemist, but this is a lot easier for me to interpret. The x axis is the temperature, the y axis is pressure, and the lines clearly show where the material switches phases, similar to cloud types. There isn't ambiguity at any point about which phase would be in place. Now try figuring out what type of cloud you would have in your diagram at 30C and 8psi.


is neural networks an online algorithm by nature?

There are three training modes for neural networks

stochastic gradient descent: Adjust the weights after every single training example
batch training: Adjust the weights after going through all data (an epoch)
mini-batch training: Adjust the weights after going through a mini-batch. This is usually 128 training examples.

Most of the time, mini-batch training seems to be used.

So the answer is:

No, the neural network learning algorithm is not online algorithm.


BERT is a transformer-based model. The pre-trained BERT can be used for two purposes:

Fine-tuning
Extracting embedding

You don't need to use an SVM once you're keyed into a BERT architecture. Your BERT model will generate embeddings and can be fine-tuned (ala ULMfit last layer) to perform a specific task. You could potentially just use the embeddings and then perform the task with another model but the performance would likely not be better.

So, how you want to use BERT still remains a choice. But if you can fine-tune the BERT model, it would generally yield higher performance. But you'll have to validate it based on the experiments.


Negative learning implementation in pytorch

One implementation is
( (loss+loss_neg) / (float((labels>=0).sum())+float((labels_neg[:,0]>=0).sum())) ).backward()

from NLNL-Negative-Learning-for-Noisy-Labels GitHub repo.


That sounds like time series forecasting. Here's a related post:

https://stats.stackexchange.com/questions/212912/forecast-time-series-data-with-external-variables


What ML techniques can be used to determine context across sentences?

What you want to do is to extract the relation within the sentence and in this case beyond - so basically a cross-sentence relation extraction.

Ambiguity and the context of words is a big deal when doing this kind of task. There might be multiple ways to get there, but it also depends on your goal (production environment, research or just doing it for fun to get into NLP).

I would recommend you looking into Graph LSTMs (e.g. this Paper)


Threshold logic unit (McCulloch-Pitts 1943) vs. Perceptron (Rosenblatt 1958)

MP model : 1) inputs are binary values; 2) has not weights
Rosenblatt model: 1) inputs can take any real numbers; 2) has weights.
Thank you, https://medium.com/@manushaurya/mcculloch-pitts-neuron-vs-perceptron-model-8668ed82c36


Count occurences of rows based one cell content with pandas

When asking code related questions, its always a good practice to at least provide a sample of the data. Since, you've not given one, I'll create it based on your question.
import pandas as pd

df = pd.DataFrame([('A','01','prod_1'),
                   ('A','01', 'prod_2'),
                   ('A','02','prod_X'),
                   ('B','05','prod_1'),
                   ('C','06','prod_2'),
                   ('C','06','prod_X')],
                 columns=['customer', 'ordernum', 'product'])
print (df)
        customer ordernum product
0        A       01  prod_1
1        A       01  prod_2
2        A       02  prod_X
3        B       05  prod_1
4        C       06  prod_2
5        C       06  prod_X

Now, you can use size() or Counter like,
val_count = df.groupby(['product','ordernum']).size()
print(val_count)
product  ordernum
prod_1   01          1
         05          1
prod_2   01          1
         06          1
prod_X   02          1
         06          1
dtype: int64

prod_order_count = df.groupby(['product', 'ordernum']).agg(Counter)
print(prod_order_count)
                  customer
product ordernum
prod_1  01        {'A': 1}
        05        {'B': 1}
prod_2  01        {'A': 1}
        06        {'C': 1}
prod_X  02        {'A': 1}
        06        {'C': 1}


Classification or regression? Which model is more accurate if I only care about being above or under the threshold?

Classification is the more direct approach and it will likely give better results. This is because the model's goal is exactly the same as your goal - i.e. predicting whether the price is above or below the threshold - and it will maximize this accuracy.

A regression model that is trying to minimize MSE, for example, could give you a model that doesn't accurately put predictions on the correct side of the threshold - which it doesn't know or care about. The model does not have the same goal as you do.

Let's construct a very simple example to see why this might be the case:

Training     Test
x      y     x      y
1.0    20    1.8    ?
2.0    20    2.4    ?
3.0    200   2.8    ?
4.0    200   3.2    ?

Here I used the rule if x<2.5: y=20, else y=200 to generate the data.

A maximum-margin classifier would get all of the test examples right, but a linear regression model fit on the training data would get all of the test examples wrong.


Training error is oscillating, cross validation error is continuously decreasing

By definition, the training error should always decrease with a proper learning rate until it reaches his plateau, then starts to oscillate because it's crossing forward and backward the minimum of the loss function.

Instead, what you expect from the validation error is that it decrease with the training error, and at some point it starts to increase, sign of over fitting.

And it is not unusual that at the begin of the training you found the validation error lower than the training error.

From your plot I would say:

Train and validation errors are simply swapped: the red line is actually the training error, and the blue one is the validation error;
Stop training just after the crossing point. This because the validation set is just a set where you tune your parameters, and not the set of the "Revealed Truth", so it is also prone to over fitting.

If you make a third dataset ("test" dataset), and you try to check your metric (accuracy?) on it, I bet that the model trained after two epochs performs better than the model trained with ten epochs.


using "OneVsRestClassifier" from sklearn in Python to tune a customized binary classification into a multi-class classification

Hi you need to implement get_params method to export all your hyper-parameters or inherit your classifier from sklearn.base.BaseEstimator since under hood of OneVsRestClassifier, every classifier is being distributed. so your classifier is being cloned.


What is wrong with my Precision-Recall curve?

Overall the performance of the model is very low, since precision and recall can be up to 1.
Since model performance is low, it will be sensitive to noise resulting in erratic behavior.


Several options you can try, all of them have different preconditions you need to consider.

1. Direct Ensemble (with high number of n, should be the first choice)

If n >> 103 you can directly use standard ML procedure different models and parameters with CV and find the best fit. Since most of the time its not the case, you may want to use ensemble tree models since they outperform other learners when number of observations are relatively small compare to number of features.

2. Clustering

If those 100 features are sourced by a common factor, you can cluster those features with optimal K (see silhouette score) and add those cluster labels as 3rd feature on your model. Finally your model becomes n observations, 3 features.

3. Fit Twice

In case of classification, you can first fit 100 features and use their probability score (predict_proba in sklearn) as 3rd feature on your model.

4. PCA + Fit

You can use PCA to 100 features and obtain p features. After that with p+2 features you can construct your model. Again common factor condition applies in here. (In direct approach you can apply PCA to 102 features as well.)

Final Comment:

All of above methods are should be tried and the best one should be picked. However, if you have n >> 102 observations you should directly fit the model with strong learners (xgboost, lgbm, nn's etc.) without clustering/twice fitting since those methods generates error by their nature in addition to final learner error.


Activity recognition in smart homes with different sources

Data-fusion is the most probable way to "fight" with all of the data sources. However, additionally, I would strongly suggest imposing an ontology and some set of rules.

Basically what you can do is the following:

-proximity beacons (to understand which room the user is at any moment) based on this sensor data you can introduce localization, decision trees may the most suitable way to go here while imposing some clear manually added rules(if the person is in this room, then goes into this room, etc). Basically, before doing the overall data fusion the localization may be really helpful in increasing the accuracy because you can introduce the ontology with specific activities that can be done in some of the rooms. (The person is not expected to be running in the kitchen or the bathroom for example).

ground pressure from the floor - this type of sensor can be really useful for localization as well.

Also, before data-fusion, the Accelerometer from wrist data readings should be carefully pre-processed. I would strongly suggest finding a nice way how to remove the bias(left handed/right-handed people) and how to remove the DC component(that is the overall gravity of the Earth gravitational field, remove this bias as well)


Neural Nets: Backpropogation from Conjugate Gradient Descent

Yes,

see the article Optimization in an error backpropagation neural network environment with a performance test on a spectral pattern classification problem


Multi-Class Text Classification: Doc2Vec performing very bad compared to Hashing Vector

If I read your model correctly, you only performed 5 epochs with the Doc2Vec model.  This is probably not be enough for the network to learn the word embeddings.  Has your loss leveled out after 5 epochs?  Try running it for 50 epochs and see if it makes any difference.  Conceivably, you could need thousands of epochs to achieve a reasonable model.


What are the prerequisites before running Holt Winters Model?

1) Holt-Winters doesn't require stationarity the way an ARIMA model does, so you don't need to perform any of the steps you mention. However Holt-Winters is specifically designed for seasonal data, so don't bother applying it unless you have good reason to think your data is seasonal.

2) In the basic formulation of Holt-Winters, no, it can't. More recent implementations of Holt-Winters use the state space approach to time series modeling, and it might be possible to do so using the state space approach - although even that is still in debate. BSTS is another state space model with a structure similar to Holt-Winters, and that approach does allow for causal factors. You should check that.

3) No, you don't need to. The whole point of Holt-Winters is that it accounts for seasonality, so you don't need to adjust for it.

4) See here. Holt-Winters can't handle missing values, but other methods can. You should be careful with Outliers, since you don't know whether an outlier is a "real" outlier, a seasonal spike, or a causally driven event. For example in a sales time series, your spike might be a true outlier, or it might be an event like Black Friday, or it might be the result of a promotion and advertising campaign.


which is the most effective(accurate) face detection method in python

You can use the Facial landmarks library, it's very easy to use
you will get 68 point and compare them from your dataSet very dirty and easy way of doing things note that

The users must take the pictures at the same position to keep the same depth


Is there any good practice to cluster 3D data array?

Given that allmpst all clustering algorithms assume data is unordered, reshaping the data into some n*p format ist indeed appropriate. If you want to take positions into account, you'll have to encode them as additional features (which can prove to be tricky because of scaling and feature weighting).

But don't treat clustering as a black box. You may have some particular goal in mind, and adequately preparing the data is a must for clustering. Consider k-means: it searches a least-squares appropximation. It's you job to prepare the data in a way that least-squares on these features is useful.


Are there any examples of neural networks that take two samples as input with a label of {same class, different class}?

Yes. This is how a face recognition algorithm might work for example, where two pictures might be of the same person or different person.

To build such a system, just pair up your data in a training set, double the input vector space and run a binary classifier that outputs "true" if the two items are the same. Any ML classifier could be adapted to this problem.

In practice, you may want more control over classification, and be robust against new classes that your algorithm has not seen before. A big problem in face recognition is the large number of potential classes, including classes not seen in training data, coupled with an equally large variance of images in the wild. This means that a naive approach as above will have poor performance in practice.

There is a more sophisticated approach: Triplet Loss. This requires that you train with three inputs for each example. Unlike the naive version:

The input is one image at a time. You train in triplets - an "anchor", and a postive match, and a negative match, in order to calculate one loss value for backprop.
The output is not a same/different class, but a vector description of the object. Similarity of objects is the distance between vectors. You do not need a label for this vector, even though this is supervised learning, thanks to how the loss function works.
The loss function is based on difference of distance from anchor example to positive example compared with a desired higher distance to negative example. This encourages learning key features of the inputs by making distance between vectors low (ideally zero) when they represent the same class, and as high as possible when they represent different classes.

In both the naive approach and triplet loss approach, you need to be careful about selecting training data. You want to make the learning algorithm work hard to learn key differences, otherwise it is too easy to get a good loss. So taking MNIST as an example, when looking at negative matches, you want to pair up more (1, 7), (3, 2), (3, 8), (4, 9) etc pairs than (0,1), (9, 5). There are ways to drive this selection based on feedback from previous training epoch.


When using an unsupervised alogirthm, what is the "learning" part since it belong to machine learning field?

You said:

  and based on my personal understanding, this "adaptation" is only possible with "supervised algorithm"

However, it is not true. Let's consider a clustering method as an unsupervised algorithm like K-means. Each iteration in K-means algorithm is a basis for the next iteration of the K-means up to the convergence of the algorithm. Hence, it means we are learning the structure of the data in each iteration up to reaching a specific target value. Although it is unsupervised, we are learning the structure of the data in each iteration. Indeed, it is a kind of (unsupervised) learning at all.


Neural Nets: unordered sets of ordered tuples as features of data

Your features are already numerical, so you can simply pass your array of numbers into a MLP.  For instance, if you have 5 sets of (x,y) coordinates per sample, your MLP will have 10 inputs.

One-hot encoding is used to convert categorical features into numerical values.  This image from Kaggle does a great job of illustrating how you might use one-hot encoding:

If you were to directly convert categorical values to numerical values, that would imply that one color is greater than another, so we need to use a boolean array instead.

As for the variable length feature vector, it is difficult to give an exact answer without knowing more details about your problem, but here are general possible solutions:

use a recurrent neural network
use distance-based regression/classification algorithms such as k nearest neighbors
transform your data into feature vectors of fixed length (which may not be reasonable depending on your data and what it represents)

A quick high level introduction to neural networks might be helpful.


Maybe you could use Orange3-Text add-on, widget Preprocess Text, Tokenization > Regexp. The source code indicates it's a Python regex, so you might be able to use a regular expression pattern such as:

(?ix)        # ignore case, ignore comments and whitespace in this RE
(?<=age:\s)  # preceded by 'age: '
.+           # characters you wish to match
(?=gender:)  # followed by 'gender:'


which forecasting models could be chosen?

Should the data be considered as a multidimensional time series?

This depends on whether the target variable (the one that you want to predict) depends on the others. If not, there is no point in doing it. A fast way of checking if the variables are linearly dependent and, therefore, multidimensional forecasting is meaningful is by checking the linear correlation of the variables. Then select only the variables that have high correlation (>0.5) with the target variable to include them in your prediction model.

  I am still thinking which methods, techniques can be chosen.

The model that I recommend for time series forecasting is a Recurrent Neural Network. This is because of its inherent ability to store previous timesteps in its memory and to incorporate them into future predictions. This is very important, because it is among the few approaches that exploit the temporal dependencies between samples.

  I'm currently looking into the direction of some kind of neural
  network and implementation in python (but I still don’t know which
  package).

The most convenient way of implementing a recurrent neural network in Python is by utilizing the Keras framework. Please go carefully through this tutorial, as it will definitely be a very good first step to attack your problem.


scikit-learn - Should I fit model with TF or TF-IDF?

In the word sampling steps in LDA the word count is used as weights for the multinomial dist. Re-weighting the TF's by its IDF's would dispropotionally increase the chance of rare words being sampled, making them have a stronger influence in topic assignment. But rare words have equally low probabilities of showing up in all documents and topics, hence flattening out the topic distribution of each document.

That is a similar reason why it is recommended to remove stop-words before running LDA.


Labels are not given for multiclass classification problem

Your problem is associated to "unsupervised" learning in machine learning. You do not have a data set that has training data - meaning that data points with correctly specified labels are not known yet.

You can try different approaches to group/label your data set using the given features. You probably have to check by yourself if your model is "auto"-labeling your data correctly.

Clustering (k-Means)
Decision Trees
Auto-Encoding with NNs

More Approaches


Advantages and disadvantages of using classification tree

One main advantage of using Decision Tree is that you can visualize your prediction more precisely than any other classification approaches.

In addition,

What rules make any specific prediction? You can generate such rules
What feature/value is crucial for what decision? You can indirectly do the feature selection.

There are some shortcomings as well; sometimes, the tree might get quite large, so you have to reduce the size of the tree (pruning) with the cost of accuracy.


Visualising K-Means clusters for 3D data in R

Is it necessary to plot all points? If not, you can use plot3d to display 6 bubbles centered at each clusters center. You can use mean within distance for each cluster to set the radius of each cluster (you will have to normalize this) and a color range to display the number of points in each cluster.


Exploratory Data Analysis and selecting good predictor variables ?

This is an interesting but broad question.

Imagine PCA. You yse it for exploring the data embedded in lower dimensional space but the first $n$ principale components are also used as the features (after projection of data on them).

Or you use correlation analysis and remove (deselect) features with high correlation with an existing feature.

You calculate the variance of each feature abd low variances tell you that there is no infirmation in this feature.

You inspect feature distributions according to target to determine how much they contribute to the prediction.

And of course much more ...


How do CNNs use a model and find the object(s) desired?

Though there can be a very detailed explanation for this question but I will try to make you understand much minimal words.

1) Cropping the images to a particular size isn't a necessary condition and neither is scaling. But put this way, it doesn't matter whether a dog is represented in a B&W image or  RGB image because a convolution network learns features in the images which are independent of colors. Scaling and resizing help to limit the value of pixels between 0 and 1.

2) Once you have trained your CNN model, it has learned all the features like edges,etc. to recognize a dog in the image. Because the model has learned the features, it acquires certain properties like translation invariance which means that no matter where you position a dog in the image, it's still a dog and have the same features. How the model recognize it? It checks for the features of a dog, learned during training, no matter what the size of the new image is or where the dog is in the image or what the dog is doing.

For getting a in-depth understanding you can refer to the following resources:

http://neuralnetworksanddeeplearning.com/chap6.html

http://cs231n.github.io/convolutional-networks/


If you have a disproportionate amount of zeros, it means you models doesn't have enough data in order to learn how to correctly classify observations. Because it sees zeros almost all the time, it probably has learned to output zeros.

The main thing you can do to solve this problem is to train your model using mini-batches, and build them by sampling 0- and 1-observations with equal probability - in other words, attributing greater weight to 1-observations compared to zeros. In that way, your model will be fed with balanced data and will learn to classify them correctly.


Time series prediction of discontinuous data

Yes, it is time series with missing values. In fact you have process $X_t$, which is room temperature in moment $t$, but you have observations in specific time intervals. So your goal is to find out missing values between non-observed intervals.  There are several methods and packages for solving problems like this, here is an implementation of some methods in R:


How to add a filter/query/condition in Plotly

Assuming your DataFrame is rwgor, you need to filter it like this:
rwgor = rwgor.loc[rwgor.attractionName >=165]
fig = px.bar(rwgor, x="attractionName", y="attraction score summary", color="attraction score summary", title="Long-Form Input")
fig.show()

or
fig = px.bar(rwgor.loc[rwgor.attractionName >=165], x="attractionName", y="attraction score summary", color="attraction score summary", title="Long-Form Input")
fig.show()

Finally, note that I used rwgor.attractionName >=165 and not rwgor.attractionName.value_counts() >=165, according your plot description.


How can I use data on customer interactions to drive communication strategy?

I would do the following:

First, cluster your customers into several groups, based on age, location, gender etc. Second, for each group, use different combination of interaction as features to predict 'yes/no', and select which combination of interaction performs best.


VAR model ValueError: x already contains a constant

Your two columns X0 and X4 are constants, i.e. they contain a single value throughout.

The model will be trying to find a constant during its fit, so probably has a check that you're not including one.

Once you have your entire dataframe (e.g. called df), you can remove constant columns like the two above by using:

df_no_constants = df.loc[:, (df != df.iloc[0]).any()]

Then try putting that into your model as before.


TL;DR

Pass the data twice in HDBSCAN.

Cluster based on the tag using word embeddings and cosine distance.
Sub-cluster the cluster from step 1 with your existing method (using the remaining 12 features).

Explanation

I suggest you do it in two steps and not give a weight to the tag feature, because the distance metric used is different. For word-embeddings, you will need to use the cosine distance between embeddings. Where as for the other 12 features you are currently using another distance (Euclidean I assume).

The first step should cluster your data on the semantical characteristic of your tag. This should cluster things like fruits, meats, vegetables, pastas...

Then, the second step can sub-cluster the data with your other 12 features. However, given your example

  cluster 1: Banana, Apple

  cluster 2: Pasta, Spaghetti

I don't see why this second step is necessary. You could instead of clustering a second time, just use the 12 features for ordering the data points for the purpose of your exercise. E.g. getting the top "fruits", as clustered at step 1, that people "enjoy" the most.


Can AI (NLP) convert user questions (text) into database SQL queries?

Yes it is possible. You can train end to end an RNN over the training data. It means input the user query as an input and set the output for the sql query.
Definitely the most challenging part is preparing the training set as RNN needs a lot of data to be learned.

Also, for the training set, it might you can use a GAN to generate more queries for the users' query dataset that you have. Notice that, as you want the query generator for the specific schema of the database, you need your own training data and it could not be generalized over all kinda schemas.


Visualizing a large graph (10'000 nodes)

Gephi may be able to handle it. Graphistry definitely can but you may need to pay. Ultimately though, unless it fits some specific graph structures, most if not all major algorithms will produce an unintelligible mess.


Named Entity Recognition (NER) would extract names of people, organizations and such. Example:

"Penalty missed! Bad penalty by <person>Felipe Brisola</person>  - <organization>Riga FC</organization> -  shot with right foot is very close to the goal. <person>Felipe Brisola</person> should be disappointed."

So it could be helpful for the "person" field, but probably not for the rest. Note that you could also train a system similar to NER in order to predict other fields, but it would require a good amount of annotated data and it's not sure to work well.


Hosmer Lemeshow Test, variable lengths differ

I think it could be that the model is being used and not the data, try:

hoslem.test(data$y, fitted(model), g = 10)


How to define quadratic weighted kappa as eval_metric in catboost classifier

Sklearn kappa has 2 variants: simple and weigthed. It seems WKappa refers to weighted (quadratic) kappa in terms of sklearn, and Kappa - to simple.


Transforming data points to match other data points

For this task, you need to decide 3 things:
(1) What are the allowed transformations.
For example do you allow an arbirtrary affine transformation (rotation, scaling, translation). Then you need a matrix $f \in \mathbb{R}^{3 \times 3}$ (using homogeneous coordinates) that maps the points from the left image to the right one. Each entry of the matrix is a variable you want to determine. You could also restrict to specific affine maps (e.g. only rotations) by setting certain entries in $\mathbf{A}$ fixed.
In principle you can use  any  map $f_{w}: \mathbb{R}^{2 \times 2} \rightarrow \mathbb{R}^{2 \times 2}$, that depends on some variables $w$.
(2) You need to think about a measure (or "loss"), which evaluates how similar the two sets are. For arbitrary sets, there is the  Hausdorff distance, which ignores any temporal information. If you consider your data as two curves over time, there is the Frechet distance.
You can have a look here to see that its a difficult problem to  using the Haussdorf distance in general. In fact,  computing only the Hausdorff distance, without any optimization is already  not trivial.
However, for polygonal curves, you can compute Hausdorff distance, Frechet distance, and discrete Frechet distance efficiently.
Finally, in your specific case, you have for each point of $A$ a corresponding point in $B$, you can measure the distance for each correspondence (this gives you simply the mean squared errors in $\mathbb{R}^{2}$).
(3) Depending on the choice of (1) and (2), you need to employ a corresponding optimizer.
If you want to minimize the pairwise distance, but you do not have the correspondences available, there is the Iterative closest point.
If you simple compute the distance between corresponding points, you have a non-linear optimization problem. You can use solvers such as Levenberg–Marquardt.
As @bogovicj points out, the topic is called point-set registration.


Is there a better way to optimize linear regression than gradient descent?

If by better you mean finding the better separator, no you can't due to the fact that the cost function for linear regression is convex which means there is just one optimal point.

If you want to optimize using different algorithms, there are different kinds of solutions. Gradient-based algorithms like Adam and RMSProp are of those. You also can use normal equation.


drop columns and rows in one line in pandas

Why don't you do this:

df = df.iloc[:3, :3]

It returns the same thing as your code.


It doesn't make sense to standardize your test set with the mean and variance computed on the test set. At best the test mean and variance will be close enough to the training mean and variance to not break your model - but why not guarantee they are the same by using the same standardization procedure for both. Typically it is best to choose one of the following:

Compute mean and variance on entire data set (train + test) and use these to standardize each set.
Compute mean and variance on train set and use these to standardize each set.

If you are not worried about leaking information from the test set then choose (1), otherwise choose (2).

If you standardize the test set with different values than the training set you might end up with train and test data from different distributions, which will lead to failure to make accurate predictions.


Word embeddings for Information Retrieval - Document search?

Doc2Vec is on possible approach. With this, model learns to "cluster" similar sentences together.

Most simplistic approach is to aggregate word vectors but that ignores order of words. Details on few of the approaches :

https://towardsdatascience.com/sentence-embedding-3053db22ea77
https://medium.com/explorations-in-language-and-learning/how-to-obtain-sentence-vectors-2a6d88bd3c8b


Increasing the sentence length might help, but not much. You can try n-grams(2 or 3) while vectorizing the data. Generally more number of sentences will help.


How to use Cross Entropy loss in pytorch for binary prediction?

Actually there is no need for that. PyTorch has BCELoss which stands for Binary Cross Entropy Loss. Please check out original documentation here. Here is a quick example:

m = nn.Sigmoid() # initialize sigmoid layer
loss = nn.BCELoss() # initialize loss function
input = torch.randn(3, requires_grad=True) # give some random input
target = torch.empty(3).random_(2) # create some ground truth values
output = loss(m(input), target) # forward pass
output.backward() # backward pass


Concept of xml files for haar cascade in object detection with opencv?

XML files are generated when you train the Haar Cascade Classifier. From the OpenCV docs:

Object Detection using Haar feature-based cascade classifiers is an effective object detection method proposed by Paul Viola and Michael Jones in their paper, "Rapid Object Detection using a Boosted Cascade of Simple Features" in 2001. It is a machine learning based approach where a cascade function is trained from a lot of positive and negative images. It is then used to detect objects in other images

More information on Haar cascades can be found here.
Some tutorials for training Haar Cascades are available.

Link 1
Link 2
Link 3


Following module provides a Python API to IMMA files:

import IMMA

You could use write()

opfile=open('some_output_file.imma','w')
for ob in obs:
    IMMA.write(ob,opfile)

More info could be found here.


Due to high demand, it is possible to start a career in data science without a formal degree. My experience is that having a degree is often a 'requirement' in job descriptions, but if the employer is desperate enough, then that won't matter. In general, it's harder to get into large corporations with formalized job application processes than smaller companies without them. "Knowing people" can get you a long way, in either case.

Regardless of your education, no matter how high demand is, you must have the skills to do the job.

You are correct in noting that advanced statistics and other mathematics are very hard to learn independently. It is a matter of how badly you want to make the career change. While some people do have 'natural talent' in mathematics, everybody does have to do the work to learn. Some may learn more quickly, but everybody has to take the time to learn.

What it comes down to is your ability to show potential employers that you have a genuine interest in the field, and that you will be able to learn quickly on the job. The more knowledge you have, the more projects you can share in a portfolio, and the more work experience under your belt, the higher level jobs that will be available to you. You may have to start  in an entry level position first.

I could suggest ways to study mathematics independently, but that isn't part of your question. For now, just know that it's hard, but possible if you are determined to make a career change. Strike while the iron is hot (while demand is high).


How to test accuracy of an unsupervised clustering model output?

Since you have the actual labels, you can compare them with the obtained labels and evaluate performance. Typically purity and nmi (normalized mutual information) are used. Read this (Evaluation of Clustering) document for detailed explanation.

If you don't have actual labels then you can use modularity or [Silhouette] for measuring clustering performance.


How to estimate probabilities of different classes for a Text

If I understand correctly, we are interested in soft multilabel classification, where a single text can have multiple correct genres.

According to your comment, we don't have any training data, just a list of keywords associated with each genre.

We can try computing the similarity between each document and each keyword list:

Normalize the document (convert to lowercase, remove punctuation, diacritics, non-alphanums, etc)
Remove stopwords
Convert the document to tf-idf vector over our genre keyword vocabulary: Each document gets an n-length vector where each entry is the frequency of the ith genre keyword in the document. Normalize this vector to magnitude 1.
Convert each genre keyword list to a tf-idf vector in the same way (again over the keyword vocabulary for all genres).
Compute the cosine similarity between the document vector and each genre vector.

For each document, this will give us a number in the range [0,1] for each genre. For example:

         Comedy Drama Fiction Romance Mythology Adventure
Text #1: 0.15   0.11  0.03    0.00    0.00      0.07

If we were doing single label classification we could normalize each row to add up to 1 and we might have a working model. However there is no such trick for multilabel classification here. We don't have a good way to calibrate these values into probability estimates.

At this point the only solution I see is to build a small training set so we can fit our model to actual data.

After gathering some training examples, we can run a multilabel regression with sigmoid activation and binary crossentropy loss with the cosine similarities as input features to get a probability estimate for each class.

Using this method our list of genre keywords will at least save us having to build a large training set to solve the problem directly with bag-of-words or similar approaches.


Are euclidian vectors and unit vectors same thing?

A unit vector in a normed vector space is a vector of length 1. A unit vector is an euclidian vector of length 1. Not every euclidean vetor has length 1.


How do you integrate nlp into an existing website?

The answer is it depends on you architecture.

Some things to consider:

The tags are attributes of words in a (presumably) text field in a database. So Boyce Codd would prescribe putting it in a seperate table referencing the primary key (and possibly the position in the text)
The tags are just a function over your data, so storing it at all should be for caching purposes. Which you might not want to do in your database, but at the presentation level, or perhaps in a separate database, or schema.
The tags are in fact the product of an approximation of a function over your data. There is no such thing as a perfect NER-tagger. This means that you might want to switch the function later buy something more suited.

It could make sense to extract the information on display time from a microservice (perhaps hosted by you), and cache the resulting presentation for better performance.


One hot encoding at character level with Keras

I think that you are looking for the keras Tokenizer with the char_level=True flag:

from keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer(char_level=True)
tokenizer.fit_on_texts(your_dataset_train)
sequence_of_int = tokenizer.texts_to_sequences(your_dataset_train_or_test)

Now that you have sequences of Integer, you can use keras.utils.to_categorical =)


How to get vector representations(or embeddings) of time series?

Maybe the framework of Neural Processes could be interesting here? It defines a  family of functions parameterized by a neural network. The parameters could serve as your embeddings, eventually after projecting into a lower dimension. See the paper Attentive Neural Processes and the preceding papers cited therein.


Clustering based on features of varied importance

If education level and income are more important than other features, you can multiply those features by a factor (greater than 1). That will allow the  clustering algo to focus more on those features than the rest.
For larger datasets, with features where the differences are not so obvious, you will need to rely on your judgement based on the end objective of the clustering. If you have a target variable in mind, you want to choose or boost those features that are highly correlated with the target


Implementation of reliable rule learning

I think this paper just compares algorithms:
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.294.9414&rep=rep1&type=pdf
If you want something specific, here's the white paper for SLIPPER:
https://www.aaai.org/Papers/AAAI/1999/AAAI99-049.pdf
Hope that helps.


normalize dependent variable with respect to one independent variable

Though continuum regression methods appear to be significantly different from binary classification methods, the crux of the difference is usually tied up in a sigmoid function.

$$S(t)=\frac{1}{1+e^-t}$$

Sigmoid literally means sigma like or "S" like, as becomes obvious in looking at its graphical representation:

The sigmoid provides a tractable means to map from continuum analytics into discrete space while retaining differentiability and integrability needed for some theoretical derivations.

The important part is that in classification methods, one has to choose a threshold value of the sigma function for mapping to a 0 or 1.  This is overwhelmingly chosen as 0.5, but there are cases of unbalanced classes where other values might work better.

That said, you should be able to proceed exactly as you have already suggested: "I would have simply divided it by the respective averages and removed the effect."  Do this, but then you have to turn the results back into 0's and 1's by choosing where your cutoff threshold lies.

Hope this helps!


Which algorithm to apply for choosing the right point

This is a fascinating problem!
Two things make it especially challenging:

How should we compare two point sets? Classical problems in Machine Learning have a fixed number of attributes, and these attributes are not interchangeable: For example, I might have data on different persons with attributes age and height (in centimeters). Every sample has one entry for each, and of course (age, height) = (22, 180) is not the same as (age, height) = (180, 22).
Neither is true in your problem. A point set has between 3 and 10 points, and the order in which we enter the points should not make a difference when comparing two point sets.
How do we make a prediction? Say we have found a way to pick point sets from our training set that are similar to your point set above. We face the problem that our prediction must be one of the 7 points in your picture; but none of these points might be contained in the similar point sets.

Let me outline an algorithm that deals with both challenges. The prediction accuracy is not very good; but maybe you see a way how it can be improved. And at least it predicts something, right?

1. Simulating samples

To be able to test the algorithm, I wrote functions that generate samples and labels.

Generating samples:
Each sample contains between 3 and 10 points. The number of points is random, drawn from a uniform distribution. Each point is of the form (x_coordinate, y_coordinate). The coordinates are again random, drawn from a normal distribution.

import numpy as np
from random import randint

def create_samples(number_samples, min_points, max_points):

    def create_single_sample(min_points, max_points):
        n = randint(min_points, max_points)
        return np.array([np.random.normal(size=2) for _ in range(n)])

    return np.array([create_single_sample(min_points, max_points) for _ in range(number_samples)])

Generating labels: As a toy example, let us assume that the rule for choosing a point is: Always pick the point that is closest to (0, 0), where 'closest' should be understood in terms of the Euclidean norm.

def decision_function_minnorm(sample):
    norms = np.apply_along_axis(np.linalg.norm, axis=1, arr=sample)
    return sample[norms.argmin()]

def create_labels(samples, decision_function):
    return np.array([decision_function(sample) for sample in samples])

We can now create our train and test sets:

n_train, n_test = 1000, 100
dec_fun = decision_function_minnorm

X_train = create_samples(number_samples=n_train, min_points=3, max_points=10)
X_test = create_samples(number_samples=n_test, min_points=3, max_points=10)
y_train = create_labels(X_train, dec_fun)
y_test = create_labels(X_test, dec_fun)

2. Comparing point sets via Hausdorff distance

Let us tackle the first problem: How should we compare different point sets?
The number of points in the point sets is different.
Also remember that the order in which we write down the points should not matter: Comparing to the point set [(0,0), (1,1), (2,2)] should yield the same result as comparing to the point set [(2,2), (0,0), (1,1)].
My approach is to compare point sets via their Hausdorff distance:

def hausdorff(A, B):

    def dist_point_to_set(x, A):
        return min(np.linalg.norm(x - a) for a in A)

    def dist_set_to_set(A, B):
        return max(dist_point_set(a, B) for a in A)

    return max(dist_set_to_set(A, B), dist_set_to_set(B, A))

3. Predicting via k-nearest neighbors and averaging

We now have a notion of distance between point sets.
This makes it possible to use k-nearest neighbors classification:
Given a test point set, we find the k point sets in our training sample that have the smallest Hausdorff distance relative to the test point set, and obtain their labels.
Now comes the second problem: How do we turn these k labels into a prediction for the test point set? I took the simplest approach: average the labels and predict the point in the test point set that is closest to the average.

def predict(x, num_neighbors):
    # Find num_neighbors closest points in X_train.
    distances_to_train = np.array([hausdorff(x, x_train) for x_train in X_train])
    neighbors_idx = np.argpartition(distances_to_train, -num_neighbors)[-num_neighbors:]

    # Get labels of the neighbors and calculate the average.
    targets_neighbors = y_train[neighbors_idx]
    targets_mean = sum(targets_neighbors) / num_neighbors

    # Find point in x that is closest to targets_mean and use it as prediction.
    distances_to_mean = np.array([np.linalg.norm(p - targets_mean) for p in x])
    closest_point = x[distances_to_mean.argmin()]

    return closest_point

4. Testing

Everything is in place to test the performance of our algorithm.

num_neighbors = 70
successes = 0
for i, x in enumerate(X_test):
    print('%d/%d' % (i+1, n_test))
    prediction = predict(x, num_neighbors)
    successes += np.array_equal(prediction, y_test[i])

For the given decision function and num_neighbors = 70, we get a prediction accuracy of 84%.
This is not terribly good, and it is of course specific to our decision function, which seems fairly easy to predict.

To see this, define a different decision function:

decision_function_maxaverage(sample):
    avgs = (sample[:, 0] + sample[:, 1]) / 2
    return sample[norms.argmin()]

Using this function via dec_fun = decision_function_maxaverage brings down prediction accuracy to 45%.
This shows how important it is to think about the decision rules that generate your labels. If you have an idea why people choose certain points, this will help you find the best algorithm.

Some ways to improve this algorithm: (1) Use a different distance function instead of Hausdorff distance, (2) use something more sophisticated than k-nearest neighbors, (3) improve how the selected training labels are turned into a prediction.


From the description of your problem, it sounds like applying a weight to the input prior to sending the data to the classifier might do the trick and would be easy to implement.  The weight could be positive or negative.


Multi-ouput Classification + Sentiment Analysis

The "low hanging fruit" approach is "bag of words".

Lemmatize the sentence and use the lemma frequencies as features.

Build two models:

Sentiment (or reuse StanfordCoreNLP)
Topic (for this you need a training sample - a set of sentences labeled with the topic).

Now you can map every sentence to topic and sentiment and this will tell you whether the customer liked or disliked (sentiment) the softness (topic).

Multi-topic sentences

Usually a sentence has a single topic, but, if you do require multiple topics per sentence, you can use predict_proba or similar and use all classes with probability over, say, 0.3.

Class proliferation

The above approach fails on sentences like

  I liked the softness but hated the design.

because the total (average) sentiment of the sentence is probably neutral and we miss out softness-good and design-bad.

It is thus tempting to triple the number of classes (topic-sentiment) to accommodate reviews like

  The table combines pleasant design, flimsy build and neutral color.

The problem is that building models for very many classes requires a lot of training data. IOW, you will need to manually annotate a lot of reviews - and even then the model quality if unlikely to meet your expectations.

Real NLP

If you really need to handle multi-topic sentences, you might want to consider a "deeper" approach.

Specifically, take a look at the parse tree of the sentence.

split it into constituent phrases
use the topic model to identify the unique topic of the each phrase
use the sentiment model on the phrase


Cluster Analysis - Comparing Same Individuals Clustered Across Different Datasets with different features

Literature

The closest line of work to your problem is Multi-View Clustering. Each data set DS is considered as a view, and views share a central entity (e.g. an individual).

2004
Multi-View Clustering is a survey on the topic. Also, here are two git projects from two papers (git 1, git 2) on the topic (they are implemented in Matlab and unfortunately are not production-ready).

A shortcut

Since most mainstream clustering methods (e.g. K-means, DBSCAN) accept a custom distance matrix $d(x, y)$ instead of explicit data vectors $x$ and $y$, partiality of features can be handled outside the method.

For example, suppose $x$ and $y$ are both present in views $1$ and $3$ but not view $2$. You may first build their vector by fusing their features from views $1$ and $3$, then computing their distance $d(x, y)$. Do the same for all pairs of individuals based on their common features, and finally feed the distance matrix $d(X, Y)$ to a clustering algorithm. This way, based on your domain knowledge, you can manually assign different weights to features of different views too. For example, you can define the distance as
$$d(x, y) := \left \| x[f_1:f_3] - y[f_1:f_3] \right \| + 2\left \| x[f_4:f_5] - y[f_4:f_5] \right \|$$
which implies that the difference in view $3$ with features $f_4$ and $f_5$ is twice as important as the difference in view $1$ with features $f_1$ to $f_3$.

Comparing two partitions

If views are clustered separately, two partitions $p_1$ and $p_2$ respectively
 over views $1$ and $2$ can be compared using metric Normalized Variation of Information (NVI). If partitions are the same, it outputs $0$, and if they are statistically independent, it outputs $1$. A similar measure (that is not a metric) is Normalized Mutual Information (NMI).

If all individuals are not present in both partitions, either (1) remove non-shared individuals from both partitions, or (2) individuals in $p_1$ that are not present in $p_2$ can be assigned to an imaginary cluster $misc_2$ in $p_2$, the same for those in $p_2$ that are not in $p_1$ using $misc_1$ in $p_1$.


How to define the adequate cash prize sizing for hosting a Kaggle or similar compeition?

This is a generic question that may need more than just cash prize.

1) Organizational Reputation
2) Dataset Size
3) Defining Problem statements with Domain knowledge
4) Time period
5) Computation required for the contest.
6) How curated/clean is the dataset
and much more.

If the problem statement is huge and the reward is in single digits, its not worth anyones time and at the same time, if the problem statement is mediocre and huge money is rewarded, it does not serve the purpose of the Kaggle competition at all.

even though Kaggle provides GPU and 16GB ram with CPU's, sometime the Kernel crashes and there is no work around unless you spend some money on AWS/Azure instances.

I hope this gives few pointers


Clustering

I think a tree is a perfectly appropriate data structure in this case. You don't need an embedding to do clustering; there are similarity-based approaches too, and defining a similarity function in your case is straightforward: I'd say it's a function of the common depth between the two items in consideration. You will probably want to define a root to "normalize" the scores. For example, given the root "a/b/" and the paths:

a/b/c/d/foo1
a/b/c1/foo2
a/b/c1/foo3
a/b/c/d/foo4

The shared depth between foo1 and foo2 and 0, discounting the root. For foo1 and foo4 it is 2. For foo2 and foo3 it is 1. You can define the similarity as a transformation of this through the function $f:x \to 1-\exp(-ax)$, where $a$ is a parameter you can use to tweak the clusters.

For implementation, in python, you can try sklearn.

General

You can featurize a path string by creating a set from its parent folders, which can then be represented as a sparse bit string by hashing the set elements. For example, "a/b/c/d/foo1" becomes ("c", "c/d"), if we let "a/b" be the root, as before. (The notion of a root is not strictly necessary here except to ensure that the baseline similarity is zero.) The path similarity is simply then the set similarity or, after conversion to bit strings, the $L_p$ distance.


I am not sure if I understand your thinking, but my understanding is following - maybe it can help you see it from another perspective.

The multi-label classification problem with n possible classes can be seen as n binary classifiers. If so, we can simply calculate AUC ROC for each binary classifier and average it. This is a bit tricky - there are different ways of averaging, especially:

  'macro': Calculate metrics for each label, and find their unweighted
  mean. This does not take label imbalance into account.

  'weighted': Calculate metrics for each label, and find their average,
  weighted by support (the number of true instances for each label).

as explained here.

There is another option (which may be the closest to how you see it):

  'micro':
  Calculate metrics globally by considering each element of the label indicator matrix as a label.

which is implemented here.

There is also the fourth option, which I don't really understand:

  'samples':
  Calculate metrics for each instance, and find their average.

I have been using myself weighted averaging.


RBF kernel can classify two classes as in figure?

Your understanding that an RBF kernel can make points linearly separable only if they are located in perfect circular way is not correct. For your dataset, is very easy to use the RBF kernel to separate the two classes.
For simplicity let's assume you are using a SVM classifier.
It is true that the RBF kernel looks spherical, not elliptical. However, the decision boundary is also determined by the support vectors. Unless you only have one support vector for one of the classes, the decision boundary is usually NOT spherical.
You can easily visualize the decision boundary given by SVM. For example, I simply modified the code here (using sklearn.svm) to use your dataset, and set C=10, the result looks like:

So a more general question is:
How does the "shape" of kernel affect the shape of the decision boundary?
We can visualize the effect with some self-defined kernels. In the figure below, in the upper row we plot the shape of the kernels used (same color denotes the same kernel value), while the lower row shows the SVM decision boundary with that kernel.

So we can see that the "shape" of the kernel does affect the shape of the decision boundary, but only its details. The overall shape is still determined by the actual supported vectors. BTW in this case all 17 data points are supported vectors.


Choosing the right parameters to train a Tf-Idf vectoriser

There are no silver bullets. But here are some suggests:

Use a better stopwords vocabulary. If you still have words like "to" and "at", then you are either not removing stopwords or using a lousy vocabulary. Try using the Spanish stopwords from nltk:

from nltk.corpus import stopwords
stopwords.words('spanish')

Use max_df < 1. This will truncate words that appear in more than that percentage number of documents.
The TF-IDF part that punishes common words (transversal to all documents) is the IDF part of TF-IDF, which means inverse document transform. Several functions may be used as your IDF function. Usually, IDF=$\log\frac{\text{#documents}}{\text{#documents where word appears}}$ is used. You could try a more punitive IDF function. sklearn does not seem to allow to specify it, but you can use nltk or gensim or easily implement your own TF-IDF vectorization. It needs no more than five lines of code.

I would try each of these suggestions in this order, and stop when it is good enough. It sounds like using a better stopwords vocabulary will be good enough for you.

EDIT: I forgot to mention, but of course you can add more uninteresting words to the stopwords list, if you have the need to do so.

You may also want to first evaluate the impact TF-IDF is having in your counts. I think this will work:

m = TfidfVectorizer().fit(docs)
c = [m.transform([word]) for word in m.vocabulary_]
for i in np.argsort(c)[::-1][:20]:
    print(m.vocabulary_[i], c[i])

You may have to adjust the code. Try to use with IDF disabled and enabled, and other tweaks.


Why/When should I use VGG16 to do fine-tuning?

I think you mixed up the terms that you do here. The task is called transfer learning and you FINETUNE the model.

The reasoning is that after experimentation we found that earlylayers of CNN captures the general features of images such as edges,lines, etc, and later layers capture more specific features e.g. faces, shape of object. We only wish to alter the few layers or even only the output classifying layer.
Pretrained model is trained with datasets that are most likely bigger than your dataset. Also because of reason number 1 and we have trained it with bigger dataset, we have first few layers that is able to generalize well and hence helps you combat overfitting.
You save time and resource by doing this. First because you are only backpropagating within last few layers.
Architecture-wise, people spend time researching how every features interact with image and hence it is recommended to use this architecture and published/publicly known result implies that it gives good result.

So in general it is recommended practice to do transfer learning for image classification unless you have a very strong reason not to do so. There are many resources that tells you how to do it.

Imo, VGG16 is outdated, it is also heavy and inefficient. You might want to check efficientnet.

Medium source regarding CNN filter. Medium source


what does an auto-encoder do?

The simplest auto-encoder takes a high dimensional image (say, 100K pixels) down to a low-dimensional representation (say, a vector of length 10) and then uses only those 10 features to try to reconstruct the original image. You can imagine an analogy with humans: I look at someone, describe them ("tall, dark-haired, ...") then after I've forgotten what they look like, I try to sketch them using only my notes.

what are we trying to learn?

In other words, why bother? A few reasons:

dimensionality reduction: 10 features are a lot more convenient than 100K pixels. For example, I can perform classification by clustering in the 10-dimensional space (while clustering in the 100K-dimensional space would be intractable).
semantic meaning: if all goes well, each of the 10 features will have some obvious "explanation" -- e.g., tweaking one value will make the subject look older (though it's normally not so simple). As opposed to pixel values, which are impacted by translation, rotation, etc.
Exception recognition: if I train my auto-encoder on dogs, it should normally do a good job encoding and decoding pictures of dogs. But if I put a cat in, it will probably do a terrible job -- which I can tell because the output looks nothing like the input. So, looking for places where an auto-encoder does a bad job is a common way to look for anomalies.

I have seen examples where an image goes from a non smiling to a smiling face, or to a black and white image to a colored image.

There are many different types of auto-encoders. What I described above is the simplest kind. Another common type is a "denoising" auto-encoder -- instead of reconstructing the original image, the goal is to construct an image that is related to the original image, but different.
The classic example of this is denoising (hence the name): you can take a clean image, add a bunch of noise, run it through an auto-encoder, and then reward the auto-encoder for producing the clean image. So, the input (noisy image) is actually different from the desired output (clean image). The examples you give are similar.
The challenge in designing these types of auto-encoders is normally the loss -- you need some mechanism to tell the auto-encoder whether it did the right thing or not.

about the VAE, it uses a probabilistic approch, so we have to learn the mean and covariance of a gaussian.

A VAE is a third type of auto-encoder. It's a bit special because it is well-grounded mathematically; no ad-hoc metrics needed. The math is too complicated to go through here, but the key ideas are that:

We want the latent space to be continuous. Rather than assigning each class to its own corner of the latent space, we want the latent space to have a well-defined, continuous shape (i.e., a Gaussian). This is nice because it forces the latent space to be semantically meaningful.
The mapping between pictures and latent spaces should be probabilistic rather than deterministic. This is because the same subject can produce multiple images.

So, the workflow is this:

You start with your image as before
As before, your encoder determines a vector (say, length 200).
But that vector is not a latent space. Instead, you use that vector as the parameters to define a latent space. For example, maybe you choose your latent space to be a 100-dimensional Gaussian. A 100-dimensional Gaussian will require a mean and a standard deviation in each dimension -- this is what you use your length-200 vector for.
Now you have a probability distribution. You sample one point from this distribution. This is your image's representation in the latent space.
As before, your decoder will turn this vector into a new "output" (say, a vector of length 200K).
But, this "output" is not your output image. Instead, you use these 200K parameters to define a 100K-dimensional Gaussian. Then you sample one point from this distribution -- that's your output image.

Of course, there's nothing special about a Gaussian, you could just as easily use some other parametric distribution. In practice, people usually use Gaussians.
This sometimes gives better results than other auto-encoders. Further, you sometimes get interesting results when you look between the classes in your latent space. An image's distance in the latent space from the cluster center is sometimes related to uncertainty.
Moreover, there is the nice property that these high-dimensional Gaussians are probability distributions in a rigorous mathematical sense. They approximate the probability that a given image belongs to a given class. So, there is some thought that VAEs will be able to overcome the "hand waving" of deep learning and put everything back on a firm Bayesian probabilistic grounding. But of course, it is only an approximation, and the approximation involves a lot of deep neural networks, so there is still plenty of hand waving at the moment.
By the way, I like to use this question during interviews -- an astonishing number of people claim to have experience with VAEs but in fact do not realize that VAEs are different than "regular" AEs.


What is the definition of knowledge within data science?

The definition of knowledge varies based on the context, but can be broadly defined as actionable information based on further analysis (statistical, heuristic, or otherwise) of those patterns, associations, or relationships identified in the raw data.

Put another way, knowledge is defined by what information will most prove actionable based on the needs of the client or user (sometimes called the semantics).


If you're working in R, Carson Sievert's tutorial on using LDA to model topics in movie reviews is an excellent starting point:

http://cpsievert.github.io/LDAvis/reviews/reviews.html

This tutorial makes use of LDAvis, an interactive visualization of topic and word distributions that can really aid intuition.

Also, although not short, David M. Blei's lectures on topic models are a great resource for understanding the meaning behind the parameters: http://videolectures.net/mlss09uk_blei_tm/


Grouping/clustering similar words python

You won't be able to just solve this with an algorithm.

Instead you will need to solve this more data, about language and abbreviations. This is also known as world models background data etc.


I would recommend to look into character level named entity recognition. For example: Kuru et al, CharNER: Character-Level Named Entity Recognition, Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (2016)

The authors evaluate on many highly inflected languages including Turkish, so this should be adequate for your Finnish use case

The code is here: https://github.com/ozanarkancan/char-ner

You should hopefully be able to download and get it running out of the box for training. Of course I am assuming you have a tagged NER corpus in Finnish, which you would need to preprocess to get into the same format as the CSV file that they use for Czech in the repo.


calculating the precision and recall for a specific threshhold

y_pred = log_reg.predict_proba(X)[:, 1] > 0.63
print(precision_score(y_true, y_pred), recall_score(y_true, y_pred))


Here are my two cents.

1) While there are some things that you may want to do differently (see part 2), what you observe is normal. If you had a trivial problem with a simple landscape, all of the methods would converge on exactly one subset of features. Feature selection in the real world, however, is rarely a trivial problem (as long as you are looking for "true" optimality and not a "good enough" solution). You are facing a search space of size $2^{60}$, which is impossible to assess in its entirety, so navigation methods will vary. RFECV is a top-down approach that attempts to shed as many weak features as possible based on the CV scores, SelectFromModel relies on estimator's internal objectives to do the selection, while SelectKBest in its default form is a simple filter that computes ANOVA F-values without considering feature interactions. Since the criteria are so different, you are bound to end up with different feature subsets -- each method operates on a landscape that is "distorted" by its objective function. The fact that the subsets you have found intersect suggests that the common features possess some sort of optimality (when all used approaches are considered). While you are at it, consider sequential feature selection methods, especially the floating approaches (both bottom-up and top-down), as they can be a great deal more effective than plain RFE / KBest.

2) You are using SMOTE to balance the training data. From your problem statement it seems that you are introducing synthetic data points before feature selection. If this is the case, then please consider the implications: for RFECV (if you do something like rfecv.fit(X_train, y_train)), the validation sample is not drawn from the original distribution, but from the one already affected by synthetic data, so the CV scores that drive the selection are not based on the true qualities of the features. This discrepancy may be too small to actually affect anything, but the fact remains. In your case, the class balance is 70/30, which is not that bad. Do you actually observe any increase in performance after balancing? As far as metrics go, the area under Precision-Recall curve is stable irrespective of the balance.

3) Decision trees (extensively used in many flavors, including XGBoost and CatBoost that you mentioned) have feature selection embedded into the process of learning. This means you can just plug in full featured data into the learner and have it figure out the most important features completely on its own. It's very effective and usually sufficient. However, in order to have variety of opinions within the ensemble, forest methods (most of them, anyway) will give all features an equal chance of being used, so it's very likely that you will end up with a situation where all features have some importance -- you will still have to decide which ones to keep. Weak features with noise that correlates with your labels by pure chance can affect your model's ability to generalize and they can even trick you into keeping them as useful. Also, when your sample size is limited (and it always is), increasing the number of features does not help the confidence of your assessments, either. Without enough observations, how can you be sure that the label is what it is because of this one feature and not ten others? So delegating it all to the forest without any preliminary or followup steps is not necessarily the best course of action.

The answer to the general question of the post: How to Maximize recall for Minority class?

This is quite straightforward. Define recall on minority class as your objective, i.e. let it be the output of the scoring function that you supply to your feature selection algorithm. For sklearn's feature selection methods it's the score_func parameter, for mlextend's SFS it's scoring. The resulting feature subsets are likely to contain something of interest.

Only after performing a thorough search for optimal features, selecting the best kind of model for your problem and having tuned its hyperparameters, if you still for whatever reason want to push your precision/recall further, you may want to look into stacking -- this will squeeze out the last bits of performance juice at the expense of model complexity. But the chances are that it's really not worth it.


better confussion matrix higher LogLoss ? Is that possible>

Log loss is a measure of confidence of the model in its predictions. Lower log loss implies higher confidence and vice-versa. Better confusion matrix with worse log-loss implies that few of the misclassifications, but with higher probability score. Since the accuracy is calculated with a fixed probability threshold, you might not observe this directly. Plot the ROC curves for both the models with is based on varying the decision threshold. This gives you the idea about which model is better.