Using deep-learning on graph data for binary classification

Feeding arbitrary graphs as inputs to any current general purpose ML algorithm, unless your problem and graphs are very specific (e.g., all graphs on a handful of nodes, or the size of your training set is of the same scale as the number of possible graphs of that size, or there is some very simple dependency - e.g. output is determined by the presence of some particular edge, etc) seems a rather pointless approach.

You can encode all NP-complete problems or e.g. the halting problem by a graph or a directed graph and few other inputs, and a 0/1 label.

One of very few successful ML algorithms that applied neural networks for a combinatorial problem was AlphaGo/AlphaZero, but it relied heavily on some specific properties of the game, the possibility to generate infinite amount of training data via self-play and enormous resources.

What you did so far (trying to construct features based on your graphs and your particular problem) makes much more sense in practice, I would explore this path further. There is also some recent literature that tries to assign graph nodes vectors of numbers, or "node embeddings", but this might work better for a specific type of graphs (sparse networks, where some additional data is available per node).


In order to do clustering, you only have to define a distance measure. When you have defined a distance, you can apply K-means, hierarchical clustering or other algorithms. In your case, I would define the following distance function:

$ d(a, b) = 1 - \frac{\text{number of common elements of a and b}}{\text{maximum between lengths of a and b}}$

As $d(a, a) = 0$ and $d(a, b) > 0$ hold, this can be a distance. I don't think the triangle inequality holds, so this will not be a well-defined distance. However, you can still try to work with this function, it might give good results for your aim.

Edit: sklearn implementation

In order to implement this using scikit-learn, the way to make it work is to use DBSCAN, set metric='precomputed' and pass the distance matrix of our data as an argument. We have to use the precomputed distance matrix because scikit-learn does not allow (now) to use custom distances for clustering.


Why does an attention layer in a transformer learn context?

To provide a simplistic and less mathematical reasons. You can assume like this:
In a simple feed-forward neural network (a black-box of course), you shall learn the set of weights, learning a function to map inputs to outputs.
But, in the transformers based architecture, you have Attentions. Here, the weights are structured into Query, Key and Value (Q,K,V). These 3 set of weights drivers of attention and are responsible to learn the context. How precisely it works still remains a black-box like feed forward networks. But yeah, it works something like this, every token's embedding is transformed its query, key and value vectors using their respective weight matrices. For a given token, its query vector is multiplied with all the other tokens' key vector to obtain a value vector. These values determines the importance of every token with respect to the query token. Thus, with back-propagation, you try to optimize these Q, K, V weights, and thus learn it to better map the relationship between tokens.


Predicting probability from scikit-learn SVC decision_function with decision_function_shape='ovo'

Your link has sufficient resources, so let's go through:

  When you call decision_function(), you get the output from each of the pairwise classifiers (n*(n-1)/2 numbers total). See pages 127 and 128 of "Support Vector Machines for Pattern Classification".

Click on the "page 127 and 128" link (not shown here, but in the Stackoverflow answer). You should see:

Python's SVM implementation uses one-vs-one. That's exactly what the book is talking about.
For each pairwise comparison, we measure the decision function
The decision function is the just the regular binary SVM decision boundary

What does that to do with your question?

clf.decision_function() will give you the $D$ for each pairwise comparison
The class with the most votes win

For instance,

  [[ 96.42193513 -11.13296606 111.47424538 -88.5356536 44.29272494 141.0069203 ]]

is comparing:

  [AB, AC, AD, BC, BD, CD]

We label each of them by the sign. We get:

  [A, C, A, C, B, C]

For instance, 96.42193513 is positive and thus A is the label for AB.

Now we have three C, C would be your prediction. If you repeat my procedure for the other two examples, you will get Python's prediction. Try it!


How to insert two features in a model when a feature only applies to a certain group in the model

Remember that any model will have the relationships between the features (X) and the label (y), so your ideas are correct. What is better will depend on your data, so try both, and try more ideas if you can. The best model will be the one that generalize better for new data, so take the final decision with data.
Please, note that your features are at least 3, goals, goals_conceded and position, so this last will help with the problem.
And I suggest not to fill NaN with 0, but -1 (or any other imposible/invalid integer number), because a newbie player will have 0 in the feature(s), and it's correct, and you want to identificate this.


Name parsing does not appear to built-in to Stanford CoreNLP.
.

One option is writing a series of Regular Expression using Stanford TokensRegex to parse and label name tokens.

Another option is using a third party package, such as nameparser in Python.


Cross validation for highly imbalanced data with undersampling

You should always do your evaluation of model performance on data that has not been over/undersampled. You can setup a pipeline with scikit-learn to perform your undersampling on the training set and then evaluate on the non-undersampled fold of data for each iteration as you described.


It's generally treated as a binary classification problem, often called credit scoring. You are trying to know whether a specific entity will default with you or not. The fact that the entity has defaulted with another company can be used as feature of your problem. As mentionned by others, there are other approaches, but I honestly think they are in theoretical infency, and can't seem to find any literrature on their practical implementation to credit scoring.

Then you need to build an historical database to learn on, with an associated label. There are two main approaches : either you construct a table based on periodic (yearly) data and set an horizon for the default. It allows you to build a label by observing what happened in the past. Say you have yearly data and consider a 1 year horizon. You will have multiple line for a company : one for each year, say 2010 to 2018, the label will be built on their status the following year, 1 if they defaulted respectively in 2011, ..., 2019, 0 otherwise. Another approach is to consider life-time default, it is better in theory, and probably better match what you want to do, but it is unlikely you will be able to do that, notably because it require complete data and you usually don't keep those that defaulted.

You can learn on that database, but this is a complex process including variable selection, model selection, model training, model validation. Pretty much standard data science, there is a lot of introductory material you should look at, starting with : an introduction to statistical learning, which is free). Then you can go look at ressources oriented towards credit scoring like Credit Scoring and its application, then to supervised learning applied to credit scoring like deep learning for mortgage risk: https://arxiv.org/abs/1607.02470). This will help you measure the risk associated with your current population. More so if you calibrate your model in probability such that the output of your model reflect the probability of default.

However if you want to use that model for making decisions about lending, then the problem will be made more complex, because :

1) Your initial decision will have an impact on the credit situation of said entity. So you actually would need to assess the impact of both legs of your decision on the caracteristic of the entity before putting it in your prediction model.

2) You need to take costs into account. The costs are usually asymetrical and hard to predict. For loans for exemple, not doing the loan will cost you interests on the nominal (it loops to (1) as the % interest you may ask will modify the contract and hence the situation of the entity), but also associated returns (a bit of money for maintaining their account for exemple), while a default would cost you a proportion of the outstanding amount, which is another order of magnitude, and that proportion is often rather difficult to predict (given insurances for exemple).

I am not really aware for advanced literature for points 1) and 2), despite working in the field, I even suspect there is a complete black-out on those topics as they are the one that provide competitive advantages to a given credit lender.

3) You want to avoid social discrimination, mostly for ethic and legal purpose, which is counterproductive to statistical discrimination. There is not much literature on the topic in general, and fewer in credit scoring, except maybe : https://arxiv.org/abs/1610.02413. Depending on your jursidiction this may give you some headaches.


Can I create random forest with RandomForestClassifier which will consist the same trees?

You want to create a random forest where all decision trees are the same. (I am trusting you are trying to do this as an educational exercise, not as something serious. :))

You tried to do this by setting random_state and cited my answer from

Why `max_features=n_features` does not make the Random Forest independent of number of trees?

Indeed, as you discovered, my answer is not fully correct.

I have edited it to add

  While this removes the stochasticity component of the training, the decision trees would still be different. The thing is that sklearn ensembles generate a new random seed for each child based on the random state they are given. They do not pass along the same random_state.

  You can see this is the case by checking the _set_random_states method from the ensemble base module, in particular this line, which propagates the random_state across the ensembles' children.

As you can see, sklearn uses the random_state you give RandomForest to create new random states for each child,

for key in sorted(estimator.get_params(deep=True)):
    if key == 'random_state' or key.endswith('__random_state'):
        to_set[key] = random_state.randint(MAX_RAND_SEED)

Therefore they will be different. This is done presumably because heterogeneity is what makes an ensemble powerful. Notice this is not specific to RandomForest, but to any sklearn ensemble.

Sorry for my incorrect answer to the question you cite. If you want, you can use the random forest implementation. That one is completely homogeneous.


Convolution Neural Networks on microcontrollers

What you are looking for is often called an inference engine. Tensorflow Lite for microcontrollers was just announced. I have not seen anyone use it on PIC32, but it should be possible to port it to that platform.

For ARM Cortex-M microcontrollers there are a couple more options, for instance uTensor and CMSIS-NN. The latter is low level implementation using integer math only, which could be a good inspiration if your microcontroller does not have hardware support for floating point.


Tensor Flow Time Series Tutorial Question

When building any Machine Learning model, the only observable data you have is training data. Test data is supposed to be unobserved data, meaning that even though you might have it now, you need to act as if you didn't. When you apply normalisation, you first observe the data to get the parameters you need. As you are only supposed to be able to observe the training data, you can't use the test data to calculate those values. Doing so would be like cheating, as you are accomodating your parameters to new unobserved data (how can you observe unobserved data?).

Imagine you build a model today and you want to make predictions tomorrow. You can't use tomorrow data to build your model since you don't have it yet. You are not supposed to know tomorrow's mean and std, though your hope is that they will be similar enough. That is why when you normalose/standarise you get the parameters with the training data and then use them to transform both train and test data, so you can use them as inputs for your model.


How are the positions of the output nodes determined in the Kohonen - Self Organizing Maps algorithm?

The distance is calculated according to a distance function (euclidian, manhattan, mahalanobis and so on. As this is an unsupervised model (hence the self-organizing part in the name) there are no output neurons. Would you clarify the term?

I'll just recap the training a bit:

Inititalizing: We have a grid (let's assume 2d) of neurons $n_i = (w_i, k_i)$, where $w_i$ is a randomly initialized weight and $k_i$ the position on the grid.

We'll now pick a training sample $x_i$ randomly. For this instance $x_i$ we pick the neuron $n_m$ ($m$ for minimum) where the distance between the weight vector and the instance  is minimal given a distance function $d$, so $n_m = \text{argmin}_{n_j} \; d(x_j, W(n_j))$ where $W(\cdot)$ gives the weight for the respective neuron.
We now pick a set of neurons for which we will adapt the weight vector given a neighbourhood function (for example your gaussian, or a cone). The weights that should have their weights updated are given by $N^{+t} = \{ n_i = (w_i,k_i) \mid d_A(k_m,k_i) \leq \delta^t \}$, where $t$ is the step in time (I'll mention that later), the "reach" $\delta^t$ the neighbourhood should have and of course the position on the grid $k_m$ for the winning neuron.
We then update the weights of the neurons in the neighbourhood of the winning neuron $n_m$ according to some updating rule. This can be interpreted as moving the weights closer to the input we currently look at (because we chose the neuron with the closest weight vector). A possible update rule could be: $w_s^{t+1} = w_m^t + \epsilon^t \cdot h_{mi}^t \cdot (x_j - w_m^t)$, where $\epsilon^t$ is a time-dependent learning rate and $h_{si}^t$ weights the distance from the winning neuron to the neuron we are changing right now, also time-dependent.

In order to somewhat guarantee a topological mapping two things should be considered now:

The learning rate, which is part of the update rule, has to be decreased over time, starting with a relatively high value.
The neighbourhood "radius" has to be decreased as well, also starting from a relatively high value.

Let's visualize this on this picture real quick. The red nodes are inputs. The dark grey is the winning neuron for the input that is considered right now (the picture does not make it clear that we look at one example/input at a time!). The light grey nodes are the ones that are within the neighbourhood of the winning neuron and hence are updated slightly according to some updating rule.

Disclaimer: this was from the top of my head right now, please check any formulas or hypotheses I propose here before using it in any kind of work. (Which you should always do!)
.
See also wikipedia, they also have some explanation of the algorithm that might better suit you.

Image source
Additioanl source (German Wikipedia)


Extremely stochastic validation loss/accuracy

There can be a few reasons for this behavior as already pointed out in "Why is the validation accuracy fluctuating?":

Size of train / validation stes: Fluctuations may become stronger the smaller your validation set is, especially during the early stage of training where predictions are closer to random predictions.

Overfitting: Train loss seems to be low and accuracy over 90%. If you're overfitting, the predictions on the validation set become almost random and you observe such fluctuations.

Random validation set: If you're validation set is not constant across all epochs, the validation loss will be different and losses from different epochs cannot be compared anymore. Especially if you have imbalanced classes.

Weighted loss function: Using a weighted loss function for imbalanced data sets makes your loss appear more stochastically, especially in the early stage of training and if the distributions of the training and validation sets mismatch.

Maybe you can share additional information to your data, preprocessing, training, model etc. so it is easier to spot the cause of this more precisely.


How to deal with classification problem where labels are non uniformly distributed?

Welcome to DataScience.SE! You need to ensure that your training distribution is similar to your test distribution in order to get the best results. This can be done through stratified sampling. It might be that there is no problem, and that the classifier is simply allocating its resources to best classify the majority case, though I'm a bit confused by your description because you say that B and C dominate and the "other two classes" (A and D) are "mapped to A and B quite often". One option is to use more discriminating features, if you can think of any. Another is to partition the large classes into (B becomes the union of $B_i$ and likewise for C) if applicable. This can help if the shape of the class is complex by breaking it up into learnable pieces. It helps to plot your inputs, colored by class, to understand which classes overlap or have complex shapes. Finally, you can modify your loss function if getting certain classes wrong is more important than others.


I tried loading my saved .h5 model and predicting with that model, i'm getting error list index out of range

Try these:

import everything from tensorflow keras API.
tf.keras.models.load_model("./saved_models/our_model.h5", compile=False) (make sure you have model and weights both in the file)


Mass convert categorical columns in Pandas (not one-hot encoding)

If your categorical columns are currently character/object you can use something like this to do each one:

char_cols = df.dtypes.pipe(lambda x: x[x == 'object']).index

for c in char_cols:
    df[c] = pd.factorize(df[c])[0]

If you need to be able to get back to the categories I'd create a dictionary to save the encoding; something like:

char_cols = df.dtypes.pipe(lambda x: x[x == 'object']).index
label_mapping = {}

for c in char_cols:
    df[c], label_mapping[c] = pd.factorize(df[c])

Using Julien's mcve will output:

In [3]: print(df)
Out[3]:
    a   b   c   d
0   0   0   0   0.155463
1   1   1   1   0.496427
2   0   0   2   0.168625
3   2   0   1   0.209681
4   0   2   1   0.661857

In [4]: print(label_mapping)
Out[4]:
{'a': Index(['Var2', 'Var3', 'Var1'], dtype='object'),
 'b': Index(['Var2', 'Var1', 'Var3'], dtype='object'),
 'c': Index(['Var3', 'Var2', 'Var1'], dtype='object')}


How Can I Solve it? TypeError: fillna() got an unexpected keyword argument 'implace'

You have a typo, change implace for inplace (change the m for the n)


Is there any paper of object detection that detect not trained class or object?

Yes thats possible, but you need some kind of adaption. Normally you would use transfer learning to teach your model, that there are more things to detect with the features it already can extract. For that you normally take your model, and just delete the last dense layer, which is for classification. Then you freeze all earlier layers of the CNN and setup a new Dense-layer on the end with the number of classes you want to detect. Giving the network now new examples with ground truth (not that many normally) to adapt onto the new domain. This can work, but also defenitely fail. For example i tried using ResNet-backbones (51,101) trained on ImageNet (a big dataset to find humans, cars and stuff) and tried to use it on microscopy cell segmentation and classification. Yeah it found some boundarys, but couldnt get a precise classification at all.

To sum it up: yes you can transfer learn, or use tasks of the field of "UDA" -> Unsupervised domain adaption, or normal Supervised / semi-supervised domain adaption to teach models new things. Or just to strengthen their knowledge. Detecting things without having seem them ever before is also an updcoming task and is called "zero-shot-learning", often in combination with Graph-Neural-Networks but its quite in the beginning. Bridging the gap between multiple domains is one of the sota questions in ML overall, trying to find object specific features. One step in this direction are cGAN, trying to transform one image into another, by extracting the most important information of both domains, to perfom a precise transformation. But also that, is quite in the beginning , but can work surprisingly well with some effort and loss-function tuning. Some examples:

Transform green to red to make it possible for another CNN to segment the image:

Additional interesting papers on that topic are:

Noisy Student

Unsupervised Pixel–Level Domain Adaptation with GenerativeAdversarial Networks

Transfer learning


Evaluation metrics for Decision Tree regressor and KNN regressor

Generally when ever we are trying to compare between models and to choose the best one, we go for other metrics like AIC, BIC, AUC(this is not applicable as it is used for classification algorithm) etc along with $R^2$.
Now why are they important criteria because AIC tries to select the model that most adequately describes an unknown, high dimensional reality. This means that reality is never in the set of candidate models that are being considered. On the contrary, BIC tries to find the TRUE model among the set of candidates. I find it quite odd the assumption that reality is instantiated in one of the model that the researchers built along the way. This is a real issue for BIC.
Generally we use both AIC and BIC together.
you can got through this Link to understand better on AIC, BIC and which is better but conclusion is both are important.
Now when we compare between $R^2$ and AIC value:
$R^2$ and AIC are answering two different questions. I want to keep this breezy and non-mathematical, so my statements are non-mathematical.  $R^2$ is saying something to the effect of how well your model explains the observed data.  If the model is regression and non-adjusted $R^2$ is used, then this is correct on the nose.
AIC, on the other hand, is trying to explain how well the model will predict on new data. That is, AIC is a measure of how well the  model will fit new data, not the existing data.  Lower AIC means that a model should have improved prediction.
Frequently, adding more variables decreases predictive accuracy and in that case the model with higher $R^2$ will have a higher (worse) AIC. A nice example of this is in "Introduction to Statistical Learning with R" in the chapter on regression models including 'best subset' and regularization. They do a pretty thorough analysis of the 'hitters' data set.  One can also do a thought experiment.  Imagine one is trying to predict output on the basis of some known variables.   Adding noise variables to the fit will increase $R^2$, but it will also decrease predictive power of the model.  Thus the model with noise variables will have higher $R^2$ and higher AIC.
To understand more with respect to the above explanation, you can go through this Link

I have also pasted the R2 and MSE values. As per my understanding Linear regression model is stable, Decision tree is over fitting and KNN is having less error. Which model needs to be considered here?

To answer this I think you need to derive AIC and BIC values and finally you can decide on which model to choose.
To derive the AIC value for Linear Regression can be done by using OLS, as the value of AIC is directly available.
If you want to derive, you can go through this Link
AIC:
$AIC= 2k - 2ln(sse)$
where k= number of variables
BIC:
$BIC = n * ln(sse/n) + k * ln(n)$
Do let me know if have any additional questions.


How to handle non consistent time series( using LSTM )

If you have missing data you can add a Masking layer to your model which will prevent contributions from those time steps to be included.  You can see the documentation for it here: https://keras.io/layers/core/#masking

Example taken from the keras documentation:

  Consider a Numpy data array x of shape (samples, timesteps, features),
  to be fed to an LSTM layer. You want to mask timestep #3 and #5
  because you lack data for these timesteps. You can:

  set x[:, 3, :] = 0. and x[:, 5, :] = 0. insert a Masking layer with
  mask_value=0. before the LSTM layer:

With example code:

model = Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(LSTM(32))


how to do topic modeling on very huge data?

Use filter_extremes to Filter out tokens that appear in less than no_below documents (absolute number) or more than no_above documents (fraction of total corpus size, not absolute number).

keep only the first keep_n most frequent tokens (or keep all if None).

After the pruning, shrink resulting gaps in word ids.


Foreign exchange market forecasting with neural networks

The results you're seeing aren't a byproduct of your training product, but rather that neural nets are not a great choice for this task. Neural nets are effectively a means to create a high order non-linear function by composing a number of simpler functions. This is often a really good thing, because it allows neural nets to fit very complex patterns.

However, in a stock exchange any complex pattern, when traded upon will quickly decay. Detecting a complicated pattern will generally not generate useful results, because it is typically complex patterns in the short term. Additionally, depending on the metric you choose, there are a number of ways of performing well that actually won't pay off in investing (such as just predicting the last value in your example).

In addition the stock market is startlingly chaotic which can result in a neural net overfitting. This means that the patterns it learns will generalize poorly. Something along the lines of just seeing a stock decrease over a day and uniformly deciding that the stock will always decrease just because it was seen on a relatively short term. Instead techniques like ridge and robust regression, which will identify more general, less complex patterns, do better.

The winner of a similar Kaggle competition used robust regression for this very reason. You are likely to see better results if you switch to a shallow learning model that will find functions of a lower polynomial order, over the deep complex functions of a neural net.


time series prediction using arima and non linear trend and too much residuals

Long Short Term Memory (LSTM) is one option given that you have about 9 years of historical data.
You can take a model comparison approach where you split the data and see which algorithm is best at predicting the hold-out data.


It sounds like you want your model to output a probability distribution which matches a "ground truth" probability distribution.  Instead of a cross-entropy loss, you should try Kullback-Leiber divergence (keras.losses.kullback_leibler_divergence).
KL-divergence measures the difference between two probability distributions.  Minimizing KL-divergence should cause your predicted distribution to match the actual distribution.
By the way, KL-divergence is not just a workaround for Keras's cross-entropy limitations.  It's actually the better loss function for this task.  From the wikipedia page on KL-divergence (emphasis added by me):

$$D_{KL}(P \| Q) = H(P, Q) - H(P)$$
where $H(P, Q)$ is the cross entropy of P and Q, and $H(P)$ is the entropy of P (which is the same as the cross-entropy of P with itself).
The KL divergence $D_{KL}(P \| Q)$ can be thought of as something like a measurement of how far the distribution Q is from the distribution P.  The cross-entropy $H(P, Q)$ is itself such a measurement, but it has the defect that $H(P, P) = H(P)$ is not zero, so we subtract $H(P)$ to make $D_{KL}(P \| Q)$ agree more closely with our notion of distance.


I want to detect similar meaning in sentence "in my final year web based system"

The simple option is to implement it as a text search, preferably enriched with tf-idf weights and/or ignoring stop words: in theory all the possible topics (documents) are encoded as tf-idf vectors. When a query is provided it's also encoded as a tf-idf vector, then compared against all the documents vectors, typically with cosine similarity. The results are then ordered by the similarity score.

In practice various efficiency techniques can be used (the most obvious one is not to represent the vectors as sparse vectors).

There are obviously more advanced techniques, but it can get pretty complex so I would start with this simple option first and then consider improvements later.


Difference between Global Outlier and Contextual Outlier?

Let's say that your dataset is a bunch of temperatures with a bunch of features related to when/where the measurements were made.
A measurement is a global outlier if it diverges from the distribution of temperatures regardless of the features (when and where) because that measurement is far off the global distribution (100°C for example).
The example makes a good case of how looking at conditional distributions (temperature|place, time of measurement) can provide additional insights (here detecting that 28°C is a contextual outlier). Contextual outliers are within the global distribution but are outliers when looking at conditional distributions.
Now of course your dataset has some context, so in a sense you're right, everything is contextual, but the global/contextual border resides in what you have at hand : if all you have is the temperatures from one day, comparing a temperature from that day to all the measurements of that day will allow you do detect gloabl outliers ; now if you had access to temperatures all year long and that your still looked at that day precisely, that would become a conditional distribution.


What is the best way to split a sentence for a keyword extraction task?

I think the tagging approach has some merit here. The frequency drop you're observing as a consequence of using this is to be expected, I think. After all, keywords are the words that help differentiate a document from others in a corpus. If you have access to an ontology related to the subject matter of your corpus, you could try to map your rare keyword tags to the ontology, and use parent-level information for each to get a slightly more general set of keyword tags! If this is an approach that interests you, Stanford's open source protégé system is a good framework for working on this.


Customer Segmentation and Category association

Question 1.:

Encode the columns, (Labelencoding, tf-idf, feature generation etc...)<-> quantify the columns, than iterate fastly with different algos and evalute and compare the results, for example with silhuette score.

Question 2. Once you identified the best clustering algorithm, look for the categories that (uniquely or very likely) determine certain cluster/group. It wont be 1-1 but in probabilities determined.


How to count the number of missing values in each row in Pandas dataframe?

When using pandas, try to avoid performing operations in a loop, including apply, map, applymap etc. That's slow!

A DataFrame object has two axes: “axis 0” and “axis 1”. “axis 0” represents rows and “axis 1” represents columns.

If you want to count the missing values in each column, try:

df.isnull().sum() as default or df.isnull().sum(axis=0)

On the other hand, you can count in each row (which is your question) by:

df.isnull().sum(axis=1)

It's roughly 10 times faster than Jan van der Vegt's solution(BTW he counts valid values, rather than missing values):

In [18]: %timeit -n 1000 df.apply(lambda x: x.count(), axis=1)
1000 loops, best of 3: 3.31 ms per loop

In [19]: %timeit -n 1000 df.isnull().sum(axis=1)
1000 loops, best of 3: 329 µs per loop


Problem of continuous training - Supervised learning

What you are describing is called auto-adaptive learning. This is what most recommendation systems use to adapt to ever changing data and feedback. It is also known as autoML.
This https://towardsdatascience.com/how-to-apply-continual-learning-to-your-machine-learning-models-4754adcd7f7f
Article does a good job of explaining it. Based on what your data looks like, you might have to choose the appropriate retraining strategy and do a staggered deployment.


The 75^6 option is not only bad for speed, but it is a very difficult representation to train, because the NN doesn't "understand" that any of the output categories are related. You would need an immense amount of data to train such a network, because ideally you need at least a few examples in any category that you expect the network to predict. Unless you had literally billions of examples to train from, the chances are certain combinations will never occur in your training set, thus could never be predicted with any confidence.

Therefore I would probably use 75 outputs, one for each object representing the probability that it would be chosen. This is easy to create training data for, if you have training examples with the 6 favoured objects - just a 1 for the objects chosen and 0 for all others as a 75-wide label.

For prediction, select the 6 objects with the highest probabilities. If these choices are part of a recommender system (i.e. may be presented to same person as being predicted for), then you can select items randomly using the outputs as weights. You may even find that this weighted Monte Carlo selection works well for predicting bulk user behaviour as well (e.g. for predictions fed into stock purchases). In addition, this stochastic approach can be made to predict duplicates (but not accurately, except perhaps averaged over many predictions).

A sigmoid transfer function on the output layer is good for representing non-exclusive probability. The logloss objective function can be used to generate the error values and train the network.

If you want to accurately predict duplicate choices out of the 6 items chosen, then you will need plenty of examples where duplicates happened and have some way to represent that in the output layer. For example, you could have double the number of output neurons, with two assigned to each object. The first probability would then be probability of selecting the item once, and the second probability would be for selecting it twice.

The question has since been updated, and it appears there are strong relationships between items making the choice of a set of items potentially very recipe-like. That may reduce the effectiveness of the ideas outlined above in this answer.

However, using 75 outputs may still work better than other approaches, and is maybe the simplest setup, so I suggest still giving it a try, even if just to establish a benchmark for other ideas. This will work best when decisions are driven heavily by the feature data available, and when in practice there are lots of valid choices for combining items so there is a strong element of player preference. It will work less well if there is a large element of game mastery and logic in player decisions in order to combine items.


Ways to speed up Python code for data science purposes

Things I care about a lot:

list comprehensions instead of loops
use apply + lambda functions when forced to iterate operations on pandas dataframes
use @tf.function decorator on top of TensorFlow functions to speed up computation
use as much SQL as possible when importing data from databases, to avoid doing the same stuff in Python


What activation function should I use for a specific regression problem?

The issue with sigmoid and tanh activations is that their gradients saturate for extreme values of their arguments. This may occur if you do not normalize your inputs. If the learned weights of the unit are such that the gradient of its activation is close to zero, it will take longer for any updates to be reflected in the unit's weights. A final layer with no non-linearity will help you scale up your hidden layers' outputs.

In the end, the performance is application specific. You should try out both kinds of activations on a subset of your data and see which performs better.


What's the purpose of padding with Maxpooling?

The whole purpose of pooling layers is to reduce the spatial dimensions (height and width). Therefore, padding is not used to prevent a spatial size reduction like it is often for convolutional layers. Instead padding might be required to process inputs with a shape that does not perfectly fit kernel size and stride of the pooling layer.

This is an example where it perfectly fits and your pooling layer does not require any padding:

Side note: The output dimensions are calculated using the usual formula of $O=\frac{I-K+2P}{S}+1$ with $I$ as input size, $K$ as kernel size, $P$ as padding and $S$ as stride.

However, lets take another example where it does not fit as nicely:

Here you need padding since your input size is not an integer multiple of your kernel size. Therefore, you need to add padding on one side in order make it work.

So padding="same" in Keras does not mean the spatial dimensions do not change. It just means that padding is added as required to make up for overlaps when the input size and kernel size do not perfectly fit.

Also this question for a discussion what the difference between same and valid padding is for pooling layers.


Best practice for short sentences in a deep learning network

Yes, when you have different sequences with different length you can pad all the sequences so that they have all the length of the longest one. is a good practice and doesn t impact the learning mode. Another solution could be online learnig. With online learning you give to the model 1 sample at the time (batch_size =1) and in keras you can train the model with batch with different lengths.

With online learning your computation is slower, but theoretically your model need less epochs to learn.

If you want to pad you should pad with all 0s and not random values.


Workaround for word embeddings that do not "see" antonyms

You could probably use the technique to combine vectors with antonyms with some kind of weights. Below thread suggests adding POS tagger with word2vec vectors, instead of POS you can concatenate antonyms:

https://stats.stackexchange.com/questions/238016/deep-learning-word-embedding-with-parts-of-speech

https://stackoverflow.com/questions/51537441/how-to-combine-both-word-embeddings-and-pos-embedding-together-to-build-the-clas

You can identify Antonyms using below:

https://www.geeksforgeeks.org/get-synonymsantonyms-nltk-wordnet-python/


What is the difference between a hashing vectorizer and a tfidf vectorizer

The main difference is that HashingVectorizer applies a hashing function to term frequency counts in each document, where TfidfVectorizer scales those term frequency counts in each document by penalising terms that appear more widely across the corpus. There’s a great summary here.

Hash functions are an efficient way of mapping terms to features; it doesn’t necessarily need to be applied only to term frequencies but that’s how HashingVectorizer is employed here. Along with the 45339 documents, I suspect the feature vector is of length 1048576 because it’s the default 2^20 n_features; you could reduce this and make it less expensive to process but with an increased risk of collision, where the function maps different terms to the same feature.

Depending on the use case for the word vectors, it may be possible to reduce the length of the hash feature vector (and thus complexity) significantly with acceptable loss to accuracy/effectiveness (due to increased collision). Scikit-learn has some hashing parameters that can assist, for example alternate_sign.

If the hashing matrix is wider than the dictionary, it will mean that many of the column entries in the hashing matrix will be empty, and not just because a given document doesn't contain a specific term but because they're empty across the whole matrix. If it is not, it might send multiple terms to the same feature hash - this is the 'collision' we've been talking about. HashingVectorizer has a setting that works to mitigate this called alternate_sign that's on by default, described here.

‘Term frequency - inverse document frequency’ takes term frequencies in each document and weights them by penalising words that appear more frequently across the whole corpus. The intuition is that terms found situationally are more likely to be representative of a specific document’s topic. This is different to a hashing function in that it is necessary to have a full dictionary of words in the corpus in order to calculate the inverse document frequency. I expect your tf.idf matrix dimensions are 45339 documents by 663307 words in the corpus; Manning et al provide more detail and examples of calculation.

‘Mining of Massive Datasets’ by Leskovec et al has a ton of detail on both feature hashing and tf.idf, the authors made the pdf available here.


Coursera's course "How to Win a Data Science Competition"

It's a technically rigorous course.

Recommend? It depends on what your goal is.

Definitely, if your goal is to compete on Kaggle.
Not really, if you want to improve practical ML.

Remember, Kaggle is kind of Formula 1. The fastest car wins. But in the real world, you don't always need a Formula 1 car. Sometimes you need an SUV, sometimes you need a van, sometimes a Tesla ....


The mix of leaky Relu at the first layers of CNN along with conventional Relu for object detection

how we end up with a model which uses a leak at the first layers and then a conventional RELU at the end?

What matters is to add a non-linearity to the outputs of a neuron. Consequently, by employing each function that adds non-linearity, the network will work due to the derivative that can backpropagate the differentiation of the error term. If you use Relu or leaky Relu, the update terms will change but your network will work fine. The reason that leaky version is used in the mentioned papers is due to avoiding dying relu problem which can happen a lot for regression problems.

About your second question, consider that it usually suffice to use leaky relu in the first layers and due to them, the chance of deep neurons to be stuck at zero is not very much as the results of those papers show. You can use leaky version all over the network but by experience, relu is very fast to be trained!


What is the best metric to evaluate highly imbalanaced binary classifiction? (such as fraud detection in credit card)

Simple precision/recall/F-score is perfectly suitable for imbalanced data. It should be computed on the minority class of course.

precision says how often the system is correct when predicting an instance in the minority class.
recall says how often the system detects an instance which belongs to the minority class (this is usually the hardest part with imbalanced data)
As usual F-score can be used as a "summary metric" of both precision and recall.

ROC and AUC (which are also based on precision/recall) can be used only with a soft classifier, i.e. when the system predicts a numerical value instead of a binary label.

Macro/micro/weighted average (usually over F-score) are irrelevant: averaging over the two classes would take into account the majority class which is very easy to predict, so it would mask the useful information about the minority class.


Detecting smoke and bluring/filtering using CNN in python

There is a book on Methods and Techniques for Fire Detection.

Computer vision based method for real-time fire and flame detection is one of the most cited papers on this topic. However, CNN is not a widely applicable method here.

Update:

Here is implementation of EXPERIMENTALLY DEFINED CONVOLUTIONAL NEURAL NETWORK ARCHITECTURE
VARIANTS FOR NON-TEMPORAL REAL-TIME FIRE DETECTION.


CNN Architectures: These are proven configurations of the combination of CNN/FC/POOL etc layers, that seem to work in commonly defined problems.

Now from your question what I understand is you want to "Tinker" with the architecture and not to come up with an entirely new configuration from scratch. One thing that can help you in this scenario is transfer learning.

Transfer Learning: It is a process of using pre-learnt weights from a similar task on altogether different data, to your current problem and data.

In transfer learning what we do is we take some layers usually 80-90 % of the layers keep the weights the for these layers and delete the weights for remaining layers, and initialise these layers with some kind of initialisation batch norm or something like that. And train this newly created model with the data you have.

This have proven to be effective in various problems.

You can come up with innovative ways to make changes to arbitrary layers of this network train them and check for error metrics. If you follow this approach whatever you are claiming in your question can be achieved.


Performing 1D Depthwise conv using Keras 2D Depthwise conv

Indeed I was able to prove that this method works, confirmed by my Matlab golden model. The problem was that in order to read the output activations of an intermediate layer in Keras, you have to create a separate model with the specified output.
This separate model in my case did not have the same weights as the original model, leading to incorrect outputs. But now that I read the output activations correct, I can confirm this method works.


Data Scientist Prerequisites: What are the essential prerequisite statistics topics?

Everything on that list is is important material you should know. The sequence the topics are introduced looks to be chosen so that the material builds on itself as you move through (this effect is never totally perfect but seems generally right in this case), so I'd pretty much read the chapters as they appear.
The material listed is the most basic of basic statistics, so it is hard to make a case to omit something.

However, while knowledge of these topics is important, it is very common (probably 'more common') for data scientists to involved in machine learning, which is a different approach to analysing data which gives better results for prediction. An overlapping term (not completely equivalent) term that is used is 'data mining' - searching on Amazon for these terms will give good results that can be used to pick a book that matches your background.

Note that statistics and machine learning are both heavily dependent on maths, especially calculus, linear algebra and probability. The book you mention has been written for people with little background in these, but to go deeper you'll need to brush up or get new knowledge in these areas.


How to reload cell in jupyter notebook? After OSError: [Errno 12] Cannot allocate memory

One option is to add TensorFlow Checkpoint which allows for the saving and reloading of training.


What predictive model to use to impute Gender?

You can use one of the popular gradient boosting tree implementations such LightGBM and XGBoost as your predictive model. They can handle missing values during training and the results are often better than any imputation done in preprocessing.

The way they achieve this is by using the splitting of the decision trees they are built on. The missing values are ignored when finding the best split, then once it is found they are allocated to whichever side will minimize the loss function most.


Multi-label compute class weight - unhashable type

You're seeing this error because your Y_train data is a 2d array, where compute_class_weights expects a 1d array.

compute_class_weights can be used for multiclass classifications, but apparently not multi-label problems like yours.

You could try using compute_sample_weight instead, which is slightly different but handles multi-label output problems such as this.


How to perform SMOTE-N when there is no majority vote?

I don't know what "VDM" stands for, but a simple solution for tie breaking is to randomly pick one of the tied options.


How to detect blocks of texts in document images

I would approach the text block amalgamation as a clustering problem. If you define a suitable distance metric or a neighbour predicate between the individual text boxes, you could group the boxes and then determine their minimum bounding box, which is essentially what you are aiming for.

I guess DBSCAN could be a suitable candidate for the clustering algorithm, but more care would have to go into the design of the neighbor predicate - one idea could be that vertical distance could be treated differently than horizontal distance, etc.


Clustering mixed data types - numeric, categorical, arrays, and text

For the tags: Do you know how they are generated? How many unique tags do you have? If they are self-generated (ie lots of tags that can be subsets of other tags). You might need to do tag consolidation which will also help with dimensionality reduction of the word vectors. If you can provide a bit more information about what your data looks like and where it comes from, I can perhaps provide a more in depth answer.

For the text: You might want to try using word embeddings. You can use a pre-trained word2vec model.

I am not sure if it makes sense to use two different distance metrics. Your categorical data looks like its integers, is it ordinal, or are those indexes?


Low number of inputs compared to outputs (per row) in neural network

1.) Perform standard preprocessing for you input data, i.e. remove mean and scale to unit variance.

2.) No, you should train everything together. Although it is at the moment not clear, there can be relevant relationships between your outputs.

3.) You can actually use any architecture. Simply start with Dense-NN. More important is how you arange your outputs. You should have different activation and loss functions for your different outputs. For your binary outputs use (binary)-Crossentropy and for your real values use Mean Squared Error (or sth. similar). Your final output consists of course of the sum of your single losses.


interpreting multi linear regression results

There are a lot going on in a simple OLS model. I strongly encourage you to learn more about them from textbooks. One of the best place to start is the free online book An Introduction to Statistical Learning  (see Chapter 3 about Regression, in which it explains some of the elements in your model summary).

Also in this blogpost, they explain all elements in the model summary obtained by Statsmodel OLS model like R-Squared, F-statistic, etc (scroll down). There are also series of blogposts in blog.minitab, like this one about R-Squared, and this about F-test, that explain in more details each of these elements. You can even explore more and learn about Residual Plots for post analyses of your regression models (important one to check).

Please note, you do not often need to have all these elements to examine a regressor model (some of them like BIC or F-stat are just not that informative, unless someone explicitly ask for them).

The first table i.e. overall model statistics, you can learn about:

R-Squared, which is an estimate of the strength of the relationship between your model and the response variable that lies between 0 (worst) and 1 (best), is 0.278 for your model. While from experience this is quite low, but one can not jump into a conclusion by simply only looking at only R-Squared! I think you can still do better.
F-Stat: It is a statistical test that compare the fit of the intercept-only model with your model. In simple works, if P value for the F-Stat (here Prob (F-Statistic) is less than your significance level, one can reject the null hypothesis that a intercept-only model is better, meaning your model is better a better job (Not a big surprise honestly).

The second table i.e. summary of statistics of your model breakdown:

Gives a lot of information about each variable. Each coefficient with its corresponding standard error, t-statistic, p-value. For instance, for each coefficient if the t-statistic is large (or simply the p-value is ZERO or very very small < 0.05), that particular coefficient is trustworthy like discount or stockout, otherwise the error is large and unlikely any interpretation from them to be reliable like some of the sublevels of prod_function or prod_regime categorical variables in your model. *Just a rough thought about these variables in your model, maybe the type of categorical encoding you do on these is not performing well, that is why the errors here are large?!

How To Interpret Your Model:

This is an interesting part. Taking that your model is good enough (within the defined confidence interval), one can find out how each of these variables contribute to the dependant variable (herein sales). Read more about how Interpreting Regression Coefficients or see this nice and simple example.

Specifically for the discount variable, if all other variables are fixed, then for each change of 1 unit in discount, sales changes, on average, by 0.4146 units (the coefficient of the discount from your model).

Hope these points get you started!


When it is a good idea to train a model also on the test data?

The test set is there to show you how the model can perform to unknown cases and basically reassures that it is not overfitted. The selection of the "optimal" model needs to be made based on its performance on the test set and NOT on the training set, because what you need is the model with the best behavior in unknown cases.

In my humble opinion, just splitting the file in training and test set is not the best approach, as there might be bias in the accuracy calculation just because of the random way data were splitted. I would go with cross-validation in order to ensure that the model has been extensively tested. In cases with small amount of data, leave-one-out-cross-validation is often used in order to maximize the amount of available data for the training. You can read more on Cross validation and Leave-one-out-cross-validation (theory and python implementation) here.

When you have found the model with the best performance using cross-validation, I believe you can feel free to train it on the whole set of data before giving it to production.


How to deal with overestimation of small values and underestimation of high values in XGBoost?

I dont think its necessarily related to the type of algorithm performing the regression(XGBoost here) - but to inherent nature of regression algorithms.
Many loss function are aimed to reduce distance between $y$ and $\hat{y}$.
That can lead to model predictions distribution being tighter around $y$'s mean.

Couple of things I would check to verify this:
1) Compare distribution of real $y$ and predicted $\hat{y}$.
2) Verify this error pattern on others regression model.

Things I would try to improve results:
1) over sample low/high y values in training set.
2) Adjust loss function so errors on low/high y values will have more weight.
3) Look for features that emphasize low/high y values and engineer them better.


This example trains an image and a partial caption to predict the next word in the caption.

Input: [, "<BEGIN> The cat sat on the"]
Output: "mat"

Notice the model doesn't predict the entire output of the caption only the next word. To construct a new caption, you would have to predict multiple times for each word.

Input: [, "<BEGIN>"] # predict "The"
Input: [, "<BEGIN> The"] # predict "cat"
Input: [, "<BEGIN> The cat"] # predict "sat"
...

To predict the entire sequence, I believe you need to use TimeDistributedDense for the output layer.

Input: [, "<BEGIN> The cat sat on the mat"]
Output: "The cat sat on the mat <END>"

See this issue: https://github.com/fchollet/keras/issues/1029


Which learning tasks do brains use to train themselves to see?

Maybe this paper will give you an overview about (and entrance to) this topic from biological side. It is a review about the state of the art in human brain development (and its implications for clinical treatment).

The table of contents include for example

Stage 1: the first year, early maturation of vision and the structure of V1 neurobiology
Stage 2: preschool children have high variability in V1 development (1–4 years)
Stage 3: experience-dependent visual development in school aged children (5–11 years)

(V1 means "visual cortex")
Where this points all handle 3 categories: visual milestones (i.e. contrast sensitivity, contour integration), anatomical milestones (i.e. morphology) and neurobiological milestones (i.e. synapsis, but also a lot of genetics).

And as second : Maybe you could ask this question on bioinformatics.SE too, because this connection between biological example and computational reproduction is one of their fields.


It seems that the output of sklearn.metrics.pairwise.euclidean_distances is different to the formula on doc

The sklearn docs' formula says it is applying to row vectors $x$ and $y$.  When you call np.dot on the matrices $X$ and $Y$ it takes the matrix product.

EDIT (responding to question in comments):
It's not straightforward, as the row-vs-row operations needed aren't quite the usual matrix operations.  The source code for euclidean_distances does it this way (except that it does lots of input checks, operates on sparse inputs when possible, etc.):

(X*X).sum(axis=1)[:, np.newaxis] - 2*np.dot(X,Y.T) + (Y*Y).sum(axis=1)[np.newaxis, :]

That's not exactly straightforward itself, so I'll say a little more.  Say $X$ has $m$ rows and $Y$ has $n$ rows.  The middle term, by taking $Y^T$, gives us a $m\times n$ matrix whose $(i,j)$-entry is the dot product of the $i$th row of $X$ with the $j$th row of $Y$.  In the other terms, * on numpy arrays is the coordinate-wise product; summing along rows gives us the rows' squared-norms.  The newaxis is a nice trick: casting the first term to now be a $m\times 1$ matrix, adding it to the middle term's $m\times n$ matrix actually adds it to every column of that matrix (without needing to actually build the matrix of repeated columns of $X$'s squared-norms).  And of course similarly for the last term: casting to a $1\times n$ matrix makes it add to every row of the result.


Which Machine Learning book to choose (APM, MLAP or ISL)?

An Introduction to Statistical Learning by Hastie

It is like a bible for all those who want to get started at statistical analysis and Machine Learning.  It gives an excellent intuition about the theory behind all those models and algorithms, enough to pursue research and/or dive into another advanced ML book like the Applied Predictive Modeling by Kuhn and Johnson


Hint: what's the error rate of 1-nearest-neighbor on any training set?

From there, determine the test error on this dataset.


Machine Learning algorithm for detecting anomalies in large sets of events

You can also try these:

4- One Class SVM (Support vector Machine):

You can train this with the normal samples that you have and then it can label them all and detect anything that does not fall under this label. It can also scale easily, although it might get quite slow on training with large data. You can try the scikit-learn implementation:
https://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html

5- Isolation Forest:

Isolation forest is actually an Ensamble algorithm, it is basically using Random Forest (Which by itself is using Trees) to detect Anomalies or outliers in the data. At first you can try scikit-learn implementation of it:
https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html

6- Deep One-Class Classification:

The other thing that you can try is the Neural Network implementation of One class classification. You should be able to do it easily with Keras for example. You can take a look at Keras documentation here:
https://keras.io/models/about-keras-models/

Also for implementation, these might help:
https://blog.keras.io/building-autoencoders-in-keras.html ,
https://medium.com/@curiousily/credit-card-fraud-detection-using-autoencoders-in-keras-tensorflow-for-hackers-part-vii-20e0c85301bd


Do I need validation data if my train and test accuracy/loss is consistent?

You don't always need 3 separate datasets. You usually split a dataset into 3 if you are doing some parameter or hyperparameter tuning before choosing a final model. Tuning will usually add bias from the 2nd dataset into your model, decreasing it's performance.
For instance:

If you are manually tuning a model over several iterations and using
the results from the 2nd dataset to find the optimal parameters. By
doing so, you built some information from the 2nd dataset into your
model. This will make the 2nd dataset not a good, unbiased benchmark for your final
model. Therefore, you will want to use a 3rd untouched dataset, to
give you unbiased final performance measurements of your model

Some models use a validation dataset internally while building model
to evaluate loss, etc. This will cause the same problem with
including bias into the model. Example:
model.fit(
    train_features,
    train_labels,
    batch_size=20,
    epochs=20,
    validation_data=(val_features, val_labels), # <- here
    verbose=0)


Should we do Feature selection in parallel with feature engineering?

You can add features as you create them and check if the scoring improves (and by how much). Always measure the score on test data so you can detect overfitting.

When you are done with feature engineering you can perform feature selection to reduce the number of features or understand their importances in the model.


How to cluster histograms or density distributions?

There are quite a few ways to measure the difference between two distributions. Take a look at this overview article on wikipedia.

One very common way that is used often in Machine and Deep Learning is the Kullback-Leibler (KL) Divergence. It is most commonly used in minimising the cross-entropy between the distribution of your training data and the expected (generalised) distribution of the problem you are analysing. In general, a value close to 0 indicates that two distributions are expected to show similar behaviour, while a large value indicates the distributions behave very differently - so knowing the first distribution doesn't help you know anything about the second.

A more general class of measures of dissimilarity between distributions is the so called $f$-divergence, "an average, weighted by the function $f$ of the odds ratio given by [probability measures] $P$ and $Q$". The Wikipedia article contains a formal definition and a table with functions $f$ that give some popular dissimilarities/distances, including the KL-divergence.

  The following table lists many of the common divergences between probability distributions and the $f$ function to which they correspond (cf. Liese & Vajda (2006))

KL-Divergence is combined with the entropy itself, to define the cross-entropy. Take a look here, under the Information Theory View, for a bit more info.


The pattern you describe is not obvious, or strong. Nevertheless, to answer your question, use Fourier series like this:

library(forecast)
y <- ts(c(6.62,6.57,6.57,6.55,6.52,6.56,6.52,6.55,6.54,6.55,6.55,6.62,
          6.78,6.73,6.6,6.59,6.57,6.57,6.51,6.55,6.39,6.4,6.15,6.22,
          6.3,6.27,6.27,6.24,6.24,6.23,6.22,6.26,6.23,6.22,6.2,6.14),
        frequency=12)
mylm <- tslm(y ~ trend + fourier(y, K=3))
plot(y, type='p')
lines(fitted(mylm), col='blue')

This is called a harmonic regression. See https://otexts.org/fpp2/useful-predictors.html#fourier-series for the details.


You can just convert them into factors if your random forest algorithm is accepting that .Usually one hotting the categorical variables are not suggested
Here is the an article gives information about categorical variables in Random Forest


Distractor Generation for Multiple Choice Questions

How are you planning to create the distractor in the first place?

I can't comment because I don't have the repo to do so.


How to model product popularity by their prices and sales volume?

We know that the expensive one is actually more popular then the cheap one

What makes you think that? Apparently you have a reason for that, so start from there, find those attributes - is it high customer satisfaction? positive reviews? repurchase? low retention?

Gather those attributes together, evaluate them on all products to see what are the popular ones. It'd help to have a proper definition of popularity.

Sales volume can help too, give it a weight to balance the price difference.

Once you have all this in one table, you can start playing with prediction, clustering, trends etc.


Scaling label encoded values for Linear Algorithms

Sounds like you should use a OneHotEncoder instead of a LabelEncoder since you are trying to encode non-ordinal data such as missing values. Also, one hot encoded values don't get affected by the MinMaxScaler, so that should be fine.

If MinMaxScaling makes sense is dependent on the categories. If your categories are ordinal, like [1,2,3] is [low, medium, high], then it still makes sense to use a LabelEncoder with a MinMaxScaler.

But if you have non-ordinal categorical values, like [-1,0,1] for [MISSING, DOG, CAT], then it would be better to use a OneHotEncoder instead of forcing ordinality with a LabelEncoder. Otherwise the algorithms you will use will make the assumption that the distance MISSING-DOG is longer than MISSING-CAT, which makes no sense.


The word embeddings are the weights of the first layer i.e. the embedding layer and not the softmax output of the function. The embedding values represent a vector which gives the location of the word with respect to other words in a high dimensional vector space. And yes, the embedding values change according to the training corpus. However, if you are using a given language (for example English) and have a large amount of training data the final values of the vectors will turn out to be pretty close even with training corpus of different contexts.


Best way to fuse clusters smaller than a certain size in a distributed way?

My suggestion is just a greedy brute force approach. I am sure that more efficient solutions may exist, but it may be a first step towards your final solution.

Store in a table, for each cluster which size is lesser than T, the distance to its closest neighbouring cluster and the ID of that closest cluster.

Now, sort the table entries in ascending distance order. Merge the cluster in the first entry with its closest neighbouring one. If the closest cluster also had an entry in the table, and its size got larger than T, remove its corresponding entry. Finally, update all the entries of the table, and sort them in ascending distance order again.

Repeat the steps above until there are not more entries in the table.


Extra output layer in a neural network (Decimal to binary)

The question is asking you to make the following mapping between old representation and new representation:

Represent    Old                     New
0            1 0 0 0 0 0 0 0 0 0     0 0 0 0
1            0 1 0 0 0 0 0 0 0 0     0 0 0 1
2            0 0 1 0 0 0 0 0 0 0     0 0 1 0

3            0 0 0 1 0 0 0 0 0 0     0 0 1 1
4            0 0 0 0 1 0 0 0 0 0     0 1 0 0
5            0 0 0 0 0 1 0 0 0 0     0 1 0 1

6            0 0 0 0 0 0 1 0 0 0     0 1 1 0
7            0 0 0 0 0 0 0 1 0 0     0 1 1 1
8            0 0 0 0 0 0 0 0 1 0     1 0 0 0

9            0 0 0 0 0 0 0 0 0 1     1 0 0 1

Because the old output layer has a simple form, this is quite easy to achieve. Each output neuron should have a positive weight between itself and output neurons which should be on to represent it, and a negative weight between itself and output neurons that should be off. The values should combine to be large enough to cleanly switch on or off, so I would use largish weights, such as +10 and -10.

If you have sigmoid activations here, the bias is not that relevant. You just want to simply saturate each neuron towards on or off. The question has allowed you to assume very clear signals in the old output layer.

So taking example of representing a 3 and using zero-indexing for the neurons in the order I am showing them (these options are not set in the question), I might have weights going from activation of old output $i=3$, $A_3^{Old}$ to logit of new outputs $Z_j^{New}$, where $Z_j^{New} = \Sigma_{i=0}^{i=9} W_{ij} * A_i^{Old}$ as follows:

$$W_{3,0} = -10$$
$$W_{3,1} = -10$$
$$W_{3,2} = +10$$
$$W_{3,3} = +10$$

This should clearly produce close to 0 0 1 1 output when only the old output layer's neuron representing a "3" is active. In the question, you can assume 0.99 activation of one neuron and <0.01 for competing ones in the old layer. So, if you use the same magnitude of weights throughout, then relatively small values coming from +-0.1 (0.01 * 10) from the other old layer activation values will not seriously affect the +-9.9 value, and the outputs in the new layer will be saturated at very close to either 0 or 1.


What is the simplest ReLU network that can learn an upside-down v pattern?

In order to find the network, I tried a couple of different small networks:

One hidden layer with two neurons

This would be the function

$$out = \max(a_1 x + b_1, 0) + \max(a_2x+ b_2, 0)$$

$$a_1 = 1, b_1 = 0, a_2=1, b_2 = -1$$

$$a_1 = 1, b_1 = -5, a_2=-1, b_2 = 5$$

2 hidden layers, 3 neurons in total

This is pretty close:

I guess the pattern is right, but the values are just not quite right. Also, I'm not totally sure if only one hidden layer with 3 neurons (or more) might also work.

I always include biases.

The solution

    / o (ReLU) \
IN o            o (linear)
    \ o (ReLU) /

It is basically the 1 hidden layer with 2 neurons, but also using the scaling (-1?) and the bias (+5?) of the output layer.

$$(-1) \cdot (\max(x - 5, 0) + \max(-x + 5, 0))+5$$


Evaluating the performance of a random forest classifier

First of all, if you are trying to impute missing values with a RF model then take a look at the rfImpute() function.

Second, your data in unbalanced, which is why your classification is not good, your model is biased towards the majority class (small) and so it classifies a lot of you cases into the majority class. The issue of imbalance needs to be addressed.

Validating is done with a test set, as the results you have obtained from the model are already done using Cross-Validation (known as OOB scores).


How to implement a GridSearchCV custom scorer that is dependent on a training feature?

You should be able to do this, but without make_scorer.
The "scoring objects" for use in hyperparameter searches in sklearn, as those produced by make_scorer, have signature (estimator, X, y).  Compare with metrics/scores/losses, such as those used as input to make_scorer, which have signature (y_true, y_pred).
So the solution is just to define your own "scoring object" directly, and reference the passed X to do the computation you want.  See The User Guide for some more details.


What are examples for XOR, parity and multiplexer problems in decision tree learning?

Below is an example of XOR dataset for classification. As you can see, decision trees perform pretty poorly on this dataset. Reason is decision trees splits space into rectangular regions. Therefore they are not pretty good with this kind of distributions.

If you really want to use trees in that sort of situation, it is interesting to use so-called rotation trees. Rotation is about performing PCA (principal components analysis) on input features while learning trees. Using it, decision trees can then build non-rectangular regions.

Also, here is a playground to test gradient boosting algorithms including on XOR dataset. It is really interesting ! You can click on "rotate trees" to activate rotation.
Gradient boosting playground


What are the Most Dissimilar MNIST Digits?

Not sure if this constitutes a "study" but I have investigated using PCA to decompose the MNIST dataset to visualize in 2D:

pca = PCA(n_components=2)
pca.fit(ziptrain[:, 1:])
Z_train = pca.transform(ziptrain[:, 1:])

fig, ax = plt.subplots()
for digit in np.unique(ziptrain[:, 0]):
    x = Z_train[ziptrain[:, 0]==digit, 0]
    y = Z_train[ziptrain[:, 0]==digit, 1]
    ax.scatter(x, y, label=int(digit),
               alpha=1.0, edgecolors='none', marker='${}$'.format(int(digit)))

ax.legend()

This yielded the following plot:

Visually I would say that 1's and 0's are very different. I'm sure you could continue the study by clustering them & measuring the cartesian distance from the centroids of their respective clusters.
It is important to note though that your definition of what "different" means will change the outcome of such an investigation and that this is simply one approach.


Feature addition/ subtraction and SVM model accuracy

It is hard to say without trying it. But you should add features as much as you can. When you have so many features, according to your success metrics, you can do future selection / regularization for both increasing the accuracy and speeding up the convergence of gradient descent.

Also you can also try (after adding more features) other classification algorithms such as RandomForest, GBM, Xgboost etc - Tree based models are more accurate in recent years.

Hope this is helpful!


How does a Q algorithm consider future rewards?

As far as I understand, in each iteration, Q-learning algorithm predicts the future reward of next step (and next step only) using the machine learning technique in use (be it the CNN, DNN etc.).

The Q values should eventually converge to the expected sum, future, discounted reward when taking action A in state S and following the optimal policy. Breaking it down:

Expected sum is not exactly the same as "predicted", but close enough for our purposes. And it really does mean sum of the rewards, not a single reward. To differentiate, this is often called the "return" or "utility"
Future -> from the step being evaluated onwards until end of episode, or the limit as time goes to infinity for continuous tasks with discounting.
Discounted -> a discount factor is only necessary for continuous tasks.

  And we are multiplying the reward of next step (and that specific next step only) with discount rate, to make it less important than the immediate reward (with the ratio we specified).

No, there is no multiplication of the reward. Let's take a look at the line:

target = (reward + self.gamma *
                  np.amax(self.model.predict(next_state)[0]))

The reward is not being multiplied by anything.

What is being multiplied by $\gamma$ is the Q value of the next state. That value represents the total sum of all rewards following on from that point - not a single reward value at all.

  So, my question is, how does the algorithm takes even further steps (say, 5 steps) ahead into account?

It is in the Q values. The pseudocode for the code you are looking at is not:

target_for_Q(s,a) = next_step_reward * gamma

It is:

target_for_Q(s,a) = next_reward + gamma * current_value_of_Q(s',a')

Or:

target_for_Q(s,a) = next_reward + gamma * estimate_all_future_return

This is closely related to the Bellman function for policy evaluation.

Intuitively what is happening is that you start with (really poor) estimates for expected return (not expected reward), and update them by inserting observed values of next_reward, s' and a' into the update rule above. The values always represent a learned estimate of total expected return.


SPARKR, http://spark.apache.org/docs/latest/sparkr.html, is designed to support parallel processing.

Additional resources are summarized here:  https://cran.r-project.org/web/views/HighPerformanceComputing.html


Neural Network outputing the same value / normalization

If it works for your data, I would suggest normalization by:
input = (input-median(input)) / std(input)

Also try to use a smaller batch size for initial training. And triple check how your data actually looks like. The same goes for the labels.

Depending on the framework, you should also check the data type of the labels. Some unfortunate computation might cast your float probabilities to int, thereby killing any meaningful output.

I can recommend Andrew Ng's famous Coursera lectures. They give good insight into what happens within the neurons.


The default value for spark.sql.shuffle.partitions is 200, and configures the number of partitions that are used when shuffling data for joins or aggregations.

dataframe.repartition('id') creates 200 partitions with ID partitioned based on Hash Partitioner. Dataframe Row's with the same ID always goes to the same partition. If there is DataSkew on some ID's, you'll end up with inconsistently sized partitions.


Which algorithm to use to match two categories with n dimensions

There is no generalized algorithm to match two categories of data where the dimension is different and perhaps the attribute could also be different. However you can transform the data into similar dimension. One method I would suggest you is to take PCA of both the data.  After taking PCA you can pick the first k principal component of both the data making them virtually identical data with similar attributes. Then you can use k-nearest neighbour on the transformed data of Aisles to train your model. The model could be easily used to test the transformed data of products.


I used this post to learn how to model something similar in PowerBI/DAX. It summarizes how to use generate to create a calculated table that has one row per date and order that shows the status.  When the users creates relationships to other tables (dates and statuses in this example) then it’s relatively straightforward to get information about number of open orders or some other information you included in the calculated table (I used contract price in my set up).


ImageDataGenerator for multi task output in Keras using flow_from_directory

Have you tried creating a custom Data Generator for your use case? The structure for it can be found below. You will need the __data_generation method such that you return y as an array with 2 elements, with the first element being your labels for task 1 and the second element for task 2.

import numpy as np
import keras

class DataGenerator(keras.utils.Sequence):
    'Generates data for Keras'
    def __init__(self, list_IDs, labels, batch_size=32, dim=(32,32,32), n_channels=1,
                 n_classes=10, shuffle=True):
        'Initialization'
        self.dim = dim
        self.batch_size = batch_size
        self.labels = labels
        self.list_IDs = list_IDs
        self.n_channels = n_channels
        self.n_classes = n_classes
        self.shuffle = shuffle
        self.on_epoch_end()

    def __len__(self):
        'Denotes the number of batches per epoch'
        return int(np.floor(len(self.list_IDs) / self.batch_size))

    def __getitem__(self, index):
        'Generate one batch of data'
        # Generate indexes of the batch
        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]

        # Find list of IDs
        list_IDs_temp = [self.list_IDs[k] for k in indexes]

        # Generate data
        X, y = self.__data_generation(list_IDs_temp)

        return X, y

    def on_epoch_end(self):
        'Updates indexes after each epoch'
        self.indexes = np.arange(len(self.list_IDs))
        if self.shuffle == True:
            np.random.shuffle(self.indexes)

    def __data_generation(self, list_IDs_temp):
        'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)
        # Initialization
        X = np.empty((self.batch_size, *self.dim, self.n_channels))
        y = np.empty((self.batch_size), dtype=int)

        # Generate data
        for i, ID in enumerate(list_IDs_temp):
            # Store sample
            X[i,] = np.load('data/' + ID + '.npy')

            # Store class
            y[i] = self.labels[ID]

        return X, keras.utils.to_categorical(y, num_classes=self.n_classes)

You can then use your custom generator with the Model object you have created as shown below:
training_generator = DataGenerator(partition['train'], labels, **params)
validation_generator = DataGenerator(partition['validation'], labels, **params)

# Design model
model = Sequential()
[...] # Architecture
model.compile()

# Train model on dataset
model.fit_generator(generator=training_generator,
                    validation_data=validation_generator,
                    use_multiprocessing=True,
                    workers=6)

The codes have been taken from https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly


Can a linear regression model without polynomial features overfit?

It sure can!
Throw in a bunch of predictors that have minimal or no predictive ability, and you’ll get parameter estimates that make those work. However, when you try it out of sample, your predictions will be awful.
set.seed(2020)

# Define sample size
#
N <- 1000

# Define number of parameters
#
p <- 750

# Simulate data
#
X <- matrix(rnorm(N*p), N, p)

# Define the parameter vector to be 1, 0, 0, ..., 0, 0
#
B <- rep(0, p)#c(1, rep(0, p-1))

# Simulate the error term
#
epsilon <- rnorm(N, 0, 10)

# Define the response variable as XB + epsilon
#
y <- X %*% B + epsilon

# Fit to 80% of the data
#
L <- lm(y[1:800]~., data=data.frame(X[1:800,]))

# Predict on the remaining 20%
#
preds <- predict.lm(L, data.frame(X[801:1000, ]))

# Show the tiny in-sample MSE and the gigantic out-of-sample MSE
#
sum((predict(L) - y[1:800])^2)/800
sum((preds - y[801:1000,])^2)/200

I get an in-sample MSE of $7.410227$ and an out-of-sample MSE of $1912.764$.
It is possible to simulate this hundreds of times to show that this wasn't just a fluke.
set.seed(2020)

# Define sample size
#
N <- 1000

# Define number of parameters
#
p <- 750

# Define number of simulations to do
#
R <- 250

# Simulate data
#
X <- matrix(rnorm(N*p), N, p)

# Define the parameter vector to be 1, 0, 0, ..., 0, 0
#
B <- c(1, rep(0, p-1))

in_sample <- out_of_sample <- rep(NA, R)

for (i in 1:R){

  if (i %% 50 == 0){print(paste(i/R*100, "% done"))}

  # Simulate the error term
  #
  epsilon <- rnorm(N, 0, 10)

  # Define the response variable as XB + epsilon
  #
  y <- X %*% B + epsilon

  # Fit to 80% of the data
  #
  L <- lm(y[1:800]~., data=data.frame(X[1:800,]))

  # Predict on the remaining 20%
  #
  preds <- predict.lm(L, data.frame(X[801:1000, ]))

  # Calculate the tiny in-sample MSE and the gigantic out-of-sample MSE
  #
  in_sample[i] <- sum((predict(L) - y[1:800])^2)/800
  out_of_sample[i] <- sum((preds - y[801:1000,])^2)/200
}

# Summarize results
#
boxplot(in_sample, out_of_sample, names=c("in-sample", "out-of-sample"), main="MSE")
summary(in_sample)
summary(out_of_sample)
summary(out_of_sample/in_sample)

The model has overfit badly every time.
In-sample MSE summary
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  3.039   5.184   6.069   6.081   7.029   9.800

Out-of-sample MSE summary
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  947.8  1291.6  1511.6  1567.0  1790.0  3161.6

Paired Ratio Summary (always (!) much larget than 1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  109.8   207.9   260.2   270.3   319.6   566.9


How to improve a regression prediction model for data with parameters which are out of sample range?

I have encountered case in which potent tree learners acted like Nearest Neighbor variants: They would learn to divide the search space so that only examples remained that where close on some meaningful axis (in my case lat and long :)). This could still generalize to examples in the test set that shared those features but not to genuinely unseen examples. Or to put it in another way: The method is potent enough to find rules that work well, but do not generalize in the way you want it to (aka over fitting, but the maybe in a specific way). What helped me were two things: First of all I tested this characteristic of the domain by just using KNN,only feeding it subsets of the feature space (and sure enough KNN over lat, lng worked like a charm). This helped me understand that examples that were in the same space time coords shared certain history, so the second action was to eliminate those examples in the train space and evaluate on an a set of examples totally outside of the coordinates (space and time) of the train set. It is hard to say where this behavior could lie in your case without having a peek at the code, but perhaps my anecdote helps.


NLP to detect duplicates for very technical language

If you think fuzzy matching does not work, you basically have two options:

Unsupervised: Try topic modeling to find „similar“ products. The problem is that you will need to pre-define the number of products (groups, aka „topics“), which can be a problem. Also topic modeling will only work well if there are sufficient differences in the descriptions. https://github.com/Bixi81/R-ml/blob/master/NLP_topic_modelling.R

An unsupervised approach seems to be a good option for you, judging from what you wrote about your problem.

Supervised: If you have a good set of descriptions with a label (e.g. product 1, product 2, etc), you can use some NLP model to predict which text belongs to which product. Not sure how much you know about it: You could start with a „Bag of Words“ and try things like Lasso, Boosting, or even Neural Nets on this.

Here is an example on how to apply Neural Nets to text classification. But be aware: in real world applications, text classification can be very tricky, especially if you have many classes (products in your case) and not so many observations per class (descriptions in your case).


Found the fix by creating this calculated field:
{EXCLUDE [ColumnName] : COUNTD([ColumnName]) = 23}
This would exclude all of the customers when all were shown. Then I added this to the Filters card and the color marks card.


Python - Todoist API - Store data into Dictionary

At first remember that a dictionary is just bunches of keys and value pairs...

So, if the key is $a$ and the value is $10$ , you can access the value by doing d[a] (d is the dictionary)

Beaware that there is no order to storing the values, you can use OrderedDict for that..

Also you can have two values for the same key in a simple dictionary.

Here's the complete answer


What is the intuitive meaning of "leaf weight" in xgboost

I would read the paper (section 2.2 specifically) for a more concise presentation and formal definition of the terms.

You need to define the loss function $\mathcal{L}$ for your problem. Then $g_i$ and $h_i$ are the first and second derivatives, respectively, of $\mathcal{L}$. The optimal weights are then (equation (5) or slide 27).
$$
w_j^* = -\dfrac{\sum_{i \in I_j}g_i}{\sum_{i \in I_j}h_i + \lambda}
$$
where $\lambda$ is a regularization parameter.

So those weights give you the minimum of (the second order approximation to) $\mathcal{L}$.

Edit:

Since the weights are given in the example the hard work is done (see Figure 1 in the paper). Following the definitions, $f =w_{q(x)}$ and $q(x)$ maps an instance to a leaf node. Putting this together gives:
$$
f = \begin{cases}
      2 & \mbox{age} < 15 \mbox{ and } \mbox{male} \\
      0.1 & \mbox{age} < 15 \mbox{ and } \mbox{not male} \\
      -1 & \mbox{age} \geq 15
   \end{cases}
$$


Linear Regression with vs without polynomial features

Occam’s razor principle [ref]:

Having two hypotheses, that has the same empirical risk (here, training error), a short explanation (here, a boundary with fewer parameters) tends to be more valid than a long explanation.

Hence, complexity must add additional comparable accuracy. Otherwise, we should go with the simple model
Let's say you have 50-50 chance of Linear and Non-Linear data. With your approach, you will end up using regularization in 50% of the model. First, make a complex model and then regularize to make it simple.
Overall, you will increase the complexity which might impact future activities e.g Maintenance
Interpretability
 Many businesses demand a reason for the decision e.g. Credit card related decision. Linear regression will give you a simple explanation for each feature
Also, read these SE answers


You are getting blocked because people do not want to waste server bandwidth on someone who is trying to exploit it without bringing significant profits.

Try to make your crawling less predictable.

Slow down the frequency with which you ping the server and vary the actions of your crawler. This will make it harder to detect as it will act less predictably and may be wrongfully identified as being a very quick human.


How to fit an odd relationship with a function?

The definition you gave is the definition of the function. This is called the Heaviside Step Function. There is not a simple analytic way to express it (like as a ratio, product, or composition  of trigonometric functions, exponentials, or polynomials). Note that it is neither continuous nor differentiable at x = 0.

There are a couple of cool ways to represent it. The coolest and most intuitive way is as an integral of a Dirac Delta Function:

$$
H(x) = \int_{-\infty}^x { \delta(s)} \, \mathrm{d}s
$$

Note, though, that a Dirac Delta Function is itself not an "official" function, since it is not well-defined at x = 0. Check out Distribution Theory for some cool info on weird "functions" like this.

Now, I think you may be trying to approximate this function, because you asked how to "fit" it. Taken straight from Wikipedia:

For a smooth approximation to the step function, one can use the logistic   function

$$
H(x) \approx \frac{1}{2} + \frac{1}{2}\tanh(kx) = \frac{1}{1+\mathrm{e}^{-2kx}},
$$

where a larger k corresponds to a sharper transition at x = 0.


Depends on how you define "hubs". In Network Science, a hub is simply a node with high degree i.e. those node who contribute to the power-law nature of the degree distribution the most. But you can also find other definitions for instance according to the information flow in the network where hubs are defined as those node that are critical in the process information flow (also called central nodes).

My Suggestions

Degree Distribution: The simplest approach would be to choose high degree nodes as hubs. To ensure your results a bit more I recommend to calculate the summation of all correlations correspond to each node and have a look at these numbers as well. In this case you are looking for nodes which have high degrees and contain a larger value of correlation.
Centrality Measure: or Betweenness introduced by Linton Freeman which again somehow measures the influence of a node in the process of information flow over the network. Calculating the centrality for a vertex $v$ in a graph has basically 3 steps:

For each pair of vertices $(s,t)$, compute the shortest paths between them.
For each pair of vertices $(s,t)$, determine the fraction of shortest paths that pass through the vertex in question (here, vertex $v$).
Sum this fraction over all pairs of vertices $(s,t)$.

For more information read this carefully and in case you need any help (specially for implementation) just drop me a line in the comments :)

Good Luck!


PCA or cluster table of experimental fitness scores

PCA is a dimensionality reduction algorithm - it projects your high dimensional data onto a lower dimensional plane. This is useful for either visualisation (if you reduce to 2 or 3 dimensions to plot), or for training machine learning models.

You say you want to find patterns in your data - I’m not quite sure what you mean by this. Do you want to visualise your data, train a model on it and make some prediction, or something else?

To visualise high dimensional data, you could use either the tSNE (t-stochastic neighbour embedding) algorithm, or PCA.

Depending on the type of data you have, you can “find patterns” in different ways.

If your data is unlabelled (you don’t know the classes of each sample or there is no dependent variable) you can use unsupervised learning algorithms such as K-means clustering, K nearest neighbours, Gaussian mixture model. If your data has dependent variables, depending on whether your dependent variable is categorical or continuous, you could use classification algorithms for the former and regression algorithms for the latter. Classification algorithms include logistic regression or decision trees. Regression models include linear regression.


Different optimizers for generator and discriminator in GAN

Maybe you can share the link to this advice, so we get the context or arguments, because it doesn’t make a lot of sense to me.

You specifically ask why it’s possible to choose different optimizers. The only requirement that the relation between the discriminator and the generator must meet, is that the discriminator must be able to handle the generator output. Everything else can be different, including the optimizer, prediction algorithm, any regularization, and so on. You could have a neural net for the generator and Mechanical Turk could be your discriminator.

There is something to be said for having different algorithms or architectures between the generator and the discriminator, essentially so the discriminator is able to pick up on the weaknesses of the generator. (Because they won’t share the same weaknesses, hopefully.) But to have everything the same and change only the optimizer? I don’t see the point in that.


Standardizing features by one specific feature

If this is a pandas DataFrame:

vel_mean = df.velocity.mean()
 vel_std = df.velocity.std()
 df = df.apply(lambda x: (x - vel_mean) / vel_std)

In case it is a numpy array:

data = (data - np.mean(data, axis=0)[0]) / np.std(data, axis=0)[0]


I have no experience in hmmlearn.

But if you want to receive the labels for the underlying states of the process you have to do 'decoding' by using the 'Viterbi' algorithm. It calculates the most probable state sequence for your data. So if you only have two hidden states it calculates a 0/1 label for each of your time-stamps.

If instead you're trying to discern between two different processes and recognize them, you have to train two different HMMs for the two processes and calculate the foward probabilty up to time t. Then you can choose the process with the higher probability.


Problem reconstructing Keras's IMDB reviews

Nevermind, it wasn't very clear to me that the defaults for load_data() actually shift the sequence indices by 3, and so to obtain the proper indices for the dictionary you either need to offset those indices during lookup or shift the dictionary values, e.g.:

ind = imdb.get_word_index()
ind = {k:(v+3) for k,v in ind.items()}

And also every review is padded with a 1 at word [0], and so reconstructing the review means starting at [1] or passing start_char=None in load_data()


What are the audio features to best describe a music?

Every cepstral coefficients can be considered as one of the best features for defining a musical piece.

Most famous being the Mel Scale, as I can see you are already extracting MFCCs, you are good to go. Although you should have mentioned, which MFCC are you extracting, from experience (a little bit) first 15 are usually the most useful cause they have a positive value. You can also work on more intrinsically robust GFCC.

They can then be used to obtain Spectrograms and so on.

I would suggest not using kNN, prefer Random Forrest, moreover audio signals require a lot of preprocessing. DCT and STFT are a must.


Why do you want to build a learning model?

Your problem seems to be a common assignment problem, all informations are known, absolute and deterministic. You don't need a ML algorithm for this.

Just vectorize your data as one-hot-vectors, define a distance function (you can look to the Hamming's distance).

Compute the distance vector beetween user/task and pick the one who minimize this distance.


Regression - predict n numbers based on another n numbers

I would recommend using an encoder-decoder structure based on LSTM-RNN for sequence-to-sequence learning; it will provide you with exactly what you want. A link that I find very helpful is:

https://machinelearningmastery.com/define-encoder-decoder-sequence-sequence-model-neural-machine-translation-keras/


Common Techniques to Generate from a Regression Neural Network Model

Let $E_\phi : D\rightarrow \mathbb{R}$ be your trained differentiable regression model, where $D$ is the data space, e.g. images. Let $G : \mathbb{R}^d\rightarrow D$ be some generative model or decoder from a latent space (e.g., GAN or decoder from a VAE).

Suppose we want to find $x\in D$ such that $x = \min_y E_\phi(y)$. Then there are two obvious ways:

Gradient descent in the data space: i.e., solve $x = \min_y E_\phi(y)$ via iterating $ x_t = x_{t-1} - \eta\nabla E_\phi(x_{t-1}) $.
Gradient descent in the latent space: i.e., solve $z = \min_u E_\phi(G(u))$ via iterating $ z_t = z_{t-1} - \eta\nabla (E_\phi(G(z_{t-1}))) $.

Both are very common.
The main benefit of (2) is that (a) you are more likely to get a "reasonable" $x=G(z)$ because the generator $G$ was trained to give you one (i.e., $x$ should look like it was from $D$, which is not guaranteed in (1)), (b) $d$ is often lower dimensional than $|D|$, making the optimization easier, and (c) it may be impossible to even do (1) (for instance, in molecule generation, we can do (2) but not (1) due to the discreteness problem). Nevertheless, there is a large drawback: the need to learn $G$, which is itself non-trivial to obtain. (Hence why for e.g. generating adversarial examples, (1) is more popular.)

One might also want to avoid gradient-based optimization in some cases, in which case I often see Bayesian optimization approaches being used.


Matplotlib and Seaborn are graphical plotting libraries, there is no straightforward way to display data visualizations in the terminal.

The best option is to run Jupyter Notebook with --no-browser flag on the remote instance. Then, paste the URL from the remote instance in your local computer browser. You make have make sure the ports and permissions are correctly set. Connecting to a remote Jupyter server from a local computer allows plotting on the remote instance.