It would be a bad idea to use one hot encoding because this would make the problem multi-label. This would make the problem much more complex for the model and would very likely lead to lower performance, or it would require much more data in order to reach the same performance as regular classification. This is because there are much more combinations possible in the multi-label problem: every instance can have any number of classes, from zero to all of them. Your training data would not have these cases, but the algorithm would assume that they can happen. So in your case this higher level of complexity is pointless since there can be only one class.
With scikit the standard way to encode the target in a multiclass task is to use the LabelEncoder.


I don’t know why your question was voted down! Fuzzy matching is such a common challenge. The best approach I’ve seen is this one: https://towardsdatascience.com/fuzzy-matching-at-scale-84f2bfd0c536 It gives similar results to something like Levenshtein distance but it’s much faster. If you augment the matching approach with hand-coded regex features to spot e.g. female versus male teams, you should be able to match teams pretty well.


Usually to find an optimum you set the derivative of the function equal to 0. In your case that gives

$$ \frac{dG_j}{dw_J} = G_j + (H_j+\lambda)w_j = 0 $$

leading to

$$ w_j = \frac{-G_j}{H_j+\lambda}$$


Can't understand Output shape of a Dense layer - keras

Keras is applying the dense layer to each position of the image, acting like a 1x1 convolution.

More precisely, you apply each one of the 512 dense neurons to each of the 32x32 positions, using the 3 colour values at each position as input. That's why you have 512*3 (weights) + 512 (biases) = 2048 parameters.

As a consequence, for each neuron in each position you generate an output, and that's why you have 512 outputs for each position, i.e., 32x32x512.

This all happens for each example in the batch: that's the None part.


What is the meaning of a quadratic relation when r = 0?

Below I have graphed a linear and quadratic function, $y = 50x + 3$ and $y=5x^2 - 1000$, respectively.

Before calculating anything, we can observe that $x$ and $y$ are related to one another in some form. If we describe in words, we can say for the left plot that as $x$ increases so does $y$.

Similarly, for the right plot, we can say that as $x$ moves towards 0, from the left, $y$ decreases towards 0, and as $x$ moves away from 0 towards the right, $y$ increases.

In fact, $x$ and $y$ have a perfect linear relationship in the left plot, while $x$ and $y$ has a perfect quadratic relationship in the right plot. We can say this because the points, in both plots, lie on the red line

If we take it a step further and calculate the correlation between x and y.

We observe for the linear relationship that $x$ and $y$ has a correlation of 1. In contrast, for the quadratic relation, $x$ and $y$ has a correlation of 0.

Below you will find R code for the plot above, in addition, code to calculate the correlation coefficients

set.seed(1)
library(ggplot2)
library(dplyr)
library(gridExtra)

# Linear Function
linear = function(x){(50)*x + 3 }

# Quadratic Function
quadratic = function(x){(5)*x^2 - 1000}

# Create Data Frame
df = data.frame(x = c(-20:20),
                y = c(sapply(-20:20, quadratic),sapply(-20:20, linear)),
                type = c(rep('quadratic',41),rep('linear',41)) )

# Plot Functions
lp = ggplot(subset(df,type == 'linear'), aes(x,y)) + geom_point() +
  stat_function(fun=linear, colour="red") + ggtitle('Linear Relationship')

qp = ggplot(subset(df,type == 'quadratic'), aes(x,y)) + geom_point() +
  stat_function(fun=quadratic, colour="red") + ggtitle('Quadratic Relationship')

grid.arrange(lp, qp, ncol=2)

# Calculate correlation coefficients
df %>% filter(type =='linear') %>% select(x,y) %>% cor %>% (function(x){x[1,2]})
df %>% filter(type =='quadratic') %>% select(x,y) %>% cor %>% (function(x){x[1,2]})


How to input LSTM output to MLP with concatenate?

In Keras, there is no need to give the number of samples in your training data to your model.
Also, the way you defined your LSTM means that you consider each data sample as a timestep for the LSTM, instead of your t-1, t-2 and t-3 values.
So you can remove train_X_LSTM.shape[0] in your Input layers, and give X=[train_X_LSTM, train_X_MLP] and y=train_y_LSTM to model.fit so it matches what your model expects.
#lstm input shape
lstm_input = Input(shape=(train_X_LSTM.shape[1], 1))
#lstm units
hidden1 = LSTM(10)(lstm_input)
hidden2 = Dense(10, activation='relu')(hidden1)
#lstm output which will be predicted var1 at t=t
lstm_output = Dense(1, activation='sigmoid')(hidden2)
#mlp input with additonal 3 variables at t=t
mlp_input=Input(shape=(train_X_MLP.shape[1]))
#combine the lstm output which is predicted var1 at t=t and key in var2,var3,var4 at t=t
x = Concatenate()([lstm_output, mlp_input])
#mlp model output which is predicted var1 at t=t
mlp_out = Dense(1, activation='relu')(x)
#final output of combined model which is predicted var1 at t=t
model = Model(inputs=[lstm_input, mlp_input],outputs=mlp_out)
#compile the model
model.compile(loss='mae', optimizer='adam')
#fit the model
model.fit([train_X_LSTM, train_X_MLP], train_y_LSTM, batch_size=64, epochs=10, validation_split=0.2)

In addition, from my understanding, you want to input (var1, var2, var3, var4) values vector at timesteps  t-3, t-2, t-1 to the LSTM.
In that case, you should also reshape the data you feed to the LSTM
train_X_LSTM=train_X_LSTM.reshape(-1, 3, 4)

This way, you will feed a sequence of 3 vectors containing (var1, var2, var3, var4) at the corresponding past timesteps.
Then, update the LSTM Input layer
lstm_input = Input(shape=train_X_LSTM.shape[1:])

Full code
import numpy as np
import pandas as pd
train_df = pd.DataFrame(np.random.randint(0,100,size=(100, 16)))
train_df.columns=['var1(t-3)','var2(t-3)','var3(t-3)','var4(t-3)','var1(t-2)','var2(t-2)','var3(t-2)','var4(t-2)','var1(t-1)','var2(t-1)','var3(t-1)','var4(t-1)','var1(t)','var2(t)','var3(t)','var4(t)']
train_X=train_df.drop(['var1(t)'],axis=1)
train_y=train_df[['var1(t)']]
#subset the 3 previous timesteps of the 4 variables for the time sries part
train_X_LSTM=train_X[train_X.columns[:12]].reset_index(drop=True).values
train_X_LSTM=train_X_LSTM.reshape(-1, 3, 4)
#target is always var1(t)
train_y_LSTM=train_y.values
#take the current timestep fatures which are var2,var3,var4 which are realised at t=t
train_X_MLP=train_X[train_X.columns[-3:]].reset_index(drop=True).values
#target is always var1(t)
train_y_MLP=train_y.values

from tensorflow.keras.layers import Input, LSTM, Dense, Concatenate
from tensorflow.keras.models import Model
#lstm input shape
lstm_input = Input(shape=train_X_LSTM.shape[1:])
#lstm units
hidden1 = LSTM(10)(lstm_input)
hidden2 = Dense(10, activation='relu')(hidden1)
#lstm output which will be predicted var1 at t=t
lstm_output = Dense(1, activation='sigmoid')(hidden2)
#mlp input with additonal 3 variables at t=t
mlp_input=Input(shape=(train_X_MLP.shape[1]))
#combine the lstm output which is predicted var1 at t=t and key in var2,var3,var4 at t=t
x = Concatenate()([lstm_output, mlp_input])
#mlp model output which is predicted var1 at t=t
mlp_out = Dense(1, activation='relu')(x)
#final output of combined model which is predicted var1 at t=t
model = Model(inputs=[lstm_input, mlp_input],outputs=mlp_out)
#compile the model
model.compile(loss='mae', optimizer='adam')
#fit the model
model.fit([train_X_LSTM, train_X_MLP], train_y_LSTM, batch_size=64, epochs=10, validation_split=0.2)


Tensorboard is an interactive visualization tool that reads log files and helps you track experiment metrics.

For it to work with Keras's model.fit() method you need to add Tensorboard callback to model.fit() method.

log_dir="logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(
                                      log_dir=log_dir,
                                      histogram_freq=1)
model.fit(x=x_train,
          y=y_train,
          epochs=5,
          validation_data=(x_test, y_test),
          callbacks=[tensorboard_callback])

When you have your own training loop you can define a summary writer and use it to write a summary like this:

current_time = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
train_log_dir = 'logs/gradient_tape/' + current_time + '/train'
test_log_dir = 'logs/gradient_tape/' + current_time + '/test'
train_summary_writer = tf.summary.create_file_writer(train_log_dir)
test_summary_writer = tf.summary.create_file_writer(test_log_dir)

for epoch in range(EPOCHS):
for (x_train, y_train) in train_dataset:
        train_step(model, optimizer, x_train, y_train)
      with train_summary_writer.as_default():
        tf.summary.scalar('loss', train_loss.result(), step=epoch)
        tf.summary.scalar('accuracy', train_accuracy.result(), step=epoch)

In any case, you will need to provide a log directory to run Tensorboard. (It doesn't pop up by itself when you train your network. You will have to run it separately. You can run it while training to monitor metrics at the end of each loop).

tensorboard --logdir yourmodel/logs/dir

If you want to make it work with Jupyter notebook you will have to load the Extention and then run magic command %tensorboard --logdir yourmodel/logs/dir

The code is taken from Tensorboard's getting started link. I hope it helps you understand how it works. Otherwise, you can visit these links. 1, 2


What algorithms can be used to predict the outcome of a cricket match?

Sharing what sort of outcome you are most interested will be helpful in directing you towards a proper answer. However, consider this:

If you are interested, in exploring the likelihood of will win or lose think classification algorithms:

Linear Machine Learning Algorithms
 - Logistic regression
 - Linear Discriminant Analysis

Non-linear Machine Learning Algorithms

K-nearest neighbors
Naive Bayes
Support Vector Machines

If you are interested in exploring the scores of each team in a given game think regression algorithms:

Linear regression
Logistic regression
Stepwise regression
Multivariate Adaptive Regression Splines (MARS)
Locally estimated Scatterplot Smoothing (LOESS)

In my judgement, giving us more direction on your desired outcome we can give you more direction with a solid answer.


It is definitely possible, but obviously to a certain extent. What you need, is a sequence to sequence model trained on questions and answers data of a domain. Denny Britz has this amazing blog post on impelementing a  retreival based chatbot trained on ubuntu dialog corpus using tensorflow. Go through the post to understand the difference between retreival based and generative model chatbot. The efficiency and efficacy of such a bot, is another question all together and rigorous research is going on to improve such bots. IBM watson does it pretty well as mentioned in the comments.


Tensorflow ArgumentError: argument --model_dir: conflicting option string: --model_dir

The example you linked is from version 0.12 but the current tensorflow version is 1.2. The example underwent a lot of changes from 0.12 to 1.0. Actually, the very line you are referencing is no longer there in the last version.

This way, it's possible that you are using a version of tensorflow that is no longer compatible with the APIs used by the example. Ensure that the version of tensorflow you are using is the same as the example.


Aggregate discount sequences are used to define the stacking logic for aggregate discounts. This logic can define:

The order in which aggregate discounts are applied.
Which aggregate discounts are mutually exclusive.
Which aggregate discounts can be "stacked," or applied in conjunction with one another.

You can use aggregate discounts within a quote or an order only if they are part of an aggregate discount sequence. Therefore, you must define an aggregate discount sequence even if you have only one aggregate discount. An aggregate discount can be a part of many aggregate discount sequences.

Aggregate discount sequences are linked with price list and discount matrix line items (like volume discounts and attribute adjustments). At run time, the pricing procedure picks the aggregate discount sequence to be used and evaluates the aggregate discounts to be applied.

The order in which aggregate discounts are applied within an aggregate discount sequence can affect the resulting price. For example, if aggregate discount (a) gives a discount of $100, and aggregate discount (b) gives you 10% off the price of the product, the resulting price would be lower if aggregate discount (b) is applied before aggregate discount (a).

To apply all the aggregate discounts in a sequence, you attach the sequence to either a price list or a product with component line items. A sequence includes all the aggregate discounts that apply to one price list or to one product with components in a price list.

Use the following procedure to define the aggregate discount sequence. This task is a step in Process of Creating Aggregate Discounts.


Weight training breakdown in machine learning

There is a situation called "exploding gradients" where very large error gradients accumulate and result in very large updates to neural network model weights during training.

This can result in an unstable model that cannot learn from the training data or even result in NaN weight values that can no longer be updated.


It turns out that the problem was the MAP estimation. Commenting out that line makes the code works.

Details in  this thread in the PyMC Discourse forum.


Class imbalance in one hot encoding for CNN

You might want to consider using metrics such as the f1-score in order to measure how well your model does.

Notice that if, in real life, the inbalance will be of a similar nature, then your model has to account for it anyway.


Do you need a virtual machine for data science?

Do you need a VM?
You need to keep in mind that a virtual machine is a software emulation of your own or another machine hardware configuration that can run an operating systems. In most basic terms, it acts as a layer interfacing between the virtual OS, and your own OS which then communicates with the lower level hardware to provide support to the virtual OS. What this means for you is:
Cons
Hardware Support
A drawback of virtual machine technology is that it supports only the hardware that both the virtual machine hypervisor and the guest operating system support. Even if the guest operating system supports the physical hardware, it sees only the virtual hardware presented by the virtual machine.
The second aspect of virtual machine hardware support is the hardware presented to the guest operating system. No matter the hardware in the host, the hardware presented to the guest environment is usually the same (with the exception of the CPU, which shows through). For example, VMware GSX Server presents an AMD PCnet32 Fast Ethernet card or an optimized VMware-proprietary network card, depending on which you choose. The network card in the host machine does not matter. VMware GSX Server performs the translation between the guest environment's network card and the host environment's network card. This is great for standardization, but it also means that host hardware that VMware does not understand will not be present in the guest environment.
Performance Penalty
Virtual machine technology imposes a performance penalty from running an additional layer above the physical hardware but beneath the guest operating system. The performance penalty varies based on the virtualization software used and the guest software being run. This is significant.
Pros
Isolation

One of the key reasons to employ virtualization is to isolate applications from each other. Running everything on one machine would be great if it all worked, but many times it results in undesirable interactions or even outright conflicts. The cause often is software problems or business requirements, such as the need for isolated security. Virtual machines allow you to isolate each application (or group of applications) in its own sandbox environment. The virtual machines can run on the same physical machine (simplifying IT hardware management), yet appear as independent machines to the software you are running. For all intents and purposes—except performance, the virtual machines are independent machines. If one virtual machine goes down due to application or operating system error, the others continue running, providing services your business needs to function smoothly.

Standardization

Another key benefit virtual machines provide is standardization. The hardware that is presented to the guest operating system is uniform for the most part, usually with the CPU being the only component that is "pass-through" in the sense that the guest sees what is on the host. A standardized hardware platform reduces support costs and increases the share of IT resources that you can devote to accomplishing goals that give your business a competitive advantage. The host machines can be different (as indeed they often are when hardware is acquired at different times), but the virtual machines will appear to be the same across all of them.

Ease of Testing

Virtual machines let you test scenarios easily. Most virtual machine software today provides snapshot and rollback capabilities. This means you can stop a virtual machine, create a snapshot, perform more operations in the virtual machine, and then roll back again and again until you have finished your testing. This is very handy for software development, but it is also useful for system administration. Admins can snapshot a system and install some software or make some configuration changes that they suspect may destabilize the system. If the software installs or changes work, then the admin can commit the updates. If the updates damage or destroy the system, the admin can roll them back.
Virtual machines also facilitate scenario testing by enabling virtual networks. In VMware Workstation, for example, you can set up multiple virtual machines on a virtual network with configurable parameters, such as packet loss from congestion and latency. You can thus test timing-sensitive or load-sensitive applications to see how they perform under the stress of a simulated heavy workload.

Mobility

Virtual machines are easy to move between physical machines. Most of the virtual machine software on the market today stores a whole disk in the guest environment as a single file in the host environment. Snapshot and rollback capabilities are implemented by storing the change in state in a separate file in the host information. Having a single file represent an entire guest environment disk promotes the mobility of virtual machines. Transferring the virtual machine to another physical machine is as easy as moving the virtual disk file and some configuration files to the other physical machine. Deploying another copy of a virtual machine is the same as transferring a virtual machine, except that instead of moving the files, you copy them.

Which VM should I use if I am starting out?
The Data Science Box or the Data Science Toolbox are your best bets if you just getting into data science. They have the basic software that you will need, with the primary difference being the virtual environment in which each of these can run. The DSB can run on AWS while the DST can run on Virtual Box (which is the most common tool used for VMs).
Sources

http://www.devx.com/vmspecialreport/Article/30383
http://jeroenjanssens.com/2013/12/07/lean-mean-data-science-machine.html


In Hadoop framework, there are multiple way to analyze the data. This depends on your use case, expertise and preference. Hive EXTERNAL tables are designed in a such way that other programmer can also share the same data location from other data processing model like Pig, MapReduce Programming, Spark and other without affecting each other work.

In case of external table you will not loose the data if you have accidentally dropped your table  as you already know it only drops the meta data and delete the schema and data remain untouched.

This method is useful if there is already legacy data in HDFS on which the user wants to put some metadata so that the data can be queried and manipulated using Hive. Or you are loading data on HDFS from other ETL tools.

Since EXTERNAL table doesn't delete the data and you are loading file again you are getting the count difference. if you are on your own to do all operation like load, analysis, drop etc, Hive support the INTERNAL table as well. If you want to delete the data when you drop table you can use Hive INTERNAL table. To create internal table you just need to remove the EXTERNAL keyword from your query and when you will drop this table it will delete the data as well.


ROC curve shows strange results for imbalanced dataset

Davis and Goadrich have explained the relationship between ROC and PR Curves in their paper. It is always recommended to use PR curve over the ROC curve in the presence of highly imbalanced data.

Back to the behavior of your ROC curve, It seems that you don't have more threshold points! I would also agree with Dan and do K-fold CV.

  Davis, J. and Goadrich, M., 2006, June. The relationship between
  Precision-Recall and ROC curves. In Proceedings of the 23rd
  international conference on Machine learning (pp. 233-240). ACM.


Difference of sklearns accuracy_score() to the commonly accepted Accuracy metric

Your "commonly agreed on" and "standard" accuracy is meant for binary classification, in which case it agrees with the other formula from sklearn.  In that case, "positive/negative" refer to the two classes, so this is also a little different from your version.

Your version of it is a sort of average of (the "mediant" of) the implicit one-vs-rest classifiers.  As such, your score is meaningful, but will generally be larger than the actual common multiclass accuracy metric.  For a balanced problem, a constant classifier will get a mediant-of-OVR-accuracy score of $(n-1)^2/n^2$ but an accuracy score of just $1/n$.  (Back to the binary case, to compare your method, you'd have to interpret e.g. "Sum(TN)" as including both diagonal entries, so the "accuracy" there is actually $1/2n$, which agrees with the mediant-of-OVR score.)

As such, your metric is similar to macro-averaged scores (though I've never heard of that for accuracy, only precision/recall/Fbeta).  Micro Average vs Macro average Performance in a Multiclass classification setting

Finally, as an opinion, since accuracy measures the probability of getting the prediction right, it's easier to interpret; your metric gives credit for not misclassifying a sample into each class it's not put in, hence the inflation.  Of course, this also perhaps makes a multiclass model's accuracy sound terrible (sklearn says "this is the subset accuracy which is a harsh metric since you require for each sample that each label set be correctly predicted").


Build a Generalised Regression Model Containing Linear and Non Linear Predictor Variables with a Target Variable in Python

I suggest using "Generalised Additive Models". These type of models are linear but can treat wild non-linearity. The idea is - e.g. with regression splines - that a number of linear regressions are "stacked", so that they can jointly account for highly non-linear effects.

Here is a Python implementation: https://pygam.readthedocs.io/en/latest/

When you are bound to linear regression (OLS), you can add polynomials to the regression. In this case you simply generate a new "column" in your data frame, containing e.g. $x^2$. You can add this variable to the regression directly because linear regression is additive:

Example:

$$ y = \beta_0 + \beta_1 x + u $$

...can be augmented by a squared term for $x$...

$$ y = \beta_0 + \beta_1 x + \beta_2 x^2 + u $$

... and this also works for $x^3$ (and so on) or you can take $log()$ etc.

With GAM you don't have to decide on how to model non-linearity. That is the great advantage of GAMs. When you stick to OLS, you need to check if the non-linearity (imposed by you) really helps to improve fit and/or prediction.

GAM are very well explained in "Introduction to Statistical Learning" have a look at Chapter 7. There also is Python code for the Labs in the book.


Training an acoustic model for a speech-to-text engine

This paper has details on how to prepare Audio data (and merge it with Language Models) for speech-to-text :

http://slazebni.cs.illinois.edu/spring17/lec26_audio.pdf

Slide 16 has very high level description of feature engineering.

Following papers provide more detailed analysis of audio processing for speech to text :

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.701.6802&rep=rep1&type=pdf
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.689.4627&rep=rep1&type=pdf
https://www.ijariit.com/manuscripts/v4i3/V4I3-1646.pdf


Which algo you should use is up to debate and sometimes comes down to the person

If you're going for accuracy then random forests are a great place to start (as mentioned in the comments) but personally I like logistic regressions as my starting place for this type of problem as they have decent accuracy and are comparatively explainable.

But if this is your first Machine Learning problem I'd recommend doing some wider reading first on how to formulate and approach machine learning problems.

It'll give you the perspective to decide which model to use on your own. Furthermore, you'll likely need to explain your model choices further down the line and using the "some dude from stack overflow told me to use it" excuse won't cut the mustard.

Links:

How to approach a ML problem

Intro to ML in Python


For very simple linear regression can we quantify the prediction accuracy hit between using one hot encoding and simple numerical mapping?

The issue with numerical encoding in this context is you are enforcing that your input variable X is ordinal when it's likely not. This is telling your model that the order in which you encode your inputs are either increasing or decreasing monotonically with your target. Let's say you encoded your data like this:
[North] - 0
[East] - 1
[West] - 2
[South] - 3
[North, East] - 4
...
[North, East, West, South] - 15

If you train a linear regression model with this encoding you are telling your model that [North] either indicates a higher or lower target than [North, East, West, South], and that [East], [West], [South], and [North, East] are somewhere in between. What if [West] typically has a lower target than either [North] or [North, East, West, South]? In this case you would be enforcing some constraint on your model which is not true.
To test this you could split your input data in a 70/30 train-test-split and evaluate how your numerical encoding performs against one-hot-encoding on the test set. Another alternative would be to look into Target Encoding - this would allow you to keep a small feature space (an issue with one-hot-encoding) while attempting to keep your input and target increasing/decreasing monotonically. I would recommend testing all three encoding methods to figure out the ideal solution for your given problem.


What does the relative sum of squares error signify in neural network analysis in SPSS?

Check this:

  A value closer to 0 indicates that the model has a smaller random error component, and that the fit will be more useful for prediction.


Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder

About the need for tgt_key_padding_mask
While padding is usually applied after the normal tokens (i.e. right padding), it is perfectly fine to apply it before normal tokens (i.e. left padding). For instance, fairseq supports parameter left_pad to specify precisely this.
For left padding to be handled correctly, you must mask the padding tokens, because the self-attention mask would not prevent the hidden states of the normal token positions to depend on the padding positions.
About the meaning of each argument
I think the best documentation is in MultiHeadAttention.forward:

key_padding_mask – if provided, specified padding elements in the key will be ignored by the attention. When given a binary mask and a value is True, the corresponding value on the attention layer will be ignored. When given a byte mask and a value is non-zero, the corresponding value on the attention layer will be ignored
attn_mask – 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all the batches while a 3D mask allows to specify a different mask for the entries of each batch.

With that information and knowing where keys, values and queries come from in each multi-head attention block, it should be clear the purpose of each parameter in nn.TransformerDecoder.forward. Also, the documentation of MultiheadAttention.forward contains info about the expected shapes.


Just looked into the book. It was an example of the data that is presented in Fig 1.4. Numpy is a good package to derive the sum of squares fit to the data. This is an optimization problem. Look at the notes section of numpy.polynomial.polynomial.polyfit. I think the data is not given (you can digitize it!). The book is showing some qualitative behavior to give some intuition.


Can hadoop with Spark be configured with 1GB RAM

So if 4GB of RAM isn't sufficient, 1GB isn't going to be. That is really too little to run an HDFS namenode, a datanode, YARN, Spark driver alone, let alone leaving room for your workers.

Much more reasonable is to simply run Spark locally on that instance without Hadoop at all.

But I would question whether Spark is the right choice if you are definitely limited to such a small machine.


To demonstrate how this can be done, I created some sample/dummy records. See my answer below with comments explaining each step:

(1) Prepare data and create a column to capture all ID's of active customers

import pandas as pd
import numpy as np
from dateutil.relativedelta import relativedelta

# Create a dataframe with five customers from three cohorts
df = pd.DataFrame({'txn_new_cif': [1, 1, 2, 3, 4, 5],
                   'CohortGroup': [np.datetime64('2018-01-01'), np.datetime64('2018-01-01'),
                                   np.datetime64('2018-02-01'), np.datetime64('2018-02-01'),
                                   np.datetime64('2018-02-01'), np.datetime64('2018-03-01')],
                   'Txn_date': [np.datetime64('2018-01-30'), np.datetime64('2018-07-01'),
                                np.datetime64('2018-05-01'), np.datetime64('2018-08-01'),
                                np.datetime64('2018-06-01'), np.datetime64('2018-12-30')]})

# Count the number of days between each transaction date and the cohort date
days_diff = df['Txn_date'] - df['CohortGroup']

# Convert into integer
df['Diff_days'] = days_diff.dt.days

# If the transaction occured during the first 120 days (30 days of cohort month + 90 days)..
# Then those customers are considered as 'active'
# Capture the ID of those active customers
df['Active_cif'] = df.loc[df['Diff_days'] <= 120]['txn_new_cif']

# Find the date of each of the following three months after the end of the cohort month
df['First_month_after'] = [x+ relativedelta(months=2) for x in df['CohortGroup']]
df['Second_month_after'] = [x+ relativedelta(months=3) for x in df['CohortGroup']]
df['Third_month_after'] = [x+ relativedelta(months=4) for x in df['CohortGroup']]

# Calculate the cut-off date to determine whether the txn occured with the 1st/2nd/3rd month after
days_diff_1 = df['First_month_after'] - df['CohortGroup']
days_diff_2 = df['Second_month_after'] - df['CohortGroup']
days_diff_3 = df['Third_month_after'] - df['CohortGroup']

df['First_month_days'] = days_diff_1.dt.days
df['Second_month_days'] = days_diff_2.dt.days
df['Third_month_days'] = days_diff_3.dt.days

# Based on these cut-offs, determine if the activity took place within the 1st/2nd/3rd month after
df['Active_month'] = np.where(df['Diff_days'] <= df['First_month_days'], df['First_month_after'],
                              np.where(df['Diff_days'] <= df['Second_month_days'], df['Second_month_after'],
                                      np.where(df['Diff_days'] <= df['Third_month_days'], df['Third_month_after'], None)))

df['Active_month'] = pd.to_datetime(df['Active_month'])

df.head()

(2) Aggregate data by cohort

# Count the total number of customers in each cohort
cohort_cust_ct = pd.DataFrame(df.groupby(['CohortGroup', 'Active_month'])['txn_new_cif'].nunique()).reset_index()

# Count the total number of customers who were active in each cohort
cohort_act_ct = pd.DataFrame(df.groupby(['CohortGroup', 'Active_month'])['Active_cif'].nunique()).reset_index()

# Combine both results
df_grp = cohort_cust_ct.merge(cohort_act_ct, left_on=['CohortGroup', 'Active_month'], right_on=['CohortGroup', 'Active_month'],
                             how='outer')

# Calculate churn rate for each cohort
df_grp['Churn_rate'] = 1 - (df_grp['Active_cif'] / df_grp['txn_new_cif'])

# View results
df_grp.head()


What parameters to use when normalising training, validation, and testing data?

I am not exactly sure what you mean by "what parameters should be used when normalizing datasets."
However, it is important to note:
Normalization is a preprocessing step that you do to some or all of the parameters of your model before constructing the model.
But in answer to your question:
You always normalize the same parameters used in both the train and the test set (otherwise how would you be able to compare the results?).


How many models shall be used in ensemble modelling?

There are no restrictions/guidelines on the number of models. You can start even from 3 models. You can keep the number of models as a hyperparameter if the training cost is less. Typically, you will observe the slanted 'L' shaped curve for MSE vs # of models plot. You can take the elbow point as the final # of models.


Optimization function returns the same optimal parameters for two labels

The error stemmed apparently from the formula for the gradient: there should be no minus sign (-) at the start of it. The correct code for this would be:

function [jValue, gradient] = costFunction(theta, m, X, Y)
    V = X*theta;
    jValue = (-1/m)*sum(Y.*log(sigmoid(V)) + (1-Y).*log(1-sigmoid(V)));
    gradient = (1/m)*X'*(sigmoid(V)-Y);
end


Clustering time series based on monotonic similarity

The way Ward's linkage is computed really only makes sense with squared Euclidean type of measures. Only then the Konig-Huygens theorem can be used.

Why don't you consider average linkage? Why Ward?


Does AWS Sagemaker continue running when I go offline?

I couldn't stay at work forever, so I left the Sagemaker notebook running, closed my computer and commuted home.  That evening when I returned to my Sagemaker Notebook, the work had completed running and all the cells and their variables continued to be accessible in memory.

Long story short, if you disconnect from Sagemaker but leave your notebook running, it will continue to process for you.


How to prove that a function is sigmoidal

I'm not so sure about how close this is to what you want but one way to prove if a function $y = f(x)$ is the sigmoid equation, is to check if it's a solution of the ordinal differential equation:

$$
\frac{dy}{dx} = y \; (1-y)
$$

with initial condition $y(0) = 1/2$.


Evaluate prediction from multiple classification model

Based on the description of your question, it seems that you want the probability of outcome of each class (in multiclass-classification).

I would suggest you to use XGBoost to get output based on your requirement. By setting the value of objective parameter to multi:softprob, you can get probability of prediction of each and every class. If you set the value of objective parameter to multi:softmax, then you will only get the class with maximum probability among other classes.

Here, I am writing a example for your reference and to explain this description in a better way. You can get output by printing y_test_preds.

import xgboost as xgb

xgb_class = xgb.XGBClassifier(**params)
bst = xgb.train(params, dtrain, num_rounds)
y_test_preds = bst.predict(dtest)

By the following way, you can set the parameters for XGBoost. I would strongly suggest you to modify these parameters (except objective) based on your data and requirements.

params = {
     'objective' : 'multi:softprob',
     'max_depth' : 6,
     'silent' : 1,
     'eta' : 0.4,
     'num_class' : 3,
     'n_estimators' : 500,
     'learning_rate' : 0.1,
     'num_rounds' : 15
}

Note: In XGBoost, you have to use DMatrix instead of DataFrame. You can also get the DMatrix from DataFrame by this way.

dtrain = xgb.DMatrix(X_train.values, label = y_train.values)
dtest = xgb.DMatrix(X_test.values, label = y_test.values)

If you are new to XGBoost, then I would recommend you to go through this link once. https://xgboost.readthedocs.io/en/latest/get_started.html


How to find appropliate algorithm to bulid a model for natural language based two data

There's no algorithm intended specifically for this task, you need to design the process yourself (like for most tasks btw).
Given that the goal would be to use a person's name as an indication, I'd suggest you represent a name as a vector of characters n-grams in the features.
Example with bigrams ($n=2$):
"Braund" = [ #B, Br, ra, au, un, nd, d# ]

Intuitively the goal is for the model to find the sequences of letters which are more specific to a nationality. You could try with unigrams, bigrams or trigrams (the higher $n$, the more data you need for training).
Once the names are represented as features this way, you can train any type of supervised model, for example Decision Tree or Naive Bayes.


If you are just starting out, it is pretty unlikely that you come across such big datasets that require massive parallelization either in computing power or data storage means.

When you start pushing the limits of what a single machine can do, that's when you deploy a big data solution like Spark and Apache HDFS storage and start writing parallelizable models that scale well with bigger data.

In a small scale it might be actually way slower to run some jobs at Spark, because it adds some overhead to the calculations. Spark is best used for large enough, well parallelizable and iterative problems.

It is also worth mentioning that it requires some patience to configure the whole cluster to work well.


Principal components analysis with compositional data

I'm in the same situation as you.  PCA seems to have some difficulties for compositional data.  I've listed some papers below that have been helpful to me.  I'm still trying to understand everything, but these papers have a good overview I think.  Or googling "Compositional Data" or "Compositional Data Principal Component Analysis" returns a lot of other papers that are freely available and informative.

Principal component analysis of compositional data J. Aitchinson

A Concise Guide to Compositional Data Analysis J. Aitchinson

Principal components analysis for compositional data with outliers P. Filzmoser, et al.


Machine Learning, Imputing values that should be blank

Well, it seems that you are dealing with sparse data, however imputation is a difficult and often an attempt of imputation can add trivial amount of difference. You may look out on for this link for some approaches like Gharamani and Jordan.
These are variants of SVM, focused with Sparse nature.


How to get an intuitive value for regression module evaluation?

To get an intuitive contrast, I came up with:

Refer to
$$R^2=1-\frac{\sum ( y_i - \hat y_i)^2} {\sum (y_i - \bar y)^2}$$

My method is:
$$My Value=\frac {\sum |y_i - \bar y|} {\sum | y_i - \hat y_i|} $$
Indicating how many times the model better than other model, always predicts mean value.


order of features importance after make_column_transformer and pipeline

Your assumption is correct. Usually after column Transformation columns lose their names and get default values corresponding to their orders.

Additional Info:
You may try Eli5

from eli5 import show_weights,show_prediction
show_weights(model)
show_prediction(model,data_point)

The later function shows the impact of every features for predicting a data_point.


Choosing a model for dataset with categorical variables

I know you ask about the model choice here, but it is worth to discuss about your input data first. Data with many categorical features is still an active research; so it is not that straightforward. I suggest you first look at this very similar post, where I discuss some techniques to convert categorical variables to numerical values.

Since you provide little information about your categorical variables, for example how many levels each categorical variable have or how you do label encoding (just out-of-the-box method?) it is hard to give better guidelines. Generally speaking though, label encoding is mostly used for encoding  target variables (dependent variable) if it is categorical, while you mention you use label encoding for other 6 independent variables (inputs). Here your target is rather numerical values (regression), so I would go beyond label encoding to convert your independent categorical variables.

Image the following very simplified scenario (just for educational purposes) to understand why label encoding is not a good idea where there is more than 1 independent categorical variables:

   Color   Year
0    Red   2010
1   Blue   2011
2  Green   2012

Now label encoding this would converting this to (naively speaking):

  Color  Year
0   1    1
1   2    2
2   3    3

There ways like constructing your own dictionary for each categorical variable and values associated to its levels while label encoding to prevent this obvious mistake. Apart from that, since you hardly go beyond RMSE value of 0.11, one could think of your input values (maybe not encoded properly!) and choice of your model.

Maybe try using this code for handling your independent categorical variables. Please note here, as a general rule of thumb, avoid using one-hot encoding although it is widely used in XGBoost (explained in the post). Maybe try "Target Encoding"!

Now speaking of the choice of model:

Have you tried regularization (L1, L2)?
Have confident you are about the independency of your independent variables? In simple regression you are assuming this holds, but may it is not which is easily the case that your independent variables are correlated. Maybe you should analyze this first using confusion matrix (see here for a brief tutorial)
Have you tried gradient boosting decision trees (GBDT) for regression?
Note1: If you want to go with GBDTs, do not leave that other 1 independent variable alone. You said you have 6 independent variables out of 7. I assume the other one is continuous numerical values? If that so, you have to handle that properly in GBDTs like binarizing (put in bins).
Note2: You may try either XGBoost, or Catboost, and a nice tutorial using Catboost. Each has its pros and cons. For the former hyperparameter tuning could be challenging and there is no ways to automatically account for categorical variables but have larger community. While the latter is recently released (mid last year), it is much easier in terms of hyperparameter tuning (often defaults works great), and there are ways to automatically account for categorical features without explicitly encode them, but less support and community is smaller.
Note3: The good thing about GBDT is that you do not need to worry about the correlation of features per se. Algorithm automatically ignore the coupling of strongly correlated features.
Note4: You may easily overfit, although there are built-in regularization. Catboost for example have a interactive learning curve for training and validation to control overfitting (see the last tutorial).

I strongly believe taking care of your inputs together with a proper choice of algorithm would boost your RMSE error.


Should there be a flat layer in between the conv layers and dense layer in YOLO?

It doesn't matter, with or without flattening, a Dense layer takes the whole previous layer as input. The spatial structure information is not used anymore. Some Neural Network implementations might not be able to map a spatial structure directly into a dense layer, which is why you would need the Flatten in between. Mathematically it is exactly the same in this case.
EDIT: As I expected, in the source of Keras it mentions that if the rank is higher like with convolutions, they implicitly Flatten it first.


In text classification, how can I use a neural network on word embeddings?

Okay. These are the steps to follow.

Pad your sentences to a fixed length. Use the maximum length. If you are comfortable with creating buckets, you can pad the sentences to maximum length for individual bucket. To pad, add  PAD tokens to each sentence till they become the size of the fixed sequence length you want to use.
Extract the unique words out of the padded corpus and assign a numeric id to every word. You can use there index in the unique list as IDs. Filtering out rare words and replacing them with unknown token will prove useful.
Now if you had a sentence like "I am John", and your maximum sequence length was 5 the sentence will become, "I am John PAD PAD." Once you extract the unique words, you can represent the sentence as IDs like [1,2,3,4,4] Now if you had 100 sentences you will have a matrix of size 100 X 5.

Now you can go in two ways, represent each word as a one hot vector i.e.a vectors of zeros with a one at the index it represents or a simpler way would be to train a word embedding.

For details look into this RNN tutorial by Denny Britz, this is from scratch.

If you are of the deep learning persuasaion then check out Denny Britz's CNN text classification tutorial where he uses tensorflow and trains his own word embedding. The first blog will give you all the information you need on how to prepare your text dataset.


Which classifier is efficient in dealing with test query which belongs to no trained class?

You have trained a model to recognize or discriminate between several particular classes. So when having a new test sample (that according to your knowledge) does not belong to any of this class, the model will fit it to the most similar class.

This is, off course, a very general way of putting it. The behavior and things to consider wether you are using a probabilistic graphical model (like Naive Bayes), linear or non linear classifiers, etc, will vary. However, the principle is the same: the model has learnt the relation among the features of the training data you have used to match particular classes.

More specifically, if you are using a probabilistic approach you can  use the probability of belonging to a class and define a threshold. So if a probability higher than certain value (let's say 65%) is not given, then your confidence on the result will be low, and you might say "I cannot say wether it belongs to any of this 5 classes with certainty".

Other non probabilistic methods have interesting approaches for scoring a new example in terms of probabilities. Check this link for SVM (https://stats.stackexchange.com/questions/55072/svm-confidence-according-to-distance-from-hyperline), which is actually the approach used in Python scikit-learn: http://scikit-learn.org/stable/modules/svm.html#scores-and-probabilities

If you use Random Forests you could use the voting of each tree to define a rate of confidence in a similar way (https://stats.stackexchange.com/questions/94845/how-to-estimate-confidence-level-for-svm-or-random-forest)

A similar thing can be done with K-Nearest-Neighbors by analyzing the distance of the K or J (where J < K) nearest neighbors and just trust the result if they are close to your test sample in some ratio (very generally said, this can be done in many ways).

Now, methods based on deep learning are trying to learn features in an unsupervised way, so your problem could solved in a more interesting way. However, it would not be strict classification anymore, nor you would have the amount of data and servers required to even try it (but just saying :) )


Is the probabilistic cutoff in random forest flexible?

Yes, you are exactly right. 0.5 is just a heuristic, ROC curve and precision-recall curve give a much better idea of what the cut off should be. You can then use predict_proba, extract the probabilities and do the classification based on the cut-off you have inferred from ROC curve and.  precision-recall curve


How to combine two time dependent datasets?

If I merge on dates, I'd have multiple repeated rows with fires occurring at the same time in different places, would that be the best way?

Probably not, since you don't want to lose the location information. You should probably find a way to map the latitude/longitude to borough/county between the two datasets, so that you obtain a semantically consistent dataset (list of fires by both date/time and location).

The problem I see is that the locations where there was no fire would not be represented, so I'd have to add a bunch of blank rows for all locations where there was no fire to balance the set.

This depends what you plan to do with your data, but given how your first dataset is currently structured (list of fires) it doesn't make a lot of sense to add locations where there was no fire. You might want to create a dataset which lists for every place and every time whether there was a fire or not, for example.


Is there any advantage in using Particle Swarm Optimization for clustering than K-Means?

K-means makes locally the optimal decision. Most of the time (in particular when this objective works reasonably well at all) this works quite well to find a good local optimum. I doubt that using an approach such as PSO gives you any advantage here - the problem is just too simple, and k-means has the speed advantage, while PSO is unlikely to find other optima.
And in particular the better k-means algorithms such as Hamerly, Elkan, ... are so fast, they'll be able to run with hundreds of random restarts in the time needed for just one PSO.


There are quite a few tools that serve this purpose (E.g.: Tableau , QlikView are expensive but feature rich; Kibana / Dash have Open Source versions).

You can start with Kibana. It will take a week to learn it. Advantages are :

Part of ElasticSearch stack, it is easy to ingest data (Say that you need to update data from ERP every hour / day /week etc.)  'Logstash' component isused for ingesting / updating data
Widely used, so tutorials / documentation are easy to look up
You can customize dashboards with JS / html
As usage grows, you can add features (Such as access control, periodic email of reports as PDF / Excel etc)

Some tutorials / Documentation  :

https://en.wikipedia.org/wiki/Kibana
https://www.elastic.co/products/kibana

https://www.youtube.com/watch?v=gQ1c1uILyKI


What is difference between final episodes of training and test in DQN?

What is difference between running in final episode of training mode and running in test mode in DQN?

As Q learning is an off-policy method, and DQN is based on Q learning, then the agent effectively has two different policies:

A behaviour policy that explores actions regardless of whether data so far suggests that they are optimal. The simplest and perhaps most commonly implemented behaviour policy is $\epsilon$-greedy with respect to action values $\hat{q}(s,a,\theta)$.
A target policy that is being learned, which is the best guess at an optimal policy so far. In Q learning that policy is fully greedy with respect to action values $\hat{q}(s,a,\theta)$.

When assessing how well the agent has learned the environment after training, you are interested in the target policy. So with Q learning you need to run the environment with the agent choosing actions greedily. That could be achieved by setting $\epsilon = 0$, or with a different action selection routine which doesn't even check $\epsilon$ (which to use is an implementation detail).

  Is there any difference more than after training and tune the hyper-parameters, we test for one episode and without any exploration?

Essentially yes to second part - without exploration. But . . .

  Why in some test code of DQN, they test for multiple episodes?

That is because one episode is often not enough to get reliable statistics.

If the environment is completely deterministic, including having only one start state, then one episode would be sufficient to assess an agent.

However, if the environment is stochastic, or you are interested in multiple possible start states, then multiple episodes are necessary in order to have low mean squared error bounds on the average total return, or any other metrics you might be interested in. Otherwise it would be hard to compare results between different hyperparameters.

How many episodes are enough? There is no fixed number, it depends on how much variance there is in the environment and how accurately you want to know the values. In some cases - e.g. OpenAI gym, there are suggested values. For instance, LunarLander suggests a target score over 100 episodes, so they are suggesting measurements are accurate enough over 100 episodes. However, the advice only applies to that problem. In general, you can measure the variance in the score, like any other statistical measure, and calculate error bounds on your metric.


Can I use euclidean distance for Latent Dirichlet Allocation document similarity?

Euclidean distance -by which in this application, I assume you mean the euclidean distance in an $n$-dimensional space defined by the distribution of document contents among $n$ topics considered, is a valid measure to use in comparing the topics represented within two documents.

What you're doing by applying this method is quantifying a topic frequency difference within this newly defined space, and so interpretation of these quanta will require analysis of the space. For example, what euclidean distance indicates that documents are relatively similar?

In distiction, the normalized result of something like the hellinger distance provides an easily interperable framework by which to evaluate the results- a score of 0 indicates no overlap in the distribution over the topics in question of the two documents, and a 1, perfect overlap.

For the efficiency concerns, it's not clear to me why you couldn't truncate your topics considered to the crucial topics and then calculate any of the metrics on the distributions over ony those topics, rather than the entire universe of considered topics.


What you are exactly looking for is a modification to the DEDICOM Algorithm (page 4). the DEDICOM itself gives you measure for relation between different components of a directed graph. You just need to be a bit creative to use it for converting a graph into DAG. Read the paper and if further help needed just drop me a comment.


If you read the following sentence at the first line of section 6:

  The debiasing algorithms are defined in terms of sets of words rather than just pairs, for generality, so that we can consider other biases such as racial or religious biases.

$D_1, D_2, \ldots, D_m$ are the set of words, in general, to be considered in the de-biasing algorithm, instead of considering a pair of words such as Computer Programmer and Houseworker for the de-biasing genders of man and woman. Hence, one of $D$s could be {'Computer Programmer', 'Houseworker'}, here, as an example.


Dealing with pre-trained model for grayscale images

print(grayscale_batch.shape)  # (64, 224, 224)

rgb_batch = np.repeat(grayscale_batch[..., np.newaxis], 3, -1)
print(rgb_batch.shape)  # (64, 224, 224, 3)


how to create outputs for key points of bounding boxes on image in Neural network in Python

Your network can be improved in many ways :
Adding a few convolutional layers could help reduce the dimension as you have a lot a weights on your first FC layer, and this will improve the speed of your training.
But your main problem is your loss function, cross entropy is used for classification, which means your network is considering all your 5 ouputs as classes (a softmax function is applied to all 5 of them before computing cross entropy).
For your task, i would feed the outputs of the convolutional network into 2 networks:

One is the classifier that tells you if it is a cat or a dog (1 output).
The second outputs the bounding box (4 outputs).

These networks can be trained independently using different losses and will give you coherent results.


Attributing causality to single quasi-independent variable

I would suggest you to consider either direct dimensionality reduction approach. Check my relevant answer on this site. Another valid option is to use latent variable modeling, for example, structural equation modeling. You can start with relevant articles on Wikipedia (this and this, correspondingly) and then, as needed, read more specialized or more practical articles, papers and books.


neural network with large numbers of inputs

One-hot encoding is the normal approach, and yes you would end up with 500 features for your group alone. Depending on how much training data you have this does not need to be a problem. If you have a lot of other features without direct interactions between the individual products you could use an Embedding layer before adding them to the rest, which maps your sparse categorical one-hot encoded features into a dense space via backpropagation. This will reduce the amount of parameters significantly.

If you don't have enough training data to do this, you could look at gathering statistics about your product, that say something about other features for the group, or about the target for other rows (be sure not to use the current row, which would introduce target leakage) which would allow you to drop the category all together.

Turning them into IDs and then using this as a numerical feature is a bad idea, since there is no inherent structure in the numbers, which means there is no signal in that and it would need a very high complexity to learn how to distinguish 1 and 5 being very similar but 2 and 4 very different.


Handling nominal category features in decision tree

I would not say that using One-Hot-Encoder to handle nominal features is an optimal way. Even it might affect adversely your Decision Tree model in terms of performance.
Theoretically, a Decision Tree does not need a one-hot-encoding for categorical features, besides it will increase the number of computations and will result in a relatively inefficient model. Unlike non-tree algorithms (e.g. regression algorithms), a Decision Tree cannot be biased by the label encoding of categorical features because it uses splits to reach the optimal result.
Experimental results show that one-hot-encoding gives a rise to sparsity in the trees which might drastically drop the performance of the model. Here is a wonderful analysis and explanation of the effect of one-hot-encoding on Tree-based algorithms. Also, check this useful source.


What are some standard ways of computing the distance between documents?

There's a number of different ways of going about this depending on exactly how much semantic information you want to retain and how easy your documents are to tokenize (html documents would probably be pretty difficult to tokenize, but you could conceivably do something with tags and context.)

Some of them have been mentioned by ffriend, and the paragraph vectors by user1133029 is a really solid one, but I just figured I would go into some more depth about plusses and minuses of different approaches.

Cosine Distance - Tried a true, cosine distance is probably the most common distance metric used generically across multiple domains. With that said, there's very little information in cosine distance that can actually be mapped back to anything semantic, which seems to be non-ideal for this situation.
Levenshtein Distance - Also known as edit distance, this is usually just used on the individual token level (words, bigrams, etc...). In general I wouldn't recommend this metric as it not only discards any semantic information, but also tends to treat very different word alterations very similarly, but it is an extremely common metric for this kind of thing
LSA - Is a part of a large arsenal of techniques when it comes to evaluating document similarity called topic modeling. LSA has gone out of fashion pretty recently, and in my experience, it's not quite the strongest topic modeling approach, but it is relatively straightforward to implement and has a few open source implementations
LDA - Is also a technique used for topic modeling, but it's different from LSA in that it actually learns internal representations that tend to be more smooth and intuitive. In general, the results you get from LDA are better for modeling document similarity than LSA, but not quite as good for learning how to discriminate strongly between topics.
Pachinko Allocation - Is a really neat extension on top of LDA. In general, this is just a significantly improved version of LDA, with the only downside being that it takes a bit longer to train and open-source implementations are a little harder to come by
word2vec - Google has been working on a series of techniques for intelligently reducing words and documents to more reasonable vectors than the sparse vectors yielded by techniques such as Count Vectorizers and TF-IDF. Word2vec is great because it has a number of open source implementations. Once you have the vector, any other similarity metric (like cosine distance) can be used on top of it with significantly more efficacy.
doc2vec - Also known as paragraph vectors, this is the latest and greatest in a series of papers by Google, looking into dense vector representations of documents. The gensim library in python has an implementation of word2vec that is straightforward enough that it can pretty reasonably be leveraged to build doc2vec, but make sure to keep the license in mind if you want to go down this route

Hope that helps, let me know if you've got any questions.


How can I extract the residual array from the Scikit Learn PCA routine?

I'm not super familiar with PLS/NIPALS, but I think you need to construct it yourself. Multiply the scores by the loadings and subtract the resulting matrix from the data to get the residual matrix. See here for a more detailed discussion: https://learnche.org/pid/latent-variable-modelling/projection-to-latent-structures/how-the-pls-model-is-calculated


Suggestion of a model for these type of data?

Any kind of classification algorithm should be alright for your data.

Logistic Regression is probably the first thing you will learn in a ML tutorial.

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html


What are the limitations while using XGboost algorithm?

I think you should be more specific about what you mean by "fail". As an example, a practitioner could consider an xgboost model as a failure if it achieves < 80% accuracy.

Nevertheless, there are some annoying quirks in xgboost which similar packages don't suffer from:

xgboost can't handle categorical features while lightgbm and catboost can.
xgboost can be more memory-hungry than lightgbm (although this can be mitigated).
xgboost can be slower than lightgbm.


Why continuous features are more important than categorical features in decision tree models?

It could be the way that you encode categorical variables.

If you do One Hot Encoding (dummy) each encoded feature will only have two possible values [0,1]. Binary variables normally have less importance in Decision trees given how it is computed the feature weight.

Let's say per example, that you are trying to predict the condition of a patient at the hospital Alive==0, Dead ==1. Imagine that you have a feature is called Head_Shot[0,1], that is really rare, it only appears a few times in the dataset.

The linear model will assign a lot of weight to this coefficient since it is crucial for the target variable. If this happens the rest of the features has no meaning.

For the decision tree, it could do a split in just one of the tree and since it calculates the importance of a feature weighted time the number of times it appears it wouldn´t such a relevant weight.

I am assuming you are doing one-hot encoding. With other techniques, it will be different. And I also assume the way that you calculate the feature importance. So this is far from a scientific answer.

Continuous variables can have more importance in decision trees because each tree can do several splits along its way.

Sorry for the example, it is a bit drastic but I believe it makes the point.


Recover a integer valued function with *-learning

If I understand correctly, the goal will be for the model to be provided with a value $n$ and predict the optimal $p$ which minimizes $f(n,p)$.
From this point of view, it looks like a simple regression problem. You could probably train a regression model: for every  point in your training set, the predictor is $n$, the response variable is the $p$ value which minimizes $f(n,p)$ for this $n$. Note that all the points which don't correspond to the minimum are irrelevant in this context.
I'd suggest you start by plotting this function from your data (not the one with a fixed $n$, the one which maps $n$ to the optimal value of $p$). It would be useful  to know what it looks like, in particular to choose an appropriate regression method.
[edit] About dealing with noise.
The standard statistical way to deal with noise is to do nothing at all :)
How much data do you have? Because if you have enough data, statistically the noise should balance itself out. For instance if you have 100 such graphs for a fixed $n$, the mean optimal $p$ should be close enough to the true optimal $p$. Even if for a few values of $n$ the data doesn't contain the optimal value of $p$, across sufficiently many values of $n$ this noise is unlikely to be significant. What matters is for the regression model to correctly represent the general trend, and not to capture the small variations due to noise.
I would start with a graph made of one boxplot for each value of $n$ and varying $n$, where the set of values represented in a boxplot are all the mnimum $p$ in the data for this value of $n$.
[edit 2] Another interesting visualization: you can represent your full data as a heatmap with $n$/$p$ on the X/Y axis and the colour based on the value of $f(n,p$)$.


What is bad, good and excellent metric score for time series model?

I do not think there is a silver bullet for metric value, because it is highly problem-specific, one way or another you need to compare your result to baseline in order to understand how viable it is.
For example, in time series common baseline for the next point predicted is a previous point or some moving average. You can get an error metric like MAPE (but be careful with it, since it has its downsides) and compare your models MAPE to Baseline MAPE (say moving average) and with this comparison, you can argue about how your approach is better.
Also, check this article regarding MAPE and other metrics.


This question may be more suited to CrossValidated. However:

"Bagging" is short for "bootstrap aggregating", meaning that a random sample with replacement is taken from the overall set. The key here is "with". For example, suppose this is your dataset:

{1, 2, 3, 4, 5}

and you are interested in obtaining 4 samples of size 3. You could end up with the following:

{1, 4, 4}

{1, 4, 5}

{2, 3, 4}

{3, 3, 4}

That is, you can have repeated elements within the same bootstrap sample (e.g. the first result, {1, 4, 4}), and each bootstrap sample could contain the same element (e.g. notice the value '4' in each sample). You can see more on this here.

Yes, it is possible for there to be a correlation among the bootstrapped samples, but if you're interested in, for example, a random forest, then the trees are necessarily NOT correlated. If you were to train 100 trees on the same data set, then you could of course end up with correlated trees. However, in a random forest, the trees are uncorrelated. You can see more on this here.

Hope that helps!


What is the use of torch.no_grad in pytorch?

The wrapper with torch.no_grad() temporarily sets all of the requires_grad flags to false. An example is from the official PyTorch tutorial.
x = torch.randn(3, requires_grad=True)
print(x.requires_grad)
print((x ** 2).requires_grad)

with torch.no_grad():
    print((x ** 2).requires_grad)

Output:
True
True
False

I recommend you to read all the tutorials from the link above.
In your example: I guess the author does not want PyTorch to calculate the gradients of the new defined variables w1 and w2 since he just want to update their values.


Clustering using both text and numerical features

One way to achieve this is to use a clustering method based on a custom similarity/distance measure. For example you could defined the similarity measure between two instances as:
$$sim(\langle v_1, n_1\rangle,\langle v_2, n_2\rangle)=\frac{1}{2} cosine(v_1,v_2)\ +\ \frac{1}{2} \left(1-\frac{|n_1-n_2|}{max(n_1,n_2)}\right)$$
This measure gives the same weight to the similarity between the vectors ($v_1$ and $v_2$) and the similarity between the numerical values ($n_1$ and $n_2$). Note that since this similarity measure is normalized, you can convert it to a normalized distance measure: $d=1-s$.
Of course you should define the exact measure based on what the values represent, this is just an example.
You could use this measure with a hierarchical clustering method or a graph clustering method (with edges based on similarity value).


How the squared Euclidean distance is an example of non-metric function?

Let $x, y \in \mathbb{R}^n$. The Euclidean distance $d$ is defined as
$$
d(x,y) = \sqrt{\sum_{i=1}^n (x_i - y_i)^2}.
$$
The squared Euclidean distance is therefore
$$
d(x,y)^2 = \sum_{i=1}^n (x_i - y_i)^2.
$$

We know that Euclidean distance is a metric.
Let us check whether squared Euclidean distance is also a metric. I will use the definition from Wikipedia (Ankit Seth's definition is equivalent).

Non-negativity: $d(x,y)^2 \ge 0$. This one is obvious.
Identity of indiscernibles: $d(x,y)^2 = 0$ if and only if $x=y.$ This is true because $d$ is a metric, and $d(x,y)^2 = 0$ if and only if $d(x,y) = 0$.
Symmetry: $d(x,y)^2 = d(y,x)^2$. This is again true because $d$ is a metric and therefore $d(x,y) = d(y,x).$

(You can also see that 2. and 3. are true directly from the definition of $d(x,y)^2$).
Looks fine so far. But:

Triangle inequality: Do we have $d(x,z)^2 \le d(x,y)^2 + d(y, z)^2$ for all $x, y, z \in \mathbb{R}^n$? No. Pick an arbitrary $x \in \mathbb{R}^n \setminus \{0\}$ and set $y = 2x$ and $z=3x$. Then
$$
d(x,z)^2 = \sum_{i=1}^n (x_i - 3x_i)^2 = 4 \sum_{i=1}^n x_i^2
$$ and
$$d(x,y)^2 + d(y,z)^2 = \sum_{i=1}^n x_i^2 + \sum_{i=1}^n x_i^2 = 2 \sum_{i=1}^n x_i^2.
$$
Since $4\sum_{i=1}^n x_i^2 > 2 \sum_{i=1}^n x_i^2$, we have found a counterexample that shows that the triangle inequality does not hold.

What does it mean that squared Euclidean distance gives the same ranking as Euclidean distance?
Suppose we have $x, y, z$ such that $d(x,y) < d(x,z)$.
Then $d(x,y)^2 < d(x,z)^2$ as well: It ranks points in the same way as Euclidean distance.

This is good to know. For instance, this tells us that the $k$-nearest neighbors classifiers gives the exact same results for squared Euclidean distance and Euclidean distance.


Are machine learning models good at autoregressive forecasting of time series?

Raw regression is usually not as good performance-wise but there are variations of said algorithms to accommodate time series more effectively like Long Short Term Memory (LSTM) neural networks, Gated Recurrent Units Neural nets among others (GRU) as well as other non neural net approaches and closed to graph theory like Dynamic Bayesian Networks.


Predict set elements based on other elements

Maybe you could use some algorithm that solves "Market Basket Analysis". The problem is explained here:

  Market Basket Analysis is a modelling technique based upon the theory
  that if you buy a certain group of items, you are more (or less)
  likely to buy another group of items. For example, if you are in an
  English pub and you buy a pint of beer and don't buy a bar meal, you
  are more likely to buy crisps (US. chips) at the same time than
  somebody who didn't buy beer.

http://albionresearch.com/data_mining/market_basket.php

One example of such an algorithm is: https://en.wikipedia.org/wiki/Association_rule_learning


Matrix factorization how to initialize weights and biases?

For matrix factorization I usually see it being initialized by a uniform distribution from [0, 1) like in this (pytorch) or a truncated normal with mean=0.0 and std=1.0 as in this (tensorflow).


Derivation of Ward's formula for agglomerative clustering

I will be using these facts without proof (but the proofs either follow directly from definitions or are straightforward): The vector of the centroid of a set of points is arithmetic mean of all vectors of the points in the set. Also, the centroid of the union of two sets of point is on the straight line connecting centroids of these two sets, and it divides that straight line into two parts whose ratio is determined by the ratio of number of points of in these sets.

Let's consider following series of identities:

$\sum_{i\in A} \|\vec{x}_i-\vec{m}_{A\cup B}\|^2 =$

$$= \sum_{i\in A} \Big\Vert \vec{x}_i-\Big(\vec{m}_{A} - \frac{n_B}{n_A+n_B} (\vec{m}_{A} - \vec{m}_{B})\Big)\Big\Vert^2$$

$$= \sum_{i\in A} \Big\Vert\big(\vec{x}_i-\vec{m}_{A}\big) + \frac{n_B}{n_A+n_B} \big(\vec{m}_{A} - \vec{m}_{B}\big)\Big\Vert^2 $$

$$= \sum_{i\in A} \Big\Vert\big(\vec{x}_i-\vec{m}_{A}\big)\Big\Vert^2 + 2 \cdot \sum_{i\in A}\big(\vec{x}_i-\vec{m}_{A}\big) \cdot \frac{n_B}{n_A+n_B} \big(\vec{m}_{A} - \vec{m}_{B}\big) + n_A \cdot \Big\Vert\frac{n_B}{n_A+n_B} \big(\vec{m}_{A} - \vec{m}_{B}\big)\Big\Vert^2\big)$$

$$= \sum_{i\in A} \Big\Vert\big(\vec{x}_i-\vec{m}_{A}\big)\Big\Vert^2 + n_A \cdot \Big\Vert\frac{n_B}{n_A+n_B} \big(\vec{m}_{A} - \vec{m}_{B}\big)\Big\Vert^2\big)$$

$$= \sum_{i\in A} \Vert\vec{x}_i-\vec{m}_{A}\Vert^2 +  \frac{n_A n_B^2}{(n_A+n_B)^2} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$

Similarly:

$$\sum_{i\in B} \|\vec{x}_i-\vec{m}_{A\cup B}\|^2 = \sum_{i\in B} \Vert\vec{x}_i-\vec{m}_{B}\Vert^2 +  \frac{n_A^2 n_B}{(n_A+n_B)^2} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$

By adding:

$$\sum_{i\in {A\cup B}} \|\vec{x}_i-\vec{m}_{A\cup B}\|^2 = \sum_{i\in A} \Vert\vec{x}_i-\vec{m}_{A}\Vert^2 +\sum_{i\in B} \Vert\vec{x}_i-\vec{m}_{B}\Vert^2 + \frac{n_A^2 n_B + n_A n_B^2}{(n_A+n_B)^2} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$

Or:

$$\sum_{i\in {A\cup B}} \|\vec{x}_i-\vec{m}_{A\cup B}\|^2 - \sum_{i\in A} \Vert\vec{x}_i-\vec{m}_{A}\Vert^2 - \sum_{i\in B} \Vert\vec{x}_i-\vec{m}_{B}\Vert^2 = \frac{n_A n_B (n_A + n_B)}{(n_A+n_B)^2} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$

Or:

$$\Delta \big(A, B\big) = \frac{n_A n_B}{n_A+n_B} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$


Replace method in pandas not giving expected result

This is pure(almost) python:

list(map(lambda x: x[0] + '.'.join(['X' * len(c) for c in x[1:].split('.')]), my_df['IP']))

Explanation:
 - Use the map to iterate over each row in my_df['IP'] column.
 - Per each IP value, split into first char and others using the x[0], x[1:]
 notation.
 - Get each part in x[1:] using the split method.
 - For each part get its length and accordingly create a string made of X's.
 - Rejoin this X's string into one string with '.' between them.

If using python2 you dont need the list.


Which, if any, machine learning algorithms are accepted as being a good tradeoff between explainability and prediction?

Are there machine learning models commonly accepted as representing a
  good tradeoff between the two?

I assume that by being good at prediction you mean being able to fit nonlinearities present in the data while being fairly robust to overfitting.
The tradeoff between interpretability and being able to predict those nonlinearities depends on the data and question asked. There really is no free lunch in data science and no single algorithm can be considered to be the best for any set of data (and the same applies for interpretability).

The general rule should be that the more algorithms you know the better it is for you as you can adopt to your specific needs more easily.

If I had to pick my favorite for classification task that I often use in business environment I would pick elastic-net for logistic regression. Despite strong assumption about the process that generates the data it can easily adopt to data thanks to the regularization term maintaining its interpretability from basic logistic regression.

  Is there are any literature enumerating the characteristics of
  algorithms which allow them to be explainable?

I would suggest you to pick a well written book that describes the commonly used machine learning algorithms and thier pros and cons in diffrent scenarios.
An example of such book can be The Elements of Statistical Learning by  T. Hastie, R. Tibshirani and J. Friedman


Loss function returns x whereas tensorflow shows validation loss as (x+0.0567)

On further investigation was able to narrow it down.

The loss value is not only calculated by the response from loss function, but also the various regularizers added in the intermediate layers.

And one of the FC layers in my model was indeed having a L2 regularization term as below:

model.add(layers.Dense(1024, activation='relu', kernel_regularizer=regularizers.l2(0.1))

If I remove the l2 regularization, the loss goes indeed goes to zero if the loss function returns a zero.


Can an entire data frame be used as a prediction variable?

First question; is it possible to pass an entire data frame as a variable to a model?

No, each feature must be a single value. In other words you could provide the data frame as a vector containing all the values, assuming the size is fixed: each column would correspond to a specific cell in the original data frame.

But I think an even better option in your case is to look into methods which take (chronological) sequences into account. Conditional Random Fields might be a good option, assuming you need to predict the target variables for the whole sequence?


To see which attributes influence your model will depend on the type of model you are using. For Decision Trees, Random Forests, and Gradient Boosting based on trees you can plot feature importance metrics.

In linear models such as Logistic Regression, theoretically the coefficients will indicate how much the model will be influenced by their features.

Are you using Python? Let me know which models are you using so I can help you more.


Converting the categorical data into numerical data isn't really meaningful. Different mappings will give you different solutions.

There is an extension of K-means algorithm for categorical data called k-modes. You can read about K-modes in detail here. This article explains the difference between K-modes and converting data into numeric vectors and then running K-means.


What model should I use to find a common pattern for a specific user group based on the other dimensions?

The most common approach is to create business rules handmade, based on the univariate and multivariate analysis of the variable.

Basically, do some frequency count, see if you could isolate some subset of your data just looking at one or two variables.

Then when you have your labels, create a linear or so model with this new variable as output. For exemple, a linear discriminant analysis. The analysis will supply you with new insights on your group.

If you want to rely on an algorithm, two solutions:

As you don't seems to have a lot of variables, a non-supervised segmentation could do the job. For exemple, a k-Nearest Neighbor or a decision tree are basic and good aproachs.

With a few more variables, I like is to do a principal component analysis then a non-supervised classification to define your group on the result of the PCA.
Note that a PCA + handmade rules based on the analysis of your PCA result may be enough.

Each time, in the end, a discriminant analysis and a profile of your groups to asses the quality of your results.


Keras: extreme spike in loss during training

Sorry I couldn't comment as it requires 50 Reputation. On Epoch 44 there is a huge spike in the loss. It is entirely possible that the model may have come across new data and it may have learned a few tricks up its sleeve. Try to plot loss of train & validation vs epoch to see if it underfits or overfits.


Typically you want a smaller RMSE and without getting into detail it should be sufficient to just take the smaller one.  However, I am concerned because you state that the models were ran on the same dataset but at different timeframes.  Since RMSE scale depends on the dependent variable scale, it's entirely possible that these two timeframes are scaled different.  A somewhat contrive example would be energy consumption.  I would expect a model trained on daytime consumption to have a higher RMSE than for one trained between 1am and 3 am.  In that case, comparing the RMSE may be meaningless.  You can try to normalize your data and RMSE to help with this, but i'm unsure if AWS provides this ability.

As for your second questions, you really won't get a 75% accurate number for regression.  You can look at the deviations of the residual or do cross valiadtion and see how well the model performs.

Again this may not be possible in AWS.

edit:  I juse realized that the histograms were residual plots.  Do three things  Increase bin size.  Check to see if the residuals are centered around 0 and then check if there is skewness in the data.  If the data is centered around 0 and symmetric then you can say the model error is basically random and does not favor over or under predicting.  If the data is not centered around 0 and there is skewness, then the errors can be systematic and then in that case considering adding more variables.


What is the meaning of the sparsity parameter

When we implement penalized regression models we are saying that we are going to add a penalty to the sum of the squared errors.
Recall that the sum of squared errors is the following and that we are trying to minimize this value with Least Squares Regression:
$$SSE = \sum_{i=1}^{n}(y_i-\hat{y_i})^2$$
When the model overfits or there is collinearity present, the estimates for the coefficients for our least squares model may be higher than they should be.
How do we fix this? We use regularization. What this means is that we add a penalty to the sum of squared errors thereby limiting how large the parameter estimates can get.
For Ridge Regression this looks like this:
$$SSE_{L2 norm} = \sum_{i=1}^{n}(y_i-\hat{y_i})^2 + \lambda \sum_{j=1}^{P}\beta_j^2$$
Notice what is different with this model. Here, we add a L2 regularization penalty to the end of the SSE. What this does is add the multiplication of $\lambda$ by the square of the parameter estimations as a penalty to the SSE. This limits how large the parameter estimates can get. As you increase the "shrinkage parameter" $\lambda$ the parameter estimates are shrunk more towards zero. What is important to note with Ridge Regression, is that this model shrinks the values towards zero, but not to zero.
You may also use the LASSO Regression technique as shown below:
$$SSE_{L1 norm} = \sum_{i=1}^{n}(y_i-\hat{y_i})^2 + \lambda \sum_{j=1}^{P} \lvert{\beta_j^2}\rvert$$
Notice here that the change by adding the L1 penalty is very similar. However, this difference here is that we are now penalizing the absolute value of the coefficients. This allows shrinkage to zero and can be considered a form of feature selection.
In either case both methods penalize model complexity and are parsimonious!
Two things to note:

In answer to your question, no, $\lambda$ cannot be negative. Why? This would make no sense. $\lambda$ is multiplied by either the L2 or L1 norm to add a penalty to the SSE. If instead, you had a negative $\lambda$ you would actually be rewarding the model's complexity not penalizing it.

When you have a value of $\lambda = 0$ you have no penalty and just have regular least squares regression!


Which Loss function is correct for binary mapping?

Yes, use binary cross-entropy loss. In case you are using Keras, this has been already implemented as a standard loss function for binary outputs.


Meaning of variance in machine learning models

Variance actually measures the variability of the model prediction (say, for simplification, for a particular sample instance) if we would retrain the model multiple times (on different subsets of the data).

To gain an intuitive feeling of variance, suppose you have 100 training examples in your dataset (150 examples in total where 50 examples are reserved for a 50:50 split between validation and test sets).

Pick a specific example from this set of 150 examples, say ${x}_{i}$. You are required to classify this in any one of the classes: ${k}_{1}, {k}_{2}, ..., {k}_{n}$.

Now randomly pick 100 samples from your original set of 150 samples (excluding your ${x}_{i}$). Now suppose you choose a hypothesis that has many higher degree terms, for instance: ${a}_{1}{x}^8 + {a}_{2}{x}^{7} + ... {a}_{0} = 0$ and do not regularise it, it will fit the model nicely and you get a low error on your training set.

You now run this model on ${x}_{i}$ and it predicts wrongly. You repeat the above process of having 100 samples five times and each time you get a different result, say ${k}_{a},    {k}_{b}, {k}_{c}, {k}_{d}$, and  ${k}_{e}$.

This variability in your prediction (coupled with the low error on your training set) implies your model is overfitting the data it is given. That's why it is not able to follow the general pattern in your data and is being affected by outliers more than it should.

I guess that answers your question about declaring a model as highly variant when

  the predicted points are too prolonged


How do I get started with machine learning and image recognition?

What are the advantages of using a pre-trained model vs training my own?

The main advantage of pre-trained models is that it greatly reduces required data for your model. You can see it as a better way to initialize your model.

  Where would I find pre-trained models? At what point would it be worth it to train my own models, and how can I do so?

Keras has a variety of pretrained models for image classification. If your data consists of rare objects or uncommon situations, it might be better to train your own model, say for very specific task. Pretrained models only work if patterns in your data share similarities to patterns learned by the pretrained model.

  Which library should I be using as a beginner? TensorFlow, scikit-learn, etc.

Scikit-learn and Keras are pretty straightforward. They both have toy datasets for experimenting. Deep Learning with Python by François Chollet has a nice chapter on computer vision, which has a section on using pretrained models.


What are the use cases for Apache Spark vs Hadoop

Hadoop means HDFS, YARN, MapReduce, and a lot of other things. Do you mean Spark vs MapReduce? Because Spark runs on/with Hadoop, which is rather the point.

The primary reason to use Spark is for speed, and this comes from the fact that its execution can keep data in memory between stages rather than always persist back to HDFS after a Map or Reduce. This advantage is very pronounced for iterative computations, which have tens of stages each of which is touching the same data. This is where things might be "100x" faster. For simple, one-pass ETL-like jobs for which MapReduce was designed, it's not in general faster.

Another reason to use Spark is its nicer high-level language compared to MapReduce. It provides a functional programming-like view that mimics Scala, which is far nicer than writing MapReduce code. (Although you have to either use Scala, or adopt the slightly-less-developed Java or Python APIs for Spark). Crunch and Cascading already provide a similar abstraction on top of MapReduce, but this is still an area where Spark is nice.

Finally Spark has as-yet-young but promising subprojects for ML, graph analysis, and streaming, which expose a similar, coherent API. With MapReduce, you would have to turn to several different other projects for this (Mahout, Giraph, Storm). It's nice to have it in one package, albeit not yet 'baked'.

Why would you not use Spark? paraphrasing myself:

Spark is primarily Scala, with ported Java APIs; MapReduce might be friendlier and more native for Java-based developers
There is more MapReduce expertise out there now than Spark
For the data-parallel, one-pass, ETL-like jobs MapReduce was designed for, MapReduce is lighter-weight compared to the Spark equivalent
Spark is fairly mature, and so is YARN now, but Spark-on-YARN is still pretty new. The two may not be optimally integrated yet. For example until recently I don't think Spark could ask YARN for allocations based on number of cores? That is: MapReduce might be easier to understand, manage and tune


Recall score for each sample in multilabel classification

This metric is usually referred to as a sample-based or example-based score and can be applied in multi-label cases (to recall and other scores too). You can find a brief explanation here.

Scikit learn has an implementation for it (see here for recall) when setting  average='samples' for recall_score:

  'samples': Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from accuracy_score).