What is the meaning of the term "pipeline" within data science?

A pipeline is almost like an algorithm, but at a higher level, in that it lists the steps of a process. People use it to describe the main stages of project. This could include everything from gathering data and pre-processing it, right through to post-analysis of predictions. The pipeline is essentially a large chain of modules, which can be individually examined/explained.
Here is an example image (source: DataBricks)

There is actually a nifty module (class, actually) in Scikit Learn for building your own machine learning pipelines, which is literally called Pipeline. You can specify processing steps, models and other transformations, then wrap them into a pipeline, which carries out the start-to-finish process for you. This makes it much easier to work in a modular way and alter parameters, while keeping things organised.

In the documentation, they use an ANOVA (analysis of variance) model to select variables, which are then fed into an SVM (support vector machine) to perform classification.

In the context of what might be considered a single model, a pipeline may refer to the various transformations performed on data. This might include dimensionsality reduction, embeddings, encoding/decoding (GAN example), attention mechanisims, and so on.

Here is an example of what might be referred to as a pipeline: the Spatial Transformer Network:

Images are passed through a pipeline with three parts:

a localisation network
a grid generator
a sampling mechanism

these three parts might be akin to one of the parts in the MLlib Pipeline displayed above.

Another area in which pipeline is used extensively is within data management. In this case, it refers to how and where data is transferred, and perhaps by which frequency. There are large packages dedicated to building such pipelines, e.g. :

Apache Spark - can do a lot more than just pipelines these days (use case example)
Luigi - manages complex batch processes (developed by Spotify)

Note: this question was asked and removed just before I posted my answer, so am providing the content via a self-answered question


It would be nice show a few example data points to make sure that my answer is what you may be looking for, but anyways,

1) I think so, you may wanna look at feature weighting for k-means in this paper

3) For the libraries commonly used for this, you may wanna look at this answer or this code

2) I'm unclear what you mean here, maybe you could elaborate, or probably someone's going to pick up on it and answer you on this point.


Logistic regression with high cardinality categorical variable

So I believe you're building a model on the binary outcome {lose, win}:= {0, 1}, correct?

I'd recommend just using a one-hot-encoding or a sparse matrix to store these inputs, then the model should run just fine. This is very straightforward in R (sparse.model.matrix) or Python (pd.get_dummies(sparse=True)).

Here's a quick demo of how to build a sparse matrix in R out of sampled categories and select a subset of them with at least 5 observations.

library(MASS)
require(glmnet)
n <- 1000
x1 <- sample(paste(letters,1), n, replace=T)
x2 <- sample(paste(letters,2), n, replace=T)
x3 <- paste(x1,x2,sep='-')
xdf <- data.frame(x1,x2,x3)
xs <- sparse.model.matrix(~.-1, data=xdf)
vars <- colnames(xs)
colsmry <- colSums(xs)
colsubset <- colsmry > 4
xs_ss <- xs[,vars[colsubset]]
dim(xs)
dim(xs_ss)


What is the difference between Dilated Convolution and Deconvolution?

In sort of mechanistic/pictorial/image-based terms:
Dilation: ### SEE COMMENTS, WORKING ON CORRECTING THIS SECTION
Dilation is largely the same as run-of-the-mill convolution (frankly so is deconvolution), except that it introduces gaps into it's kernels, i.e. whereas a standard kernel would typically slide over contiguous sections of the input, it's dilated counterpart may, for instance, "encircle" a larger section of the image --while still only have as many weights/inputs as the standard form.

(Note well, whereas dilation injects zeros into it's kernel in order to more quickly decrease the facial dimensions/resolution of it's output, transpose convolution injects zeroes into it's input in order to increase the resolution of it's output.)
To make this more concrete, let's take a very simple example:  Say you have a 9x9 image, x with no padding. If you take a standard 3x3 kernel, with stride 2, the first subset of concern from the input will be x[0:2, 0:2], and all nine points within these bounds will be considered by the kernel. You would then sweep over x[0:2, 2:4] and so on.
Clearly, the output will have smaller facial dimensions, specifically 4x4. Thus, the next layer's neurons have receptive fields in the exact size of these kernels passes. But if you need or desire neurons with more global spatial knowledge (e.g if an important feature is only definable in regions larger than this) then you will need to convolve this layer a second time to create a third layer in which the effective receptive field is some union of the previous layers r.f..
But if you don't want to add more layers and/or you feel that the information being passed on is overly redundant (i.e. your 3x3 receptive fields in the second layer only actually carry "2x2" amount of distinct information), you can use a dilated filter. Let's be extreme about this for clarity and say we'll use a 9x9 3-dialated filter. Now, our filter will "encircle" the entire input, so we won't have to slide it at all. We will still however, only be taking 3x3=9 data points from the input, x, typically:
x[0,0] U x[0,4] U x[0,8] U x[4,0] U x[4,4] U x[4,8] U x[8,0] U x[8,4] U x[8,8]
Now, the neuron in our next layer (we'll only have one) will have data "representing" a much larger portion of our image, and again, if the image's data is highly redundant for adjacent data, we may well have preserved the same information and learned an equivalent transformation, but with fewer layers and fewer parameters. I think within the confines of this description it's clear that while definable as resampling, we are here downsampling for each kernel.

Fractionally-strided or transpose or "deconvolution":
This sort is very much still convolution at heart. The difference is, again, that we will be moving from smaller input volume to a larger output volume. OP posed no questions about what upsampling is, so I'll save a bit of breadth, this time 'round and go straight to the relevant example.
In our 9x9 case from before, say we want to now upsample to 11x11. In this case, we have two common options: we can take a 3x3 kernel and with stride 1 and sweep it over our 3x3 input with 2-padding so that our first pass will be over the region [left-pad-2:1, above-pad-2: 1] then [left-pad-1:2, above-pad-2: 1] and so on and so forth.
Alternatively, we can additionally insert padding in between the input data, and sweep the kernel over it without as much padding. Clearly we will sometimes be concerning ourselves with the exact same input points more than once for a single kernel; this is where the term "fractionally-strided" seems more well-reasoned. I think the following animation (borrowed from here and based (I believe) off of this work will help to clear things up despite being of different dimensions. The input is blue, the white injected zeros and padding, and the output green:

Of course, we're concerning ourselves with all of the input data as opposed to dilation which may or may not ignore some regions entirely. And since we're clearly winding up with more data than we began, "upsampling".
I encourage you to read the excellent document I linked to for a more sound, abstract definition and explanation of transpose convolution, as well as for learning why the examples shared are illustrative but largely inappropriate forms for actually computing the represented transformation.


Is it advisable to rerun LASSO multiple (2) times?

Wow I'm surprised that no one has given this one a go. I'm very very late to the party but I'll attempt an answer just for information's sake.

Before I attempt to answer your questions I'd like to say that any situation in which the number of features exceeds the number of observations by 50% there may be some instability in the results.

What I'd recommend is to attempt a number of feature selection approaches and compare the results. Personally I'd start with a PCA to reduce the number of features such that n > p and go from there. But that's just me.

1) I'm not surprised by more estimates being zero at all. In the first lasso regression with this many features a large amount of these features are likely to be noise (no prediction power) meaning that any feature with a small capability of prediction would be selected in the first cut of variables. Running a secondary lasso regression might not be completely advisable (logically, can't really think of any theoretical issues at current) but it's likely that some of the stronger predictors are being uncovered on this second iteration.

2) The estimates in lasso regressions are quite difficult to interpret (apart from zero and non-zero values obviously), but what I'd recommend is to check whether you've scaled your features before running the lasso reg? As my understanding is that a lasso reg is a penalised version of the ridge regression, and the ridge regression needs the features to be scaled as otherwise features represented by larger numbers are given higher coefficients. If you have just ignore this, but if you want to interpret the estimates I'd switch to a Linear Reg/GLM not a lasso reg.

Hopefully this helps.


I can give you some basics but the proper way to do this is to learn the prolog language, and it's very different from any other programming language.
In Prolog you define predicates (similar to functions) which contain conditions for the predicate to be satisfied, for example:
dt(Feat1, Feat2, Feat3, labelA) :-
    Feat1 =< 3,
    Feat2 == 'green',
    Feat3 >= 0.54.

dt(Feat1, Feat2, Feat3, labelB) :-
    Feat1 =< 3,
    Feat2 == 'green',
    Feat3 < 0.54.

dt(Feat1, Feat2, _Feat3, labelC) :-
    Feat1 > 3,
    Feat2 == 'green'.

dt(_Feat1, Feat2, _Feat3, labelB) :-
    Feat2 == 'blue'.

Here the predictate is dt and it has 4 clauses. In order for dt to be satisfied at least one of the clauses needs to to be true (i.e. all the conditions it contains must be true).
Variables are represented with a capital, for instance Feat1, Feat2, Feat3. Prolog will try to find an instanciation of the variables which satisfies one of the clauses (it will actually try all the possibilities until one of them works, it's a solver).
In each clause above the label is the last parameter of the predicate. Basically each clause means "if all the conditions after :- are satisfied, then the label is labelX".

Assuming the 4 clauses above have been saved in a file dt.pl, one can use for instance the SWI Prolog interpreter like this:
?- consult('dt.pl').
true.

Now the way to use this toy decision tree would be:
?- dt(4,'green',0.3, Label).
Label = labelC .

?- dt(4,'blue',0.3, Label).
Label = labelB.

Note the capital L for Label: it means that now we let prolog find a value of Label which satisfies the predicate dt given the values provided for the features.

This is the basic idea. A more advanced and more generic representation of the DT would be possible using recursion and unification, but this would require an advanced understanding of Prolog.


Iris data set comes with a label, it is a supervised learning problem. The task is to predict the species of the flowers.

Your problem need not have a label, it really depends on your task and data. If your goal is to predict the gender and you want to see if you can see a pattern in lower dimensional space, you can give it a try to treat gender as a target.

If your problem is an unsupervised one, You can also just plot out your data and see whether you see any clustering in your dataset.

Remark: You might want to be careful about the $0$ values, $0$ doesn't mean the movie is good or bad. It really means no feedback.


Multiclass classification task where each class is present only once in the test set

When talking about the test data, we can have any number of examples of each class for inference. It doesn't matter if it has 20 classes with one example each.

You can use a CNN classifier for this type of a problem but if one book can be written by more than one author you should use sigmoid activation in the last layer rather than using softmax.

You can see this article which address an usecase similar to yours - Link


It is just a type of namespacing, because $a$ is already assigned the chosen action. There are two contexts of action being considered in the equation, so there needs to be a symbol for each context. Using $a'$ is an obvious choice as the letter $a$ is implicitly linked to representing an action already.

The sum over $a'$ is a sum over all possible actions in state $s$, irrespective of the chosen action $a$.

So both $a$ and $a'$ represent actions. $a$ is the current action, supplied on the LHS of the equation. $a'$ represents the iterator of a sum over all actions $[\forall a' \in \mathcal{A}(s)]$, only used in the calculation on the RHS. Sometimes you will see a completely different letter chosen, or some subscripting or other way to show these represent different actions.

It is also quite common to see $a$ representing current action, and $a'$ representing the next action (taken when in state $s'$). But that is not what is happening here.


Searching for a dataset that targets difficult words

On the basis that difficult words are difficult because they are not commonly used, I think something as simple as TF-IDF would work well.


Logic behind pre-trained weights and transfer learning

If we classify new objects using transfer learning:
1.We delete the top Dense layer of the pre trained neural network.
2.Now suppose you have to classify 5 classes, so your final dense layer will contain 5 nodes.
3.Also you will add some dense layers prior to your new 5 node Dense layer, so that you can train the model with new data.
4.All the layers prior to the few dense layers you added will be frozen.
5.Now you will train your new model with your dataset.

The Logic behind these steps are:
1.As CNN uses hierarchical based learning, meaning the initial layers will try to learn basics features in a dataset ie. lines, corners, shapes etc. So the frozen layers of your model have already learnt the basic features which are common to your dataset as well as the dataset it has been previously trained on.
2.The few Dense layers you added, will try to learn high level features specific to your dataset when you train your model with your dataset.


How to get out of local minimums on stochastic gradient descent?

Stohastic gradient descent loss landscape vs. gradient descent loss landscape

  I don't know how using the training data in batches rather than all at once allows it to steer around local minimum in the example, which is clearly steeper than the path to the global minimum behind it.

So, stochastic gradient descent is more able to avoid local minimum because the landscape of batch loss function is different than the loss function of whole dataset (the case when you calculate the losses on all data and then update parameters). That means the gradient on the whole dataset could be 0 at some point, but at that same point, the gradient of the batch could be different (so we hope to go in other direction than the local minimum).

Neural network architecture and loss landscape

In order to escape the local minimum, your neural architecture can also help. For example, see this work: Visualizing the Loss Landscape of Neural Nets. It shows that skip connections can smoothen your loss landscape and, hence, help the optimizers to find the global minimum more easily.

Local minimums vs global optimum

Finally, there are some works suggesting that the local minimums have almost the same function value as the global optimum. See this question and answer.


I figured out the answer to my question and here is the code that builds on the above answer.

from keras.layers import Input, Dense
from keras.models import Model
from keras.utils import plot_model

A1 = Input(shape=(30,),name='A1')
A2 = Dense(8, activation='relu',name='A2')(A1)
A3 = Dense(30, activation='relu',name='A3')(A2)

B2 = Dense(40, activation='relu',name='B2')(A2)
B3 = Dense(30, activation='relu',name='B3')(B2)

merged = Model(inputs=[A1],outputs=[A3,B3])
plot_model(merged,to_file='demo.png',show_shapes=True)

and here is the output structure that I wanted:


If I have got different routes - a series of (lat, lng) points, how to get the similarity of different routes?

I would go about this problem using the following approach:

Create a poly-line for the baseline trip.
Create a buffer zone (closed polygon) around this poly-line using a small radius, say 20 meters.
Using the points in the second trip, calculate the fraction of points that lie outside the buffer zone.

A fraction of zero means that all points from the second trip lie inside the first trip's buffer zone, while a fraction of one means that all lie outside.

You can use this fraction as a measure of dissimilarity, where 1 means that both trips are completely dissimilar and 0 means completely similar (within the given buffer radius). For completeness, you might want to reverse the poly-line roles and calculate the "reverse" similarity. Your final similarity score could then be the product of both.

I usually implement my algorithms in C# using .NET and for these purposes I use Microsoft's System.Spatial NuGet package. Here you can find methods like STBuffer and STContains that will greatly help to implement this.


California housing prices dataset on Kaggle

It looks like the dataset is aggregated from another raw dataset, e.g. the mean house price has been calculated from the raw data listing every house individually.
The creator of the dataset perhaps made a mistake, or intentionally left out the number of houses, perhaps not thinking this feature was relevant for the use-case in mind.
Either way, the dataset is made from a census taken in California in 1990, so perhaps you need to search for the raw census data instead and calculate it yourself.


you can use "frbs", "sets" and fugeR packege for fuzzy logic model building.


I will assume your text is in two strings like this:

In [1]: import pandas as pd

In [2]: text1 = "9 10 13 110 14 16"

In [3]: text2 = "12 1 6 1 1 2"

A one-liner solution would be:

In [4] df = pd.DataFrame.from_records(zip(text1.split(" "), text2.split(" ")))

A Pandas Dataframe can be created by passing it one list (or tuple) for each row that you want in the table. This is done by using the from_records() method you see above.

So the steps that make the above line work:

split() each string on the spaces, to get a list of strings - one per value.
create each row that we want in the dataframe, which is each matched pair from the two lists of values. zip does exactly that for us.
Put the result into the from_records() method.

The final result:

In [7]: df
Out[7]:

0   1
0    9  12
1   10   1
2   13   6
3  110   1
4   14   1
5   16   2

Because we just gave the dataframe lists of strings, the values are still strings in teh dataframe. If you want to actually use them as number, you can use the astype() method, like this

df_integers = df.astype(int)    # now contains integers
df_floats = df.astype(float)    # now contains floats, i.e. decimal values


Of course cosine similarity is not proper for searching a specific features in documents! To do this, you can exactly using dot product, as it will ignore zero features in query vector from documents.

Cosine similarity, in the current context, can be used to finding similarity between two documents. So, all features can be important to finding similarity. It means, if there is not a feature in query vector, but there is in  a document, or vice versa, these two are 100 percent similar. Hence, it makes sense.


Is (manual) feature extraction outdated?

In the general case, this is by no means true. Let's break down the case for different data scenarios:

For discriminative image models (e.g. image classification/labeling) this is true for some scenarios. You just throw some convnets (even pretrained models) at your data, and that's it. Nevertheless, convnets themselves profit from the "expert knowledge" that information locality is important and so is hierarchical information processing. For some other scenarios, applying domain knowledge (e.g. specific data transformations) may give the edge to reach the needed level of quality in the results.
For many image processing problems, neural networks work best when infused with some kind of inductive bias, e.g. attention.
For Natural Language Processing (NLP) problems, a good amount of craftsmanship is needed nowadays, especially in the data preprocessing stage.
For "typical data science" problems, it is also crucial to do feature extraction. You can have a look at Kaggle competitions to verify this.
For time series problems, it is also normal to rely on expert knowledge to understand which models fit best based on the nature of the data.

However, I think that the trend of the areas where deep learning is applicable (i.e. tons of available data) is to try to devise systems that are trained end-to-end, with the least possible ad hoc processing. Nevertheless, many times this is achieved by infusing the expert knowledge into the network in the form of inductive biases.


Does it make sense to use a tfidf matrix for a model which expects to see new text?

The problem is not really "new text", since by definition any classification model for text is meant to be applied to some new text. The problem is out of vocabulary words (OOV): the model will not be able to represent words that it didn't see in the training data.

The most simple way (and probably the most standard way) to deal with OOV in the test data is to completely remove them before representing the text as features.

Naturally OOV words can be a serious problem, especially in data such as Twitter where vocabulary evolves fast. But this issue is not related to using TF-IDF or not: any model trained at a certain point in time can only take into account the vocabulary in the training data, it cannot guess how future words are going to behave with respect to the class. The only solution for that is to use some form of re-training, for instance semi-supervised learning.


Do I need to standardize my one hot encoded labels?

NO, you do not standardize labels
The purpose of standardization is to bring features with disparate ranges into a standard range. When the data is not standardized, features with large numerical values will tend to have a larger influence (weight) than those that are smaller numerically.

Consider the Automobile Data Set (https://archive.ics.uci.edu/ml/datasets/automobile)  from UC Irvine. Among the features, it has height, lenght, width, weight, # cylinders, along with a number of other label and numerical features. Height ranges from from 47.8 to 59.8 and weight ranges from 1488 to 4066. You'll want to standardize them, but not label features such as body-style or engine-type.


Proper Understanding of Condensed Nearest Neighbor

Condensed Nearest Neighbors algorithm helps to reduce the dataset X for k-NN classification. It constructs a subset of examples which are able to correctly classify the original data set using a 1-NN algorithm.

It is returning not the array of misclassified points, but a subset Z of the data set X.

CNN works like that:

1) Scan all elements of X, looking for an element x whose nearest prototype from Z has a different label than x

2) Remove x from X and add it to Z

3) Repeat the scan until no more prototypes are added to Z

Z used instead of X for kNN classification.

An advantage of this method is decreasing of execution time, reducing a space complexity


How should I convert Logistic Regression's coefs into action strategy?

To understand the coefficients you just need to understand how the logistic regression model that you fit uses the coefficients to make predictions. No, it does not work like a decision tree. It's a linear model.

Really, predictions are based on the dot product of the coefficients and the values from some new instance to predict. This is just the sum of their products. The higher the dot product, the more positive the prediction.

So you can understand it as computing something like -2.40477246e-10 * MATCH_HOME + -5.57611571e-02 * MATCH_AWAY + ... (I don't know what coefficients go with what feature in your model.)

That generally means that inputs with bigger coefficients matter more, and inputs with positive coefficients correlate positively with a positive prediction. That's most of what you can interpret here.

The first of those conclusions is only really valid if inputs have been normalized to be on the same scale though. I'm not clear that you've done that here. You should also in general use L1 regularization if you intend to interpret the coefficients this way.


Face Recognition using Eigenfaces and SVM

Take a look at the code that you linked to:

# Download the data, if not already on disk and load it as numpy arrays

lfw_people = fetch_lfw_people(min_faces_per_person=70, resize=0.4)

fetch_lfw-poeple is a routine that loads the data and is detailed here.

Hope this helps!


Can we use Binary Cross Entropy for Multiclass Classification?

Use categorical_crossentropy when it comes for Multiclass classification, Because multiclass have more than one exclusive targets which is restricted by the binary_cross_entrophy.

binary_cross_entrophy is used when the target vector has only two levels of class.

In other cases when target vector has more than two levels categorical_crossentropy can be used for better model convergence.


XGBRegressor hyperparameter optimization using xgb cv function

It seems to be complaining about the folds.split part, indicating that folds is an integer.  If your cv_folds object is just the number of folds, pass that as nfolds instead; folds expects the actual sklearn KFold object rather than a number.


Do you mean a class label that the algorithm has never seen before? Then no, it is not possible. If you dont have labelled data for all your samples you can run a LDA, to get some topics and then assign labels based on the topics obtained. Even this approach wouldnt be really great. Since your problem is classification, I would recommend restricting your domain and using only samples for which the label is known. For better classification accuracy you can look into CNN.


CNN Multi-Class Model Only Predicts 1 class for all test images

73 millions trainable parms

- When using Transfer learning we first freeze the base model
- Train it till you reach good accuracy
- Then unfreeze it and train for just few epochs. Keep LR small
Other probable issues -
- If your labels are not One-Hot coded, please use sparse_categorical_crossentropy
- Add validation_split in fit method
- Suggest you add a keras.layers.GlobalAveragePooling2D after the base model and before flattening it

"setting include_top=False: this excludes the global average pooling layer and the dense output layer"

Can use this code as guidance
base_model = keras.applications.xception.Xception(weights="imagenet", include_top=False)
model = keras.layers.GlobalAveragePooling2D()(base_model.output)
output = keras.layers.Dense(n_classes, activation="softmax")(model)
model = keras.Model(inputs=base_model.input, outputs=output)

for layer in base_model.layers:
    layer.trainable = False

optimizer = keras.optimizers.SGD(lr=0.2, momentum=0.9, decay=0.01)
model.compile(loss="sparse_categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"])
history = model.fit(train_set, epochs=15, validation_data=valid_set)

for layer in base_model.layers:
    layer.trainable = True

optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, decay=0.001)
model.compile(loss="sparse_categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"])
history = model.fit(train_set, epochs=5, validation_data=valid_set)

Code Ref - Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition


Good introductory reference for Bayesian Non-parametric (Dirichlet Process / Chinese Restaurant Process)

I would recommend Tamara Broderick's 3-part tutorial series from MLSS. She explains both modeling + inference using these methods from the ground up and focuses on the intuition.

Links to part 1, part 2, and part 3.


How to define a custom resampling methodology

Random subsampling seems appropriate, bootstrapping is a bit more generic, but also correct.

Here are some references and synonyms: http://www.frank-dieterle.com/phd/2_4_3.html


How to find slope of curve at certain points

The numpy calculation is the correct one to use, but may be a bit tricky to understand how it is calculated
Your custom calculation is accidentally returning the inverse slope, the x and y values are reversed in the slope function (x1 -> y[i], etc). The slope should be delta_y/delta_x
def slope(x1, y1, x2, y2):
v=slope(y[i], x[i], y[i-1], x[i-1])

Also, you are calculating the slope at x = 1.5, 2.5, etc but numpy is calculating the slope at x = 1, 2, 3
In the gradient calculation, numpy is calculating the gradient at each x value, by using the x-1 and x+1 values and dividing by the difference in x which is 2. You are calculating the inverse of the x + .5 values.
x=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, ]

y=[512, 256, 128, 64 , 32 , 16 , 8  , 7  , 6  , 5  , 4  , 3  , 2  , 1  ]

numpy answer:
[-256. , -192. ,  -96. ,  -48. ,  -24. ,  -12. ,   -4.5,   -1. ,-1. ,   -1. ,   -1. ,   -1. ,   -1. ,   -1. ]

each calculation in numpy function:
-256 = (-256)/1 there is no x-1 value so the default is to use x
-192 = (128-512)/2
-96 =  (64-256)/2
...


in TensorFlow 2.0, what is the different between LSTM and LSTMCell objects?

You are right - the difference is minimal. The base LSTMCell class implements the main functionality required, such as the build method, whereas the LSTM class only container an entry point: the call method, as well as a bunch of getters to retrieve attribute values. LSTMCell is the base class, which is used as a cell that is used inside the LSTM class.

All links point to relevant parts of tensorflow.keras source code.

My advice would be to use the standard LSTM class in your model as a normal layer. If you have a GPU at your disposal, you might want to use the version of the layer that is optimised with CUDA for execution on a GPU. As per the documentation:

  Note that this cell is not optimized for performance on GPU. Please use
    tf.keras.layers.CuDNNLSTM for better performance on GPU.

There is also a GRU layer as well as a CuDNNGRU layer.

If you want to tweak how things work under the hood, you might create a class and inherit from the LSTMCell, or even the base class:

from tensorflow.python.keras.engine.base_layer import Layer

class MyLSTM(Layer):
    pass

But you would have to implement many things for yourself.


Python - Get FP/TP from Confusion Matrix using a List

Looks like you're using scikit-learn. So why not explore a bit more? Scikit has a metrics module, that can be of use for your problem. Essentially, what you need is to have two separate arrays - one with real labels and another with predicted labels. And then you're good to go, you could call metrics.classification_report or metrics.confusion_matrix or metrics.accuracy_score, all of them use the real labels and the predicted labels.

There's nothing wrong with using clf_rf.score(test_features, test_target), but it will only give you a single value. If you look at the source code, what happens is that the score method calls a predict method with test_feature for prediction of labels, which occurs behind the scenes.

It's better to actually capture those predicted labels, so that you can reuse them.

clf_rf.fit(training_features, training_target)
predicted_target = clf_rf.predict(test_features)
accuracy = sklearn.metrics.accuracy_score(test_target, predicted_target)
cnf_matrix = sklearn.metrics.confusion_matrix(test_target, predicted_target)
class_report = sklearn.metrics.classification_report(test_target, predicted_target)

And then you could do whichever you like with the calculated metrics, print, plot, etc.
Have a look at the examples that are included for each model/metrics you're using 1, 2, 3, 4, etc.


Choosing input variables for Linear Regression for higher accuracy

I am using Linear Regression in trying to predict the Value using Site Code, Site Longitude and Site Latitude

Unfortunately none of these three variables can go directly into linear regression.

Site Code looks like a numerical variable, but it is actually categorical. For example Site Code = 10 is probably orthogonal to Site Code = 5, and should not be interpreted as twice as significant. The correct way to handle this is to create a boolean dummy indicator variable for each possible site code. You can also use this method for some of your other object variables which also appear to be categorical.

Latitude and Longitude will not work well in a linear regression because their relationship is highly non-linear. For example two points can have the same latitude/longitude but be very far apart. One typical method is to convert (lat,lng) pairs into predefined zones, and treat the zone as a categorical variable.

The final type of variable you have is a date. You could convert this directly to a categorical variable, but it might be better to take only the month to reduce the number of categories and generalize seasonal effects better.


Data prediction using scikit-learn and a list

There are two popular approaches:

Model it with traditional supervised binary classification algorithms (e.g., logistic regression or Random Forest) and generate features (e.g., look back 1 steps, look back 2 steps, …)

Model it with a Hidden Markov Model (HMM). The hidden state is the machine on or off. The observed state is temperature. Given the observed state, what is the most likely hidden state that would generate it?


how to split available data into training and testing (Information security)

You should check out k-fold cross validation techique. Its a quality control technique which makes sure, the prediction/classification done by a model would generalize well to unseen test data. The idea is to split the dataset into k times and use it recursively to train and cross validate your network which at end can be used on the test data to report the actual accuracy.

Here is a link for reference :
https://machinelearningmastery.com/k-fold-cross-validation/


Can you apply PCA to part of your dataset?

I plotted the heatmap for the 14 continuous variables and found that most of them are weakly correlated with the response variable but highly correlated with each other

You absolutely can select specific columns [continous data] from your original data and apply PCA on them, PCA1, PCA2 eigenvectors will show you the amount of correlation between each feature. However, you should use all of the data points or rows when applying PCA as PCA calculates the maximal variance between data points and its best to use all of them for accurate results.

So in short, you should select column[feature] wise but not row[data points] wise.

  Is it ok to do so? Or should I one-hot-encoding / label encoding the categorical variables and do PCA to the entire dataset?

No need to do this, and it doesn't make sense in case of PCA as it only works on continuous data points.


Understanding policy gradient theorem - What does it mean to take gradients of reward wrt policy parameters?

We want to find the gradient of policy "return" $V$ wrt. parameters of the policy $\theta$. Where the return $V$ could be written as "how good is an action $Q$ $\times$ probability of taking that action $\pi$".

Consider the policy gradient, $\nabla_\theta V = \sum_a Q \nabla_\theta \pi + \pi \nabla_\theta Q$

The first term tells us to adjust the action probability proportionally to how good it is. To me it could read "if an action yields good, take more". That is to move the peak of $\pi$ to match the peak of $Q$. This is a reasonable thing to do. But of course since $Q$ cannot directly guide us toward its peak, it is up to our $\pi$ to luckily stumble upon the high peak of $Q$. This emphasizes the importance of exploratory nature of $\pi$.

The second term is vice versa. That is to move the peak of $Q$ to match the peak of $\pi$. This is much harder a task because $Q$ is a function of both action and policy, $Q_{\pi_\theta}(s, a)$. We clearly don't have this in a differentiable form i.e. we don't have a universal $Q$ function over the space of all possible $\pi$.

We now have a partial gradient from the first term but we have yet to estimate the second term.

Turns out, the second term can be recursively written solely in the form of the first term but with subsequent actions and states.

$$
\nabla_\theta V_0 = \sum Q_0 \nabla_\theta \pi_0 + \sum Q_1 \nabla_\theta \pi_1 + \sum Q_2 \nabla_\theta \pi_2 + \dots
$$

That is to get good policy i.e. policy gradient we only need to move the peaks of $\pi$ to match the peaks of $Q$ not only the first (state, action) but also for all subsequent (state, action)'s. This yields the same result as if we differentiate through the $Q$.


How i can generate the probabilistic graph for my dataset?

Søren Højsgaard has many quality resources for graphical models in R. He has a tutorial "Graphical Models with R" and a list of CRAN packages.

Additionally, mclust is one of the best clustering packages in R.


Why results of statsmodel logreg is different from scikit-learn logreg?

As far as I can tell, you use a "normal" Logit in one approach and a Logit with L1 penalty in the other case (penalty': ['l1']), which is called "Lasso". In Statsmodels L1 penalty would be implemented like stated in the docs.

Lasso and "normal" Logit are two different approaches. In the first case, (some) parameters are shrunken and can be set to zero, while with normal Logit, no parameter is shrunken. Lasso is often used to improve fit with noisy $x$, to do feature selection, or in case of high dimensional data (lot of $x$). Compare Chapters 4.3 and 6.2 in Introduction to Statistical Learning.

ISLR comes with Python labs. This can give you a good starting point how to apply one or the other method in a proper way.


Beyond one-hot encoding for LSTM model in Keras

You are looking for top-k categorical accuracy. This is actually implemented in keras, but you might want to change the k, which can be done with partial function. This is the related link.


When to use Standard Scaler and when Normalizer?

They are used for two different purposes.
StandardScaler changes each feature column $f_{:,i}$ to $$f'_{:,i} = \frac{f_{:,i} - mean(f_{:,i})}{std(f_{:,i})}.$$
Normalizer changes each sample $x_n=(f_{n,1},...,f_{n,d})$ to  $$x'_n = \frac{x_n}{size(x_n)},$$   where $size(x_n)$ for

l1 norm is $\left \| x_n \right \|_1=|f_{n,1}|+...+|f_{n,d}|$,
l2 norm is $\left \| x_n \right \|_2=\sqrt{f^{2}_{n,1}+...+f^{2}_{n,d}}$,
max norm is $\left \| x_n \right \|_\infty=max\{|f_{n,1}|,...,|f_{n,d}|\}$.

To illustrate the contrast, consider data set $\{1, 2, 3, 4, 5\}$ which consists of 5 one dimensional data points (each data point has one feature),
After applying StandardScaler, data set becomes $\{-1.41, -0.71, 0. ,0.71, 1.41\}$.
After applying any type of Normalizer, data set becomes $\{1., 1., 1., 1., 1.\}$, since the only feature is divided by itself. So Normalizer has no use for this case.
Also, when features have different units, e.g. $(height, age, income)$, Normalizer is not used as a pre-processing step; although, it might be used as an ad-hoc feature engineering step similar to what a neuron does in a neural network.
As mentioned in this answer, Normalizer is mostly useful for controlling the size of a vector in an iterative process, e.g. a parameter vector during training, to avoid numerical instabilities due to large values.


What is the correct file extension for Orange image embedding?

As r.kfr said: Import Images works with folders, not individual images. Place all images in a folder and select it for import.
Note that if you have images in different folders, Orange will consider each folder name as a class label for images.

Alternatively, you can create an Excel/csv file and define the path to the image. Mark it with meta type=image in the third header row. Here is how to define the header.


How to measure model success in production

Short-term targets and more out-of-time samples.

Work with the business and your data to develop relevant short-term targets. For example, if a sale takes 6 months after first-contact and the sale is the modeled target, set first-contact as short-term target for measurement. Set other steps in the funnel as a short-term target. Then run your analysis with these. Similar to predicting a charge-off of a loan. If a model is used to give a loan and the loan is 2 years, we do not wait 2 years from model deployment to get the first day of measurements. That could be a lot of bad loans. Perhaps half of the people who typically charge off do so within 12 months. And then 75% charge off within 20 months. You need to examine what is relevant and how predictive those short-term targets are of the modeled target. Use your training data and SMEs.

Also, if your process takes time, then you did not use the most up-to-date data in modeling. For example, if a loan has a 24 month modeled target, the model could not have used the latest 24 months, since that does not have a mature target (assuming this is not a survivor model - then this paragraph does not hold). But now the model can be evaluated on that data as each month progresses. The model did not score that data for action, but the model can be evaluated. Perhaps some loans would not have been given with the new model, but this is better than waiting 2 years to determine that the model is bad.


Yes, you will have to convert everything to numeric. That requires thinking about what these attributes represent accordingly you can use either the below 3 options.

There are three options:

One-Hot encoding for categorical data
Arbitrary numbers for ordinal data
Use something like group means for categorical data (e. g. mean
prices for city districts).

You have to be careful to not infuse information you do not have in the application case.

I'm expanding on option 1 and 3, if you want to know about option 2 you can go through the links attached at last.

One hot encoding

If you have categorical data, you can create dummy variables with 0/1 values for each possible value. Similarly you could implement for children, smoker.

E. g.

id  Sex
0   Male
1   Feamle

to

id   Male   Female
0      1       0
1      0       1

This can easily be done with pandas:

import pandas as pd

data = pd.DataFrame({'Sex': ['Male', 'Female']})
print(pd.get_dummies(data))

will result in:

        Sex_Male  Sex_Female
0           1            0
1           0            1

Using categorical data for groupby operations

This is an additional usecase but in your case it is not necessary to use this but if you feel so, you can try implementing this as well

You could use the mean for each category over past (known events).

Say you have a DataFrame with the last known mean prices for cities:

prices = pd.DataFrame({
    'city': ['A', 'A', 'A', 'B', 'B', 'C'],
    'price': [1, 1, 1, 2, 2, 3],
})
mean_price = prices.groupby('city').mean()
data = pd.DataFrame({'city': ['A', 'B', 'C', 'A', 'B', 'A']})

print(data.merge(mean_price, on='city', how='left'))

Result:

  city  price
0    A      1
1    B      2
2    C      3
3    A      1
4    B      2
5    A      1

For better understanding you can go through this Link-1, Link-2


Bagging vs Dropout in Deep Neural Networks

Bagging and dropout do not achieve quite the same thing, though both are types of model averaging.

Bagging is an operation across your entire dataset which trains models on a subset of the training data.  Thus some training examples are not shown to a given model.

Dropout, by contrast, is applied to features within each training example.  It is true that the result is functionally equivalent to training exponentially many networks (with shared weights!) and then equally weighting their outputs.  But dropout works on the feature space, causing certain features to be unavailable to the network, not full examples.  Because each neuron cannot completely rely on one input, representations in these networks tend to be more distributed and the network is less likely to overfit.


Using regex in redshift to find dollar values

I am getting that you want extract numbers with either '$' or even not without a dollar.

Why are not using just [0-9]+ for that field? The results will be like

    some $1 something >> ['1']
    some other 115 >> ['115']
    $115 some thing >> ['115']

If you want to include '\$' sign, then '\$' can be repeated zero or once or by typo more, then regex would be

    \$*[0-9]+

which outputs

    some $1 something >> ['$1']
    some other 115 >> ['115']
    $115 some thing >> ['$115']

If this is not what you want, comment.

Hope this helps.

I would recommend using online regex matchers like here. They have short and sweet references which really help us get to the context.


How to predict value in every 120 minutes using LSTM in python

if you want to predict exactly at a particular timestamp, & you do not want to train the model again and again then may be you can run a crontab on a python file to load the model and predict the value, with cron job starting at 6 AM & execute again at every 120 minutes

Even if you have a stream of data coming continuously you can re-train the model in between those 120 minutes.


Training a LSTM/any other deep learning model with temporal as well as non temporal attributes

Like you said, you can proceed your temporal data with a recurrent network like LSTM. I suggest you simply concatenate your other features to the output of the LSTM (with return_sequence = False). Then add some dense layers (1 could be enough) : this will output the probability of belonging to your classes.


Classification problem: custom minimization measure

I think what you are dealing with is a type I vs. type II error discussion. It seems that you want to avoid Type II error.

One way to go is to consider $F_\beta$ for your performance metric. It is a modified version of $F_1$. As seen here : https://en.wikipedia.org/wiki/F1_score, $F_\beta$ can be formulated in terms of type I /type II error. You then need to select a $\beta$ based on how much avoiding type II error is important to you. This can be done by considerations on the cost of type II errors compared to the cost of type I error.


I have the same problem as you, and I just used pip to reinstall the 3.17 version.

pip install Orange3==3.17


How to split a large set of coordinates into smaller sets using something like nearby selection?

Have a look into KD-Trees. They work by partitioning your space into discrete blocks, via a (wait for it...) tree. Here is the wikipedia example for a 3d space:

Each of the cuboids is represented by a leaf or node in the tree. It works via binary splits (splitting the space into two via a criterion that you could choose yourself). Here is a short intro to the algorithm.

There is an implementation in Scikit-Learn. You build a tree and then you can query the data for a given point and radius, returning all points within that radius/distance.

This has the benefit that you could also test your algorithms over a group of subsets by querying a selection of coordinates - and you only need to construct the tree once.

Small demo

I will generate some random points random points on a 2d integer grid, so values between 0 and 10. Some points on the grid could therefore be empty.

I put them into a Pandas Dataframe just to make things a little easier to manage and plot:

In [1]: import numpy as np

In [2]: import pandas as pd

In [3]: from sklearn.neighbors import KDTree         # could use cKDTree

In [4]: import matplotlib.pyplot as plt

In [5]: data = {"x": np.random.randint(0, 11, 200), "y": np.random.randint(0, 10, 200)}    # 200 random coords

In [6]: df = pd.DataFrame(data)

In [7]: df.head()
Out[7]:
   x  y
0  1  7
1  9  8
2  6  1
3  3  2
4  4  3

Now I create the KDTree and query the coordinate (5, 5) in the centre - I want all points back that lie within a radius distance of 3:

In [8]: tree = KDTree(df.values)

In [9]: ix = tree.query_radius([(5, 5)], r=3)[0]

Now I use the returned indices (ix) to filter out those points I want - and just plot them here in red, along with the original complete dataset in blue:

In [10]: ax = df.plot.scatter("x", "y", c="b", alpha=0.5);   # blue base points

In [11]: df.iloc[ix].plot.scatter("x", "y", c="r", ax=ax);   # query results

In [12]: plt.scatter(5, 5, c="g");    # query point

In [11]: plt.show()

The queried point is green (5, 5)
Original points are blue
Query results are red

Faster please!

There is also the cKDTree class, which works very similarly, but is implemented in C and so should be faster in many cases. Have a look here at some differences. the cKDTree is the best choice if you want to do simple querying like in my example!


TensorFlow Sigmoid activation function as output layer - value interpretation

In short, value of model.predict() function is interpreted as mentioned in option 2.

In order to clarify, let's assume we are talking about spam detection application. Label 0 represents that text/email is not spam and label 1 represents that text/email is spam.

Suppose, after running the function model.predict(), we get value 0.9899. Then we can interpret it as:

chance of given text/email being spam is 98.99%
chance of given text/email not being spam is 1.01%

Usually, when we talk about a binary classification problem, we will often use labels 0 & 1 to represent the classes. So the predicted value being closer to 0 means it is more likely to fall under the class 0 (in our example class 0 represents text/email not being spam.). Likewise, if the predicted value being closer to 1 means it is more likely to fall under the class 1 (in our example class 1 represents text/email being spam.)


You could change in this case with command

game_results$winner[1] <- "Dad"

But this copies the whole table, so i will recomend you to create an empty data in another way

game_results <- setNames(data.table(matrix(nrow = 10, ncol = 3)), c("winner","round","winnerscore"))

and assignment by reference in the way you prefer (hole column or specific cell), here instructions for both cases.


How can I run SVM on 500k rows with 81 columns?

If you want to stick to SVC(sklearn) , you can use Ensembling
we have many ways of ensembling
good guide is here : http://mlwave.com/kaggle-ensembling-guide/
for your problem you can train multiple SVC models on different subset of your data,and as quick solution you can combine them whit BaggingClassifier like this :

clf = BaggingClassifier(SVC(C=1.0,
        cache_size=200,
        class_weight=None,
        coef0=0.0,
        decision_function_shape=None,
        degree=3,
        gamma='auto',
        kernel='rbf',
        max_iter=-1,
        probability=False,
        random_state=None,
        shrinking=True,
        tol=0.001,
        verbose=False,
        ))

If you want to use each record is used only once for training in the BaggingClassifier, set the bootstrap parameter to False.
now you can parallel each of this models in on core of your cluster.


Is there any augmentation tool for images and bounding boxes?

I think you should look into imgaug. It supports most image augmentation and does have support for bounding boxes.

Docs: https://imgaug.readthedocs.io/en/latest/


How to distinguish between normal fluctuation and outliers in ARIMA model?

What you have is the fundamental question of statistics, what is noise and what is signal. It’s true that from a modeling perspective it’s easier to model something that’s been nicely smoothed and trimmed, but is that what you want? That’s a hard question to answer in general. I will say that with time data it is quite normal to have seasonal components which I do see in the original but do seem absent in the cleaned data so I err on the side of too clean. I would look for some a more sensitive method than tsclean.


How does a Recommender System recommend movies to a New User?

In general, a safe default option for recommender systems is to recommend the most popular product, so two options for the recommendation:

simply recommend the most popular movie in a time window around the current moment (maybe taking into account momentum),
do the same but marginalizing over account info, e.g. age, gender, country, language; also, IP location is an option to get such type of info.

Confidence assessment can be done based on recommendation success rate over the results of the same recommendation to previous users with the same profile characteristics, if such data is available.


How to penalize for empty fields in a DataFrame?

This heavily depends on the domain knowledge. A general approach would be to place

A multiplicative of the worst or average consistency at each circuit $c$, i.e. $(1 + m)\text{max}(\sigma_c)$ or $(1 + m)\text{avg}(\sigma_c)$ respectively, for the null values at that circuit, or
A multiplicative of the worst or average consistency of each driver $d$, i.e. $(1 + m)\text{max}(\sigma_d)$ or  $(1 + m)\text{avg}(\sigma_d)$ respectively, for their unfinished races, or
A multiplicative of average of driver and circuit average consistencies, i.e. $(1 + m)[\text{avg}(\sigma_d) + \text{avg}(\sigma_c)]/2$, for unfinished race of driver $d$ at circuit $c$, or some other combinations.

No matter which approach to choose, the choice of coefficient $m$ affects the final ranking and could be determined either

Subjectively by looking at the rankings from an expert point of view and selecting the one that makes more sense, or
By trying a range of values like $m \in \{-0.2, -0.1, 0, 0.1, 0.2, .., 0.5\}$ and averaging the consistencies $\sigma_d$ or rankings $R_d$ for each driver $d$. An advantage of this approach would be that when rank of a driver has a low variance over different values of $m$, it implies that driver's rank is insensitive to the choice of $m$, i.e. it is less controversial, and when rank changes a lot with different choices of $m$, the average rank is more controversial.


Why transform embedding dimension in sin-cos positional encoding?

First, let's reason why positional embeddings are needed at all:
A multi-head attention layer of the Transformer architecture performs computations that are position-independent. This means that, if the same inputs are received at two different positions, the attention heads in the layer would return the same value at the two positions.
Note that this is different from LSTMs and other recurrent architectures which, apart from the input, receive the state from the previous time step.
The role of positional embeddings is to supply information regarding the position of each token. This allows the attention layer to compute results that are context-dependent, that is, two tokens with the same value in the input sentence would get different representations.

Second, let's clarify why having a fixed formula to compute the positional embeddings:
Positional embeddings can be handled as "normal" embedding matrixes and
therefore can be trained with the rest of the network. These are "trainable positional embeddings". With this kind of positional embeddings, after each training step, the positional embedding matrix is updated together with the rest of the parameters.
However, we can obtain the same level of performance (translation quality, perplexity, or whatever other measure being used) if, instead of training the positional embeddings, we used the formula proposed in the original transformer paper.
This saves us from having to train a very big embedding matrix.

Now, about why using the "embedding size dimension" in the formula:
We need different values in each position of the embedded vector. Having the same value in each position of the vector would leave us with an "effective" embedding size of 1, as we are wasting the other $d-1$ positions.
In order to compute different values for each position of the embedded vector, we need an independent variable that we use to compute the value at each position based on it. We don't have any other suitable variable but the position itself. That's why it is used in the formula.

Old answer:
Making the embedding vector independent from the "embedding size dimension" would lead to having the same value in all positions, and this would reduce the effective embedding dimensionality to 1.
The formula uses the embedding size dimension to be able to provide different values within each embedded vector.


How close is close enough, with regression?

I think it is a far question, since we should always try to translate any metric result to a useful interpretation.
When talking about regression, we can judge the success of our model by checking whether that MAE or RMSE exceed what we are willing to accept.
For instance, if we want to predict the temperature for the next days, and our result (in this case in a time series problem type) is a MAE of about 2ºC (after re-scaling our data to the natural units like celsius degrees), is it enough for our goal? Is a mean (absolute) error of 2ºC above or below the real temperature enough for the case being studied? Maybe it is for an approximate model, but maybe it is not a good result if we want to use it in an environment where tempreature could have more impact on whatever being studied...
Another additional chance you have is estimating confidence intervals (see this answer) for your model coefficents and, more precisely, prediction intervals (see this other validated answer) to have an estimate for your prediction together with an interval.


NAB benchmark (Numenta Anomaly Benchmark)

The anomallies sometimes are not marked as anomallies, the task consists in checking by yourself which are anomallies, but you have nothing to compare to.


Regression: how to interpret different linear relations?

What you are looking for is the Analysis of Covariance (ANCOVA) analysis, which is used to compare two or more regression lines by testing the effect of a categorical factor on a dependent variable (y-var) while controlling for the effect of a continuous co-variable (x-var).

Here is an example for carrying out the ANCOVA analysis using R.


Separating styles of numbers for simple digit classification

Interesting question. You are right in assuming that some 1s may be confused for 7s, same with 8s and 3s for instance.
Generally creating different classes as you suggest doesn't happen, simply because it would require more annotation.
There are multiple ways to handle this.

stacked models

Anything labelled as a 1 or a 7 would be given to a model fine-tuned to differentiate these two specific digits

weighted training

It is possible to teach your model that some mistakes are worse than others. In your case, 1s for 7s is bad, so you could increase the cost of making that mistake.


How to utilize user feedback due to miss-classification when correct class label is unknown?

This can be accomplished by a modification to multi-class cross-entropy.

We are faced with two types of supervision. First type is "data $i$ belongs to class $k$" denoted by $y_{ik}=1$, and second type is "data $i$ does not belong to class $k$" denoted by $\bar{y}_{ik}=1$. For example, for 3 classes, $y_i=(1, 0, 0)$ denotes that point $i$ belongs to class $1$, and $\bar{y}_{i}=(0, 0, 1)$ denotes that point $i$ does not belong to class $3$. Let $y'_{ik} \in [0, 1]$ denote the model prediction. The original cross-entropy for $K$ classes is:

$$H_y(y')=-\sum_{i}\sum_{k=1}^{K}y_{ik}log(y'_{ik})$$.

This objective assigns loss $-log(y'_{ik})$ to $y_{ik} = 1$ to encourage the model to output $y'_{ik} \rightarrow 1$ leading to $-log(y'_{ik})\rightarrow 0$.

On the other hand, for the second supervision $\bar{y}_{ik}=1$, we want to encourage the model to output $y'_{ik} \rightarrow 0$. Therefore, loss $-log(1- y'_{ik})$ can be used to have $-log(1- y'_{ik})\rightarrow 0$.

Accordingly, second supervision can be combined with first one as follows:

$$H_{(y,\bar{y})}(y')=-\sum_{i}\sum_{k=1}^{K}y_{ik}log(y'_{ik})+\bar{y}_{ik}log(1-y'_{ik})$$

Note that supervision "data $i$ does not belong to classes $1$ and $2$" is also supported. For example, $\bar{y}_{i}=(1, 1, 0,...)$ activates both $-log(1 - y'_{i1})$ and $-log(1 - y'_{i2})$ to encourage the model to output less probabilities for classes $1$ and $2$, i.e. $y'_{i1} \rightarrow 0$, and $y'_{i2} \rightarrow 0$.


In what stage should you plot validation and learning curve

I understand that with "learning curve" you are referring to the plot of the loss function over the training data (or subsets of it) when optimizing with iterative methods, like gradient descent. This is sometimes referred to as the training loss plot.
You use a training loss plot during training, to evaluate the convergence of your training algorithm and model. Usually, together with the training loss plot, you visualize the validation loss, that is, the loss function computed on a held-out dataset, to evaluate whether your model is overfitting your training data.
Normally, the training loss gets a new point after each optimization step. For instance, in stochastic gradient descent, the training loss points are the loss evaluated at each minibatch. On the other hand, the validation loss curve gets a new point after each epoch, or at least after a few optimization steps have been performed.
This is different from a ROC curve, which is evaluated once the model is fully trained, to understand the behavior of the trained model.


What is the difference between global and universal compression methods?

Consider the following chunk of data:

1010010110100101

Universal - these are generic compression algorithms that are data agnostic. A crude version of run length encoding would fall into this category. The advantage is that it is very fast to compress and decompress. The downside is that it may be extremely ineffective based on the data to be compressed.

1111111111111111 -> 16 1 (lucky case)

1010010110100101 -> 1010010110100101 (unlucky case)

Local - this method would consider smaller segments of a fixed length, say 4, look for patterns and compress them. Eg. This data only contains these two types of patterns - 1010 and 0101. These patterns can be represented as 0s and 1s and the overall data will be a table representing the mappings, and something like 0101. This has the potential to result in a much smaller compressed size.

1010010110100101 -> 1010 0101 1010 0101 -> 0101 (0=1010,1=0101)

Global - this method would look at the entire data and find the optimal / much better patterns to compress the data. The example data contains just one pattern 10100101 and represent it as 00 along with the mapping table. This has the potential to obtain the smallest possible compressed size, but is also computationally the heaviest.

1010010110100101 -> 10100101 10100101 -> 00 (0=10100101)


Convolution neural network: small dataset affecting accuracy

If I understand the question correctly, you have 36 class and 1 sample/class.
Then I believe it's impossible to use Machine Learning for this dataset to predict the label.

Reason: When you split train/test set, there will some classes that only appear in test set and not train set. The model doesn't learn about them at all. A major assumption in Machine Learning problems is that the train set and test set must have the same statistical distribution.


How to filter out a portion of the text data with re.match or any other regex method with Python?

nice that you solved it. This would also work:

import re

def eventdetails(text):
   match =re.search(r'Event Details :(.*)',text, flags=re.S)
   return match

result = eventdetails(text)
print(result.group(0))


Gradient flow through concatenation operation

For concatenation, the gradient values during back propagation split to their respective source layers. There is no direct interaction between gradients in either of the source layers.

The layer immediately after the concatenated layer does interact with both networks, and it will have some weight parameters that multiply outputs from network A and some that multiply outputs from network B. There will not be any parameters that multiply outputs from both layers (unless you are forcing them to be the same through weight sharing, but that won't be the case if for example you are stacking features from both starting networks).

The only issue you might have is clearly identifying which parameters link to each original network. That is an implementation detail, so you would need to share your code so far in order to debug that if it goes wrong.


How to interpret my logistic regression result with statsmodels

Something's wrong with your feature selection tool: p-value is NaN, confidence interval includes $0$. Confusion matrix shows that all observations are predicted as Class 1. How many explanatory variables do you have? Try using all of them instead of just one. Are you sure
logit_model = sm.Logit(y_train, X_train).fit()

is correct? Shouldn't it be the other way around,  logit_model = sm.Logit(X_train, y_train).fit()?


Keras Loss Function for Multidimensional Regression Problem

To answer:

1) When calculating the loss function for multivariate outputs, keras calculates it as a mean across all your outputs: https://github.com/keras-team/keras/blob/2.0.4/keras/losses.py#L12

Hence, if one output is doing really badly and others not, it could influence your loss result.

2) In the source code there are no mentioning about scaling the outputs for the calculation of loss function and, thus, I would conclude that the loss function will depend highly on the boundaries of each of your Y features.

To avoid this, what you could do is to write a custom loss function that scales your Y outputs before outputting loss value:

scaler = MinMaxScaler()
data_trans = scaler.fit_transform(y_true)

def custom_mean_absolute_error(y_true, y_pred):
    y_true = (y_true - K.constant(scaler.min_)) / K.constant(scaler.scale_)
    y_pred = (y_pred - K.constant(scaler.min_)) / K.constant(scaler.scale_)
    return K.mean(K.abs(y_pred - y_true), axis=-1)


How is the Beeswarm plot better than a histogram?

They are quite similar but :

Histograms results of aggregation into bins (loss of information due to aggregation: high-resolution point (2.324) is reduced to bins resolution [2,3])
Beeswarm plot displays the exact value of points on the axis. It is a 1D-scatter plot.

However, if two points are very close to each other they will overlap if the point size is big enough. To avoid this, the beeswarm plot will slightly shift one point to the side so it does not overlap.

They will often look similar because the more points you have locally, the higher the probability of overlapping. Then, high-density areas will often result in "peaks" as in histograms.

The beeswarm plot is a more faithful representation

Gap in the data: depending on the binning (both placement and size), the histogram may not represent it (as on point 1 in the figure).
Min and max: Imagine values are limited at 5 but the histogram last bins continues to 6, it might give the impression that the max value of the variable is 6.

In both cases, you can see that beeswarm is a more faithful representation.

Because of this, it is not highly scalable (it has to represent every single point with no overlap). On the seaborn.swarplot function docs :

  This function is similar to stripplot(), but the points are adjusted (only along the categorical axis) so that they don’t overlap. This gives a better representation of the distribution of values, but it does not scale well to large numbers of observations. This style of plot is sometimes called a “beeswarm”.


How to get significance level for ranked features?

1. univariate feature importance (that is c) in your list)

You are correct. Univariate statistics to estimate feature importance does not capture feature interactions. But they are fast and simple.

2. model-based feature importance (that are a) and b) in your list)

On the other hand model-based feature importance estimates can capture interactions as long as the model is capable of doing so (see "Introduction to Machine Learning with Python"; Mueller, Guido; 2017; p. 238/239). Which is not the case for linear regression.

For model-based feature importance estimates using trees there are ways to derive p-values. And at least R does have some implementations for that. Have a look at section "2.5 Importance testing procedures" in this paper The revival of the Gini importance?.


Text classification and predictive model

As a person trying to do text classification myself,Let me try to help you to get started here. Please don't bother reading ahead if your question is about finding models which are able to fit the "italian" language features and find co-relation and you already know how to do text classification in general. If not then-

It is recommended to formulate your problem first properly i.e. what exactly are you trying to solve here. "..........classifying them between True and False." It doesn't give a lot of clarity. This should be clear for you.
What kind of data you are dealing with, saying "30000 of texts" does that mean 30000 lines of text? documents? books?
Now comes the part where we can do EDA which might include pre processing of data also. try to plot your data with respect to your labels and get some insights on how to go about the problem, if there are trends , patterns you can utilize.
There are many techniques to get the features out of text data. there is feature selection, feature extraction and there are various methods to do that.
Then you can decide on which classification models will fit the data, i.e.model selection, null hypothesis if I might suggest. You can try various models to fit the data and compare its performances and choose the one which gives you the best results.there is cross validation for example do help you decide on this.There is model evaluation which shall help you decide if the model is able to generalize well on unseen data.

Here is the point about percentage of data used for training and testing. Read about cross validation and hold out method to help you understand it. Generally it s a good idea to 70/30 ratio to train and test the model and also training data can be split into training and validation set.

However, here comes the famous bias-variance problem w.r.t. training size, generally more data leads to better generalize which avoids over-fitting. With less data there are more chances of over-fitting. Hence you should try run the models and plot the graph to understand if its over-fitting or under fitting(if you think you have way low data for any model to fit and predict).
https://scikit-learn.org/stable/modules/learning_curve.html

With the best selected model you can start predicting the results.
After you have got the best fit model you can tune its hyper parameters to get the better performance.

Now these are just rough list for you to go on. Please try below link to read about text classification with sk-learn which might be a good starting point if you haven't done this before-

https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html
https://www.nltk.org/book/ch06.html


It looks like you are training this as a multiclass classifier, to represent a binary choice. In which case, your Y value is wrong:

Y = [[0, 1],
    [1, 1],
    [1, 0],
    [0, 1]]

Here your second label is not self-consistent, and thus it is impossible to predict using a softmax output layer (where the sum of all outputs must equal 1). The best it can do is [0.5, 0.5] to match that label and you can see actually it got close to that in your test.

You want this instead:

Y = [[0, 1],
    [1, 0],
    [1, 0],
    [0, 1]]

A few asides . . .

Your example inputs all have the same third column (=1). This is redundant data, and you could drop it.
Your network is more complex than it needs to be for this task. A single hidden layer with only a few neurons in it should be sufficient.
For this specific task you could have chosen a single output neuron using a sigmoid activation (and need only one column in Y).


Analytics term for turning row values into column names and count its assigned values


Steps to fit a Machine learning model for prediction of up and down market movement

Why do you want to use SVM+feature engineering in 2021? If you have a time series dataset, use LSTM/GRU/biLSTM for binary classification (last layer is fully connected). Loss function is binary cross-entropy, accuracy metrics overall accuracy or F1 or precision-recall.


How L2 Regularization penalizes weights in TensorFlow?

L2 loss is based on the square of the weights of your Network. As a given weight increases in size, the loss will increase exponentially (quadratically, to be precise). Neural Networks are therefore "pushed" to spread the weight values more evenly across each layer, since for such a quadratic loss factor it's better to have a lot of smaller weights, rather than few very big ones. The exponential relationship between weight value and loss is the reason why large weights get penalized. L2 loss is typically useful when your model suffers from overfitting, since weights redistribution across layers is good to prevent few neurons from staling all the "explanatory power" and generating overfitting.

(The main drawback of L2 loss is that the model becomes more sensitive to outliers, therefore less robust.)

Hope this helps, otherwise let me know.


Training Deep learning and validation loss

Using validation loss to determine when to stop iterating is a good strategy when you have a lot of labeled training data. However, in medical contexts, it's often the case that labeled data is expensive and/or difficult to come by. You mention that you only have 1000 samples -- this is a pretty small amount of data to be training a deep net with! Setting aside 100 or 200 images to form a validation set will probably hurt the model. You can use a validation set to get an estimate of how many epochs you should train for, and then train on the whole dataset for that many epochs.

In regards to your batch size query: I would recommend working with smaller batch sizes of around 32-128 images. In my experience, the networks converge more quickly when using smaller batch sizes.


Data Science Noob - Customer Scoring based on conversion probability

Your question could be closed for too broad, but let's give a try. You want the enrolment probability, this sounds like a logistic regression for me. Neither categorical nor continuous data type should present a problem. You may model the additional information, applications who don't have one simply be assigned a NA category. You can include the NA category in your model.

I recommend you read the book Applied Predictive Modelling, I think it has a section on credit-card applications, which is close to what you are doing. You should try to learn from it.


Normalisation results in R^2 score of 0 - Lasso regression

Standardizing/normalizing is generally the right thing to do, but it will make little/no difference with just one independent variable if you also adjust the regularization strength.

With more than one independent variable, standardizing assures that the lasso's penalty applies more equally to all the variables.  Without standardizing, a large-scale variable will be penalized less than small-scale ones (because the coefficients will already tend to be small).

In your case, the original model needs a fairly small coefficient on X because its scale is so large, and so you don't receive much penalty from the lasso.  After you standardize, now the model wants a larger coefficient on X, but that means the lasso penalty makes more of a contribution.  That you get an $R^2$ of zero suggests that the lasso penalty is large enough now to push the coefficient to zero, so that the model is just a horizontal line.  If you reduce the regularization strength a bit, you should be able to recover the old model (exactly maybe, with the right adjustment? If I get some spare time I'll look into this).


We use column vectors because it offers efficient memory access in column-major format.
For example: CUDA, which is commonly used to accelerate libraries such as tensorflow,  operates most efficiently when you coalesce memory. Strided memory access in parallelized applications cause significant slow-downs; threads (which are themselves sequentially indexed) should operate on sequentially indexed memory, rather than striding by a number of elements equal to the matrix row size. This is true even for cpu applications, mostly because of CPU caching. You want to make as few transactions as possible, which of course only column-major column-vectors or row-major row-vectors can offer.
Neither column-major nor row-major is universally "superior" and often they can be arbitrarily decided upon. However, there are a few reasons row-major is used in some applications but isn't used in others. In particular, relational databases are inherently row-major when you consider that each entry must be a single row. The C programming language is also inherently row-major because a 2D matrix is an array of arrays that stack row upon row when you consider the first dimension is the row index, as you do in math. The reasons for column-major format are to reflect mathematical interpretations, which also ties into some legacy integration; Fortran, for example, uses column-major where C is row-major. Fortran is still used in scientific computing applications, and although most have moved away from it, some of the most popular libraries originally written in Fortran have maintained a consistent API regardless of the language. BLAS, for example, was initially written in Fortran, but its many implementions in other language (including C/C++ for cuBLAS) have typically maintained its API. Many other popular mathematical programming tools, such as MATLAB, use column major format. It makes sense coming from a math-perspective where you consider the first dimension as the row index.


Which one I should choose for random forest?

If you are doing a classification task, use random forest classifier. If you want probabilities use predict_proba or you can use predict directly to get classes. You might be getting correct results but understand that the random forest regressor works on a different cost function, and is not constrained to give outputs between 0 and 1, so what you are getting out of the regressor are not really probabilities. If you run it on enough datasets it might start giving you outputs greater than 1. Hence, bottom line stick to classifier. Also I will recommend reading up a bit on the differences of regression and classification tasks.


When using KNN, how do I know which order of the Minkowski distance to use?

Unfortunately, there is no rule (either general or rule of thumb) for using a given distance rather than another. Minkowski distance of n-th order could work greatly for some task and some datasets, and extremely bad for others.

The reason Euclidean distance is used as default is due to its simplicity and the fact that there is no reason to consider more complex formulae as better a priori.

I suggest you to try multiple measures of distance (or multiple Minkowski orders) and check which works best for your current task. Please consider the risk of overfitting. I would run models more than once, for example using techniques for keeping overfitting under control, such as k-fold Cross Validation, and/or triple train-validation-test splits.


Validation data for multi-series stateful LSTM

Using a stateful model, am I correct in assuming that you have time-series data?

If so, it would perhaps make sense (at least for your test accuracy) to always validate against the time-step that immediately follows a test batch. This is what you would actually want to be doing with the model once it is trained.

Have a look at this answer, where I explain the idea in a lot more detail. The main idea is to split data into a kind of rolling forecast pattern, whereby you would have a test batch e.g. with something like:

window_size = 100     #whatever makes sense for your data
val_size = 15

train_batch1 = data[0:window_size]
val_batch1 = data[window_size:window_size + val_size]

train_batch2 = data[1:window_size+1]
val_batch2 = data[window_size + 1:window_size + val_size + 1]

...

That is just pseudo code to make it as clear as possible. Here is the diagram I created for the post linked above, which visualises the same notion:

In the image, it just has train/test sets... you could of course add validation into the mix.

If this doesn't seem to make sense for you data, temporal relationships must not matter too much. In that case, perhaps a stateful model is also not entirely useful.


K-means clustering on the data frame having only one column

Here is how to fit k-means to single dimensional text data in Pandas:

import pandas as pd
from sklearn.cluster import KMeans
from sklearn.feature_extraction.text import TfidfVectorizer

df = pd.DataFrame({"corpus": ["I am Sam. Sam-I-am",
                              "That Sam-I-am! That Sam-I-am! I do not like that Sam-I-am",
                              "Do you like green eggs and ham?",
                              "I do not like them, Sam-I-am. I do not like green eggs and ham"]})

x = TfidfVectorizer().fit_transform(df.corpus)
km = KMeans(n_clusters=2).fit(x)
km.labels_.tolist() # Results in a list similar to this: [0, 0, 1, 1]


Should we denormalize our data after normalization?

This can be done in Python using scaler.inverse_transform.

Consider a dataset that has been normalized with MinMaxScaler as follows:

# normalize dataset with MinMaxScaler
scaler = MinMaxScaler(feature_range=(0, 1))
dataset = scaler.fit_transform(dataset)

# Training and Test data partition
train_size = int(len(dataset) * 0.8)
test_size = len(dataset) - train_size
train, test = dataset[0:train_size,:], dataset[train_size:len(dataset),:]

# reshape into X=t-50 and Y=t
previous = 50
X_train, Y_train = create_dataset(train, previous)
X_test, Y_test = create_dataset(test, previous)

# reshape input to be [samples, time steps, features]
X_train = np.reshape(X_train, (X_train.shape[0], 1, X_train.shape[1]))
X_test = np.reshape(X_test, (X_test.shape[0], 1, X_test.shape[1]))

Upon generating the predictions from LSTM, the same can be converted back using scaler.inverse_transform as follows:

model = Sequential()
model.add(LSTM(4, input_shape=(1, previous)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(X_train, Y_train, epochs=100, batch_size=1, verbose=2)
trainpred = model.predict(X_train)
testpred = model.predict(X_test)
trainpred = scaler.inverse_transform(trainpred)
Y_train = scaler.inverse_transform([Y_train])
testpred = scaler.inverse_transform(testpred)
Y_test = scaler.inverse_transform([Y_test])
predictions = testpred


When you are training one batch, the GPU memory is full, i.e. it is generally needed for calculation. Time of one batch processing is like 0.1 second or less. What is your CPU-GPU bandwidth, 2 GB/s? So without loosing speed you could send at maximum 0.1 GB back and forward... Or in terms of "I want anyway", you can extend your memory by 1 GB for the price of 10x slowdown.

There are techniques of throwing out some intermediate layers' activations and recalculating them during back-propagation. It's much better idea to sacrify say half of (enormous) GPU calculation power to save some memory. I saw reports that people actually use them. But don't know how and didn't see any supports in ML frameworks


Custom loss function with multiple outputs in tensorflow

def model(input_shape=4, output_shape=3, lr=0.0001):
    """
    single input and multi-output
    loss = custom_loss(out_1_true, out_1_pred)+mse(out_2_true, out_2_pred))
    """
    input_layer = Input(input_shape)
    layer_1 = Dense(units=32, activation='elu', kernel_initializer='he_uniform')(input_layer)

    # STEP-1:   name your outputs
    y_1 = Dense(units=output_shape, activation='softmax', kernel_initializer='he_uniform', name='actions')(layer_1)
    y_2 = Dense(units=1, activation='linear', kernel_initializer='he_uniform', name='values')(layer_1)

    # STEP-2:    Define the custom custom_loss. Note that the args should be y_true, y_pred
    def custom_loss(y_true, y_pred):
        entropy_loss = -y_pred * K.log(y_true + 1e-10)
        return entropy_loss

    # STEP-3:   Define the dicts for loss and lossweights with keys as the name of the output layers
    LossFunc    =     {'actions':custom_loss, 'values':'mse'}
    lossWeights =     {'actions':0.5, 'values':0.5}

    Network_model = Model(inputs = input_layer, outputs = [y_1, y_2])

    # STEP-4:   complie using LossFunc and lossWeights dicts
    Network_model.compile(optimizer=RMSprop(lr=lr), loss=LossFunc, loss_weights=lossWeights, metrics=["accuracy"])

    return Network_model

#training example:

#model
Network_model = model(4,2)
# input
S = np.reshape([1.,2.,3.,4.], (1, -1))
#true outputs
action, value = np.reshape([0.1], (1, -1)), np.reshape([0.1, 0.9] , (1, -1))
Y = [action, value]

Network_model.fit(S, [A_t, V_t])

1/1 [==============================] - 0s 2ms/step - loss: 6.7340 - actions_loss: 1.1512 - values_loss: 12.3169 - actions_accuracy: 0.0000e+00 - values_accuracy: 1.0000


SPARK 1.5.1: Convert multi-labeled data into binary vector

Spark only recently implemented CountVectorizer, which will take the labels (as strings) and encode them as your 100-dimensional vector (assuming all 100 labels show up somewhere in your dataset).  Once you have those vectors, it should be a simple step to threshold them to make them 0/1 instead of a frequency.


Converting Text Entries into values for machine learning - KNN

It is not recommended to use the method you suggested of converting text to integer values (unless you are using decision trees or XGBoost as the predictive model). If you are usin KNN (and many other models) you should use a technique called one-hot encoding. The reason for this is that (using your example above) Install is 1 and Repair A is 2, but Install isn't necessarily quantitively twice as small as Repair A but you are assigning them 1 and 2 which are quantitively related (2 is twice as big as 1). So your model will interpret it that way.

That is why you need to use One Hot encoding.

I'm not privy to ML in JS but in python you could do:
With pandas:

import pandas as pd

df = pd.DataFrame({
      'A':['a','b','a'],
      'B':['b','a','c']
    })
df
Out[]:
   A  B
0  a  b
1  b  a
2  a  c

# Get one hot encoding of columns B
one_hot = pd.get_dummies(df['B'])
# Drop column B as it is now encoded
df = df.drop('B',axis = 1)
# Join the encoded df
df = df.join(one_hot)
df
Out[]:
   A  a  b  c
0  a  0  1  0
1  b  1  0  0
2  a  0  0  1

Maybe someone else can give you specific code in JS.


How to implement feature selection for categorical variables (especially with many categories)?

Some algorithms will perform feature selection inherently - e.g. elastic net regression, random forest - so you will not necessarily need to do this prior to running the algorithm.

You've identified one issue with one-hot encoding - it may create exceptionally wide data sets. Another issue is that a feature selection technique based on one-hot encoded variables will identify important feature levels and not important features. I would argue you might want to keep your features "together" and not evaluate individual levels.

Have you considered some basic statistical tests? Sometimes simple methods are just as useful as complex approaches. Useful tests include ANOVA (continuous response, categorical predictors) and Chi-squared / Cramer's V tests of independence (categorical response, categorical predictors).


R - How do I remove a varying number of digits from a date-vector

assuming your strings are of the form [NUMBER][WHITESPACE][STUFF-TO-KEEP]

you can use regex, for example with the stringr package.

# construct strings
strings <- paste(c(1,11,111,1111), "KEEP")
strings
[1] "1 KEEP"    "11 KEEP"   "111 KEEP"  "1111 KEEP"

stringr::str_remove(strings, "[0-9].* (?=[a-zA-Z])")
[1] "KEEP" "KEEP" "KEEP" "KEEP"

the regex "[0-9].* (?=[a-zA-Z])" expained:

'[0-9].* ' # (including the whitespace!)
# matches any number of digigs from 0 to 9 followed by a whitespace

(?=[a-zA-Z])
# this is a lookahead group, so an additional requirement is,
# that a match is only used when the above match is followed
# by a letter in the alphabet, small(a-z) or large (A-Z)

of course, str_remove removes the matches string.


Note the instruction is to view the utility matrix as boolean. That is if it is positive, then view it as $1$ and $0$ otherwise.

Example, to compute the Jaccard similairty between $a$ and $b$.

Consider the component of $a$ to be $(1,0,1,1,0,1)$ and component of $b$ to be $(0,1,1,0,1,1)$.

To compute $|a \cap b|$, look at how many position where both of the values are $1$. This is satisfied at the third and the sixth coordinate.  Hence the value is $2$.

To compute $|a \cup b|$, look at how many position where at least one of the value is $1$. This is satisfied at every position.  Hence the value is $6$.


Does click frequency account for relevance?

Depends on the user's intent, for starters.

Users normally only view the first set of links, which means that unless the link is viewable, it's not getting clicks; meaning you'd have to be positive those are the best links, otherwise the clicks are most likely going to reflect placement, not relevance. For example, here's a click and attention distribution heat-map for  Google search results:

Further, using click frequency to account for relevance is not a direct measure of the resource's relevance. Also, using clicks is problematic, since issues like click-inflation, click-fraud, etc. will pop-up and are hard to counter.

That said, if you're interested in using user interaction to model relevance, I would suggest you attempt to measure post-click engagement, not how users respond to search results; see "YouTube's head of engineering speaking about clicks vs engagement" for more information, though note that the size itself of the content is a factor too.

Might be worth noting that historically Google was known for PageRank algorithm though it's possible your intent is only to review click-streams, so I won't delve Google ranking factors; if you are interested in the Google's approach, you might find a review of Google's Search Quality Rating Guidelines.


What does the term 'Facet' in Seaborn FacetGrid imply?

According to the facetGrid documentation :

This class maps a dataset onto multiple axes arrayed in a grid of rows and columns that correspond to levels of variables in the dataset. The plots it produces are often called “lattice”, “trellis”, or “small-multiple” graphics.

Here levels mean  « categories ». The main goal is to not display the full set of data, but multiple subsets of data that fit conditions on categorical variables.
Definition of facet found here :

facet : a particular aspect or feature of something.

In the following example, the vertical axis (column of graphs) represents the facet/feature/attribute Smoker/Not-Smoker and the horizontal axis (line of graphs) represents the attribute Time. So each graph is the conditional AND between the two attributes :

Smoker AND Lunch
Smoker AND Dinner
Not Smoker AND Lunch
Not Smoker AND Dinner

(source: pydata.org)


Using sklearn knn imputation on a large dataset

You could use a memmap
import numpy as np
from tempfile import mkdtemp
import os.path as path
filename = path.join(mkdtemp(), 'newfile.dat') # or you could use another dat file that already constains your dataset

# supposing your data is loaded in a variable named "data"

fp = np.memmap(filename, dtype='float32', mode='w+', shape=data.shape)
fp[:] = data[:]

You can check full documentation on this page, the code above is based on the documentation.
This way you:

only change the declaration of your data matrices, keeping you code as clean as possible
uses numpy built-in ndarray subclass without having to explicitly manage data retrieval from the disk


Automatically Remove highly correlated features using Variance Inflation Factor?

I might oversimplify this, but Pandas allows to drop correlated features based on a threshold. E.g. correlation >0.95

https://chrisalbon.com/machine_learning/feature_selection/drop_highly_correlated_features/


Implementing U-Net segmentation model without padding

$\hspace{3cm}$
If we follow the definition of each arrow.
Gray => Copy and Crop

Every step in the expansive path consists of an upsampling of the
feature map followed by a 2x2 convolution (“up-convolution”) that halves the
number of feature channels, a concatenation with the correspondingly cropped
feature map from the contracting path, and two 3x3 convolutions, each followed by a ReLU. The cropping is necessary due to the loss of border pixels in
every convolution. Paper

So, believe(I have added 3 coloured circles)

Blue - 28x28 is upsampled and become 56x56, 1024 is halved to 512
Red - 64x64 is cropped to 56x56. Then Concatenated along FM axis.
Black - 3x3 convolutions, followed by a ReLU


Change Year to a factor and add group=1:

g <- ggplot(Total_Emmisssions,aes(x=factor(Year), y=Emissions, colour=Type, group=1))

you can leave the rest the same (you'll also prbly want to change the xlab).


Let's assume that you are trying to minimize the following loss for a given task, $$\ell(y, \hat{y}) = \frac{1}{2}\| y - \hat{y}\|^2, $$ where $\|
\cdot\|$ is the Eucledian distance, and the predicted output is $\hat{y} =\hat{x}$ for simplicity. The gradients then can be computed as follow:

$$\Delta b = - \frac{\partial\ell}{\partial\hat{x}} \cdot \frac{\partial\hat{x}}{\partial b}, \hspace{20pt} \Delta \omega = - \frac{\partial\ell}{\partial\hat{x}} \cdot \frac{\partial\hat{x}}{\partial \omega}$$

Now, the gradient of $\hat{x}$ with respect to the bias $b$ is $$\frac{\partial\hat{x}}{\partial b} = \frac{\partial}{\partial b}(x-E[x]) = \frac{\partial}{\partial b}\left((u+b)-E[(u+b)]\right) = 1 - \frac{\partial}{\partial b}E[(u+b)]$$

Ignoring the fact $E(x)$ is dependant on $b$ makes the above gradient equals to 1, and hence keeps updating the bias as follow:

$$\frac{\partial\hat{x}}{\partial b} = 1 -  \frac{\partial}{\partial b}E[x] = 1 -0
 = 1$$
and then

$$\Delta b = - \frac{\partial\ell}{\partial\hat{x}} \cdot (1), \hspace{20pt} b \leftarrow b + \Delta b $$

Otherwise, if you wanna consider this dependency, the gradient becomes 0, and hence no update as follow:

$$\frac{\partial\hat{x}}{\partial b} = 1 -  \frac{\partial}{\partial b}E[u+b] =  1 -  \left(\frac{\partial}{\partial b}E[u]+\frac{\partial}{\partial b}E[b]\right)
 = 1 - (0+1) = 0$$
and then
 $$\Delta b = - \frac{\partial\ell}{\partial\hat{x}} \cdot (0), \hspace{20pt} b \leftarrow b + 0 $$

In both cases, regardless bias update, the loss function will remain fixed,

$$u+(b+\Delta b)−E[u+(b+∆b)]=u+b−E[u+b], $$

however, in the first case, the bias will grow indefinitely.


How can I provide an answer to Neural Network skeptics?

Neural networks are essentially a black box, especially big ones. You could know even how it is designed and how it is training, but you really do not know how it is working in the end. In my work lots of people want to understand the model instead of using "black box" models. This is the reason why companies choose to use linear regressions and polynomial models instead of using stronger machine learning algorithms, like LightGBM and Neural Networks.

I never found a true answer to this question. Some engineers are taught that you cannot use models that you cannot understand. Therefore every model that is a "black box" is not usable for them. This means that most of machine learning models are magic and heresy for them. Though take this with a grain of salt, this is my subjective experience. Sometimes as the time passes these people are more willing to use data science methods because it becomes mainstream. They start to trust the methods, because others use them.

The situation is different on the higher level. For high-tier managers it matters less how to interpret the model, but more what results it could give you. They are more willing to try, especially if there is a hype of something, like "artificial intelligence", "data science".

As a result, I could only give you an advice to find some good support higher in the hierarchy of the company. Someone who believes in data science more and who has more power in the company.

In data science community the performance of the model on the test dataset is one of the most important things people look at. Just look at the competitions on kaggle.com. They are extremely focused on test dataset and the performance of these models is really good.

The only problem with performance on the test dataset is that it depends on the data in the test dataset. If in real life you will have completely different data that will be outside of the bounds of the test dataset, then the test dataset will not be able to give a good approximation of the performance of the model in real life.


I think what they were doing is just gradient clipping. It keeps the gradient of softmax layer between [-200,200]


Do I use actual data or data difference to train machine learning model?

I would use absolute temperatures. Knowing it's 37.2C, I'm more likely to say the temperature will decrease just from the context. This context would be lost if you only use the differences.

But like most data science questions, the proof is in the pudding. Try both and see what works best for you. You may even try using both absolute and difference as two feature columns.