This technique is called saliency maps [1][2].

From the documentation of Keras-Vis:

  The idea is pretty simple. We compute the gradient of output category with respect to input image. This should tell us how output category value changes with respect to a small change in input image pixels. All the positive values in the gradients tell us that a small change to that pixel will increase the output value. Hence, visualizing these gradients, which are the same shape as the image should provide some intuition of attention.

  The idea behind saliency is pretty simple in hindsight. We compute the gradient of output category with respect to input image.

  $\frac{∂output}{∂input}$

  This should tell us how the output value changes with respect to a small change in inputs. We can use these gradients to highlight input regions that cause the most change in the output. Intuitively this should highlight salient image regions that most contribute towards the output.


What is the difference between Missing at Random and Missing not at Random data?

Definitions of missingness process are tricky.
Missing completely at random occurs when the missingness is really at random (MCAR; e.g. when conducting a survey there are error in the data entry process).

Missing at random (MAR) occurs when the missingness is not really at random, but when it could be considered at random conditioning on what is observed in the rest of the data (e.g. males are less likely to express their opinion in a survey but this is completely not related to their attitude as customers).
Missing completely at random occurs when the missingness is really at random (MCAR; e.g.
when conducting a survey there are error in the data entry process).
Missing not at random (MNAR) occurs when the missingness depends on the value of the variable (those who buy more tend to not answer survey questions).

I think that it is difficult to figure out in what category are your missing data patterns. In your case seems to me that MAR is a reasonable assumption (which is impossible to test statistically) and that multiple imputation with chained equations could be an option (mice or miceadds in R).

If you adopt a mice approach, a crucial part in the application is to carry out sensitivity analyses. what happens if you:
- impute only some of your missing data pattern?
- vary the number of imputed sets
- perform a complete case analysis.


Facing this issue while predicting "CountVectorizer - Vocabulary wasn't fitted"

You have to give your training set to the model to be trained

_= pipe.fit(triningSet.data, triningSet.target)

I don't see any training dataset here. you have to fit the CountVectoriser to your data set.


Human Height Estimation using walking stride

I did not find scientific data, but interesting ratio is mentioned on several sports sites like this:

Women have ratio 0.413 and men 0.415 and it is used by multiplying height with ratio and we get the stride length.

My lenght 182cm,
ratio 0.415
stride length 182 x 0.415 = 75.53cm


why is MSE of prediction way different from loss over batches

From my understanding, you are comparing the prediction error on the test set versus the training loss error. When the training error is greater than the test error, that means your model is under-fitted.

Under-fitting occurs when there is still room for improvement on the test data. This can happen for a number of reasons: If the model is not powerful enough, is over-regularized, or has simply not been trained long enough. This means the network has not learned the relevant patterns in the training data. There are number of ways you can resolve under-fitting problem:

Increase the time of training procedure
Increase number of parameters in your model or the complexity of your model


To answer your question, you need not transform the numeric to binary variable(you meant binning right?).

I will try to explain it with the above example:

Let's start with row-1 where A = 1 , B =0 , C=0 so it means that you are talking about A. So the value of CreditAmount_Numeric belongs to A and the retired binary value also belongs to A. Similarly, you can deduce for the rest records.

Conclusion is that you need not do any transformation on your numeric variable.


Layer notation for feed forward neural networks

Well, the image you sent is not nicely denominated.

The first layer in the image is $x_0$ which is the input consisting of d dimensions, it is actually the first sample of the training set. Here its dimensions are $x_{01}, x_{02}, x_{03}, x_{04}$ (the green nodes in the left, hence, d equals $4$). Then the next layer which is called $x_1$ is the first hidden layer and subsequently $x_2$ is the second hidden layer and $x_3$ is the output of this feed-forward network.

By this definition, $x_0$ is the input with d dimensions $x_{01}, x_{02}, x_{03}, x_{04}$ and for calculating each node in the proceding hidden layer which is called $x_1$ here, we should do:

consider the most up node in hidden layer $x_1$ as the node we want to figure its value. We call it $x_{11}$, first we compute a linear computation of weights and inputs and then we apply some activation function $\sigma$ to it:
$$x_{11} = \sigma(x_{01} \cdot w_{11} + x_{02} \cdot w_{12} + x_{03} \cdot w_{13} + x_{04} \cdot w_{14})$$

also, an offset might be added to this expression.
consider that hidden layers can be of any size.

  Each one of the $d$ features in every example in the training set. In this case, $n_0 = d$ and $x_0$ is $(d \times 1)$.

$n_0 = d$ and $x_0$ is $(d \times 1)$ is right and in the first layer, yes each node is depicting a single one of d features of the input. but not for the hidden layers.

  Each example of the training set, which is $d-$dimensional. In this case, $n_0$ is the number of examples and $x_0$ is $(d \times n_0)$.

No as I mentioned, this is an architecture that depicts the process for a single training set. Hence, each node is not a sample of the training set.
You set $n_0$ for the number of nodes in the first layer which is input. so $n_0$ here equals d and $x_0$ which is the input equals $(x_{00}, x_{01}, x_{02}, ...x_{0d})$, $0$ showing that this is the first sample of training set.

In the backpropagation process, we have the same architecture. Then by calculating the gradient of each node, we update each weight. this process is done so many times to find the most optimal weights. there are various approaches for this weight updating thing like batch, etc updates.


GLOSH works with regard to local outliers, thus if a region is very dense and then has a few sparse points near it GLOSH will rate those sparse points as more outlying than other noise points in sparser regions. This is because it is considered particularly odd for a point to be near, but not in a very dense region -- local to the region the point is a significant outlier.

With regard to your question: if you care about local outliers (things that are odd relative to the density of the nearby region) then GLOSH is a good choice. This would be useful, for example, in detecting network intrusions since an attacker will be trying to behave like normal users, but will be slightly outside the dense normal user cluster. If, on the other hand, you are more interested in generally outlying points then merely selecting the noise points from the clustering will pick out points in sparser regions of the space.

Finally note that selecting noise points gives you only a single threshold; in contract GLOSH provides a range of scores and you can tune the threshold to find the number of outliers you desire.


How to get nearest 5 points mean with Nearest Neighbours?

First of all you train/ fit a model on the training set, may be cross validate using a training set. But it is not meant to predict itself. That's what test set is for.

You can achieve your objective by predicting each observation in a Leave One Out manner (also used for cross validation):

For every observation (i) in your dataset, fit a KNN regressor using all data except (i), evaluate your prediction for (i).


What is major difference between different dimensionality reduction algorithms?

A detailed answer would require many pages of explanation, but I think a brief answer may point to the right direction for further research.
First of all the choice of dimensionality reduction algorithm depends on the problem and data at hand. There is no golden standard. Your problem requirements dictate the best option(s) to try out.
The main concept of dimensionality reduction is to find alternative representations for the data that have less "dimensions" while at the same time keeping most of the original information contained in the data.
Right from the start, one sees that there is some trade-off between space/dimensions used and information kept. The less space/dimensions kept means also that the less original information contained in the data is kept.
The trick is to try to leave out redundant and useless information for the problem at hand. This is why the choice of algorithm depends crucialy on the problem and data at hand. Then one can indeed reduce space/dimensions of the data while at the same time keeping all relevant information.
In order to do that and depending on the nature and characteristics of data one can try some approaches:
1. PCA and variants
This factors the data to "principal/decorrelated components" and keeps those with most variance and discards the rest (as irrelevant and noise). This decomposes the data according to statistical variance (ie 2nd-order statistics are used) usualy performed by variants of EVD/SVD on correlation matrix of data.
PCA is the best, in the mean-square error sense, linear dimension reduction technique.
2. ICA
This factors the data into "independent components" which means that it uses higher-order statistics than 2nd-order which imply decorrelation only. However depending on algorithm, some data may not work well with ICA, since they must not be normally distributed (since for normal random variables decorrelation implies independece as well). Note: PCA is a pre-processing step in most ICA algorithms (eg JADE, ..)
ICA is a higher-order method that seeks linear projections, not necessarily orthogonal to each other, that are as nearly statistically independent as possible. Statistical independence is a much stronger condition than uncorrelatdness.
3. Dictionary and variants
Note that the above algorithms result in a set of "basic" components which can form the basis for all the data. Like the basis vectors of a vector space, each datum is a specific combination/sample of the basic components. Or like a dictionary each datum is a combination/sample of elements from this dictionary. But what if one knows the basic dictionary before hand for a certain problem. Then one can try to find the optimum representation of each datum wrt this basic dictionary. Or can try to learn adaptively this basic dictionary using some adaptve learning method.
4. Factor analysis
Also note that the first 2 approaches extract a set of basic factors from the data (ie they are equivalent to data factors). But what if one assumes a more general probabilistic setting for extracting (linear or non-linear) data factors (factor analysis). For example PCA/ICA can be seen as a specific example of  factor analysis where factors are required to be uncorrelated/independent. This approach is the generic probabilistic form.
5. Other data factorisation methods..
One can get the idea, if the data have certain properties that can be exploited in learning the optimum minimum dimension for represetnation one can try variations of data factorisation methods exploiting those data properies.
6. Unsupervised clustering methods..
Being able to find optimum representations of data automaticaly is a great advantage. Unsuperivised clustering algorithms can be used to this end, in the sense that they can try to cluster the data in an unsupervised manner (no prior information given) and then the cluster representatives can be chosen as the dictionary or basis factors that best represent the data as a whole. This results in dimensionality reduction (eg k-means, vector quantisation, ..)
References for further research:

A survey of dimension reduction techniques

Advances in data collection and storage capabilities during the
past decades have led to an information overload in
most sciences. Researchers working in domains as diverse as
engineering, astronomy, biology, remote sensing, economics,and
consumer transactions, face larger and larger observations and simulations on a dailybasis. Such datasets, in contrast with smaller, more traditional datasets that have been
studied extensively in the past, present new challenges in
data analysis. Traditional statistical methods break down partly because of
the increase in the number of observations, but mostly because of the
increase in the number of variables associated with each
observation. The dimension of the datais the number of variables that are
measured on each observation. High-dimensional datasets present many
mathematical challenges as well as some opportunities, and are bound to give
rise to new theoretical developments. One of the
problems with high-dimensional datasets is that, in many cases, not all the
measured variables are "important" for understanding the
underlying phenomena of interest. While certain computationally expensive
novel methods can
construct predictive models with highaccuracy from high-dimensional data, it
is still of interest in many applications to reduce the dimension of the
original data prior to any modeling of the data.

A survey of dimensionality reduction techniques

Experimental life sciences like biology or chemistry have seen in the
recent decades an explosion of the data available from experiments.
Laboratory instruments become more and more complex and report
hundreds or thousands measurements for a single experiment and
therefore the statistical methods face challenging tasks when dealing
with such high dimensional data. However, much of the data is highly
redundant and can be efficiently brought down to a much smaller number
of variables without a significant loss of information. The
mathematical procedures making possible this reduction are called
dimensionality reduction techniques; they have widely been developed
by fields like Statistics or Machine Learning, and are currently a hot
research topic. In this review we categorize the plethora of dimension
reduction techniques available and give the mathematical insight
behind them.


What is the MapReduce application master?

MapReduce Application Master coordinates the tasks running the MapReduce job. It is the main container for requesting, launching and monitoring specific resources. It negotiates resources from the ResourceManager and works with the NodeManager to execute and monitor the granted resources.

For MapReduce program, you do not need map and reduce functions. You can have map-only jobs and reduce-only jobs.

MapReduce Application Master appropriately implements the code in the main().


Would it make a difference to clean the data before using R findCorrelation()?

In most cases, you will want to run findCorrelation after cleaning. The reason being the cleaning process is either removing unwanted data points (that you deem irrelevant anyway) or computing some transformation that you'll want findCorrelation to be aware of.

Now, that said, I've often run it twice, once before cleaning and once after if my transformation could alter the correlation structure of the data.

Fortunately, since you're using caret, it is a simple matter to estimate your models both ways, then compare the ultimate differences in accuracy.  Unfortunately, the correct answer to many questions in ML is 'do it both ways then compare'


To use both groups and stratification, it seems that you should write your own code. Note that you will inevitably loose samples in training and/or in testing set (unless you are lucky).
One way of achieving it is to:

do your split by groups (you could use the GroupKFold method from sklearn)
check the distribution of the targets in training/testing sets.
randomly remove targets in training or testing set to balance the distributions.

Note: It is possible that a group disappear using such algorithm. You might prefer to not randomly remove the targets when balancing the training/testing sets.

Here is a sample code
import pandas as pd
import numpy as np
from sklearn.model_selection import GroupKFold

df = pd.DataFrame({
    'person': ['a', 'b', 'c', 'aa', 'bb', 'cc', 'aaa', 'bbb', 'ccc'],
    'group': [10, 10, 20, 20, 20, 20, 20, 30, 30],
    'target': [1, 2, 2, 3, 2, 3, 1, 2, 3]
})

X = df['person']
y = df['target']
groups = df['group'].values
group_kfold = GroupKFold(n_splits=3)
group_kfold.get_n_splits(X, y, groups)

# First split by groups
for train_index, test_index in group_kfold.split(X, y, groups):
    print("Groups split: TRAIN:", train_index, "TEST:", test_index)
    y_train_grouped, y_test_grouped = y[train_index], y[test_index]

    final_train_index = []
    final_test_index = []
    # Then balance the distributions for each target
    for target in df['target'].unique():
        target_train_index = y_train_grouped[y_train_grouped == target].index.tolist()
        target_test_index = y_test_grouped[y_test_grouped == target].index.tolist()
        n_training = len(target_train_index)
        n_testing = len(target_test_index)
        print("Target:" + str(target) + " - n_training:" + str(n_training) + " - n_testing:" + str(n_testing) +
              " | target_train_index:" + str(target_train_index) + " - target_test_index:" + str(target_test_index))

        # Shuffle to remove randomly
        np.random.shuffle(target_train_index)
        np.random.shuffle(target_test_index)

        # Check if we need to remove samples from training or testing set
        if n_training > n_testing:
            while n_training > n_testing:
                target_train_index.pop(0)
                n_training = len(target_train_index)
        if n_training < n_testing:
            while n_training < n_testing:
                target_test_index.pop(0)
                n_testing = len(target_test_index)

        # Append new indexes to global train/test indexes
        final_train_index.append(target_train_index)
        final_test_index.append(target_test_index)

    # Flatten for readability
    final_train_index = [item for sublist in final_train_index for item in sublist]
    final_test_index = [item for sublist in final_test_index for item in sublist]
    print("FINAL split: TRAIN:", final_train_index," TEST:", final_test_index, "\n")

EDIT
Using stratified cross-validation seems not to be mandatory (see links below), so you might reconsider using it.
You might find this links usefuls:

Why use stratified cross validation? Why does this not damage variance related benefit?
Understanding stratified cross-validation
A study of cross-validation and bootstrap for accuracy estimation and model selection

Regarding over/undersampling, I think it is hard to answer without more details of the data distribution and how imbalanced your classes are.


Prediction problem across a wide space of Clash Royale card games

Given the domain is a digital card game, deck value and game play strategy can be simulated. The wide space of options can filled in with synthetic data.
Classic machine learning algorithms, such as logistic regression and boosted trees, will have limited success given the sequential nature of the problem. It would be more useful to frame it as reinforcement learning problem which has built-in modeling concepts of sequential decision and "exploration" of different strategies.


weighted number of samples reaching node j

We have to compare the Parent to two children, then the children's impurity must be weighted.
Let's say the Parent has 100 samples and impurity=5
Children has the same data as [20, 2] and [80, 4]
Dip = 5 - (0.2 * 2 + 0.8 * 4)
=> 5 - 3.6
=> Dip = 1.4
"Weighted" means applying the appropriate ratio i.e. 0.2 and 0.8 here.


When would you choose certain CNN pretrained models?

There is a list of models and their performance and parameter count here. I prefer to use the MobileNet model initially. It is about as accurate as VGG but has about 4 million parameters versus the 140 million for VGG. Thus from a computational standpoint and thus training time it is far more efficient. If you modify it to use the callbacks ReduceLROnPlateau and ModelCheckpoint (documentation is here) you can in most cases achieve very good performance. I have found using the Adamax optimizer works very well, documentation for that is here. The code to use MobileNet is shown below

mobile = tf.keras.applications.mobilenet.MobileNet( include_top=False,
                                                           input_shape=(image_size,image_size,3),
                                                           pooling='avg', weights='imagenet',
                                                           alpha=1, depth_multiplier=1)
x=mobile.layers[-1].output
x=keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001 )(x)
predictions=Dense (len(classes), activation='softmax')(x)
model = Model(inputs=mobile.input, outputs=predictions)
for layer in model.layers:
    layer.trainable=True
model.compile(Adamax(lr=lr_rate), loss='categorical_crossentropy', metrics=['accuracy'])

If the model does not train well on your data set you can add more dense layers but it is always best to keep the model as simple as possible initially and try to optimize the hyper-parameters before making your model more complex.


Multiclass classification with high number of classes, high number of features and small sample size

You can try dimensionality reduction techniques such as PCA, which will reduce your numbers of features and maximizing the information describing your objects.

Moreover, if you want to classify the age which is a continuous variable, you may want to try regression instead of classification


Stanford CoreNLP has a very good implementation of NER for date/time.

https://nlp.stanford.edu/software/sutime.html   (demo: http://nlp.stanford.edu:8080/sutime/process)

Though this is written in Java, there are quite a few Python wrappers for this library (Such as : https://github.com/FraBle/python-sutime).  List of such libraries : https://stanfordnlp.github.io/CoreNLP/other-languages.html


Modelling regressor of historic data on basic features test set

Well, what you have here is a time series problem.

In a time series problem, you don't have the data parallel to the variable you are trying to predict. That means your data is over at the time $T$ and you have to predict based on the data from times $1$ to $T-1$.

Also, Shop_id and Item_id are not regressor which have a coefficient, because they are IDs, and using IDs to predict causes you model to learn-by-memory your data for the ID's, it gives a overfitted model.

Imagine the real-life situation: The store asks you to predict the sales of the next month, of course, you don't have data like the average price of the next month either, so you have to predict with historical data of both (all) variables.

The solution you are looking for is an ARIMAX model: The ARIMAX model allows you to use variables like Sales, Avg price, and others with information from the past.

$$Sales_t = \alpha_1 Sales_{t-1} + \alpha_2 Sales_{t-2}+\dots+\beta_1AvgPrice_{t-1}+\dots+\beta_mAvgPrice_{t-n}$$

The model is an example of how your predictions could be made, the use of the IDs can be "a model for every store". And every $Sales$ variable in the model could be a set of products.


From this SO answer

https://stackoverflow.com/a/53832130/1727543

import xml.etree.ElementTree as ET
import os

def read_content(xml_file: str):

    tree = ET.parse(xml_file)
    root = tree.getroot()

    list_with_all_boxes = []

    for boxes in root.iter('object'):

        filename = root.find('filename').text

        ymin, xmin, ymax, xmax = None, None, None, None

        for box in boxes.findall("bndbox"):
            ymin = int(box.find("ymin").text)
            xmin = int(box.find("xmin").text)
            ymax = int(box.find("ymax").text)
            xmax = int(box.find("xmax").text)

        list_with_single_boxes = [xmin, ymin, xmax, ymax]
        list_with_all_boxes.append(list_with_single_boxes)

    return {'filename':filename,'box_list': list_with_all_boxes, 'box_len':len(list_with_all_boxes)}

result = []
for file in os.listdir("path_to_Annotation_xml_folder"):
    result.append(read_content(file))

Extract more relevant features using 'EelementTree' to add your analysis.


The answer depends less on the classifier and more on the nature of the variable. In your case One Hot Encoding might be the best answer.

Label Encoding (Replacing categorical variables with integers) is useful when the variable is ordinal, i.e. it has a sense of order. For example the days of the week or the months of the year. Since they follow a fixed order, you can encode January as 1 and February as 2 and so on. The classifier would interpret Feb as being greater than Jan in some way (which is okay for a task like weather prediction and so on).

Can your countries be considered to be ordinal? If not, One Hot Encode them.


I would just like to add Microsoft's interpret to the list of explainers which uses boosting, and has some really cool graphics for explaining a model's behavior.


Multivariate Time Series Anomalous Entry Detection

If too many entries come in a sequence that have similar values for either id1 or id2, I want to classify them as anomalies and flag them (...) I'd appreciate any help w.r.t. to what direction to look in

General Approach

Here's how I would approach this:

Select the time windows, e.g. using pandas.rolling
For each time window, count the number of occurances for each string (per id1, id2)
Compare occurances to usual occurances. Usual here could be in comparision to say the moving average of occurances of the same string in the same window, across many windows, in comparison to the same window/range on the previous day etc.
Identify the anomalies. Anomalies would be those values that occur less or more frequently as "usual" by some threshold (say occurances within 2 standard deviations of the mean is normal, anything else is an anomaly)

A more elaborate and extensive treatment of possible approaches is given in Detecting Anomalies in System log files by Tim Zwietasch.

What algorithm to use?

This can be done with or without a machine learning algorithm:

LSTM might be an option if the process generating these logs can be modelled as some distribution so that an LSTM is able to generate the most likely sequences. In this case an anomaly would be a sequence that has a low probability of being generated by the model.
A simpler ML option would seem to use a "classic" anomaly detection algorithm such as one-class SVM, KNN, K-Means or LOF. A good overview is given in Introduction to Anomaly Detection by datascience.com (I have no affiliation)
Build your own "classifier" using simple statistics.

Personally I would start out with building my own, using a subset of the data, just to understand the data and its patterns better. Then, perhaps, use machine learning for robustness and scalability.

Encoding the log files

Whatever your choice, the key will be in encoding the data. Any algorithm will model the distribution of your data in one way or another (i.e. occurances of values per time window), so you have to present the data in a way suitable to detection of anomalies as per your criteria (anomaly relative to what?).

One encoding might be as follows:

for every time window, build one row of input data
for every value, have one column as the number of occurances of this value
add more columns for any other factor that could be influecing what is "normal" e.g. hour of day, day of week, week of year

Dealing with similar values

Note I did not yet treat "similar values" but assumed that the occurance counts are for exact values. However it should be relatively easy to extend the above to similar values by creating columns of occurance patterns as opposed to specific values. A pattern in this case could be e.g. all strings that contain some substring, say 'aaa' - or you could use a machine learning model to create clusters of value. Then use counts by patterns/clusters as the input instead of counts per value.


Strong overfitting accompanying strong class imbalance

With complex data it's rare not to have any overfitting (or underfitting). Ideally one wants to avoid strong overfitting, and if given a choice between two models it's clearly safer to use one which isn't overfit. But if it's impossible to avoid, from a practical perspective there's no reason not to use a model for this reason, in my opinion. The problem with an overfit model is simply that it's going to perform poorly on fresh instances, so the worst that can happen is the performance that you observe on the test set.
Just an idea: I don't know if it makes sense with your data but it might be worth trying one-class classification on the minority class. This could be a way to avoid the imbalance issue, and could be followed by a second stage of regular binary classification.


Difference between n-dimensional of length m and m-dimensional of length n?

They are just transposed version.

Often input shape wont matter for informational (performance purposes yes) what will matter is the information inside these vectors and wether its discriminative enough.


Is it ok to concat train and test sets to for fit_transform before splitting them?

They way you tried to do it is correct and no, you should not concatenate your train and test set to get around this.

Unfortunately it seems that the transformer you are trying to use just does not implement the functionality you need. From the docs:

  There are some imputation algorithms that are inductive, meaning they
  can be applied to new data after a call to solver.fit or
  solver.fit_transform. Currently only IterativeImputer supports the
  full scikit-learn API: fit, fit_transform, and transform, but we are
  actively looking for contributions that extend other imputers to
  support induction. At least the KNN and SimpleFill imputers can be
  extended in a straightforward manner.

Getting around this by concatenating your train and test set together will cause a data leak. This happens when your model gets information from the test set during training time, which invalidates the test evaluation.

And data leaks aside, what imputer will you use when you want to make predictions on new data?

As I see it your options now are:

Find another KNN imputation implementation that supports transform on new data
Use another inductive imputer
Implement KNN imputation yourself


Does traning the Word2Vec model multiple times affect `min_count` parameter?

The question has been answered in google groups by Gordon mohr.

Normally there's one read of the corpus to build the vocabulary (which includes initializing the model based on the learned vocabulary size), then any number of extra passes for training. It's only after the one vocabulary-learning scan that word counts are looked at (and compared to min_count for trimming).
If you supply a corpus (as a restartable iterator) as one of the arguments to the initial creation of the Word2Vec model, all these steps are done automatically: one read of the corpus (through the build_vocab() method) to collect words/counts, then one or more passes (as controlled by the 'iter' parameter and done through the train() method) for training. Still, only the count for the single pass over the supplied corpus matters for frequency decisions.
If you don't supply a corpus at model-initialization, you can then call build_vocab(…) and train(…) yourself. It's only what's passed to build_vocab() that matters for retained frequency counts (and the estimate of corpus size). You can then call train(…) in other ways, or repeatedly – it just keeps using the vocabulary from the one earlier build_vocab(…) call.
(Note that train(…) does try to reuse the single-pass corpus size, remembered from the vocab-scanning pass, to give accurate progress-estimates and schedule the decay of the training-rate alpha. So if you give a different-sized corpus to train(…), you should also use its other optional parameters to give it a hint of the size.)


sklearn.decomposition.PCA explained_variance_ratio_ attribute does not exist

This is untested, but I believe the error is occurring because you're calling explained variance on the fit_transform object, as opposed to simply just the results of fit.

Try:

df = pd.read_csv('Input.csv')
df = df.dropna()
df_transform = StandardScaler().fit_transform(df)
pca = PCA(n_components=2).fit(df_transform)
new_df = pca.transform(df_transform)
var_exp = pca.explained_variance_ratio_


Information leakage when train/test are truly i.i.d.?

Well, keep in mind that when you standardize/impute data you're estimating parameters. Given the conditions that you've defined and having enough data such that the estimates are good, then I don't think it should matter to use the training data or all the data (as a matter of fact, the estimate of the parameter using training data should be very similar to the parameter using all the data).
However, when the datasets are small these estimates may have high variance, and if you want your cross-validation to be reliable you might want to take it into account. In general, I think for scaling and missing imputation this should not be an issue, but if you're doing some things that are more prone to overfitting like target encoding, or imputing using a model instead of a single parameter, then you should be more careful.
In order to not even think about having enough data or not, the cross-validation framework tries to treat the test data as what the serving data should be when putting the model into production, and then you'll always be alright.


Normalized metric for comparing regression models performance

A couple suggestions

MAE represents your mean error. This is essential to recognize as having an error of even half of your MAE on the low end of your spectrum ($34,900) is huge. However, to your point, at the high end of your spectrum it is quite small.
To solve the above, you should plot a graph to identify the patterns in your error. You'll plot your predicted y value vs. your actual y value and see where your predictions deviate from the y=x line.

Normalized metrics:

you may use an r-squared value. This might be ideal because its looking at the percent of variance explained by your regression and thus is a relative measure of fit. Your r-squared value will always be between 0 and 1 (assuming your model is not worse than always guessing your mean value) where 1 is perfect. Implement as follows:

from sklearn.metrics import r2_score
y_pred = model.predict(X_train)
print(r2_score(y_true, y_pred))


Can we define a data partitioning in K clusters, by cutting the branches of the tree at some levels in the tree below the root node?

Yes absolutely, in fact that's exactly how one obtains actual clusters with hierarchical clustering.
With hierarchical clustering it's possible to decide the number of clusters (i.e. the level of "precision") after obtaining the tree (dendogram).


How to find the values that never matched in a full join

What you appear to want is a left outer join on NULL, this will give you only the rows of the left table which do not have a match with the right table:

SELECT    F.Store
FROM      `project.campaigns`  F
LEFT JOIN `project.visitors`   C
ON (
      F.Store = C.Store
  AND F.Date  = C.Date
  )
WHERE C.Store IS NULL


What does it mean when an Actual vs Predicted plot is like this assuming you are using the best model?

It's not said taht you are doing something wrong, some datasets don't contain highly predictable data. However in this case it seems there is a pattern in your data that your models is not capturing. I have a couple of questions about it:

Is there a big difference between train and test performances? If so, you must use some regularization techniques, such as dropout, in order to reduce overfitting.
What is the architecture of your Neural Network? And did you try different ones?


How to restructure my dataset for interpretability without losing performance?

What may be happening is that your attribute predictors are weak predictors, they are noisy. Meaningful decision trees can't be made out of product attribute features by xgb.

When you are adding name as a predictor, xgb finds some signal wrt your target variable - rating and thus you get a better score. So your name plus attributes model may be performing better than attributes only model for this reason.

So if you from domain experience know product attributes are very weakly related to rating then you can conclude that this feature set of attributes is not going to help you make accurate predictions. Or instead of relying  on d omain expertise, you can use correlation or relevant statistical tests to understand attributes relation to rating and if found that relationship is non existent or very weak you can conclude model isn't possible.

So may be add more relevant features if possible if you want to make a reasonably good model.

Regards
Vik


How to forecast time series analysis for more then one dependent variables?

I think you do not need to "predict" counter_id and country_code as variables, what you need is to produce results to every counter and country (and sometimes these results will be zero).

The technique you are looking for is called VAR (Vector Autoregression)

VAR

The vector autoregression (VAR) model is one of the most successful, flexible, and easy to use models for the analysis of multivariate time series. It is a natural extension of the univariate autoregressive model to dynamic multivariate time series.

The equation you propose is solvable by VAR:

$Y_t = C + A_1 Y_{t-1} + A_2 Y_{t-2} + ... + A_n Y_{t-n}$

Where $Y_t = \begin{bmatrix} Y_{i=1,j=1,t} \\ Y_{i=1,j=2,t} \\
\vdots \\
Y_{i=1,j=m,t} \\
\vdots \\
Y_{i=n,j=m,t} \end{bmatrix}$

Where, $i$ is every country, $j$ is every counter.

The usage is very similar to univariate time series analysis, which you are probably familiar with.


Segment 5-7 min audio into sentence wise audio clips for creating speech recognition dataset

The typical approach is to just cut the clips into consecutive sections, and run the model on each such section. Sometimes a bit of overlap is used, say 10%. then you have to decide what to do with potential conflicts in these overlaps.
A good model is usually robust against silence, otherwise you can try to cut silence in start and end of your 10-second window.

librosa.util.frame is a practical way of doing this in Python.


Available data about 'likers' as a page on Facebook

Want to wish you good luck. Some time ago faced with the same problem, but didn't find any satisfying solution.
First of all, there is no way to get list of users, who "liked" a particular page. Even, if you are an administrator of this page (I was). One only can get list of last 3 or 5 hundred users.

Friendships data for most of the users is also inaccessible. Looks like gender is the only thing from your list, that you can get.

Data about pages, that exact user "likes", should be available (as it's written in docs), but in reality, through API you can collect something only for friends and FoF. Even though this data is available through web interface. So the only way is to try dirty trick with parsing and scraping (but remember, that I didn't advise it ;) ).


First question: computing $\Sigma$

Actually, you perform the same computation but using a matrix operation instead of a scalar operation. You might be mislead by your notation of the feature matrix $X$ that you write $x$ because in fact
$$X = (x_j^{(i)})_{i,j} =\Big(x^{(1)} ... x^{(i)} ... x^{(n)}\Big)^T$$
that is to say, $i$-th row of the matrix $X$ contains $x^{(i)}$ the $i$-th sample.

So, when you want to compute the empirical covariance matrix $\Sigma$ (which is indeed a $n \times n$ matrix), you have:
$$\Sigma = \frac{1}{n} \sum_{i=1}^n x^{(i)} {x^{(i)}}^T = \frac{1}{n} X^T X$$
you can check that this is exactly the same computation.

In your code, you are actually computing directly $\Sigma$ using the matrix operation (i.e. $\Sigma = \frac{1}{n} X^T X$) and this does give you a $n \times n$ matrix. So, both your implementation and the formula is correct.

Second question: reconstructing $X$

Your new feature matrix $Z$ is a $n \times 3$ matrix according to your code. In fact, as your code does not show the original size of the feature space, we will see the two cases here:

$X$ is also a $n \times 3$ matrix, then you do not perform any kind of dimension reduction and you should have $X_{new} = X$ (at least in theory, in practice you might have a very small numerical approximation error but it will basically be the same)
$X$ is a $n \times d$ matrix with $d > 3$, then you do perform a dimension reduction and in this case you get rid of some information contained in the original data and in this case $X_{new} \neq X$

In your case, I guess that you do have $d > 3$ and so you're in the second situation.

Third question: when to apply PCA

Well that depends in fact...

First of all PCA performs an SVD which can become very expensive if you have a lot of features. In fact, there are several way to compute PCA, your way is the closer to the mathematical theory but in practice computing an SVD directly on $X$ avoids computing $X^T X$ which is expensive and allows to retrieve the same space. But in any case, in some situation, it can be expensive and time consuming and thus not practical.

As PCA sorts the new feature space according to the value of the eigenvalues, i.e. of the variance of the new directions, removing the last direction might in some cases remove some noise which can help. However, at the same time, it can throw away valuable discrimative information, this is why other methods such as LDA can be interesting.

So to summarize, the answer to the question is no, it is not a good choice to use it everytime, it depends of your problem.


How to separate overlapping curves more effectively?

The problem here isn't your chosen neural network architecture, it's that you're working with literally unseparatable data.

There's isn't any practical way to distinguish observations near the mode of multiple distributions that share a mode. The best you can do is approximate the distance from the mode at which the high variance distribution is more likely than the low variance distribution, and classify observations relative to that. You can do that analytically, and your neural network isn't going to do any better.

Have you tried visualizing your data to better understand what you're tasking your model with here? This is a bit more extreme than "kind of overlapping" distributions.


I had the same problem, and I'm not proud of the very "hacky" solution, but the bottom line is it's working now. I'm working on a Mac, so I'm sure your solution will be a bit different.

The normal pip install wasn't working for me, so to install I did...

git clone https://github.com/fukatani/rgf_python.git
cd rgf_python
python setup.py install

I was then able to find the executable file your error message is complaining about in the new rgf_python directory, specifically...

rgf_python/include/rgf/bin/rgf

From there, I tried to change the exe_location in my .rgfrc file to point to the above file, but it didn't work so I went to the /site-packages/rgf/sklearn.py file mentioned at the top of your error message and changed a line in the "_get_paths" function.

For me it was line 64/66. I suspect yours may be 54/56 if you're on Windows. I commented out the old declaration of "exe" and changed it to point to the rgf executable file I found earlier.

I realize how messy this is, so I hope someone can find a better solution, but in the meantime, this did the trick for me. Good luck, and let me know if you have any questions.


What's a good method for combining data of different deep learning models?

Actually you can merge two different DL model into a big one.

I'm using Keras+Tensorflow for deep learning, and there's an example about models with multiple inputs(outputs). In this example, tweets are fed into a RNN, while the extra data are fed into a fully-connected layer along with the output of this RNN.


When to use cosine simlarity over Euclidean similarity

When to use cosine similarity over Euclidean similarity

Cosine similarity looks at the angle between two vectors, euclidian similarity at the distance between two points.

Let's say you are in an e-commerce setting and you want to compare users for product recommendations:

User 1 bought 1x eggs, 1x flour and 1x sugar.
User 2 bought 100x eggs, 100x flour and 100x sugar
User 3 bought 1x eggs, 1x Vodka and 1x Red Bull

By cosine similarity, user 1 and user 2 are more similar. By euclidean similarity, user 3 is more similar to user 1.

Questions in the text

I don't understand the first part.

  Cosine similarity is specialized in handling scale/length effects. For case 1, context length is fixed -- 4 words, there's no scale effects. In terms of case 2, the term frequency matters, a word appears once is different from a word appears twice, we cannot apply cosine.

This goes in the right direction, but is not completely true. For example:

$$
\cos \left (\begin{pmatrix}1\\0\end{pmatrix}, \begin{pmatrix}2\\1\end{pmatrix} \right) = \cos \left (\begin{pmatrix}1\\0\end{pmatrix}, \begin{pmatrix}4\\2\end{pmatrix} \right) \neq \cos \left (\begin{pmatrix}1\\0\end{pmatrix}, \begin{pmatrix}5\\2\end{pmatrix} \right)
$$

With cosine similarity, the following is true:

$$
\cos \left (\begin{pmatrix}a\\b\end{pmatrix}, \cdot \begin{pmatrix}c\\d\end{pmatrix} \right) = \cos \left (\begin{pmatrix}a\\b\end{pmatrix}, n \cdot \begin{pmatrix}c\\d\end{pmatrix} \right) \text{ with } n \in \mathbb{N}
$$

So frequencies are only ignored, if all features are multiplied with the same constant.

Curse of Dimensionality

When you look at the table of my blog post, you can see:

The more dimensions I have, the closer the average distance and the maximum distance between randomly placed points become.
Similarly, the average angle between uniformly randomly placed points becomes 90°.

So both measures suffer from high dimensionality. More about this: Curse of dimensionality - does cosine similarity work better and if so, why?. A key point:

Cosine is essentially the same as Euclidean on normalized data.

Alternatives

You might be interested in metric learning. The principle is described/used in FaceNet: A Unified Embedding for Face Recognition and Clustering (my summary). Instead of taking one of the well-defined and simple metrics. You can learn a metric for the problem domain.


How to plot an image from a series of bits inPython?

You could use at least these two options:

PIL library:

from PIL import Image

img = Image.new('1', (width, height)) #width and height of the original image
data=[int(x) for x in x]  #x being your array
img.putdata(data)
img.save('my_handwritten_number.png')
img.show()

You can see here an example with the whole process, from reading a hand written image file through converting it to an array and displaying it again to an image:

import numpy as np

np_img=np_img.reshape(28900, )

width = np.sqrt(len(np_img))

from PIL import Image

img = Image.new('1', (int(width), int(width)))
data=[int(np_img) for np_img in np_img]
img.putdata(data)
img.save('my_hand_written_digit.png')
img.show()

getting as a result this reconstructed digit:

nevertheless, I see your array is a 1D array whose square root is not an integer, so it seems it does not come from an square image; you need to be able to convert it to a matrix (via the Image.new method).
If I do it with your array, this is what I get:

another option you have is with Keras:

import tensorflow.keras as keras

keras.preprocessing.image.array_to_img(np_img.reshape(170, 170, 1),
data_format=None, scale=True, dtype=None)


How to distinguish between different values of a hyperparameter in communication?

TLDR: Basically, you have answered your question with your quote from Wikipedia: the common terminology is "values".

The question on SE SO which you have quoted discusses the terminology from a software development perspective. But there is a difference between the use of the word "parameter" in software development and machine learning. This is how Wikipedia defines parameter from a software development perspective:

In computer programming, a parameter or a formal argument, is a special kind of variable, used in a subroutine to refer to one of the pieces of data provided as input to the subroutine. These pieces of data are the values of the arguments (often called actual arguments or actual parameters) with which the subroutine is going to be called/invoked. An ordered list of parameters is usually included in the definition of a subroutine, so that, each time the subroutine is called, its arguments for that call are evaluated, and the resulting values can be assigned to the corresponding parameters.

Let's make an example:
n_estimators = 100
model = RandomForestRegressor(n_estimators=n_estimators)

From a software development perspective n_estimators is a parameter, 100 is the parameter's value and n_estimators=n_estimators is the argument.
Howevever, in machine learning terminology you'd consider n_estimators a hyperparameter and 100 its value. Going back to your examples I would just do something like this:
hyperparameter_dict = {
    "learning_rate": 0.001,
    "dropout": 0.25,
}

and
for hyperparameter, value in hyperparameter_dict.items():
    ...

Introducing a new terminology which is not common in machine learning (like "hyperarguments") would only cause confusion and not make your code or communication any clearer.


how to calculate similarity between users based on movie ratings

Use euclidean distance
1
$d(p,q) = \sqrt{\sum_{i=1}^{n}(q_i - p_i)^2}$
and check main user vs all other users.


Given the specific context of detecting abnormal changes in the amount of free space, I'd suggest that you use the variation over time instead of the raw amount. For instance:

date                   free_space  variation
2019-05-15 09:00:00    102.65      NA
2019-05-15 09:05:00    102.69      0.04
2019-05-15 09:10:00    103.11      0.42
2019-05-15 09:15:00    102.58      -0.53
2019-05-15 09:20:00    102.55      -0.03

Whatever method you use, the variation is a much more relevant information to detect an unusual change than the raw size. You could also use a time window, e.g. calculate the variation over the last 30 minutes.

Personally I would simply use a heuristic for something like this: if the absolute value of the variation is higher than a threshold then label as outlier. The threshold could be a percentage of the size of the disk, e.g. 5%.


How to generate higher quality jpg plots on R

I think if you save the plot as SVG (Scalable Vector Graphics) you can scale the image to any size. See this answer for an example.
Second, if you want to print larger, you might need to tweak the DPI (Dots per inch), as done here.


R data frame create a new variable which corresponds to one of the existing one

If my assumptions about CommArea_Num are correct, then:

data$community_area_clean = data$community_area_clean[data$CommAreaNum]

should remap that column. Note that the AREA_KM fields will remain in the order currently shown.


Breaking down a column in Pandas into a separate CSV for display in Tableau

If you don't need other columns, here is a solution. It splits the column, stacks in vertically and combines with "permalink" column

df.set_index('permalink').category_list.str.split('|', expand=True).stack().reset_index('permalink').rename(columns={0:'category'})

permalink   category
0   /organization/-qounter  Application Platforms
1   /organization/-qounter  Real Time
2   /organization/-qounter  Social Network Media
0   /organization/-the-one-of-them-inc- Apps
1   /organization/-the-one-of-them-inc- Games
2   /organization/-the-one-of-them-inc- Mobile
0   /organization/1-4-all   Entertainment
1   /organization/1-4-all   Games
2   /organization/1-4-all   Software
0   /organization/1-800-publicrelations-inc-    Internet
1   /organization/1-800-publicrelations-inc-    Marketing
2   /organization/1-800-publicrelations-inc-    Media
3   /organization/1-800-publicrelations-inc-    Public Relations
0   /organization/1-mainstream  Apps
1   /organization/1-mainstream  Cable
2   /organization/1-mainstream  Distribution
3   /organization/1-mainstream  Software

Then you can save it to csv.


How to do Predictive Modeling for Different Companies

Interesting question and the answer is: it depends.

You would have to provide more information about the exact inputs that are needed for the model and you would have to be 100% certain that each input means the same thing for each company. So, for example, you would have to be sure about things like timelines - if you are looking at "sales" and one company measures sales monthly, but the other quarterly then that's a problem for your model.

Also, you would have to consider the scale of the inputs. If one company has sales in the millions of dollars while a smaller competitor has sales in the thousands of dollars, that's also a problem for your inputs.

If you can think about issues above and still decide that it's OK to continue, then yes, you can use the same model. This is called transfer learning and you can take a trained model and then "update" it with the data from the new company.

But if you do run into issues with things like the above (or some other variation on issues with inputs) then I would recommend that you use your third option and just develop a model for each. If you have a framework that works for one company (i.e.  you have a model and you have a sense for the data you need) you should find that you are able to create similar accuracy numbers, even with brand new data.

Personally, I think there is a 4th option. You could just handle all of this from the data side. I would develop a new model for each company, but I would develop Company Y's model based on Company Y data but then also augment my training with Company Z data. You can choose the mix you want, but by including records like this you are still leveraging the power of writing a single model for a company with the added industry/competitor knowledge.


ValueError: Input 0 of layer conv1_pad is incompatible with the layer: expected ndim=4, found ndim=3. Full shape received: [None, 224, 3]

You need to make sure to add an extra dimension for the batch size, if you are passing in a single image the batch size would be 1. You can use np.expand_dims to add the extra dimension.


Chi-Squared test: ok for selecting significant features?

I will raise several issues that could arise if you are selecting features based on chi-2 tests

Repeated use of chi-2 test can lead to spurious results unless you correct for the number of times you run it

You can include features that are correlated with each other, i.a. A is correlated with B, and both are correlated with label. Not sure, but I think, this can lead to results where model performs worse with more features.

I would try starting with all the features, remove the ones linearly correlated.  But this is just a suggestion.
Also, mutual information can be used to estimate how well any given feature describes the label.


Normalization(minmax) gives me worse results than before in KNN, follow up actions?

Is bad luck.

Look what (I think) is happening:

When you use MinMaxScaler, what you do is relativize (reduce or augment) the distance between individuals in a way it was not before the change.

Let's suppose your best model is defined by the variable $X_j$ from the variable set $X=\{X_1,X_2,...,X_n\}$. When you change the set to an scalled one: $S=\{S_1,S_2,...,S_n\}$, what you do is giving the same importance to every variable, instead of preserving what the variables said before scalling.

$X_3$ gives better information on the output than $S_3$ (and in $S$, all are scalled), $S_3$ gives less information because is mixed with all other variables.

Imagine $X_3$ as something like "age" (and you are determining probability of having cancer), the greater the age, the greater the probability. This is true for $S_3$ also, but when $S_3$ is combined with the rest of $S$, $S_3$ looses importance amongst them. With $X_3$, age keeps its relative importance (bigger values amongst lesser values).

This does not happen very often, you could find yourself on the opposite situation: When combined $S$ might be a very powerful set.

That is why I think you were simply, unlucky.


When would not normalizing input values have higher accuracy?

The thing I'm wondering is why I'm getting better results with the neural network without the normalization?

From a theoretical standpoint, normalizing your input shouldn't affect accuracy, in a sense that a NN can converge to any range of input values.

The reason why we normalize our data when using NNs is that it offers some useful properties, mainly regarding convergence and convergence speed. You can read more details here.

  Is it because the optimal hyperparameters required for the network with the normalized input are different from when it is not normalized?

Yes, I think that is your problem. Some hyperparameters might be optimized for the regular input and might not work well for the normalized one. An example of this could be a higher learning rate, which might work for a higher range of input values but not for a normalized one.


If you look at the documentation of the BCELoss you will the that they mean the following:

$w$ is the weight to use, by which the loss will be rescaled.
$x$ is the output value, which is the value that is predicted by the model.
$y$ is the target value, which is the ground truth value.


How to summarize(add) a column according to same year and plot in R?

Step 1: Changing dates to years

First you can change your column of dates (Order_Date) to simply years by using this formula:

> salesdata$Order_Date = format(salesdata$Order_Date, "%Y")

This will give your Order Dates just the year the sales was done in.

Step 2: Summarizing your sales data

After this you can create a vector with the summarizations of your sales data per year. This can be done using tapply:

> tapply(salesdata$Sales, salesdata$Order_Date, FUN=sum)

This will give you an overview of the total sales per year.

Step 3: Plotting your graph

You can use barplot() to plot your bar chart using the values you got in your tapply function.

One-liner

The function as a oneliner:

> barplot(tapply(salesdata$Sales, format(salesdata$Order_Date, "%Y"), FUN=sum))

Of course you can edit your graph to your liking.


How Linear SVM Regression and Multiple Linear Regression different in terms of the regression result?

They could potentially be very different, because linear regression penalizes using squared error while SVRs penalize absolute error. You might have learned that "outliers" can have an outsized influence on the regression line because of this--this won't be the case with SVR, however.

Furthermore, if the SVR you're talking about is like the one described here (great resource there, by the way) then choosing a larger $\epsilon$ will yield a flatter regression line/hyperplane, whereas you don't have such a tuning parameter for regular ol' linear regression. This parameter enables you to try to fight overfitting by flattening the regression line.


Timeseries VAR vs VARMA model: issue in time to fit model

It's because 80,000 samples is probably too much training data. Usually, statistical forecasting models are not trained on the entire training set like ML models. Try reducing your training data significantly, and incorporate only a certain number of seasons as your training data.

Example:
If your data is hourly, one season can be 24 samples. Then, the training data can span 14 seasons - 24*14 samples.


Data cleaning and data transformation before EDA?

Although not very helpful, the answer is probably "it depends".

I like to do data cleaning and some EDA together since EDA can highlight appropriate treatments to clean the data - e.g. influencing how to handle missing values.

I think data transformation should be done just prior to modelling; whether or not you need to do any transformation at all depends on the techniques you plan to use.


Improving a basic neural network with only 3 input columns

There is no point in looking at the prediction at this point. Your model has not learnt the data yet.
Suggestions -

When copy from a source. Try copying everything first. Run it and then start tweaking for learning Or to map with your own need. You have changed a few things as compared to what Jason has in his post

Use the wisdom of researchers. So, you can keep a few things as default -
     1. Optimizer - Adam
     2. Activation - Relu for Hidden layer
     3. Loss - MSE(Regression), Cross-entropy(Classification). Beware of Cros-entropy variance.
     4. Activation for Output layer - Linear(Regression), Sigmoid/Softmax(Classification)
     3. Have Batch normalization layer when you have a deep network.

Try to know why behind every point that I have mentioned. It's will be an easy search on the Internet.

A good place to start can be - Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition, by Aurélien Géron
Further reading -
     Practical Recommendations for Gradient-Based Training of Deep Architectures, by Yoshua Bengio
    Andrej Karpathy blog
    Raúl Gómez blog


If we are using batch normalization as the first layer, can we forego standard scaling of inputs?

Scaling is a bit different from what Batch normalization does. Performing scaling creates scale indifference amongst all the data points. Ex: Values 5 and 55 will have a higher magnitude of scale difference than the log(5)=0.698 and log(55)=1.740. This is the idea behind scaling. Similarly, We scale the images with 255 which helps in faster convergence

When we pass in the scaled data to our network, due to the operations that take place in every layer of the network, the distribution of data observed in let's say 5th layer of the network is different from the distribution in the 1st layer. This is a problem because we are kind of training the first part of the network in one form of data and the latter part of the network with another form of data. So to remove this effect, We do BatchNorm in later layers of the network


Alternatives to imputation of missing values?

Supposedly you had these models, divided into subspaces, then what if in your test dataset a point is close proximity to where the missing point is in your training data. What would be the best solution? Have the average output of the two closest models? Is the output binary? And then would you have to weight each model, based on that proximity? Then I am thinking that many submodels could overfit on the subset of the data. Just some thoughts, which may be challenging to overcome.

Moreover, you could use other types of Classifiers which deal with missing data, like some decision trees implementations.

Finally, data imputation is a delicate art, and doing it right will leverage your model. For instance, you wouldn't want to impute all the missing data with the average as then you may change your data distribution. Knn techniques or ensemble regressions would be preferred.


Keras - Computing cosine similarity matrix of two 3D tensors

I know of no pairwise distance operations in Keras or tensorflow. But the matrix math can be implemented in TF/Keras backend code and then placed in a Keras layer.

Here's the matrix representation of the cosine similarity of two vectors:
$$
cos(\theta) = \frac{\mathbf{A}\cdot\mathbf{B}}{\|\mathbf{A}\|_2 \|\mathbf{B}\|_2}
$$

I'll show the code and a test that confirms that it works. First, generate non-trivial test data.

import numpy as np
import keras
import keras.backend as K

# set up test data
n_batch = 100
n = 400 # number of points in the first set
m = 500 # number of points in the second set
d = 200 # number of dimensions

A = np.random.rand(n_batch, n, d)
B = np.random.rand(n_batch, m, d)

Define pairwise cosine similarity function.

# convenience l2_norm function
def l2_norm(x, axis=None):
    """
    takes an input tensor and returns the l2 norm along specified axis
    """

    square_sum = K.sum(K.square(x), axis=axis, keepdims=True)
    norm = K.sqrt(K.maximum(square_sum, K.epsilon()))

    return norm

def pairwise_cosine_sim(A_B):
    """
    A [batch x n x d] tensor of n rows with d dimensions
    B [batch x m x d] tensor of n rows with d dimensions

    returns:
    D [batch x n x m] tensor of cosine similarity scores between each point i<n, j<m
    """

    A, B = A_B
    A_mag = l2_norm(A, axis=2)
    B_mag = l2_norm(B, axis=2)
    num = K.batch_dot(A_tensor, K.permute_dimensions(B_tensor, (0,2,1)))
    den = (A_mag * K.permute_dimensions(B_mag, (0,2,1)))
    dist_mat =  num / den

    return dist_mat

Build a shallow Keras model around the function.

# build dummy model
A_tensor = K.constant(A)
B_tensor = K.constant(B)
A_input = keras.Input(tensor=A_tensor)
B_input = keras.Input(tensor=B_tensor)
dist_output = keras.layers.Lambda(pairwise_cosine_sim)([A_input, B_input])
dist_model = keras.Model(inputs=[A_input, B_input], outputs=dist_output)
dist_model.compile("sgd", "mse")

Compare to sklearn implementation

sk_dist = np.zeros( (n_batch, n, m) )
for i in range(n_batch):
    sk_dist[i,...] = cosine_similarity(A[i,...], B[i,...])

keras_dist = dist_model.predict(None, steps=1)
np.allclose(sk_dist, keras_dist)
> True


Is Oracle 11g is able to ingest big data?

There is no reason why you cannot use Oracle 11g to create large databases. The 1000-column limit is a theoretical one, not a practical one imo. The theoretical database size limit is in petabytes. You should question why you would want to put that much data in a single database and be prepared to pay for the infrastructure to support it. This is a good summary link - http://awads.net/wp/2010/02/15/oracle-database-limits-you-may-not-know-about/

Having administered and tuned dbs of 50TB and more, the limits you should be more concerned with are for query performance. Ask Oracle about licensing as well before you proceed.


True Negatives for Named Entity Recognition

By definition, true negative is non-entity tokens in the text, agreed by gold standard. So;
TN: 5 = {announced, the, release, of}
But, in practice the number of true negative tokens are quite high and this heavily dominates the metrics, and any measure using TN (like accuracy) becomes highly controversial. Instead, preferred better metrics:

Recall: TP/(TP+FN) - coverage in all correct entities, and
Precision: TP/(TP+FP) - how reliable for a positive result
F1 Score (harmonic mean of precision and recall) can be used as a balanced metrics.


xgboost feature selection and feature importance

1) There are no guidelines for new features for xgboost, or even linear regression. It is not necessarily true that features have to be linearly related to the outcome in a linear regression, one can use transformations and interactions to capture non-linear relationships. The problem? In a linear regression we have to manually model the non-linearity, which is a pain to find if you have more than just a handful of features and not much prior information. Xgboost (hopefully) finds these non-linear relationships automatically.

The "guidelines" for new features, regardless of your model, is really driven by the actual problem at hand. That is, what knowledge do you have about the possible relationship between a variable X and Y, and furthermore, are there better ways to express the relationship using the current information that you do know? Here is an example; suppose I have a problem where I wish to predict the probability of some customer defaulting on a loan I provide to them. I am given their past payment history, provided as a series of payments (if made). I can use external knowledge from my background in finance to know that there is an explicit formula to calculate the given interest rate in the past, given a series of payments at known intervals. I can then use this calculated interest rate as a variable to get a better representation for the relative risk of a client (as opposed to just aggregating the payments themselves) since I know that riskier people get higher interest rates due to a higher implicit probability of defaulting. Another example: in insurance, people in my country typically get into fewer accidents in the winter than the summer. Perhaps, if I were trying to predict accident occurrence over time I would strive to include some seasonality indicator variables.

I have read some guidelines in this book here by Kuhn and Johnson regarding interaction features, namely that more important features are generally more likely to be better candidates to include as interactions (product of two or more predictors are interactions, in case you were wondering).

This process is commonly referred to as feature engineering, where we essentially manipulate our current data such that the model can learn easier.

2) Maybe. For some problems yes, for other problems, no. The curse of dimensionality is real and definitely can lead to overfitting. One can use the feature importances from xgboost to drop variables and improve their model but often you can overfit to the training set when doing this and you may actually make your model worse than if you had done nothing at all. Hence, it is incredibly important that you wrap this process in a cross validation scheme to see if the feature selection process actually improves your model. Two popular algorithms that make use of feature importance scores for variable selection are recursive feature elimination and permutation methods (so called "null importances").

3) This topic is unfortunately not as well covered in literature though feature selection, extraction, and engineering are very fruitful topics being researched right now. That being said, this aforementioned book might be of use to you (current a work in progress, so it might not be free forever):

http://www.feat.engineering/


What brings the performance difference in Deep Learning with different data augmentation strategies?

In general, you have to be careful when using data augmentation.

For example, doing rotation for this kind of image makes sense, we expect to see any of these images as potential 'real-life' example :

However, doing rotation for this kind of image is less meaningful. We don't expect to see this in 'real-life' example :

And GAN potentially makes generated image meaningless.
If your GAN produce 'thrash' augmented-data, then your network will train and learn 'thrash', which you don't want.

When you are training your model on GAN generated images, you're actually training your model to recognize GAN-generated image, not real-life example.

Sources:
 - towardsdatascience
 - quora question


Mean estimation for nested location data

I don't know if that is the case, but if some kind of continuity assumptions are realistic, you could try to move away from categorical variables (block) to continuous variables (longitude and latitude). Then, if you have information on two neighboring blocks, you could interpolate those values with say a spline.

Of course, this can also be fitted into a machine learning model with predictors such as average income of blocks with distance < x.
And if you don't have data of nearby blocks, then your state average might be the next best approximation.

Your state level data can serve as a predictor and also as validation.

Also, plotting your data always helps get some kind of intuition.


vgg16 needs less epochs than resnet ,why?

For some reason VGG might be better suited for cifar10 (maybe kernel sizes etc.). Generally speaking, however, this isn't the case. I've trained VGGs much slower than even the largest resnets (i.e. 150 layers).


Machine learning to predict apps (recomendation)

Much depends on the data available to you.  Perhaps you can be more specific about the scale and scope.

Modelling time is the straightforward bit.  To understand how to conceptualise time as useful features, see this excellent answer on Machine learning - features engineering from date/time data

Modelling the user is more complicated.  You will likely not have enough data on each user, but you can build some user models.  (Too few, then the system will make similar predictions for all users, without nuance.  Too many and there will be sparsity, overfitting, and generally the same problems as having no profile models at all, ie one model per actual user.)

This can be done supervised or unsupervised, finding representative clusters.
(Search for user profile categorisation, user models, user model clustering)


Using z-score for neural network normalization

One of the reasons z-scores are useful in neural nets is that they can allow gradient descent to converge faster. I don't think taking these z-scores and scaling them further to be between -1 and 1 will add any extra benefit, as you already have your features expressed on a common scale.


Instead of focusing on the technology term, will provide generalized answer and use the term “containers”.

Containers run only what it supposed to run, assuming anything unknown is untrusted so what only reside in container only for the life of container, so modification of a code in database to test will be same approach in VMs(sandboxing) or containers(docker) and biggest difference is kn resource consumption and time for provisioning VMs vs spinning up containers/pods in couple seconds for application.

  More details:

  Part1_ From application point of view

Containers are very important when does come to data science world from following points:

Minimizing conflicting library versions for different development projects(each project has own container/pod to run).
Consistency across environments and developers against specific design criteria.
Rapid Prototyping with Edge Node On Demand.
Avoiding a need to reinstall everything on new hardware after a refresh or failure.
Portability of your compute environment, develop in your machine and if you need more resources like CPU, RAM or GPU you can port your environment to cloud, powerful clusters, powerful machine, group of containers that can be orchestrated with Kubernetes or Dockerized cluster called Swarm.
Maximize collaboration clarity across groups. Consistent libraries, result sets, etc.
Extending on-premises autonomy/agility to cloud scale and reach.
Project workspaces powered by Docker containers for control over environment configuration. You can install new packages or run command-line scripts directly from the built-in terminal.
Resources used by containers are very few comparing with VM or Bare metal ... you can have multiple containers running in VM or Bare metal so this will greatly reduce license cost for OS and projects/apps will scale based on resources needed and not full resources of the host( here to note: you will run multiple apps/models in a machine when using containers comparing without containers you’ll use full machine resources for 1 single app/model and this is waste of resources. In a cluster environments you can spin up multiple pods/containers(apps/models across the cluster and with zero cost of the pod comparing with VMs you pay for OS of each host and you run all resources to that task).
easy to have new image with new packages installed and share with everyone instead to deal with virtual environments into VMs and have conflict of packages and many more(would like to see virtual environments to be moved from traditional approach to containers, that will be a good improvement and will help a lot for Data scientists to bypass the configuration of each project individually with all conflicts and activate/deactivate then search for requirements when putting in production when with containers all you need is a config file, this is my point of view from what I see daily in Data Science world).
Infrastructure independent platform allows data scientists to run the the analytics applications on the infrastructure best optimized for the applications
Empower data scientists to build models with the tools and packages best suited for the research without worry of application and environment conflicts.
Most important Research Reproducibility: The SCALE provided by containers, you scale easily with containers and will be identical across all environments and you don’t care about the host OS, ensuring repeatability of analysis and research with immutable containers and eliminating issues with different environments
Secure Collaboration: Integrated security in the platform and lifecycle allows for collaboration without risk of tampering and data integrity(more details in Part2&3.

Here comes in play Cloudera with  Cloudera Data Science Workbench  and Anaconda with Anaconda Enterprise both does use containers so can bring quick results for the business and deploy easy the models from dev to QA and ultimately to production.

Why is important last statement? is to have portability from dev to prod without any changes to environments and without cost DevOps part to operationalize.

  Part2_ From security point of view

One notorious security advantage is that the host OS security is separate from the container, meaning that is patched separated, patching host OS wouldn’t impact your containerized application(how many times we have the issue when we do patch the OS and does affect the app  and service in that OS(paths, ports, services etc)?

For example: if you have a malicious library on your app/code than that malware will reside only on that container for the time of container being live, containers does run as endpoint all the time and didn’t saw a case that will spread the malware into network.
monitoring a container node is robust and you can put add-on or service to monitor behaviors of application or node, and will only replicate new node/container only and only based on config file.

Comparing VMs vs containers: With containers is a different story, you take care of OS separate from containers(containers is separate task when security is in place).

Docker security does provide detailed information of major security points.

Docker standards and compliances provides a full list of security compliances and standards available for containers.

"Docker container with a well crafted seccomp profile (which blocks unexpected system calls) provides roughly equivalent security to a hypervisor."

Folder sharing. With containers, you can share a folder by setting up a shared mount, and since the Docker/kernel enforces file permissions that is used by containers, the guest system can't bypass that restrictions. This is very important for Data Science applications and users because in corporate is used most of the time sensitive/restricted data and multiple layers of security or restrictions are in place, one approach to solve this security problem is using VDI or VMs that with AD group in place to restrict/share data access and does become problematic and costly to maintain and allocate resources.

When does come to debugging applications or OS with all services and logs generated I think containers are winning and evolving now to NLP approach: An experimental natural language processing (NLP) utility is also included, for proofreading security narratives.

  Here I will quote Jianing Guo from Google: A properly secured and
  updated VM provides process level isolation that applies to both
  regular applications as well as container workloads, and customers can
  use Linux security modules to further restrict a container’s attack
  surface. For example, Kubernetes, an open source production-grade
  container orchestration system, supports native integration with
  AppArmor, Seccomp and SELinux to impose restrictions on syscalls that
  are exposed to containers. Kubernetes also provides additional tooling
  to further support container isolation. PodSecurityPolicy allows
  customers to impose restriction on what a workload can do or access at
  the Node level. For particularly sensitive workloads that require VM
  level isolation, customers can use taint and toleration to help ensure
  only workloads that trust each other are scheduled on the same VM.

On top of all, kubernetes cluster has following additional security features:

Use Transport Layer Security (TLS) for all API traffic
API Authentication
API Authorization
Easy remediation of security flaws and minim impact on application/environment compared with VMs or dedicated Bare metal.

  Part3_CIS benchmarks for hardening containers: Docker & Kubernetes.

First step making containers more secure is to prepare the host machine that is planed to use for executing containweized workloads. By securing containers host and following infrastructure security best practices would build a solid and secure foundation for executing containerized workloads.
Stay up to date on Docker updates, vulnerabilities in the software.
Have a specific dedicated directory for Docker related files and allocate just enough space to see for containers to run(default path is /var/lib/docker but change to other mounting point and monitor at OS level using auditd or aide services for any changes or size/unwanted workload run, keep the logs and configure according to the needs.

Note: The best part about step 2 is that with VMs you need to monitor way more locations for your data science project(libraries in different location, multiple versions of packages, locations/path for even python, run cron jobs or systemd to ensure some process does run, log all logs etc but with containers is single point for all this jobs to run and monitor only a path instead of multiple ones).

Verify all the time the users into docker group so you prevent unauthorized elevated access into the system(Docker allows the share of directory between the Docker host and a guest container without limiting the access rights of the container) so remove any untrusted users from the docker group and do not creat a mapping of sensitive directories frim the host to container volumes. Here I would say to use a separate user for the installation and specific container tasks "NEVER use root for containers run dedicate a PID only for this task(will have elevated access but will be task based, I use gravitational for the cluster and when installing I NEVER use root).
Audit all Docker daemon activities(be aware that does take space the logs in containerized "world" so prepare separate partition with decent space to hold the logs and required configuration(rotation and period to store the logs).
Audit all Docker files and docker.service , in etc, var and what else is aplicable.
Restrict all inter-container communication, Link specific containers together that are require communication(best will be to create a custom network and only join containers that need to communicate to that custom network). This hardening approach will prevent unintended and unwanted disclosure of information to other containers.
All applications in containerized infrastructure should be configured or at least to have th eoption to Encryp All Sensitive Information(this is very importan for Data Scientist because most of the time we do login to platforms to get data, including sensitive data for the company.
Have option to have Encrypted All Sensitive Infomration in Transit.
Uses only specific approved Ports, Protocols and Services, VMs has more open surface when does come to run an app/project, with containers you specify only what will be used and not wondering for all other ports listening, protocls and services that does run at OS level to protect or monitor, this does minimize the "attack surface".
Sensitive information stored on systems is encrypted at rest and does require secondary authentification mechaninsm , not integrated into operating system, in order to access the information.
Does allow to be enabled Operating System Anti-Exploitation Features/Deploy Anti-Exploit Technologies: such as Data Execution Prevention(DEP) or Address Space Layout Randomization (ASLR).
The best simple security difference between VMs and Containers is: when updating or running a project you don't need elevated access to do it over entire VM or network, you just run as a defined user and if is elevated access then does exist only for the time of the container and not shared across of the host(here does come Data Science libraries install, update, executing projects code etc).

  Part4_ More resources(on top of links incorporated in Part1-3) related to containers for Data Science:

Data Science on docker
Conda, Docker, and Kubernetes: The cloud-native future of data science (sponsored by Anaconda)
https://devblogs.nvidia.com/making-data-science-teams-productive-kubernetes-rapids/
Why Data Scientists Love Kubernetes
A very good book  to understand docker usage for Data Science: Docker for Data Science: Building Scalable and Extensible Data Infrastructure Around the Jupyter Notebook Server.


Service Request classification, questionnaire filling and call logging

He reads this and in another application he logs a call. There he has to fill the request type and a number of questionnaires.

This is a "multi class" classification problem.

You can start with a library that makes it easy to mix text , categorical and numerical attributes.

Text : "issue experienced" attribute   (Library should be able to apply NLP on text )
Categorical : "type of request" and similar attributes
Numerical : "price" and similar attribute (if applicable)

One such library is : Uber Lugwig

If you have data in CSV, it should not take more than a few hours to train the model.

Documentation : https://uber.github.io/ludwig/user_guide/

Introduction : https://hackaday.com/2019/02/25/ludwig-promises-easy-machine-learning-from-uber/


What are the best sites to display your data science skills & projects?

LinkedIn
Github
Personal Website (of course)
Coderbits (not popular, but you can display your profile link on the resume/LinkedIn)
Kaggle


After some study, I figured out the answer and want to share with people if someone also finds it helpful. The loss function of Perceptron is hinge loss or

$J(w) = max(0, -yw^Tx)$.

Adding a constant to the loss function does not change the function value, as it does not change the sign of the decision. In other words

$J_2(w) = max(0, -\alpha yw^Tx) = J(w)$.

If we do gradient descent using $J_2$, we have

$\partial(J_2)/\partial(w) = 0$,  if $J_2 = 0$;

$\partial(J_2)/\partial(w) = -\alpha yx$,  otherwise.

So the update function of gradient descent is

$w_{new} = w_{old} \pm \alpha x$.

As long as $\alpha > 0$, it does not change Perceptron decision in any step. This is why for Perceptron, you only need to set learning rate to be 1.

Specifically answer the question, when people say "the learning rate only scales $w$", they are referring to $J_2(w) = max(0, -\alpha yw^Tx)$ rather than $w_{new} = w_{old} \pm \alpha x$.

A related question I found very helpful is  Normalizing the final weights vector in the upper bound on the Perceptron's convergence


This is because of the loss assumption is the (Mean) Squared Error $\mathcal{L} = (\hat{y} - y)^2$ and the derivative is

$$
\frac{\partial}{\partial \hat{y}} \mathcal{L} = 2 (\hat{y} - y)
$$

which is then passed "backward" for use in the chain rule.


Yes, this is what the Save Model widget does under the Model tab. Create your model, then click save model and your model will be saved to a pickle file. Then just load your pickle file in Python.

Here is a like to a similar question on Stack Overflow:

https://stackoverflow.com/questions/45598627/orange-save-model-in-python-script


Hyperparameter Tuning Time Series in Production

Some ideas:

Number of previous observations to use: depends on the process you are modelling. If the target is related in some way to many of the previous values, you may need to use more data in your tuning process. On the other hand, if the target is largely unrelated to previous observations you can use fewer. You will need to check the relationship - lag plots and auto-correlation plots may be useful.
Frequency of parameter tuning: again, depends on your model and process. It lay be a good idea to monitor the prediction accuracy of your model, and to only perform more parameter tuning if the model accuracy drops below a certain threshold.


Clustering based on partial information?

You may need to treat this as an optimization problem.

Without a formal notion of what should be a cluster, you will not be getting far. A blackbox algorithm will not solve your problem, you need to open the box (understand the algorithm) and adjust it as necessary for your problem.

Once you have this notion, you can try to match it to algorithms. For example, if you can quantify similarity (e.g. if distance times attribute dissimilarity matches your desires) then you can try HAC. Or if you have thresholds (distance at most 100mi, attribute difference at most y) then you can run DBSCAN and merge neigh ors transitively into clusters.

But don't expect anything automatic.


How to represent target variable for chess AI

See the following paper:

Giraffe: Using Deep Reinforcement Learning to Play Chess - Lai


Predicting when component will fail having its parameters data

Looks to me like a sequence labeling problem, where the class is binary indicating whether the component is still working or failed. In this option you should build a training dataset which each cycle which looks like this:

No_of_runs    Para1    Para2    Para3    Para4    Para5    status
1              100      32        45      230       86       ok
2              101      34        65      234       90       ok
3              120      24        32      242       80       ok
4              105      45        40      213       75       ok
5              90       42        54      200       77       ok
...            ...      ...       ...     ...       ...
1234           ..       ..        ..      ..        ..       fail

The order of the instances matters. After training, the model can tell you the probability of failing for an instance given its sequence of runs.

Conditional Random Fields are a standard option for such problems.


There is no valid answer. If you have a stationary process with 0 variance, then the forecast horizon has no limit.

More realistically, you have the follow rule of thumb (which is totally from experience with absolutely no theoretical base): the horizon forecast may be half the historical base. You have 8 data points, so you can forecast 4 points. You have 2 years, so you can forecast 1 year.

The key point is : check this after your forecast is done.


Verifying business ideas based on keywords derived from a description of the business idea

You need keyword extraction technics. Basically they try to find the most important words in the text. There are many types of such methods.


This feels like a bit of a hack, but I was able to infer something from an answer on another question: https://stackoverflow.com/a/46466275/6182971

It works if I change the first line above from:

ones = tf.ones(shape=(1,1))

to:

ones = layers.Lambda(lambda x: tf.ones(shape=(1,1)))(features_input)

Even though the Lambda layer is returning a constant, passing in features_input, which is the main training data connects the tf.ones constant to the network inputs, which seems to be sufficient.


metric performances varies with each training run of my deep learning model

That difference in results could be from high variance, the error from sensitivity to small fluctuations in the training set.

Generally, the best solution to high variance is to increase the size of the training dataset.


Pandas replace column values by condition with averages based on a value in another column

Get a dictionary with average age where experience < age

mapping = df[df['age']>df['experience']].groupby('age').mean().apply(list).to_dict()['experience']

Use this dictionary to replace values where experience > age

df.loc[df['age']<df['experience'],'experience'] = df[df['age']<df['experience']]['age'].map(mapping)


Is There any RNN method used for Object detection

Yes, there have been many attempts, but perhaps the most noteable one is the approach described in the paper of Andrej Karpathy and  Li Fei-Fei where they connect a CNN and RNN in series (CNN over image region + bidirectional RNN + Multimodal RNN) and use this for labeling a scene with a whole sentence. Though, this one is more than just object detection as it leverages a data set of scenes and their descriptions to generate natural language descriptions of new unseen images.

Another example is Ming Liang and Xiaolin Hu's approche where they mix a CNN with an RNN and use this architecture for better object detection. As Ming and Xiaolin explained in their paper (linked above), the RNN is used to improve the CNN:

  A prominent difference is that CNN is typically a feed-forward architecture while in the visual system recurrent connections are abundant. Inspired by this fact, we propose a recurrent CNN (RCNN) for object recognition by incorporating recurrent connections into each convolutional layer.


Classifier with a single 1 value per year

As @Emre noted in the comments, you will want to use the softmax function. After acquiring a set of scores per film, the function will squash the scores into the range [0,1] and the scores will add up to 1.

Here's a possible procedure:

Generate a feature set per image
Train a model on your feature set to output a score for a movie.  The score is a number in an arbitrary domain.
Repeat step 2 for all of your candidate movies to create a score vector.
Feed the score vector into the softmax function to squash the scores into the domain [0,1] and make them add to 1

For reference, the softmax function is defined:

$ \sigma(z_i) = \frac{e^{z_i}}{\displaystyle\sum^N_{n=1}e^{z_n}} $

And an example:

Our feature set will include the following: critic/audience ratings, revenue, cost, total tickets sold, etc.

Let's say you have historical data of the Oscars that includes the feature sets and a score that we will use for the ground truth metric.  You define the score.  For instance, if a movie was not even nominated for the Oscar award, it might have a score of zero, whereas an Oscar winning movie might have a score of 1.  A movie that was nominated but did not receive many votes might have a score of 0.50.

You train a model on your historical data so that, given a movie's feature set, it will output a score, similar to your training set.

Now you are considering three movies for prediction: Avengers Infinity War, Deadpool 2, and Venom.

You acquire their feature sets (the same categories that you used for training: critic/audience ratings, revenue, etc.)

Then you pass each feature set through your model and acquire a vector of scores:

\begin{array}{|l|c|}
\hline
Movie & Score\\
\hline
Avengers\ Infinity\ War & 0.98\\
Deadpool\ 2 & 0.82\\
Venom & 0.24\\
\hline
\end{array}

We can interpret the score results as a probability by using the softmax function:

The denominator of the function is given:

$ \displaystyle\sum^N_{n=1} e^{z_n} = e^{0.98} + e^{0.82} + e^{0.24} \ \ \ where: z=\{0.98,0.82,0.24\} $

We calculate the softmax of a given score $ x $ as such:

$ \Large\frac{e^x}{e^{0.98} + e^{0.82} + e^{0.24}} $

Therefore the softmax scores are:

\begin{array}{|l|c|c|}
\hline
Movie & Score & Softmax\\
\hline
Avengers\ Infinity\ War & 0.98 & 0.42932\\
Deadpool\ 2 & 0.82 & 0.36584\\
Venom & 0.24 & 0.20484\\
\hline
\end{array}

and we can see that:

$ 0.42932 + 0.36584 + 0.20484 = 1 $


Which metric to choose for tracking model performance?

TL;DR

I would suggest you to:

Balance your dataset (since you are not doing an anomaly detection task, that should be fine).
Measure Precision/Recall/F1 Score and argue for them depending on your problem.
Perform K-Fold Cross Validation to compare models

Balancing your dataset

You said you have a 33:67 ratio. Why keep it like that an not just make it 50:50 (e.g. by removing samples from the larger class)? In general this help your models be less biased to the larger class.

Special care is needed when doing anomaly detection, where ratios are usually around 1:99. But since you are not doing that, simple dataset balancing is advisable.

Precision / Recall / F1

F1 score is the default score to go to for classification. It should be ok to use across models as long as you keep your test set the same for all models. An even better thing to do to compare models, is to perform K-Fold Cross Validation on each model. (Sklean docs).

However, you could look at Precision or Recall instead of the F1 Score, but this will depend on the problem you are trying to solve.

  Precision = when I detect an instance of a class, I am sure it is the right class.

  Recall = I detect all instances of a class, even if I detect instances that don't belong.

For example: we are performing a binary classification of whether a patient has a deadly disease or not. Health Insurance: "let's optimize the precision, because we only want to pay treatment for patients that really have the disease". Doctor: "let's optimize for recall, because it is better to treat more people even if the don't have the disease to ensure that those that do have it are likely being taken care of".

If you think that precision are recall are equally important then that is the definition of the F1 score.