Instead of one-hot encoding, can I store the same information in one column using a single value?

I suppose you could do this, but if your goal is simply to store 15 boolean values in a single column you are complicating things unnecessarily. Instead of going to all the trouble to compute the prime factors of the stored value, why don't you just store the flags as a bit string? Your example of 15 different possible values could be stored in a single SMALLINT (2-byte) SQL column. After retrieving the value, you would just need to extract the bits of interest for your record with some basic bitwise arithmetic.


Difference between paragraph2vec and doc2vec

There may be differing implementations, but these two terms refer to the same thing.

Both convert a generic block of text into a vector similarly to how word2vec converts a word to vector. Paragraph vectors don't need to refer to paragraphs as they are traditionally laid out in text. They can theoretically be applied to phrases, sentences, paragraphs, or even larger blocks of text.

Here's one definition of a paragraph vector:

  An unsupervised algorithm that learns fixed-length feature
  representations from variable-length pieces of texts, such as
  sentences, paragraphs, and documents.

And the full paper if you are interested:

https://cs.stanford.edu/~quocle/paragraph_vector.pdf


If you are going to use the ECG time series as the input for the LSTM-network you will need to reshape your data

data = data.reshape(signal length,time step pr sample, ECG channel)

So if your data is 36000 samples long and you only use 1 channel you will do like this:

data = data.reshape(36000,1,1)

Then you can make a simple LSTM model like this:

model = Sequential()
model.add(InputLayer(input_shape=(1, 1)))
model.add(LSTM(32,activation='tanh', return_sequences=True))
model.add(Dense(1, activation='sigmoid'))

I think the same principals will apply to your encoder/decoder


GlobalAveragePooling2D in inception v3 example

Prior to GAP, one would flatten your tensor and then add a few fully connected layers in your model.  The problem is that a bunch of parameters in your model end up being attributed to the dense layers and could potentially lead to overfitting.  A natural solution was to add dropout to help regulate that.

However a few years ago, the idea of having a Global Average Pooling came into play.  GAP can be viewed as alternative to the whole flatten FC Dropout paradigm.  GAP helps prevent overfitting by doing an extreme form of reduction.  Given a H X W X D tensor, GAP will average the H X W features into a single number and reduce the tensor into a 1 X 1 X D tensor.

The original paper simply applied GAP and then a softmax.  However, it's now common to have GAP followed by a FC layer.


How to interpret the Mean squared error value in a regression model?

Whether you model is performing well or not depends on your business case, you might hive tiny RMSE or great looking score on whatever metric you are using, but it just not enough to solve the business problem, in that case the model is not performing well.

MSE is just that Mean Squared Error

Both MSE and RMSE measure by how much the predicted result deviates from actual, because of the squared term more weight is given to larger errors, and because of square root in RMSE, it is in the same units as dependent variable. MAE, Mean Absolute Error is another useful metric to look at when you are evaluating a regression model; it is also easier to interpret.

Given your data, R-squared seems fine to me.


Why does an SVM model store the support vectors, and not just the separating hyperplane?

The hyperplane is a linear combination of the support vectors. In the soft margin case, there is only a limited amount of slack; every input does not get to be support vector. In the nonlinear case, the separating hypersurface may be embedded in an infinite-dimensional space, making it impossible to store. To borrow from the Wikipedia article, the normal vector $w$ is given by

$$w = \sum_i c_i y_i \phi(x_i)$$

where $\phi$ is the feature embedding function, and $c_i$ is a Lagrangian dual variable that is zero for points on the correct side of the margin. Instead, test points are classified through a kernel function $k(x_i,x_j) = \left< \phi(x_i), \phi(x_j) \right>$ like so:

$$x \to \mathrm{sgn}(\left<w , \phi(x)\right> + b) \equiv \mathrm{sgn} \left( b+\sum_i c_i y_i k(x_i, x)\right)$$

Notice how we avoided explicitly calculating $w$.


Reward function to avoid illegal actions, minimize legal action and learn to win - Reinforcement Learning

Do you want your agent to perform illegal actions at any given time or not at all? One way to avoid illegal actions is to use a mask over your 144 action vector (so the action indices are preserved) when you calculate the softmax probabilities. Then you sample actions according to the masked probabilities. First you need to determine which actions are illegal at a given step which should depend on the dynamics of the game. One example are the algorithms used to learn to play the mini-missions in StarCraft II.

In order to enforce your agent to act with the minimum amount of steps you should introduce a small penalty (lets say for example -0.01) so your agent will try to optimize also this part of the reward.

For winning the game question, there is no answer to that. This depends on many factors, I name few here: how appropriate is the agent's architecture given the game, reward sparsity, types of observations (images, non-vector data), data p reprocessing, inputs to your agent, amount of exploration etc. There is no guarantee that if you use the X type of method will solve your task (except if your task has already been solved with a specific type of learner). You could possibly name the game/task you want to solve.


Human readable format for clusters of word vectors

The usual way is to present the top N (e.g. top 10) words for the cluster:

With distance-based clustering like K-means, the top words can be picked as the closest ones to the centroid.
With probabilistic methods such as LDA, the top words are the ones with the highest probability for the topic.


Best practice to test (unit and integration) a neural network implementation

Make your code testable. You should be able to write plenty of tests for individual components without having to mock up an entire training pipeline for each one. A "property-based testing" library like Hypothesis can be helpful for this.
Train a standard model on a standard dataset and make sure the accuracy numbers are similar.
See here for an in-depth discussion, with more and better advice than I've been able to provide.


What does the limit of xgboost max_depth=1 represent?

A decision tree model is a non-linear mapping from x to y where XGBoost (or LightGBM) is a level-wise decision-tree ensembling algorithm, so your model will still be nonlinear with max_depth = 1. But max_depth = 1 will most probably block your algorithm from your model getting complex enough to capture complex patterns from the data, since you permit only for one split of the tree. Max_depth is used for preventing overfitting, to avoid tree growing very deep.

I suggest you looking at XGBoost documentation, since tree parameters have a strong correlation with each others upon hyperparameter tuning in the way: https://xgboost.readthedocs.io/en/latest/parameter.html


Display images after augmentation in Keras

Depending on the kind of data set you are using you can use .flow (if you have data as numpy arrays) or .flow_from_directory (if you have images in file system) to run through the data generator and save the output using save_to_dir argument.


How to use weights from our data set properly? Should we use them at all?

Firstly, if a variable is imbalanced (Race in your case), you shouldn't worry about. As its a categorical variable, it will be converted to two or more dummy variables.

Secondly, if the weight represents the number of people for each entry, as the following example, you should include the weights.

hypothetical data

Race       Education    Marital    weight
------------------------------------------
White       tertiary    Married      10
White       tertiary    Single        6
Black       tertiary    Married       8
White       secondary   Single        4
...
...

For this example, I would oversample each entry by its weight. Because your real database is:

          X               y
---------------------  --------
Race        Education   Marital
-----------------------------------
White       tertiary    Married
White       tertiary    Married
White       tertiary    Married   10 records
...
White       tertiary    Married

White       tertiary    Single
White       tertiary    Single     6 records
...
White       tertiary    Single
Black       tertiary    Married
Black       tertiary    Married    8 records
...
Black       tertiary    Married
White       secondary   Single
White       secondary   Single
White       secondary   Single
White       secondary   Single

However, I should see few records of your dataset to help you better.


The outputs are the "cell state" which is transferred only to the next LSTM cell and the "hidden state" which goes to the next cell and also as an output of the layer (in the case LSTM should return sequences).


What types of features are used in a large-scale click-through rate prediction problem?

That really is a nice question, although once you're Facebook or Google etc., you have the opposite problem: how to reduce the number of features from many billions, to let's say, a billion or so.

There really are billions of features out there.

Imagine, that in your feature vector you have billions of possible phrases that the user could type in into search engine. Or, that you have billions of web sites a user could visit. Or millions of locations from which a user could log in to the system. Or billions of mail accounts a user could send mails to or receive mails from.

Or, to swich a bit to social networking site-like problem. Imagine that in your feature vector you have billions of users which a particular user could either know or be in some degree of separation from. You can add billions of links that user could post in his SNS feed, or millions of pages a user could 'like' (or do whatever the SNS allows him to do).

Similar problems may be found in many domains from voice and image recognition, to various branches of biology, chemistry etc. I like your question, because it's a good starting point to dive into the problems of dealing with the abundance of features. Good luck in exploring this area!

UPDATE due to your comment:

Using features other than binary is just one step further in imagining things. You could somehow cluster the searches, and count frequencies of searches for a particular cluster.

In a SNS setting you could build a vector of relations between users defined as degree of separation instead of a mere binary feature of being or not being friends.

Imagine logs that global corporations are holding on millions of their users. There's a whole lot of stuff that can be measured in a more detailed way than binary.

Things become even more complicated once we're considering an online setting. In such a case you do not have time for complicated computations and you're often left with binary features since they are cheaper.

And no, I am not saying, that the problem becomes tractable once it's reduced to a magical number of billion features. I am only saying that a billion of features is something you may end up after a lot of effort in reducing the number of dimensions.


How to calculate mean and standard deviation of all features in a class identified by k-nearest neighbors?

If you append the predicted neighbourhood onto your data df (let's call this neighbourhood), then using groupby and transform within a loop should do the trick.

As an example:

features = [var_1,var_2,...] # a list of the features to run over
for col in features:
  df[col+'_mean'] = df.groupby('neighbourhood')[col].transform('mean')


(Serious) Dataset of paedophilic Youtube comments (or similar)?

So your question is asking how to go about making a model which predicts whether the YouTube comment is paedophilic given the text provided in the YouTube comment itself.
Before I provide an answer, it is worth considering the ethical applications of this model and what this will be used for because models do implicitly cause bias in all stages of model production, from data collection to model implementation. So before taking this task on, I urge to read this paper on model ethics (https://arxiv.org/abs/1901.10002)
Firstly, you obviously need data/YouTube comments. The most important part to consider is labelling. Now, this stage is particularly important as you want to get multiple people involved with labelling these comments as (non-)paedophilic.
In this step, ensure that you use comments which are have a higher inter-labeller agreement.
For the model, something like a RNN/LSTM with a many-to-one configuration with a final 2 node softmax layer after the sequential architecture has been fed all of the text input. Again, before you start such an endeavour, do be aware of the ethical consequences of this model.


General speaking, you can assign the cluster to the new value using the method of that clustering method. For example, if you are using the DBSACAN algorithm, you can assign the cluster to the new value, if it is reachable from one core points of a cluster (in a tie situation, we can assign the more density cluster or many other methods).


Dealing with correlated features when calculating permutation importance

Using mutual information namely the correlation of each feature and the output is not something that can be helpful very much. The reason is that the correlation coefficient is just capable of finding whether they are linearly dependent or not. Other than Gaussian distribution, as far as I know, it is not even capable of finding out whether the inputs are independent or not which means if you see the coefficient is equal to zero, if you don't know the distribution you cannot conclude that they are independent but you can say they are not linearly dependent. In real-world applications, it scarcely happens that features just have linear dependencies with the output. Consequently, approaches like this are not very helpful because you may have two features that can have a better relationship with the output but none of them has a good separation with the output.

For cases that you have a lot of features, you can decide to choose other feature selection and extraction methods. I guess for your case wrapper method may suit better. In the wrapper method, you don't have any criterion and you just search through all possibilities of features using heuristic-based methods in order to find the best sub-features which have the best cross-validation accuracy.


Why does Lasso behave "erratically" when the number of features is greater than the number of training instances?

When p > n, the LASSO model can only sustain up to n variables (this can be proven using linear algebra, the rank of the data matrix in particular), leaving at least p - n variables out (some that might be predictive, consider a model where you use LASSO and all your variables are predictive). This is not necessairly a bad thing though, depending on your use case.

For the colinearity problem, if two variables are highly correlated (that is, they contain the same "information" in some sense) then LASSO will tend to pick one of the two randomly and drop the other one. Again, might not be an issue but can lead to "erratic" behavior in which the variables selected change quite a bit depending on the training set.

In general, I find LASSO to choose different sets of variables that change quite a bit depending on the training set, especially for small datasets and even worse when p > n. Again, not necessarily a problem but you might look to not using explicit feature selection and instead opting for ridge regression. Alternatively, the elastic net is a balance between the two methods.


How to generate 12 independent random weights which all add up to one

I am afraid I do not know the software you mention, but I can show you the principles and suggest why maybe you are getting negative numbers.

I will do this in Python, making use of the numerical library, numpy.

I import numpy and generate 12 random integers between 0 and 9 (10 is excluded as upper limit):

In [1]: import numpy as np; samples = np.random.randint(0, 10, 12)

In [2]: samples
Out[2]: array([8, 5, 8, 4, 4, 7, 2, 2, 0, 5, 9, 1])

To scale the values to a range that makes their sum equal to 1, we can do the following. First sum up all values:

In [3]: total = np.sum(samples)

Now simply divide each value by the sum (the division happens here individually for each element of samples:

In [4]: normalised = samples/total

In [5]: normalised
Out[5]:
array([0.14545455, 0.09090909, 0.14545455, 0.07272727, 0.07272727,
       0.12727273, 0.03636364, 0.03636364, 0.        , 0.09090909,
       0.16363636, 0.01818182])

We can see that the result does indeed sum to 1:

In [6]: np.sum(normalised)
Out[6]: 1.0

What you may have is a set of samples that contains some negative numbers, like the following in samples_neg, with ten integers ranging from -5 to +9:

In [7]: samples_neg = np.random.randint(-5, 10, 10)

In [8]: samples_neg
Out[8]: array([ 4,  7,  0,  4, -3,  0,  9, -3,  9,  3])

We can follow the same recipe as before, summing the values and dividing each value by the sum:

In [9]: total_neg = np.sum(samples_neg)

In [10]: normalised_neg = samples_neg / total_neg

We see that the result this time includes negative values, as you mentioned:

In [11]: normalised_neg
Out[11]:
array([ 0.13333333,  0.23333333,  0.        ,  0.13333333, -0.1       ,
        0.        ,  0.3       , -0.1       ,  0.3       ,  0.1       ])

However, this does still satisfy the constraint you originally had, which was that they sum to 1:

In [12]: np.sum(normalised_neg)
Out[12]: 0.9999999999999999        # this is 1, within rounding errors of floating point values

A suggestion would be to first normalise the values in a range of [0, 1] and afterwards, re-weight the values such that their sum is 1.


In this paper, theauthors have provided a deep explanation, basically its the exact same thing. sub-pixel uses or^2 conv kernels all at res H/r,W/r while the regular transpose conv uses o conv kernels all at res H,W the authors prove that one can rearrange the feature map (Or^2,h,w) to a feature map (O,hr,wr) without losing information.


There are traditional ways of creating a risk scorecard using linear regression techniques. It's a whole topic in itself. A good book for beginners to read in-depth about it would be this.

For sure, you can also tackle it as three-class classification problem if you have labelled data for that. Or consider it as binary classification problem and create three buckets out of class probability output by the classifier.


The most common approach is a combination of these two strategies:

Domain expertise - Given knowledge about the dataset and the goal of the model, choose the order that best manipulates the data to accomplish the goal of the project.
Empirical evidence - Permutation the order and benchmark results. Pick the permutation that has the highest performance on the benchmarks.


Perplexity is a measurement of how well a probability model predicts a sample.
Intuitively, perplexity can be understood as a measure of uncertainty. Say the real thing you want to predict is the sequence of numbers from one to six. If you predict each number in turn with a six-sided die, you will be right about one-sixth of the time. The die's perplexity with this data (and really with any data it could possibly predict - dice are dumb) is six. The perplexity of whatever you're evaluating, on the data you're evaluating it on, sort of tells you "this thing is right about as often as an x-sided die would be."
Wikipedia defines perplexity as:

“a measurement of how well a probability distribution or probability model predicts a sample."

What is perplexity? from stats.exchange.com by Henry

It gives the perplexity of a discrete distribution as
$$2^{-\sum_x p(x)\log_2 p(x)}$$
which could also be written as
$$\exp\left({\sum_x p(x)\log_e \frac{1}{p(x)}}\right)$$
i.e. as a weighted geometric average of the inverses of the probabilities. For a continuous distribution, the sum would turn into an integral.
The article also gives a way of estimating perplexity for a model using $N$ pieces of test data
$$2^{-\sum_{i=1}^N \frac{1}{N} \log_2 q(x_i)}$$
which could also be written
$$\exp\left(\frac{{\sum_{i=1}^N \log_e \left(\dfrac{1}{q(x_i)}\right)}}{N}\right) \text{ or } \sqrt[N]{\prod_{i=1}^N  \frac{1}{q(x_i)}}$$
or in a variety of other ways, and this should make it even clearer where "log-average inverse probability" comes from.

Bits-per-character and bits-per-word
Bits-per-character (BPC) is another metric often reported for recent language models. It measures exactly the quantity that it is named after the average number of bits needed to encode on character. This leads to revisiting Shannon’s explanation of entropy of a language:
“if the language is translated into binary digits (0 or 1) in the most efficient way, the entropy is the average number of binary digits required per letter of the original language."
By this definition, entropy is the average number of BPC. The reason that some language models report both cross-entropy loss and BPC is purely technical.

In the case of Alex Graves' papers, the aim of the model is to approximate the probability distribution of the next character given past characters. At each time step $t$, let's call this (approximate) distribution $\hat{P}_t$ and let $P_t$ be the true distribution. These discrete probability distributions can be represented by a vector of size $n$, where n is the number of possible characters in your alphabet.
So the BPC or average cross-entropy can be calculated as follows:
\begin{align}
bpc(string) = \frac{1}{T}\sum_{t=1}^T H(P_t, \hat{P}_t) &= -\frac{1}{T}\sum_{t=1}^T \sum_{c=1}^n P_t(c) \log_2 \hat{P}_t(c), \\
& = -\frac{1}{T}\sum_{t=1}^T \log_2 \hat{P}_t(x_t).
\end{align}
Where $T$ is the length of your input string.
The equality in the second line comes from the fact that the true distribution $P_t$ is zero everywhere except at the index corresponding to the true character $x_t$ in the input string at location $t$.

How to compute bits per character (BPC)?, from stats.exchange.com by user6952886


Memory allocating error with coxph model

If you're using a 64-bit version of R on Windows (I'm not sure about other operating systems), then you can allocate additional memory to R with memory.limit function.

memory.limit(size = 25000) ## limit is set based on Mb

32-bit version (depending on the system, whether it is 32-bit or 64-bit) is limited to
2 to 4Gb.

You can further read about memory limits in R.

All that said, I would take a step back beforehand to make sure I actually need to deal with all that data and would investigate other alternatives.


From the description of the problem, you can just choose whatever classification algorithm you like to classify the need of intervention at  $t_{0}$.

The general rule for choosing a classifier for this is to start simple--e.g., by using nearest neighbor and iterate to more powerful classifiers until you get enough accuracy.

If additional measurements at $t_{0}$ can be made a classifier can be used to determine the necessity of intervention without this additional data, then based on the probability output of this classifier and a threshold the additional measurements can be triggered. Based on this measurements another probability can be output which hopefully uses more informative features.

If you care about $\Delta t$, you should think about wether there is a real time dependency. Does the need of intervention depend upon the series of classifications? Is the order of determining interventions as necessary important for you?

In my opinion, you do not have to care about ordering for this application.
Rather, I would suggest pooling classifications for one instance according to the most frequent label or something like that.


Relatively high regularization parameters for XGBoost model only way to prevent overfitting

I support your "proof is in the pudding" sentiment.
Some of those hyperparameters are not that extreme, in my experience.  Boosted trees very often perform best with weak individual learners; your max_depth is right in line with what I'm used to seeing as best.  The score regularization penalties (alpha, lambda) don't play as important a role in my experience, but I'm used to seeing optimal parameters chosen in the high double-digits.  Your subsampling and column subsetting rates also seem reasonable, if on the lower end of what I've generally seen as being optimal.  Your gamma is quite high, but that doesn't mean something is wrong; perhaps if you shrink the max depth a bit you could relax the gamma regularization, but I don't think that's in any way necessary.
One possible explanation for this situation: your data is relatively linear and without interactions, so that xgboost doesn't get its main benefits.  And your data is noisy enough that, lacking those nonlinear trends, xgboost ends up fitting to noise readily unless you strongly regularize it.


Assuming you are doing feature scaling: method to limit the range of variables so that they can be compared on common grounds. I'm not sure about what you are looking here. If the matrix is small, you can densify it with X.toarray(). If the matrix is large, then this will probably blow your RAM. Remember that: PCA(X) is SVD(X-mean(X)). Even If X is a sparse matrix, X-mean(X) is always a dense matrix.


AUC is high but not able to represent other class properly

It does not make sense to make two separate AUC / AUROC curves for each class, these are aggregating functions, same as F1-score. If AUC is high but F-score is not then it's possible you have an imbalanced data set (classes are not equally represented in the data), since the AUC will not be able to measure this, while the F-score does. So, if you do have an imbalanced data set you should use something other then AU(RO)C, such as the F-score.


Is there a way to force a transformer to return a pandas dataframe?

Might be late but for anyone with the same question the answers (as almost everything with Scikit-lear) is the usage of Pipelines
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import FunctionTransformer
from sklearn.pipeline import Pipeline
import pandas as pd

df = pd.DataFrame(dict(
    x=[1, 2, np.nan],
    y=[2, np.nan, 0]
))

imputer = Pipeline([("imputer", SimpleImputer()),
                    ("pandarizer",FunctionTransformer(lambda x: pd.DataFrame(x, columns = ["x", "y"])))])

imputer.fit_transform(df)


Can I apply Hidden Markov Models this way?

This is a reasonable application for a hidden markov model only if you have reason to believe that the transactions have the markov property - that the next state depends only on the present state, and not on any of the previous ones. This often makes sense for models in time series (like in the weather example, in which the fact that it's already raining makes it more likely that in the next time step it will still be raining). Of course, the model can be useful even if it's not a perfect fit.

The way you've formulated the model is saying that the hidden state (suspicious/non-suspicious) of an ATM may be predictive of the emitted, or observed states (location, time, and amount of money). At least for the first two, this feels a bit off, particularly if there are multiple ATMs involved, as is suggested by the presence of a location variable. I'll try to explain what I think the way you've formulated the model suggests in terms of observed variables and how it might be nonsensical, and suggest some alternatives.

First, this seems to suggest that suspicious/non-suspicious is a shared state of all ATMs, and that that state somehow influences where an ATM is being used (each hidden state has a particular distribution from the location is drawn). It may make more sense to have a separate model for each ATM location. Maybe this is already what you were intending. If that is the case, location wouldn't really be an observed variable. Instead of thinking of the transactions switching between states, which doesn't seem to make much sense, as you say, perhaps you want to be thinking of the ATMs switching states.

Second, it's not clear how time would be a random variable that would be emitted based on a state of suspiciousness. Instead, time would probably make more sense as the way of ordering the observations you feed into the model.

If you do want to combine all transactions into a single model, maybe an SVM would be more straightforward as a binary classifier.


Confusion matrix of UNET image sgemenation model

If your masks have only one channel and in the case of binary segmentation, you can easily compute a confusion matrix from one image thanks to:

TP: ground truth and predicted pixel are of class 1 (object)
TN: ground truth and predicted pixel are of class 0 (background)
FP: ground truth pixel=0 while predicted pixel=1
FN: ground truth pixel=1 while predicted pixel=0

On a dataset, you can get the confusion matrix from chosen iou thresholds.
For example if your iou threshold is 0.5:

TP += 1 for each image where iou(ground_truth, predicted) >= 0.5
FP += 1 for each image where iou(ground_truth, predicted) < 0.5 and predicted != empty
FN += 1 for each image where iou(ground_truth, predicted) < 0.5 and gt != empty

It is just a suggestion, there is maybe better ways to deal with that.


This is a tough question to answer without more information. I'm going to assume that this is for model building, but without more detail it's hard to recommend something.

However, there are some things which should generally be known:

Population size

How large is the population? Does your 2TB of data comprise the total population, or is this a sample of a given timeframe? What frame of data are you looking at - is this 2 days worth of data that is only representative for a given subset of the population, or is this everything? You'll need to know this to know what conclusions you can draw from this dataset.

Variance

What's the variance of the sample? If it's categorical data, how many unique values are there? Having a metric around this will help determine the number of samples you'll need. If this is a low variance set, you may only need a few hundred/thousand observations.

Stratification/grouping

Is your data grouped in a meaningful way? If so, you'll need to factor this into your sample. Depending on what you're doing, you'll want a meaningful representation of the population. If the data is not grouped, but has distinct groups within it that you care about, you may need to stratify or pre-process your data.

Model and goals

All of this ends up coming down to what you're trying to do. If you're trying to classify or parse a set of unique entities, you may be better off streaming a large set of your data rather than trying to sample it. If you're trying to classify images or customers based on behavior, you may only need a small subset depending on how these groups differ.


Question about the Silhouette number for K means clustering

These are only heuristics.

The are not very reliable, and often fail. They will not detect if you did not preprocess your data well.

So instead of looking at some number, look at your data instead.


How mean and deviation come out with MNIST dataset?

mean : It is the mean of all pixel values in the dataset ( 60000 × 28 × 28 ). This mean is calculated over the whole dataset.
deviation : It is the standard deviation of all pixel values. The dataset is treated as a population rather than a sample.

  What are the uses of these values?

Mean and standard deviation are commonly used to standardize the data in this case the images. Standardized data has mean close to 0 and standard deviation close to 1. You can read more here.

  Why to standardize the data?

Standardization transforms your data in such a manner that it has unit variance.
According to Wikipedia,

  In statistics, the standard score is the signed number of standard
  deviations by which the value of an observation or data point is above
  the mean value of what is being observed or measured


An error coming out when fitting a model with keras

You are missing a flatten() layer between your last MaxPooling2D and Dense layer.


1) No. You can standardize data, but you don‘t need to. Often you don‘t want this because you like to see the effect of some $x$ on $y$ in „real units“ (not standard deviations).

2) In case you standardize $X$, the coefficients must of course change as the X‘s are now no longer real units but standard deviations. The advantage here is that you can compare coefficients in terms of the magnitude or importance (larger absolute value, more important). This is sometimes called „beta coefficients“ or „standardized coefficients".

The model fit may be a little better with standardization, but the main restriction in OLS is the linear model structure. You can relax this assumption by using "Generalized Additive Models" GAM or introducing polynomials.

3) I don‘t understand your question here and there is no code, but it is likely that you did something wrong. OLS like $y=\beta X + u$ simply requires solving

$$ \hat{\beta}=(X'X)^{-1}X'y.$$

Never drop variables due to high p-value (it summarizes confidence bands you could say). A variable may still be useful and important for the model even with high p-value. If you want to do feature/variable selection, use "shrinkage" via Lasso.


How to use a different model to deep neural network with reinforcement learning based on DQN?

Well, if you remove the DNN, I would not call that a Deep Q-Network anymore.. but it is definitely possible to remove that and still consider the approach as Reinforcement Learning.

Actually, the function of the deep neural network is just to approximate the Q-value function. The DNN is just a function in the form $Q_\theta(s, a)$ and based on samples, you adjust $\theta$ to minimize the error.

In practice, you could use any function you want. Obviously, some functions will work better than others. There are simpler approaches based on linear regression, least squares, kernel-based approaches, etc.

With respect to unsupervised learning.. mm. I guess you could force it somehow, but keep in mind that it would not totally make sense given that RL already provides you feedback so it makes sense that you use it.

Some interested advantage of neural networks is that they can find common elements in $\phi$ and rapidly abstract away some noise that if is not relevant with respect to predicting the Q-value of certain state/action.

I noticed that you would like to use something like random forests, etc. There is nothing to stop you from doing that. In particular, random forests are not ideal because they don't deal well with non-stationary. Instead they require bit batches of data. However, this does not mean, it is impossible.

In DQN, after you sample from the environment, you obtain feedback from the environment in the form of rewards associated with state/action pairs. RL combines this information with previous predictions using TD (there are some variations like TD($\lambda$), etc.). What TD gives you is a better estimate of $Q(s,a)$ and your predictor (in this case, your random forest) should learn that when the input is $(s,a)$ the correct answer is $Q(s,a)$.

It is likely that you've seen the same inputs before (specially for initial states) but because TD improves the reference value, you would need to drop these previous values and replace them with the most recent ones. Random forest would need to get this whole batch again.

In general, I accept that while both cases are supervised learning, adjusting DQN to work with non-parametric methods such as random forests can be inconvenient.

If I had to guess based on the comments, I think your main question circles around the distinction between parametric and non-parametric supervised learning. In that case, it is fair to say that most non-parametric methods are not easy to combine with DQN.


How to interpret shapley force plot for feature importance?

I've never practiced this package myself, but I've read a few analyses based on SHAP, so here's what I can say:

A day_2_balance of 532 contributes to increase the predicted output. In this area, such a value of day_2_balance would let to higher predictions.
The axis scale represents the predicted output value scale. The actually predicted value is in bold font (-2.98). I don't know if the min and max values of the scale represent true min and max of the model predicted values. They more likely result from automatic scaling of the plot.
Base value should be the mean value of the estimator over the whole input space.
Those indicate the other features of which value influence the prediction to a higher value. Features are sorted by local importance, so those are features that have lower influence than those visible.
Yes, but only locally. On some other locations, you could have other contributions
higher/lower is a caption. It indicates if each feature value influences the prediction to a higher or lower output value.

From the example plot, you can draw the following interpretation: "sample n°4100 is predicted to be -2.92, which is much lower than the average predicted value (~0.643), mostly due to the specific values of features PEEP_min(5), Fi02_100_max (50), etc., and although day_2_balance is 532".


What does it mean when the shape of input images is (600,64,64,3)?

600 can be  the number of  images of shape 64x64x3 and not  only one image.


Stackoverflow API Structure data storage

What do you want to define as your document? You could define a document as a single question, each question-answer pair, or a question with all its answers. How you define a document depends on why you want to cluster documents and how you plan to use this information. From reading your question, my guess is that you want to define a question with all its answer as a document.

Next, it sounds like you want to apply bag of words techniques to extract features (e.g. tf-idf). If so, you would first want to concatenate the question title, question body, and each answer body into one string object. Then I would suggest using a Python package like scikit learn or nltk to do the other preprocessing steps. For example in scikit learn you can then apply TfidfVectorizer to convert a string to a vector.

This is unrelated to your question, but it may be easier to use the data explorer instead of the StackOverflow API. The data explorer allows you to download questions and answers in a large batch and does not have a quota. Unless you need to collect data in real time, the data explorer may be more efficient. It is updated every week and you can use SQL queries to output data tables as csv.


Non-parametric approach to healthcare dataset?

They are not specifically referring to a plot based approach.  They are referring to a class of methods that must be employed when the data is not normal enough or not well-powered enough to use regular statistics.

Parametric and nonparametric are two broad classifications of statistical procedures with loose definitions separating them:

Parametric tests usually assume that the data are approximately normally distributed.
Nonparametric tests do not rely on a normally distributed data assumption.
Using parametric statistics on non-normal data could lead to incorrect results.
If you are not sure that your data is normal enough or that your sample size is big enough (n < 30), use nonparametric procedures rather than parametric procedures.
Nonparametric procedures generally have less power for the same sample
size than the corresponding parametric procedure if the data truly are normal.

Take a look at some examples of parametric and analogous nonparametric tests from Tanya Hoskin's Demystifying Summary:

Here are some summary references:

Another general table with some different information
Nonparametric Statistics
All of Nonparametric Statistics, by Larry Wasserman
R tutorial
Nonparametric Econometrics with Python


Are single input single output neural networks possible?

You have saturation of the weights which results in constant predictions.
There are several ways to fix that:

Replace hardlim with a standard activation function like ReLU or sigmoid
Pick a different random initialization
Normalize inputs


There's nothing out of the box that will do true online learning.  In order for a scikit-learn algorithm to support online learning it must provide the partial_fit function, which RandomForestClassifier does not.  I think the code you have given will just refit the entire forest on the subset of data it is currently looking at.

One alternative you could try is to initialize RandomForestClassifier with warm_start set to True.  Then on each subsequent call to fit, the random forest will add a new set of trees to the ensemble.  You may have to reduce the number of estimators in the ensemble because the number of estimators you end up with will be the number you initialized with times the number of chunks you loop over.

This won't behave exactly the same way as if you were to train the entire ensemble in a truly online way, but it may be close enough for your purposes.


I don't think it's trying to impute across rows; rather, it's trying to impute in the dates column.  You may want to use ColumnTransformer to select which columns to impute.


Generative adversarial networks for multiple distribution noise removal

The task is denoising, so I think you can go with many-to-one mapping between distribution, and then I would probably try changes in generatr architecture if it won't learn properly.
With multiple models, you'd have to classify the type of noise to select correct trained model for particular noise distribution.

And about the CycleGAN...

CycleGAN is for unpaired images. It tries to learn generally more complex problem, because it does not require paired images.

Since you have paired images, you should utilize that fact and go for pix2pix which needs paired images. Also, CycleGAN mas 2 generators and 2 discriminators, so you have bigger model and thus you need more computational power.


How to use a NN architecture that is too big for GPU?

I suggest a method similar to what @ignatius offered.
Since you don't need to train the first model and only the second one you could do the following:

Use the first model over the entire dataset and save the extracted results in your memory.
Train the second model using the results from the previous step.

This way you will only have to load one model at a given time.
In addition this will make the training of the second model to be faster.
Edit 1
(the op has specified he does not have enough memory for the inference results)
In this case I can suggest to parallelize between the inference of the first model and the training of the second model.
This could be achieved through prefatching.
Let's denote the original dataset as data1 and the corresponding inference results from the first model as data2.
Let's also denote the first model as M1 and the second model as M2.
Thus you can do the following:

load M1 to cpu, and M2 to gpu.
Draw a batch from data1 and use M1 to prepare a batch of data2 for M2.
Use the gpu to train M2 on the batch received from the previous step
while step 3 is being executed, prepare the next batch of data2 by executing step 2.

In this way, while M2 is being trained on a batch, the next batch is being prepared. This is called prefetching the next batch.
This will parallelize the training of M2 and the making of data2.
The method i described above has a very simple way to implement using the tensorflow "Data" api.
Here is a code example:
import numpy as np
from tensorflow.keras import backend as K
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model
from tensorflow.data import Dataset

with K.device('cpu0'):
    input = Input((10,))
    x = Dense(100)(input)
    m1 = Model(input_shape=input, outputs=x)

with K.device('gpu0'):
    input = Input((100,))
    x = Dense(5)(input)
    m2 = Model(input_shape=input, outputs=x)

features = np.random.rand(10, 2000)
labels = np.random.randint(2, size=2000)
dataset = Dataset.from_tensor_slices((features, labels))

def preprocess_sample(features, label):
    inference = m2.predict(features, batch_size=1)
    return inference, label

dataset = dataset.shuffle(2000).repeat().map(preprocess_sample).batch(32)
dataset = dataset.prefetch(1)

m2.fit(dataset, epochs=100, steps_per_epoch=2000//32)


Is there a general guideline for experience replay size, and how to store?

If yes, how can I go about this without that much RAM?

Your value of 56GB seems correct to me, assuming you include a multiplier of 4 for the "4 frames per state representation" used in the DQN/Atari paper. However you should note that in the original paper, the images were converted to greyscale and downsampled to 110×84 prior to use in state representation. This made them 16 times smaller than the frames in your problem, so the whole data set would fit into 4GB.

If you must use larger images, you could store them on disk - maybe in a database - and load on demand. That may unfortunately create an I/O bottleneck that slows learning, although you should still be able to work with it. You can parallelise fetching from the database for mini-batches with the learning process, and this is similar to the mini-batch generators used for things like ImageNet training. You can also work on improving disk performance using optimisations such as parallel disk arrays or SSDs.

You could also pre-process the frames using a hidden layer embedding from a generic computer vision network trained on e.g. ImageNet, and store that representation, not the raw pixel values. This may limit self-discovery of important low-level features by the learning agent, but then again it may still be worth a shot, as lower-level image features are often very similar across different problems, and transfer learning in e.g. image classification has been quite successful using that approach.

More likely practical answers used by RL researchers at least up to a certain scale is one of:

Store less states in replay memory. Base the size of replay on the memory you have available. Yes this may compromise the learning, but there is no special magic about the number 50,000 and if you are optimising resource use you may have to decide between how efficiently a system learns with 10,000 fast replay memory size or 50,000 slower I/O-based replay memory size.
Buy more memory. The big name research labs working in deep RL are funded well enough that they can afford to throw money at this problem. One estimate for how much AlphaGo Zero hardware cost is $25million, so you can imagine that loading a few machines with 128GB+ RAM if they thought it necessary for any reason on any other flagship project would not be a major blocker.

If you look at what OpenAI are doing at the cutting edge of video game playing, you can see that their hardware setup is equally monstrous. It is not clear whether they have an issue with storing experience as they use a different algorithm, or needing RAM in general, but if they did, it is also clear they could quite happily finance maximum RAM on their training rigs.

Do note I am not a RL researcher myself (just a hobbyist) and have not asked anyone else what they would do when faced with this problem.


Is there any way to calculate a relevance score between a title and the content of a text?

Word2Vec and cosine similarity are certainly a plausible solution. Note that the longer the text is, the more function words are in the text, therefore the average Word2Vec vectors are more similar to each other with the growing text length. This can be partially solved by removing the stop-words.

Another method to try might be cosine similarity with representation from contextual embeddings like ELMo or BERT. Also if you have some training examples (small hundreds might be enough) training a classifier based on BERT should work.

Finally, the fanciest solution is using the summarization model. Generating a title from an article content is one of the tasks that is done in summarization literature. If your data is similar to datasets that are used for training summarization models (or you have enough domain-specific data to train your own summarization model), you can use the model to estimate the probability of the title given the content. Even models that perform poorly on generation, perform well when used discriminatively.


Trying to make compelling plot for classification results with python

It's really rare that you'd show a plot of the probabilities for each example in your set. Are you sure you want to do this? A better presentation might be a confusion matrix. Here's how it works:

1) The columns are the true class labels

2) The rows are the predicted classes

3) Along the right hand side of the plot you can show the probability of correctly assigning to a class (or the classification error, if you prefer).

For example, say I have three classes in my dataset.
I have 10 examples of each class, so 30 examples total.

My classification results from my model are:

8 of the 10 examples in class 1 were correctly labelled; 1 was misclassified to class 2 and 1 was misclassified to class 3.

7 of the 10 examples in class 2 were correctly labelled; 3 were misclassified to class 3.

9 of the 10 examples in class 3 were correctly labelled; 1 was misclassified to class 2.

Then my confusion matrix looks like this:

$\begin{bmatrix} 0.8 & 0.1 & 0.1 \\ 0.0 & 0.7 & 0.3 \\ 0.0 & 0.1 & 0.9 \end{bmatrix}$

And along the right hand side of my confusion matrix I can include the classification error: 0.2 for class 1; 0.3 for class 2; 0.1 for class 3.

Notice the rows must add to 1.


How to Validate Decision Tree model by using *statistical tests*?

I am assuming you are using the D.T. for binary classification for a moment. One of the first tests I learned (and still dam good) is the 2x2 contingency table or frequency table or marginal frequencies or 2-way tables (many names means it's been around a while). It is simple to use and can branch off into many other tests and areas. Such as;

Phi Coefficient of Association
Chi-Square Test of Association
Fisher Exact Probability Test
Accuracy and precision
F-score or F-measure, ...(to name a few)

I usually like Kahn Academy way over Wikipedia(hate their explanations).


Install Spark and Hadoop in the same machine

The Hadoop stack is difficult to setup and people complain that you can't trust any answers to problems over 6-12 months old. I would recommend getting a pre-configured Hadoop/Spark setup from Cloudera or HortonWorks. Both have free community editions.


I want to say how I solved my problem for anyone who is looking for a similar question.

My categorization was a mistake. I realized later that, I gave the same photos to my model and waited for the different results. For example, I had nearly same photos in Audi A3 Hatchback/5 2009 and Audi A3 Hatchback/5 2010. When the model starts training, first it learns data. Then it predicts and validates output itself. If the output is wrong, it tries a different way to success. But wait a minute, there was no mistake. I gave you the same photos and waited for different results! My categorization failed here.

I categorized my cars by body changes like Audi A3 Hatchback 2008-2013. Except for categories that have the wrong photos, my results are great for now.

Now, we will work on better photos and better year categorization.


Explain output of a given classifier w.r.t features

Try out some of the examples in this library, which attempts to use machine learning to understand black box machine learning models in Python.


Does it make sense to train a convolutional neural network on lo-res, use on hi-res pictures?

When it comes to questions like this, you should be thinking about this in reverse. By that I mean, you want to actually use this algorithm at some point, don't you? So, the real question is: What do your images look like at prediction time?

Once you establish that, then what is the pre-processing you're going to be doing on these images? My guess is that you're not going to be changing things like the resolution. Sure, you might do some cropping or re-sizing, things like that. But the bottom line is that you should be training your algorithm based on what your prediction items are going to look like.


The given dataset has 303 observations and 76 features. This is usually considered very small. Deep learning needs plenty of data to be stable since it is highly nonlinear. Random forest is one option but simpler structured models can be considered. One important thing which have to be investigated in those situations is regularization. Fighting with over fitting looks like the main goal.


How do I calculate the range of a F1-score from a confusion matrix of 3 class,A,B,C

from sklearn.metrics import f1_score

y_true = [0, 1, 2, 0, 1, 2]
y_pred = [0, 2, 1, 0, 0, 1]

f1_score(y_true, y_pred, average='weighted')

From documentation

  Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters ‘macro’ to account for label imbalance; it can result in an F-score that is not between precision and recall.

https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html


Binary Classifier making only one prediction

I suspect a couple issues:

The cost function is unusual for classification. You would typically
use something like cross entropy which you mentioned. Make sure not to threshold the outputs when you input to the cross entropy loss. Also, I wouldn't characterize L2 loss as being more "harsh". If your predicted probability for a particular data point is 0, but its label is 1, L2 loss would be 1. But for cross entropy, the loss would be very large.
It predicts all one class on the validation set or just any particular batch?

If it is just a particular batch, maybe how you batch the data is off and all of one class is in that batch. Look at metrics like accuracy and loss.
If it is the validation set, I would look into if the training loss is consistently going down. Maybe the learning rate is too high. You are using Adam which is based off of momentum so it does not necessarily need as high of a learning rate as standard SGD.

If the weights are stabilized, it means that the network is saturated or you have 0 loss and your optimization problem is solved (probably not the latter). If the network did saturate too fast, batch norm would definitely help.


Combining textual and numeric features into pre-trained Transformer BERT

When they are talking about aggregating their outputs, they mean the final embeddings (just before the classification layer) not the output of the network itself.
You take the embeddings from both of the networks and concatenate them vertically. This concatenated embedding you use to predict the final output you wanted.
Torch has a function torch.cat for this purpose.


How to handle "year" variable for Machine Learning models

First of all I would suggest you  to analyze the data. Do you really need the year? Have you studied the correlation with the other variables? I mean, study the domain of the model you intend to create before create any model.

You can treat year as categorical variable and use some of the techniques such as One Hot Ecoding or Dummy Variables for better performance.
You could also perform a normalization of the years, treating them as numerical variables, which are between 0 and 1.

The most important thing is to know what we are going to do to make good use of the information or data.


Calculate a ranking function from classification features

Generally speaking moving from a classification space to a ranking space is not straight forward. In classification problems there is no meaning to order between labels. This means that your suggested equation might not represent the order between labels at all.

This is somewhat dependent on the feature space and classification algorithm. Some classification algorithms(tree-based for example) don't use the concept of distance in their search for best fit. The separation created from the fit(which is used for classification) does not include how far the boundaries are from one another.

Some classification algorithms (logistic regressions, SVM, others) do have this distance feature, and this can be translated to the probability of an instance belonging to each class. Using this probability as a ranking mechanism (instead of how asking how efficient, asking what is the probability of being efficient) might make sense.


Simple Markov Chains Memoryless Property Question

I think you are not doing anything wrong, the markov property is satisfied when the prediction can be solely based on the present state. I do not think you are contradicting this "memoryless" condition (which applies to the whole chain) when you predict what could probaly happen after a last state that is already just predicted.


Which definition of Likelihood function is correct?

1 - Equivalence between $\mathbb{E}_{x\sim \hat{p}_{data}}\log(p_{model}(x;\theta))$ and $\frac{1}{m}\sum_{i=1}^m \log(p_{model}(x;\theta))$:
Yes, as you mentioned in the question, the key issue here is to realize that the following expectation is with respect to the empirical distribution defined by the training data ($\hat{p}_{data}$):
$$ \theta_{ML} = \arg\max_{\theta}\mathbb{E}_{x\sim \hat{p}_{data}} \log(p_{model}(x;\theta)) $$
By definition, an empirical distribution assigns a probability of $1/m$ on each of the $m$ points $\mathbb{X} = \{x^{(1)},x^{(2)},...,x^{(m)}\}$ $\to$ Thereby the value of $\hat{p}_{data}(x_i)=\frac{1}{m}f(x_i)$ where $f(x_i)$ represents the ocurrence frequency of a value $x_i$ in our dataset.
Note that a value $x_i$ may appear more than once in our dataset $\mathbb{X}$ $\Rightarrow f(x_i)$ can be $> 1$. This is why we can express the previous expectation as:
$$\begin{align}
\mathbb{E}_{x\sim \hat{p}_{data}} \log(p_{model}(x;\theta)) &= \sum_{i=1}^m \frac{1}{m}\log(p_{model}(x^{(i)};\theta))\\
\end{align}$$
Hence, the maximization of both expressions lead to the same value of $\theta_{ML}$

2 - Equivalence between the minimization of $D_{KL}(\hat{p}_{data} \Vert p_{model})$ and $H(\hat{p}_{data}, p_{model})$
As said in the question, the KL divergence of the probability distributions $\hat{p}_{data}$ (empirical distribution of $\mathbb{X}$) and $p_{model}$ (our statistical model that we are using to fit $\mathbb{X}$) is given by the next expression:
$$D_{\text{KL}}(\hat{p}_{data} \parallel p_{model}) = \mathbb{E}_{x\sim \hat{p}_{data}} [
 \log(\hat{p}_{data}(x)) - \log(p_{model}(x;\theta))]$$
Here, we can see that $\mathbb{E}_{x\sim \hat{p}_{data}}
 \log(\hat{p}_{data}(x))$ does not depend on $\theta$ (it only depends on the data generating process), so it can be trated as a constant. Hence we can adress the same problem of finding the value of $\theta$ that minimizes the cross entropy by minimizing this KL divergence, because this is the same as minimizing:
$$ \mathbb{E}_{x\sim \hat{p}_{data}} [- \log(p_{model}(x;\theta))] = - \sum_{i=1}^m \frac{1}{m}\log(p_{model}(x^{(i)};\theta))$$
Which is the definition of the cross entropy $H(\hat{p}_{data}, p_{model})$.

3 - Equivalence between the minimization of $H(\hat{p}_{data}, p_{model})$ and the maximization of $\sum_{i=1}^m \log(p_{model}(x;\theta))$
Here, we have to proof that
$$\arg\max_{\theta}\sum_{i=1}^m \log(p_{model}(x;\theta))$$
gives the same value of $\theta_{ML}$ than:
$$\begin{align}
\arg\min_{\theta}H(\hat{p}_{data}, p_{model}) &= \arg\min_{\theta}-\frac{1}{m}\sum_{i=1}^m \log(p_{model}(x^{(i)};\theta))\\
&= \arg\max_{\theta} \frac{1}{m}\sum_{i=1}^m \log(p_{model}(x^{(i)};\theta))
\end{align}$$
Both expressions are equivalent (they give the same value of $\theta_{ML}$) because a scale factor (such as $1/m$) does not affect in a minimization/ maximization problem.

4 - Equivalence between $\prod_{i=1}^m p_{model}(x^{(i)};\theta)$ and $\prod_{j} p_{model}(x_j;\theta)^{\#\text{ ocurrences of }j}$
This is because $i$ and $j$ represent different things:

$i$ represents each individual datapoint of a dataset $\mathbb{X}\Rightarrow x^{(i)}$ is a datapoint.
$j$ represents a certain value present in our dataset $\mathbb{X}\Rightarrow x_j$ is a certain value.

Given this, if a datapoint $x^{(i)}$ has the same value ($x_j$) of other $n$ datapoints, then we can express the likelihood of these datapoints in two equivalent ways:
$$ \prod_{i=1}^n p_{model}(x^{(i)};\theta) = p_{model}(x_j;\theta)^n$$
So, if we extend this reasoning to all the possible values ($x_j$) that our dataset can have, we reach the equivalency given by wikipedia:
$$\prod_{i=1}^m p_{model}(x^{(i)};\theta) = \prod_{j} p_{model}(x_j;\theta)^{\#\text{ ocurrences of }j}$$


Matplotlib -- TypeError: 'module' object is not callable

Instead of
import matplotlib as plt

use
import matplotlib.pyplot as plt


Which classification algorithms to try for classifying text data into 300 categories

In general, a decent starting point for problems like these is Naive Bayes (NB) classification using a simple bag of words model. Here are some slides describing NB as applied to natural language processing. There's nothing especially fancy about this approach, but it's pretty easy to implement and will give you a starting point to expand from.

Once you've found some initial results assuming independence among your features and your output labels, you'll probably have a better sense of where the model is weak.  From that point forward you can apply some feature engineering (maybe TF-IDF) as well as some post processing to deal with samples that get assigned to related categories.


Is it possible to have stratified train-test split of a set based on two columns?

One option would be to feed an array of both variables to the stratify parameter which accepts multidimensional arrays too. Here's the description from the scikit documentation:

stratify array-like, default=None
If not None, data is split in a stratified fashion, using this as the class labels.

Here is an example:
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

# create dummy data with unbalanced feature value distribution
X = pd.DataFrame(np.concatenate((np.random.randint(0, 3, 500), np.random.randint(0, 10, 500)),axis=0).reshape((500, 2)), columns=["text", "subreddit"])
y = pd.DataFrame(np.random.randint(0,2, 500).reshape((500, 1)), columns=["label"])

# split stratified to target variable and subreddit col
X_train, X_test, y_train, y_test = train_test_split(
    X, pd.concat([X["subreddit"], y], axis=1), stratify=pd.concat([X["subreddit"], y], axis=1))

# remove subreddit cols from target variable arrays
y_train = y_train.drop(["subreddit"], axis=1)
y_test = y_test.drop(["subreddit"], axis=1)

As you can see the split is stratified to subreddit too:
Train data shares for subreddits
X_train.groupby("subreddit").count()/len(X_train)

gives
text
subreddit
0   0.232000
1   0.232000
2   0.213333
3   0.034667
4   0.037333
5   0.045333
6   0.056000
7   0.056000
8   0.048000
9   0.045333

Test data shares for subreddits
X_test.groupby("subreddit").count()/len(X_test)

gives
text
subreddit
0   0.232
1   0.240
2   0.208
3   0.032
4   0.032
5   0.048
6   0.056
7   0.056
8   0.048
9   0.048

Naturally, this only works if you have sufficient data to stratify to subreddit and the target variable at the same time. Otherwise scikit learn will throw an exception.


BPE and word pieces are fairly equivalent, with only minimal differences. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning.
Therefore, I understand that the authors of RoBERTa take the liberty of using BPE and wordpieces interchangeably.


There is nothing in SpaCy that you can use out-of-the-box. However, they allow you to use custom components
To solve your problem, I see at least three ways to do it.

NTLK

NLTK allows you to add known abbreviations as exceptions. See this StackOverflow post.

Use a regular expression

Since your problem is that you have some example of dots that shouldn't mean a sentence starts, you could customize a basic regular expression to include that behavior. Here is a stackoverflow answer that could get you started.

Post-processing

You could also use SpaCy's default segmentation, and then merge sentences if they end with a known abbreviation. It's not incredibly elegant, but it will work.


The Meaning Behind the Cross Validation Score in Factor Analysis

The score you get is determined by the estimator you use. In your case this is Factor Analysis. Looking at the documentation of the algorithm, you can find the explanation:

  FactorAnalysis performs a maximum likelihood estimate of the so-called loading matrix, the transformation of the latent variables to the observed ones, using expectation-maximization (EM).

and the reference:

score(X[, y])   Compute the average log-likelihood of the samples

So your number is the logarithm of the maximum likelihood estimate of the factor loading matrix.


t-SNE Python implementation: Kullback-Leibler divergence

The TSNE source in scikit-learn is in pure Python. Fit fit_transform() method is actually calling a private _fit() function which then calls a private _tsne() function. That _tsne() function has a local variable error which is printed out at the end of the fit. Seems like you could pretty easily change one or two lines of source code to have that value returned to fit_transform().


Do you have to normalize data when building decision trees using R?

Most common types of decision trees you encounter are not affected by any monotonic transformation. So, as long as you preserve orde, the decision trees are the same (obviously by the same tree here I understand the same decision structure, not the same values for each test in each node of the tree).

The reason why it happens is because how usual impurity functions works. In order to find the best split it searches on each dimension (attribute) a split point which is basically an if clause which groups target values corresponding to instances which has test value less than split value, and on the right the values greater than equal. This happens for numerical attributes (which I think is your case because I do not know how to normalize a nominal attribute). Now you might note that the criteria is less than or greater than. Which means that the real information from the attributes in order to find the split (and the whole tree) is only the order of the values. Which means that as long as you transform your attributes in such a way that the original ordering is reserved, you will get the same tree.

Not all models are insensitive to such kind of transformation. For example linear regression models give the same results if you multiply an attribute with something different than zero. You will get different regression coefficients, but the predicted value will be the same. This is not the case when you take a log of that transformation. So for linear regression, for example, normalizing is useless since it will provide the same result.

However this is not the case with a penalized linear regression, like ridge regression. In penalized linear regressions a constraint is applied to coefficients. The idea is that the constraint is applied to the sum  of a function of coefficients. Now if you inflate an attribute, the coefficient will be deflated, which means that in the end the penalization for that coefficient it will be artificially modified. In such kind of situation, you normalize attributes in order that each coefficient to be constraint 'fairly'.

Hope it helps


Keras: Prediction performance does not match accuracy

Setting shuffle=false to evaluate_generator and predict_generator fixed the issue for me.


How can I find if it is an overfitting problem?

These are some ways by which you can improve your model's accuracy and reduce overfitting.

Split your dataset into training, validation, and testing.
Add dropout.


I guess it is too late, so I will be short.
You may search the SNAP graph dataset repository. There, each dataset has a set of tags like Temporal, Directed or Weighted, for instance.
It is one of the largest such repositories, so I hope you will find what you need there.


Web page data extraction using machine learning

If I understand your question properly, this seems to be a scraping problem which you can do using Beautifulsoup in python.


Threshold to consider new feature as a new finding to a model?

A lot of questions here. Here are some thoughts.

should you be happy about a 1 point increase in AUC? Yes. An effect can be genuine, but small. A 1 point advantage is still an improvement. But do I trust that outcome? Not sure.
you need some more data. Your sample size is not large. Furthermore, cross-validation is a wonderful thing, but you have been running a lot of tests on the same small data set, so cross-validation notwithstanding -- we really don't know how your classifier will perform on brand new, unseen data.
point 3. This sounds like an issue with over-fitting.
point 4. I'm not sure what you are doing here. Cross-validation is an alternative to "train" and "test" sets, since each fold acts as the "test" set to the model fit on the "Non-fold" part of the data. Did you mean to say that you were doing CV on the train part, and then used the "test" part for a final check on performance? That would be a good thing to do, but if you have been using the same test set to check every model you have tried, then that test set is beginning to look like a training set. You will need another "test" set.
point 5. You don't need a ridiculously high AUC to be guilty of over-fitting. Over-fitting happens when you fit features to the noise in your data set. An over-fit model will have a higher mean squared error on a fresh test set than the optimal model, although it will do better on the training set. Having 16 to 18 features is too many if, in fact, the outcome is well explained with only 5.
Given that you have reduced your candidate features to 14, you now have a much smaller problem, and it should be possible to examine the features from a subject matter perspective. Which features would a SME recommend you to retain? And you can also examine correlations between the 14 features and check for redundancy that way. With a small data problem (which is what this is), you can work to understand your data directly. That approach might yield some interesting insights.
You might want to use a different method of model selection than maximal AUC. This paper discusses the merits of AUC and offers an alternative. Could be interesting.


How to reduce dimensionality of audio data that comes in form of matrices and vectors?

Interesting ideas on finger printing, but that isnt going to work here, as you don't have a definitive sound you are working with.

To me, this seems a bit more like an image processing problem. So instead of thinking how you could reduce dimensionality, could could try and combine your data together into "images" that digitally represent the sounds you are looking at. Then using standard image processing techniques. I say this waving my hands around vaguely, as I don't know of specific ones, but have enough understanding of edge detection, feature detection and applying map/reduce type methodologies which might be able to give templates for what you are after.


Combining multiple probabilities from a classifier. Propagating probabilities

If you feed the model with 20 images for testing then your output should look like array of [20x10]. For each row represents the probabilities of all classes(in your case 10). Let's see the example below, and 1-index is dog with 0.9 probability, so your model classified it correctly. If your model classifies 19 times dog-correct and 1 times cat-wrong then your model's test accuracy will be calculated as correct_preds / (correct_preds+wrong_preds)
[[0.0, 0.9, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.0],
 [0.0, 0.8, 0.0, 0.0, 0.0, 0.1, 0.0, 0.1, 0.0, 0.0],
 ..,
 [0.0, 0.75, 0.0, 0.0, 0.0, 0.1, 0.0, 0.1, 0.05, 0.0]]


Grouping already clustered data (with a pre-defined x and y)

It would be helpful to know which clustering technique are you using.

You can use

Partition-based Clustering: for example K-Means Clustering, not that good with outliers.
Hierarchical-based Clustering: Produces trees of clusters (Agglomerative, Divisive). You get a Dendogram.
Density-based Clustering: produces arbitrary shaped clusters, for example DBSCAN

If you are looking something other that a circular cluster and you need clusters within clusters, I would try DBSCAN. It locates regions of high density and separate outliers and it can find clusters within clusters.

If you are using Python you can use DBSCAN with sklearn

from sklearn.cluster import DBSCAN

I hope that helps!


clarification on train, test and val and how to use/implement it

When fitting via model.fit(X_train, y_train,..), will Keras shuffle the
data autonomously?
Yes. shuffle = True is default. So, it basically shuffles every time.

Next, in Keras, you are able to provide the validation set inside the model.fit() method as validation_data=(x_test, y_test) but there is also the possibility to provide e.g. validation_split = 0.2 What is the difference?
The difference is that you can manually provide a validation data set. It's not X_test, Y_test. Rather X_test and Y_test are used for model evaluation model.evaluate() or model.predict(). model.fit is used for the training dataset. When you say validation_set = 0.2, it takes 20% data from training dataset and provides you the validation accuracy and loss.

And after that, the test/prediction set will be taken into account just as model.fit(X_pred, y_pred,..). Right?
You need to use model.predict(X_pred, Y_pred)


it's quite easy to make! One thing to note I'm using the Keras functional API.

from keras.layers import Input, Dense

Input_1= Input(shape=(shape, ))

x = Dense(100, activation='relu')(Input_1)
x = Dense(100, activation='relu')(x)
x = Dense(100, activation='relu')(x)

out1 = Dense(1,  activation='linear')(x)
out2 = Dense(1,  activation='linear')(x)
out3 = Dense(1,  activation='linear')(x)

model = Model(inputs=Input_1, outputs=[out1,out2,out3])
model.compile(optimizer = "rmsprop", loss = 'mse')


Detecting and confirming sentiments in chat messages

You need to define best.

If best means being exactly sure of your labeling, then you'd have to manually go through all messages, use crowdsourcing and what not. This is assuming that the chat messages are in a language that you can understand.

If best means automating the process, you could use an existing sentiment analysis model to predict the sentiment of each message and see if it lines up with the existing labeling. Of course, perfect alignment is unlikely.

If best means doing it from scratch, then I would try to cluster the sentences into 3 groups and then look at the composition of the groups. Of course, this is highly dependent on the clustering and how you define the distances between the messages


You don't select 128 from 1600. Dense layer performs a matrix-vector multiplication. First, flatten layer basically flat the output of your max_pool layer(5*5*64) to a vector (1600*1) (note that 5*5*64 =1600). It is just a re-arrangment of the element. We flatten max-pool layer because we want to use the output of max-pool layer as the input of dense layer (dense layer is also called fully connected layer or multiple layer perceptron). Dense layer is nothing but a vector-matrix multiplication. Say you have an input vector(I) whose shape is 1600*1, and you want your output (O) shape to be 128*1. Then your dense layer will perform the following operation: O=WI, where W is a 128*1600 weight matrix. That is how you got 128 output.The dimension of the W matrix is specified by your input shape and output shape and thus, after you specify the input/output shape, the W matrix is implicitly auto-generated by tensorflow. Btw, your "relu" function in the denselayer block, which is called non-activation function, performs relu function calculate on each element of your output vector (128*1) so it doesn't affect your final output shape.
Normally, all layers will be optimized. The loss function defines a loss between your true label and the output of your neural network (which is the last dense layer in your case). The gradient of the loss back propagates to each layer through the back propagation algorithm(you can check it online, but basically it is just chain rule). Thus, each layer has its own error gradients. We then use these error gradients to update coefficients of each layer to make the overall loss smaller (the loss defined by your loss function). The operation is called gradient descent algorithm. Since each layer updates itself separately, you can always freeze some layers so that these coefficients don't update, which can be done in tensorflow.


Question of using gradient descent instead of calculus. I checked previous questions there are still points to clarify

If I understand correctly, your question is Why do we solve gradient descent problems iteratively rather than setting the derivative to 0 and solving for the weights which minimize the cost function?

We can't do this (in all but extremely simple problems such as linear regression) because there is no easy closed-form solution when you set the derivative equal to 0.

Let's take a look at a simple example: let's say you have a 1-layer neural net with no bias and a sigmoid activation function.  Your network output is $\hat{y} = \sigma(WX)$.  If you use L2 loss, your cost function would be $L = (\sigma(WX) - y)^2$.  (For simplicity we'll assume that we're only using a batch size of 1).

Let's take the derivative with respect to our weights $W$ and use the chain rule
:
$\begin{eqnarray*}
\frac{dL}{dW} &=& 2(\sigma(WX) - y)\frac{d}{dW}\sigma(WX)\\
&=& 2(\sigma(WX) - y)\sigma'(WX)\frac{d}{dW}(WX)\\
&=& 2(\sigma(WX) - y)\sigma(WX)(1-\sigma(WX))\frac{d}{dW}(WX)\\
&=& 2(\sigma(WX) - y)\sigma(WX)(1-\sigma(WX))X\\
\end{eqnarray*}$

At this point, we get a bit stuck.  If we set this derivative equal to 0, it still won't be easy to solve for $W$ since $W$ appears 3 times in this equation and can't easily be isolated by itself on one side of the equation.

Since we cannot solve for the cost-minimizing weights directly, instead we use gradient descent.  We take the derivative of the cost function to tell us which direction we should change each weight in order to most decrease the cost.  The actual size of the update is determined by the learning rate $\alpha$.


What is difference between attention mechanism and cognitive function?

Are you reffering to biological "cognitive function attention"?
If yes than it could be argued its different.
TL;DR
Fixed-length Context vector in the encoder decoder architecture could not remember long term dependencies. Attention could learn  these long term dependencies, hence have a long-term memory. If we speak about human attention it is usually short spaned (relative).
On the other hand I believe it was Yoshua who first talked in the terms of attention and one could argue that we could implicitly focus (our attention) on some older parts of text and hence extract deeper meaning.
TL;DR TL;DR One can make a case its actually more natural to talk about long-term memory when discussing attention-ML-mechanism


General approach on time series for customer retention/churn in retail

OK, let me try to dissect your questions:

  Q1: Should I be grouping customers like I am doing, on a monthly basis
  or I have to group them on a 2-month basis since that is how they were
  labeled?

This depends a bit on what your goal is and what kind of features you want to use. I think, with the limited things I know from your question, I would go for a daily prognosis for the next two months (so if you're doing daily you can forecast a period of two months daily data for each day). Reason being that otherwise you might loose some inactivity "features" that are implicitly part of your data. As an alternative you can group your data and create features that help you introduce the information anyway. For example: activity_in_last_5d, activity_in_last_10d, etc.

  Q2: Also, how do I model this? Do I keep customer_id as a feature of
  the model or not?

Customer_id doesn't seem to be a feature to me but rather the key for identifying which features to feed into your model.

  Is the gap in dates for each customer relevant and
  how should I deal with it (if)?

You are basically asking about imputing your data (see this for more: https://towardsdatascience.com/6-different-ways-to-compensate-for-missing-values-data-imputation-with-examples-6022d9ca0779). Gaps can cause problems in your modeling. Some models (for example ARIMA for time series) won't work at all if you have gaps that aren't handled. Looking at your use case, I think taking the last known value for a gap should work fine since a gap means your customer didn't churn on that day.

  The dates repeat for different users, should I create index out of a date but it won't be unique or should I create index out of customer_id?

Probably go for indexing the customer_id and creating features based on the date! Maybe there are some seasonality features interesting as well (for example how many people churned on the same day last year?)

  Q3: If I need to predict whether the user is going to churn by the end
  of the year for example or in the next 6 months, would that change how
  I group/arrange my date and model this?

You would have to change the size of the period you are predicting for. Changing features, grouping, etc. might be useful as well but that really depends on what features you are choosing aka are important and how your model is going to work.


Weights not converging while cost function has converged in neural networks

The weights in a model do not need to converge to stop training.

One possible explanation is that the model error surface has a big, wide valley. If that is the case, the loss function would be low throughout the valley but there would be many weight combinations that would all yield similar performance on the training dataset. Once a model has reached an acceptable loss function value there is no reason to continue training, just take any set of weight values.


It's not simple, but doable. I suggest you to create training data in the following way: take a text corpus, as large as possibile, and remove words sampled randomly. Then train an seq2seq RNN to map this "deteriorated" text with its original.

The RNN you need won't be too different from an NMT model, but it's goal is different of course.

It's the first time I encountered this kind of task, therefore I can't say what is the state of the art.


How can I improve the accuracy of my model? (Cab Cancellation Prediction)

Create addition variable: Eg: lead_time-start_time can be time to book.
Reduce variables with many classes if present (part of EDA)
standardize numeric variables - (val-mean)/sigma
Tree is a very weak classifier, you will have to do bagging or boosting (like ada boost or gbm or random forest
try parameter tuning - I am not pasting any links since I don;t know the policy for advertising over here, but just search GBM parameter tuning in google - you'll get multiple links

You have mentioned time issues when you tried ensemble methods. To solve that:

take a subset of data and try ensembling on that (once you finalize the model, run it on whole dataset)
if you are using XGBoost, you have an option of taking in another model as an input (you can run this in batches)


Parameter initialization in a genetic algorithm

In my opinion you should make the range as large as possible (to a reasonable extent) for the first random initialization.

The genetic algorithm will converge to the appropriate range eventually, but giving it a narrow range could result in a sub-optimal solution because the algorithm doesn't have any way to reach a better solution. The only downside of a large range is that it might take a bit longer (more generations) to converge.

So I would suggest you keep a completely random initialization of the values, for instance in the range [-10,10].


How important is the balance between the components of a loss value?

The cost function controls the algorithm completely - the new regularization from weight decay is likely responsible for the jump in loss. If you only added a small amount of regularization and the loss ballooned, you were likely very overfit.

The solution is cross validation. The only way to know the optimal amount if regularization is to see what does the best on held-out data.


How to use keras.utils.Sequence with multiple files

The Sequence class from keras works great with multiple files. I've personally created a custom data generator using the Sequence class to load and pre-process multiple files. You should be able to figure it out using this post, I've found it to be very easy to follow the process of creating a custom data generator step by step.


Calculate likelihood of a date being worked based on location, hours and rate of pay

If you're looking for code, I'm not familiar with C#. My answer will focus on theory.

tl;dr most machine learning-related packages have a built-in logistic regression function of some sort. That's what I'm recommending here.

More detail:

I would start with a basic model and work my way up. It sounds like this is something you can figure out using a regression model.

Based on your question: "Based on location, date (day, month), start time, end time, grade, hourly rate and hours, how can I work out the percentage chance of a shift being filled?", I understand you want:

output: probability of shift getting filled
input: location, date, start time, end time, grade, hourly rate, hours (these are $x$'s below)

If you're familiar with logistic regression:

$$log\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k$$

the output of this model is the log odds of a single shift being filled (versus not filled). Note this can be rewritten as:

$$p = \frac{e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k}}{1-e^{\beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_kx_k}}$$

Where $p$ is the probability that the shift gets filled.

I would start by fitting one logistic regression model for each shift.

Be careful: multinomial regression would give you the probability of 1 shift out of k possible shifts being filled, but I don't think that's what you're looking for.

After seeing how well that worked (comparing the model predicted results to the actual percentages in the data), if necessary I would build a conditional model to take into account the probability of filling a shift, given other shifts have already been filled. Not sure how I'd do that yet, but hopefully this gives you a starting point!

EDIT based on the answer from Vikram Murthy:

I realize I forgot to mention: your response variable in this case would be 0 or 1 indicating whether a shift was filled or not. So for each shift, you would have a column indicating whether that shift was filled. That's the column being predicted. This is similar to using "dummy variables".

For example, if you have two shifts, your columns would be:

loc, day, month, starttime, endtime, grade, hourlyrate, hours, shift1_f, shift2_f

So your data might look something like this:

loc, day, month, starttime, endtime, grade, hourlyrate, hours, shift1_f, shift2_f
A, 1, 2, 12:00, 8:00, 1, 34.25, 8, 1, 0
A, 1, 2, 12:00, 8:00, 1, 34.25, 8, 0, 1
A, 1, 2, 12:00, 8:00, 1, 34.25, 8, 1, 1
A, 1, 2, 12:00, 8:00, 1, 34.25, 8, 0, 0

In this setup, this would indicate that shift 1 was filled in the first case, shift 2 was filled in the second case, both shifts in the third case, and no shifts in the last case.

Your two logistic regression models would be set up like this:

shift1_f = loc + day + month + starttime + endtime + grade + hourlyrate + hours

and

shift2_f = loc + day + month + starttime + endtime + grade + hourlyrate + hours

The proportions would be figured out automatically by the program; it's not necessary to figure them out yourself and enter them in any application of logistic regression that I've seen.


Statistical test using Gmean to compare multiple algorithms on multiple datasets

The performance of machine learning algorithms is not commonly evaluated with null hypothesis significance testing (NHST).
Machine learning performance is evaluated with performance on a hold-out data (e.g., validation or test), regardless of the evaluation metric.


Using TF-IDF for feature extraction in Sentiment Analysis

TF-IDF is a vectorization technique used to convert documents (a single tweet in your case is a document) to vectors. After you train the TF-IDF model, the only words/vocabulary it has learnt, would be from the set of documents (aka corpus, the entire set of 3k tweets).

Since you mentioned that there were 570 unique feature words after TF-IDF, that would be the vocabulary your model has learnt. If you give this model a document with words that are present in its vocabulary, it will successfully vectorize it. However, if one or more words in your new document are not present in the model's vocabulary, those words won't be included in the vectorization at all. The words in a new sentence will be given weights only if the model had encountered those words in its training. In other words, every sentence will be vectorized w.r.t the model's vocabulary.

Ex -

Model's vocabulary - a, big, hat, have, I, mat

Input - "I have a big mat"

Vector - [ sequence of tfidf weights in the order of the model's vocabulary. If word in vocab is not present in the sentence, weight assigned is 0 ]

Input - "I have a dog"

Vector - [ sequence of tfidf weights calculated for all the words in the vocabulary, in the same order as the vocabulary ]

Since "dog" wasn't a part of the vocabulary, it's not included in the vectorization.

  What if the new tweets which this model has never seen do not have
  feature words that I have used for training, will the model fail to
  make correct predictions for them (in my case there are only 3
  possible predictions viz. positive, negative and neutral) correctly
  for them?

I can't say much about the final prediction, but your feature vectors from TF-IDF can be way off, if you're expecting it to vectorize documents with a large number of words that the model hasn't trained on. Improper vectorization can affect your prediction accuracy.

  If yes, then how should I handle this scenario

My suggestion would be to consider the following -

Pre-process the train data thoroughly. Removing punctuation, substituting abbreviations by their full forms and other steps can help the TF-IDF model train well.
If gathering more data is an option, try training the model on a large set of
tweets that fall in the context of your prediction. The more the vocabulary, the better equipped the model is, to vectorize new unseen documents.
Try using a pre-trained model like BERT.


Is it possible to cluster data according to a target?

Use constrained clustering.

This allows you to set up "must link" and "cannot link" constraints.

Then you can cluster your data such that no cluster contains both 'churn' and 'non churn' entries bybsettingn"cannot link" constraints.

I'm just not aware of any good implementations.


Its about how the neural net learn inside. Usually in deep neural network you have multiple layers, the first layers will learn the low level feature then the more you approach the output layer the more the layers will learn the high level feature.
Here its identifying faces task, that's why you see that in the first layers small patterns are learned (mainly edges ...) then faces components ...

How to obtain them : you should watch what makes neuron activated in each layer depending on the input. As you know each neuron will be activated (once the DNN is trained) for specific input combinations.
By visualizing that you can get an idea about what exactly each layer has learned in term of high-low level features.


I think you are referring to a king of image auto captioning, and why not sentence alignement ! I suggest you to go over the paper Deep Visual-Semantic Alignments for Generating Image Descriptions which is one of the references today (by M. Karpathy).
The first step is about recognizing objects in an image and align sentences to them (done with R-CNN and BRNN) and the second is about generating new captions (CNN + RNN)

Even though it may be quite difficult to deeply go through, there is some code available on GitHub and it has been remade by a a lot of people with various implementations. Here is the summary.

Good luck !


A bump in CV score curve! What does it mean?

That pattern is common in neural networks training.

Train performance is an estimate of bias, and validation performance is an estimate of variance. Initially both go down. Bias continues to go down, but variance goes up. That is the classic bias-variance tradeoff. However, in neural networks variance will start to go down again. This is called the “double descent curve”.

It is not known why neural networks training show this “double descent curve.” One idea is that this pattern might be a concatenation of the two curves. One curve might be from optimization and the other curve might be from sampling. The paper "A Modern Take on the Bias-Variance Tradeoff" goes into more detail about it.


Is there a comprehensive learning plan for a self-teaching beginner in data science with no programming background?

Short Answer:

Practice.  You already see it's both math and programming, so start practicing your math and your programming and eventually you'll be there.

Long Answer:

Data science happens at the intersection of math and programming.

A lot of people start with some intense math (biotech, physics, engineering, or if you're lucky statistics) and thus have to learn the programming from scratch.

A lot of others start with being some professional developer, and then have to learn the math.

But as you said, you know neither.  So check your motivations.  Do you really want to be a data scientist, or are you just caught up in the buzz that still surrounds it?  You really have to love it to endure the massive learning curve.  And if you love it, figure out why you're only now getting started.

Alright.  Still on board?  If so, you really have three paths as I see it.

you can follow along with some mooc or series of tutorials to spoon feed you some basic project.  It is a start.
You can first try to break into being an analyst first.  Could be a much shorter path.  Best case scenario, analysts can turn directly into data scientist.  Worst case scenario, you get stuck not quite living your dream.  If you take this route with the intention of eventually becoming a data scientist, be verrry wary of the tools you use.  Highly automated analyst tools keep you locked into what the tools can do.  And if the tools don't have any statistics or machine learning baked in, then you'll get stuck.
The hard way.  If you're serious about it, then know you need to learn both overlapping languages - math and programming.  Ask around and do your research.  Find what programming languages will work best in the niche you expect you'll most like.  A part of you "do I really want to be a data scientist?" should includes the types of things you want to do as a data scientist.  Out of your answer, pull a few examples of actual projects.  Of those projects, pick the easiest and learn the math behind it.  Like financial stuff?  Perhaps some simple time series prediction on some open financial data.  Like marketing?  Perhaps start with looking up A/B tests.  Yes, this one may include also doing #1, but with the very important extra layer of sincerity around why you're doing it.  And a LOT of the job of a data scientist is being able to take a question nobody's asked yet, and figure out how to apply your tools to solve it.  Many moocs rob you of that most-necessary layer of first framing the problem before trying to solve it.


Make column as dictionary key and row as value in pandas dataframe

Check this:

data_set.to_dict(orient = 'records')


Clarifying Probability Mass Function (PMF)

I believe they mean the following: "We will use $P$ to denote the concept of a probability mass function (pmf). $P(x)$ is the pmf of random variable $x$, while $P(y)$ is the pmf of random variable $y$. The two variables are usually the results of different random experiments, hence we use two separate letters, $x$ and $y$. As such, they will probably have different pmfs. But somethimes, it might happen that they have the same."

I think the essential source of confusion is that $P$ is not used to denote any single pmf, but is the name used to refer to a pmf. As a parallel, height_of(Stefan) and height_of(Mark) may be different, or they may be the same. height_of is just the name of the thing we are measuring, not the thing itself.

P(x,y) is thus the pmf of the joint variable (x,y). P((x, y)) would be a more unambiguous notation, but I guess is omitted for brevity. If you need more clarification on joint variables let me know. I hope this helps.


Should I apply PCA on the entire dataset or just the nominal values?

It is not advised for you to apply PCA on a dataset with nominal values. You can, but PCA translates variables in space. It is hard to find the relationship between nominal values in space. For example, how would one quantify the space between 'male' and 'female' or 'white' and 'red' or 'PC' and 'mobile phone'?

Some alternatives are

Use a tree model, like random forest, which will easily handle nominal values.
Use FactoMine in R.


I would recommend looking into FBProphet.
It's a good starting point for automating the creation of forecasts. It's very easy to use, and often produces better results than classical forecasting methods (ARIMA, Holt-Winters, etc.) right out of the box.
The default settings offer an additive or multiplicative model, comprised of trend and seasonality. This can be a combination of daily, weekly and yearly seasonality patterns, with the option to add custom patterns if required. Holiday effects can also be tracked using the built in holiday calendars, which again can be customised.
This is generally used for data with no external regressors, just a datetime and a measurement. However, external regressors can be configured, to influence the outcome of the forecast.