Deep reinforcement learning on changing data sizes

Although this problem can happen quite frequently in Reinforcement Learning (RL) projects, it is not a RL issue. The issue is due to neural network requirements, and also applies to many other parametrised machine learning solutions.

You can vary the number of parameters of a neural network in a limited fashion. However, it can only usually be done before training (or re-training), and is not a suitable approach to deal with changes happening over a small number of time steps, or during post-training prediction (e.g. when playing the game with the trained agent).

There are a few different approaches you can take. Which is suitable will depend on details of the game:

Pad the inputs

This is the simplest approach, and you can do this without needing much additional research. It relies on there being a workable maximum number of inputs you could need at any time step.

Practical considerations:

Have a simple mechanism to identify whether a particular set of inputs are present. E.g. add a ${0,1}$ flag to show present of a unit, leaving rest of inputs that represent the unit at some neutral value such as $0$.
If there are only rare occasions when the last few inputs are used, you should expect the neural network to learn about those situations very slowly. It will not generalise between units in different input "slots", unless you alter the architecture to link them.

Invert the representation from list of units to "world map"

This works nicely for a game where movement is discrete and there are rules that limit the amount of stacking possible in any single space. It also allows you to use a CNN architecture.

For example, this is how AlphaZero represents Go and Chess, which both allow varied number of pieces on the board at any one time, but where the board itself is strictly defined. In the case of Chess, this is also how AlphaZero identifies which piece to move, and how, by output on a matching grid to select the piece and a movement type (each selection/movement type combination in that case is a separate output channel, the output reprsents 1000s of potential moves all at once).

Practical considerations:

If your units have statistics that you need to track, then each of these can go in a separate instance of the grid (sometimes called a "plane" or a "channel"). So you might have one grid showing presence of a piece, another identifying its type, another its attach strength etc, all stacked into a 3D structure assuming the playing board is 2D.
If the map is very large or continuous, then it may be possible to discretise it into a grid.
Using a CNN allows for good generalisation based on similarity between pieces and their stats. However, the approach becomes limited if the grid is very large or stacks of pieces at any one grid position can become large, in addition to requiring a much larger memory footprint to represent the game state.

Feature engineering

Technically, representing the game as a map/board is just one common form of feature engineering in RL. Whether or not it will work, you know the game and may have knowledge of what factors are important to playing it well. You can use this knowledge to simplify the rerpesentation and summarise key details that are independent of the number of units.

For instance, it may be important to prioritise play based on proximity of strongest units in play. So you could sort the available units based on this heuristic and present a smaller fixed number of key units.

Practical considerations:

This can be done in combination with any other approach.
This relies heavily on your creativity and a good technical understanding of both the game and what works well in features used for neural networks.

If you want advice on possible feature engineering approaches, I suggest you open a new question with more details of the game being played.

Recurrent Neural Networks (RNNs)

A recurrent neural network architecture, such as LSTM or GRU, can work directly with mixed length sequences. This can work really well when each item in the sequence is the same kind of object but with different feature values.

Practical considerations:

RNNs are harder to understand and train than fully connected feed forward networks or CNNs.
If the two lists of units are just one part of the game representation, you will need a hybrid system that combines a RNN architecture with other neural network designs.


No result for scored labels in Azure ML Web Service

Test out the web service in the web client under configure to see if it works ...
There might be something wrong with the client.


If I use Gibbs sampling with a Bayesian model, what do I have to check is memoryless?

It is best practice to check for memoryless. It is also common to assume the process is memoryless, then use Gibbs sampling.


Identifying members who are likely to move to a lower cost alternative product

We could analyze the customer's purchase history. Here, for the sake of explanation let's assume that customers A and B. First, we'll make the purchase history of customer A,

We'll calculate the frequency of customer A buying the mid-range pan. Also, we'll need the total number of purchases made by the customer. So, for customer A you'll get a ratio which is ( number of purchases of the mid-range pan ) / total purchases.
Similarly, we'll calculate such ratios for the expensive pans both for customers A and B.
Next, we'll compare the ratios for all customers and select the highest N customers who most probably will buy the mid-range or expensive pan.

Why are using the ratios ( or precisely, the probabilities of buying a certain pan )?

We'll need customers who are frequent buyers of a particular pan. These are more chances of switching a product rather than the less frequent customers.
As they are regular customers, they will be more influenced by the discount available on the mid-range pan.

  You may use various metrics to aggregate the customers who can have a product transition. You may also have a look at Bayes Theorem which I think will be appropriate too. It will give the answer- "Which are customers which have the highest probability of buying a mid-range pan given that they have already bought a expensive pan.


regression to model exponential distribution

There was an MIT group that got a lot of attention a few years back for predicting twitter trends and doing so with really good ROC-AUC Scores.  The problem you are trying to solve is kind of reminiscent of the twitter trending problem.  The crux of the solution is that they avoided the hassle of parametric modeling through a grouping of similar time series followed by a classification.

There is a pretty good writeup of the method for the general public in Stan Nikolov's Blog post.  Notice that there is a video of a presentation by his thesis advisor also.

The basic idea of the method is to find the most common shapes in the time evolution hold those as the dominant classes.  These would be various exponential shapes in your case (from Nikolov's blog post):

Then you can take your training data and choose the best shape for each of your cases.  You can then use this data to train a classification algorithm.  Finally you can use this to predict the outcome of your new data (from Nikolov's blog post):

The full dissertation is also available online for download.

There's actually an entire company that is using this methodology and trying to sell it to companies.  They have some pretty cool promotional videos on youtube also, which won't teach you the methodology, but show how powerful it can be.

Hope this helps!


Validation loss much higher than training loss

Your model is completely overfitting. The training loss is constantly decreasing but the validation loss isn't. This means that the your current model is complex enough to 'memorize' the patterns in the training data. In such situations, you need to regularize your model.

To regularize your data you may try any or all combination of the following:

1) Reduce the layers of the neural network.

2) Reduce the number of neurons in each layer of the network to reduce the number of parameters.

3) Add dropout and tune its rate.

4) Use L2 normalisation on the parameter weights and tune the lambda value.

5) If possible add more data for training.

6) Adding more data may not always be economically viable option so 'Data Augmentation' can play a big role here. Data Augmentation depends on the type of input to the neural nets. For images they comprise of rotation, translation, affine transformation like shear etc.


I found out how to increase the size of my plot with the following code...

plt.subplots(figsize=(20,15))
sns.heatmap(corr)


regex to remove repeating words in a sentence

Since you were working with RegEx, I will ofer a RegEx solution.

I will also show that you need to also take care to first remove punctuation. (I will not go down the rabbit-hole of re-sinserting the punctuation back where it was!)

A RegEx solution:

import re
sentence = 'I need need to learn regex... regex from scratch!'

# remove punctuation
# the unicode flag makes it work for more letter types (non-ascii)
no_punc = re.sub(r'[^\w\s]', '', sentence, re.UNICODE)
print('No punctuation:', no_punc)

# remove duplicates
re_output = re.sub(r'\b(\w+)( \1\b)+', r'\1', no_punc)
print('No duplicates:', re_output)

Returns:

No punctuation: I need need to learn regex regex from scratch
No duplicates: I need to learn regex from scratch

\b : matches word boundaries
\w : any word character
\1 : replaces the matches with the second word found - the group in the second set of parentheses

The parts in parentheses are referred to as groups, and you can do things like name them and refer to them later in a regex. This pattern should recursively catch repeating words, so if there were 10 in a row, they get replaced with just the final occurence.

Have a look here for more detailed definitions of the regex patterns.

The more pythonic (looking) way

It has to be said that the groupby method has a certain python-zen feel about it! Simple, easy to read, beautiful.

Here I just show another way of removing the punctuation, making use of the string module, translating any punctuation characters into None (which removes them):

from itertools import groupby
import string

sentence = 'I need need to learn regex... regex from scratch!'

# Remove punctuation
sent_map = sentence.maketrans(dict.fromkeys(string.punctuation))
sent_clean = sentence.translate(sent_map)
print('Clean sentence:', sent_clean)

no_dupes = ([k for k, v in groupby(sent_clean.split())])
print('No duplicates:', no_dupes)

# Put the list back together into a sentence
groupby_output = ' '.join(no_dupes)
print('Final output:', groupby_output)

# At least for this toy example, the outputs are identical:
print('Identical output:', re_output == groupby_output)

Returns:

Clean sentence: I need need to learn regex regex from scratch
No duplicates: ['I', 'need', 'to', 'learn', 'regex', 'from', 'scratch']
Final output: I need to learn regex from scratch
Identical output: True

Benchmarks

Out of curiosity, I dumped the lines above into functions and ran a simple benchmark:

RegEx:

In [1]: %timeit remove_regex(sentence)
8.17 µs ± 88.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

groupby:

In [2]: %timeit remove_groupby(sentence)
5.89 µs ± 527 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

I had read that regex would be faster these days (using Python3.6) - but it seems that sticking to beautiful code pays off in this case!

Disclaimer: the example sentence was very short. This result might not scale to sentences with more/less repeated words and punctuation!


Categorical variables with multiple entries transformed to entity embedding

Why don't you try with a different kind of encoding?

One of the most typical encodings, when you have high cardinality, is Target Encoding. You can find an implementation in this library. You can find an explanation of what is it and when to apply it here.


SKLearn DT regressor - good enough score?

The quality of the score depends on the context of the project.

One approach is a model comparison approach. Define a baseline model. Then see if Decision Tree Regressor is better than the baseline.


Questions about LSTM cells, units and inputs

Is this correct?

Yes the diagrams both look correct to me. The key thing to understanding both diagrams is that the inputs and outputs of an LSTM cell are vectors.

The circles in the first diagram do represent the concept that the layer contains multiple individual artificial neurons, and that might make you assume that the second diagram is a picture of one of those neurons. Arguably there are multiple "neurons" or sub-layers with different roles inside a cell, because there are multiple places in which calculations of the form $f(W\mathbf{x} +b)$ occur, performing slightly different roles. I think the term "cell" is used to refer to this architecture of neurons. As a short-hand when I say "neuron", I tend to think of the output stage of the hidden layer.

However, in the second diagram, all of the operations shown work with vectors. Most importantly the left-to-right arrows in the second diagram represent vectors of hidden state from timestep to timestep from the whole layer. So each neuron in the cell is recurrently connected to every other neuron in that cell - twice as it happens in an LSTM, because LSTMs have both an internal cell state and a layer output.

  Is each unit in the cell independent from the others? Or do they share information?

To match your description of the diagrams, let's define a "unit" as a collection of one of each type of neuron/gate used to make up the cell, that in theory could be wired together to make a working LSTM cell layer with a single scalar cell state and output value.

These units are independent in that each has its own weight parameters. There are no shared parameters for the connections between input and the units, or for the recursive connections that forward state from one time step to the next. In that sense the units do not share information.

However, the connections do mean that on each time step, input data and hidden state plus output from last output from all other units in the cell are combined are used in calculations. Any cell unit can base its new internal state plus its output on the values of all other outputs and internal states from other units in the cell. In this sense, the units do share information. I guess from your question that it is probably this second issue that you are concerned about, as the second diagram makes you think of a wiring diagram for a single neuron, but as explained above that is not the case.

  Imagine I have the following configuration: Number of samples = 1000 Number of time-steps = 10 Number of features = 5
  In this case, each unit in a cell will take as input a vector of size 5 right?

Almost. Each neuron inside the cell will take an input of 5 from $\mathbf{x}$, plus an input of the hidden layer output, $\mathbf{h}$. So if in your case the LSTM cell size was 10, then each neuron would take a combined vector of 15. In addition, a second cell state vector is maintained, not labelled in your diagram. That is not used directly as an input to any neurons (i.e. components of the form $f(W\mathbf{x}+b)$), but does interact with the other values and can get changed itself, through the various gates. In your second diagram, it is the uppermost arrow going left to right.

  But what will be the size of the output for one unit? 1?

The cell as a whole will have the output of whatever size you have made the layer. That is what the diagram number 2 tries to show. However, using our working definition of "unit", the output of each unit will be two scalar values - the hidden layer output, and the cell state - which will be part of their respective vectors shown in the diagrams.


Does high regression coefficient for Principal components that don't explain much variance imply that my data is not a good predictor?

The variance of the data is just that, variance. Imagine I'm trying to predict y from x1 and x2, and the true underlying model is y = x1 + e, where e~N(0,.001) and x1~N(0,.01), and x2 is drawn independently from N(0,1). There is significantly more variance in x2, but it is useless in predicting y, where x1 would be an extremely good predictor.

EDIT:
We can implement this in R.

e = rnorm(100,0,.001)
x1 = rnorm(100,0,.01)
x2 = rnorm(100,0,1)
y = x1 + e
x = data.frame(cbind(x1,x2))
pca1 = prcomp(x)
pca1
summary(pca1)
model <- lm(y ~ ., data=data.frame(pca1$x))

Here is what our PCA looks like

PC1           PC2
x1 0.0003702204 -0.9999999315
x2 0.9999999315  0.0003702204
Importance of components:
                          PC1     PC2
Standard deviation     1.0466 0.01098
Proportion of Variance 0.9999 0.00011
Cumulative Proportion  0.9999 1.00000

So component PC1 represents 99.99% of our variance. Wow it must be a good predictor! Let's look at our model.

Call:
lm(formula = y ~ ., data = data.frame(pca1$x))

Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept) 7.589e-04  1.017e-04   7.463 3.64e-11 ***
PC1         3.561e-05  1.035e-04   0.344    0.732
PC2         1.002e+00  1.039e-02  96.424  < 2e-16 ***

Oh. It turns out it's useless for prediction.


I like this question!

Potential pitfalls:

1. Architecture is just not sophisticated enough to catch all nuances in the time-series image. Take a look at here what they did for earthquake-time-eries-as-images approach or take a look at this paper. You could take some guidelines from there, but you notice right away that the architecture is bigger. Or this one

2. Data, you say its predicting only one class, but what part of the image is it focusing on. Use shap values for images to inspect. I suspect following: it will be some trend that is consistent only for one class, hence you can think about pre-processing steps that can point to properties on images of other classes. Or even add some hand-crafted numeric features. In any case increase your dataset if you havent already, you want to have enough information to catch all nuances.


Find all potential similar documents out of a list of documents using clustering

As much as i understood from your description, you can just assign your labels as a new column to questions_df. There is no shuffling involved, so the following will do the trick:

questions_df["cluster"] = labels


Using NLTK in python you should first Tokenize the sentences into words, even you can use Ngram for 2-Gram or 3-Gram bags of word, the reason I am suggesting N-Gram is that let's suppose you have sentence like: I am not happy with this product, then 2-Gram tokenize it as ['not happy', 'happy with', 'with this', 'this product'] here I and am are assumed as STOPWORDS. Using HashingTF you can hash the sentence into a feature vector as ['word position': frequency of word, ...] i.e highly sparse vectors, For Hashing in PySpark check this documentation.

Here below python code will help you to tokenize in bags of word

import string

from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer

PUNCTUATION = set(string.punctuation)
STOPWORDS = set(stopwords.words('english'))
STEMMER = PorterStemmer()

example = ['Hello Krishna Prasad, this is test file for spark testing',
            'Another episode of star',
            'There are far and away many stars'
            'A galloping horse using two coconuts'
            'My kingdom for a horse'
            'A long time ago in a galaxy far']

def tokenize(text):
    tokens = word_tokenize(text)
    lowercased = [t.lower() for t in tokens]
    no_punctuation = []
    for word in lowercased:
        punct_removed = ''.join([letter for letter in word if not letter in PUNCTUATION])
        no_punctuation.append(punct_removed)
    no_stopwords = [w for w in no_punctuation if not w in STOPWORDS]
    stemmed = [STEMMER.stem(w) for w in no_stopwords]
    return [w for w in stemmed if w]

tokenized_word = [tokenize(text) for text in example]
for word in tokenized_word:
    print word

Out of the above code as:

$python WordFrequencyHash.py
[u'hello', u'krishna', u'prasad', u'test', u'file', u'spark', u'test']
[u'anoth', u'episod', u'star']
[u'far', u'away', u'mani', u'starsa', u'gallop', u'hors', u'use', u'two', u'coconutsmi', u'kingdom', u'horsea', u'long', u'time', u'ago', u'galaxi', u'far']

You can also use word2vec or countvectorizer for tokenization.


How to make a decision tree with both continuous and categorical variables in the dataset?

Decision trees can handle both categorical and numerical variables at the same time as features, there is not any problem in doing that.
Theory
Every split in a decision tree is based on a feature. If the feature is categorical, the split is done with the elements belonging to a particular class. If the feature is contiuous, the split is done with the elements higher than a threshold. At every split, the decision tree will take the best variable at that moment. This will be done according to an impurity measure with the splitted branches. And the fact that the variable used to do split is categorical or continuous is irrelevant (in fact, decision trees categorize contiuous variables by creating binary regions with the threshold).
Implementation
Although, at a theoretical level, is very natural for a decision tree to handle categorical variables, most of the implementations don't do it and only accept continuous variables:

This answer reflects on decision trees on scikit-learn not handling categorical variables. However, one of the scikit-learn developers argues that

At the moment it cannot. However RF tends to be very robust to categorical features abusively encoded as integer features in practice.

This other post comments about xgboost not handling categorical variables.
rpart in R can handle categories passed as factors, as explained in here
Lightgbm and catboost can handle categories. Catboost does an "on the fly" target encoding, while lightgbm needs you to encode the categorical variable using ordinal encoding.

Here's an example of how lightgbm handles categories:
import pandas as pd
from sklearn.datasets import load_iris
from lightgbm import LGBMRegressor
from category_encoders import OrdinalEncoder

X = load_iris()['data']
y = load_iris()['target']

X = OrdinalEncoder(cols=[3]).fit_transform(X)

dt = LGBMRegressor()
dt.fit(X, y, categorical_feature=[3])


Memory issue when trying to initiate zero tensor with pytorch

The reason the tensor takes up so much memory is because by default the tensor will store the values with the type torch.float32. This data type will use 4kb for each value in the tensor (check using .element_size()), which will give a total of ~48GB after multiplying with the number of zero values in your tensor (4 * 2000 * 2000 * 3200 = 47.68GB). What you could try to do is change the datatype from torch.float32 to something like torch.int8, which only uses 1kb for each value, reducing the memory needed by 75% to 12GB. But given that you only seem to have 8GB available this will also not solve your problem. The only solution therefore would simply be to use a tensor will fewer values.


Why do histogram bars vanish when we keep the bins value high in matplotlib?

The height of a bar shows the number of measurements within that range.

The bars get shorter when you increase the bin count, because the more bins you have in total, the less measurements fall into an individual bin.


How to arrange the sets to predict y on x in time series?

fig, ax = plt.subplots(1, 1, figsize=(24,7))
for train_index, test_index in tscv.split(X_train1_raw, y_train1_raw):
     X_train1, X_test1 = scaler.fit_transform(X_train1_raw[train_index].reshape(-1,1)), scaler.fit_transform(X_train1_raw[test_index].reshape(-1,1))
     y_train1, y_test1 = scaler.fit_transform(y_train1_raw[train_index].reshape(-1,1)), scaler.fit_transform(y_train1_raw[test_index].reshape(-1,1))
     pyplot.subplot(510 + index)
     pyplot.plot(X_train1[:, 1])
     pyplot.plot([None for i in X_train1[:, 1]] + [x for x in X_test1[:, 1]])
     index +=1
pyplot.show();

This should fix your code and this happens because the Standard Scaler expects inputs as samples where each sample is an array of n_features to transform. The reshape(-1,1) does that for you if you have got a single feature to scale.


Short answer

Yes, you can use the counts as the target you are trying to predict. I would say that you should probably normalize this, so that your target is between 0 and 1. This will help collaborative filtering converge faster.

Long answer

Given the input data you describe, you have to remember the concept of "Garbage in, garbage out"

Now I am not saying that your input is garbage, far from that. What I mean is that if all you know is how many times somebody played a song, this is also all you will be able to learn from and eventually predict.

In your case, it's more of a matter of interpreting the output. Let's imagine that your model works extremely well and is able to predict how many times someone will listen to a song. In that case, it would indeed make sense to recommend them songs that they will indeed listen to a lot because the assumption is that you will listen to songs more often if you like them. But of course, you have to remember that this might not be entirely true


Adapting ZFNet on 2244x224 image using a filter 7X7

As per the formula for the feature map dimension:
$$feature_{dim} = \frac{n+2p-f}{S} + 1$$
The values for :

n = 224
p = 0
f = 7
s = 3

$$feature_{dim} = \frac{224+2*0-7}{3} + 1 = 73.66 $$
As you've guessed this is not the size of the feature map.
Tensorflow takes it as 73.
If you're relying on the formula, you are missing out on a concept, that this should a process were in the kernel slides over the Input hence the feature map dimension should be an integer. So what happens is that the kernel while sliding with a stride of 3 leaves out the last few pixels and won't reach the other edge.
If you're trying to get a feature map that's an integer by keeping a constant filter size of 7. Then you're stride is :
$$S = 217 /(N - 1)$$
where N is you're desired output size.
If you choose N to be 8 or 32, you'll end up with a stride of 7 or 31. It's better to choose S = 7 to get most of the information. But still, it doesn't matter as TensorFlow has checks for the same to prevent errors.


Keypoint detection from an image using a neural network

You could train YOLO to detect and create bounding box around the circle and then calculate its center. Or even segmentation could help you do this precisely.


How to play with .cat.codes and get_dummies

You can convert object dtype to category using astype("Category") and then do label encoding on it using .cat.codes but it's better to use sklearn.preprocessing.LabelEncoder for it.
For basic analysis you should look into seaborn


There are so many ways to go wrong with a neural net that it's going to be difficult to debug. Also, to address your intuition, each additional hidden layer makes learning much harder. With that said, here are some possibilities:

You have added weight decay. Adding more layers adds more weights which increases your regularization cost.
The problem is simple enough that a model with a single hidden layer is sufficient. Adding more hidden layers makes it harder for the network to learn (harder optimization problem).
The optimization method is not doing a great job (I prefer climin to scipy.optimize).
You are using the sigmoid/tanh activation function. The sigmoid function causes the vanishing gradient problem which makes learning hard with more layers. Try using the ReLu function.

Training neural nets takes a lot of practice, luck, and patience. Good luck.


Clustering for mixed numeric and nominal discrete data

Taking a stab:

  I am trying to identify a clustering technique with a similarity measure that would work for categorical and numeric binary data.

Gower Distance is a useful distance metric when the data contains both continuous and categorical variables.

  There are techniques in R kmodes clustering and kprototype that are designed for this type of problem, but I am using Python and need a technique from sklearn clustering that works well with this type of problems.

I wasn't able to find an implementation of Gower Distance in Python when I searched for it about 4-5 months back. So I came up with my own implementation.

import pandas as pd
import numpy as np
from sklearn.neighbors import DistanceMetric

def gower_distance(X):
    """
    This function expects a pandas dataframe as input
    The data frame is to contain the features along the columns. Based on these features a
    distance matrix will be returned which will contain the pairwise gower distance between the rows
    All variables of object type will be treated as nominal variables and the others will be treated as
    numeric variables.
    Distance metrics used for:
    Nominal variables: Dice distance (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient)
    Numeric variables: Manhattan distance normalized by the range of the variable (https://en.wikipedia.org/wiki/Taxicab_geometry)
    """
    individual_variable_distances = []

    for i in range(X.shape[1]):
        feature = X.iloc[:,[i]]
        if feature.dtypes[0] == np.object:
            feature_dist = DistanceMetric.get_metric('dice').pairwise(pd.get_dummies(feature))
        else:
            feature_dist = DistanceMetric.get_metric('manhattan').pairwise(feature) / np.ptp(feature.values)

        individual_variable_distances.append(feature_dist)

    return np.array(individual_variable_distances).mean(0)

The link to the same piece of code: https://github.com/matchado/Misc/blob/master/gower_dist.py

With regards to the clustering technique, I haven't used the ones you've mentioned. But I've used hierarchical clustering in R along with gower distance with success in the past.

Looking into the clustering techniques available in scikit learn, Agglomerative Clustering seems to fit the bill. http://scikit-learn.org/stable/modules/clustering.html#hierarchical-clustering

  I want to build profiles of segments of individuals. meaning this group of individuals care more about these set of features.

Once you've assigned cluster labels to each row of your data, for each cluster look into the distribution of the features (summary stats for continuous variables & frequency distributions for categorical variables). This is easier to analyze visually if your number of features are manageable (<20 maybe?).

But since you have 100+ features, I suggest a more organized approach. Create a matrix with cluster labels in the columns and the summary stat of the features in the rows (I suggest using median for continuous variable and percentage occurrence of most frequent value in cluster for categorical variable)

It might look something like this.

╔═══════════════════════╦═══════════╦═══════════╦════╦═══════════╗
║        Feature        ║ Cluster 1 ║ Cluster 2 ║ …  ║ Cluster N ║
╠═══════════════════════╬═══════════╬═══════════╬════╬═══════════╣
║ Numeric feature 1     ║ 15        ║ 37        ║ .. ║ 1         ║
║ Numeric feature 2     ║ 34        ║ 56        ║ …  ║ 56        ║
║ Categorical feature 1 ║ 47%       ║ 87%       ║ …  ║ 25%       ║
║ …                     ║ …         ║ …         ║ …  ║ …         ║
║ Categorical feature N ║ 25%       ║ 91%       ║ …  ║ 11%       ║
║ Numeric feature N     ║ 0.2       ║ 0.7       ║ …  ║ 0.5       ║
╚═══════════════════════╩═══════════╩═══════════╩════╩═══════════╝


Cosine similarity between query and document confusion

You want to use all of the terms in the vector.

In your example, where your query vector $\mathbf{q} = [0,1,0,1,1]$ and your document vector $\mathbf{d} = [1,1,1,0,0]$, the cosine similarity is computed as

similarity $= \frac{\mathbf{q} \cdot \mathbf{d}}{||\mathbf{q}||_2 ||\mathbf{d}||_2} = \frac{0\times1+1\times1+0\times1+1\times0+1\times0}{\sqrt{1^2+1^2+1^2} \times \sqrt{1^2+1^2+1^2}} = \frac{0+1+0+0+0}{\sqrt{3}\sqrt{3}} = \frac{1}{3}$


Brown clustering, graph partitioning, agglomerative clustering - libraries/software

For graphs I definitely suggest Networkx and igraph libraries. They support many graph partitioning algorithms.


Why some papers write false positive rate per case instead of percentage rate?

In general a false positive rate is abbreviated FPR, so it's likely that "FP: 128.52 [/case]." is not the rate but simply the number of false positive instances. In this case they should also mention the total number of instances by case, and you can easily obtain the rate: FPR = FP/total


What is the correct formula for Jaccard coefficient with integer vectors?

The Jaccard coefficient (or Jaccard similarity) is defined on two sets $A$ and $B$:
$$
J(A,B) = {{|A \cap B|}\over{|A \cup B|}} = {{|A \cap B|}\over{|A| + |B| - |A \cap B|}}
$$
There is a single definition for this coefficient, but note that Jaccard is a general similarity measure, it is not specifically designed as an evaluation measure. So assuming one wants to use it as an evaluation measure, they need to decide (design) how. More precisely confusion may arise from the following questions:

Which pair of sets does one want to compare?
Which data structure does one use to represent a set?

Let's start with the second question: a binary vector is a common and convenient way to represent (encode) a set. Given a set of possible elements (universe) $U=\{x_1, ...,x_n\}$, any subset $s\subseteq U$ can be represented as a binary vector of length $n$ where the $i^{th}$ boolean is 1 if $x_i\in s$, 0 otherwise. It follows from this representation that the Jaccard coefficient is the number of indexes where both vectors contain 1 divided by the the number of indexes where at least one of the two vectors contain 1.
Example from scikit documentation:
>>> y_true = np.array([[0, 1, 1],
...                    [1, 1, 0]])
>>> y_pred = np.array([[1, 1, 1],
...                    [1, 0, 0]])
>>> jaccard_score(y_true[0], y_pred[0])
0.6666...

The vectors compared are [0, 1, 1] and [1, 1, 1] (meaning for instance sets $\{b,c\}$ and $\{a,b,c\}$), so the result is 2/3.
Multi-label classification is when an instance can be assigned any number of classes. In the linked scikit example they apply the Jaccard coefficient to the two sets of classes for every instance (optionally aggregating the results across instances). This is an example of a design choice as mentioned in my first question above. Note: in the case of the multiclass problem, I don't even understand how they obtain this result for the third value so I can't comment about it.
The formula $TP / (TP + FP + FN)$ is another example of what does one choose as sets. This formula only makes sense for a binary classification task: TP is the intersection of true and predicted positive instances, $TP+FP+FN$ is the union of all positive cases (whether true or predicted). This is why it doesn't work with integer vectors, since these cannot represent the result of binary classification.
Conclusion: one must define which sets they want to compare. The most simple way with integer vectors is to consider them as sets of integers, in this case they could be represented as binary vectors as follows:
[0, 1, 2, 5, 6, 8, 9] -> [1,1,1,0,0,1,1,0,1,1]
[0, 2, 3, 4, 5, 7, 9] -> [1,0,1,1,1,1,0,1,0,1]

But going through the binary representation is not a must, it's only needed if one wants to use a function which takes this format as input. One could as well define their own function using a set as data structure for instance.


Determine statistically whether new product cannibalise old product by using data

You do not have the data to directly answer the question if a ride for a given service is new customer or old customer. You need to have a customer id to properly attribute if there is new growth or just service switching.

Any statistics you run will, at best, show a correlation between new service and increased revenue. There can be no causation attributed. For example, the changes could be due to seasonal differences or other factors unrelated to the new service.


What is the best architecture for Auto-Encoder for image reconstruction?

I don't know about an architecture being definitively the best, but there are some best practices you can follow. Check out these papers:

Learning to Generate Images with Perceptual Similarity Metrics
Push it to the Limit: Discover Edge-Cases in Image Data with Autoencoders
Walking the Tightrope: An Investigation of the Convolutional Autoencoder Bottleneck

To sum it up, residual blocks in between downsampling, SSIM as a loss function, and larger feature map sizes in the bottleneck seem to improve reconstruction quality significantly. How that translates to the latent space is not entirely clear yet.


How to get the weights of a linear model by solving normal equation?

Hey I know this is a year old but I wanted to provide the answer in case you haven't got it. So page 124 says how we derive the normal equations, however the following few paragraphs explains how to deal with a bias term.
You've got our dataset right, however we need to add an extra 1 to take the place of our bias in the weight vector. I'll append the ones at the end.
$\mathbf{X} = \begin{bmatrix}
0 & 0 & 1 \\
0 & 1 & 1 \\
1 & 0 & 1 \\
1 & 1 & 1 \\
\end{bmatrix} $
You've got our $y$ vector correct though, i'll write it down below for completeness sakes
$\mathbf{y} = \begin{bmatrix}
0 \\
1 \\
1 \\
0 \\
\end{bmatrix} $
Now we compute solution to the normal equations, $w = (X^{T}X)^{-1}X^{t}y$, below
$\mathbf{w} = \begin{bmatrix}
0 \\
0 \\
\frac{1}{2} \\
\end{bmatrix}$
Now the element in the 3rd position of our vector is the bias term and hence $b = \frac{1}{2}$ and the first two element of $\mathbf{w}$ is our weight vector equalling zero.
Hopefully you find this useful!


Estimating sales volume of new drugs to a therapeutic class

The first approach is a supervised machine learning model. If you have historical data, you can label members who transfer to the new drug when a new drug came out as label one, otherwise label zero. Then you have a supervised machine learning setting. Just use your other information as features. In the end, you count how many members are predicted to take the new drug.

The second approach is an unsupervised machine learning model. Using clustering to classifies members into two groups. You then count the group whose members will take new drugs. The problem of this method is that you need to have couple samples's label determined so that you can know which group is the group you need.

The above two methods all require that you have historical data.

I am thinking if there are other methods that can give an expected value and don't require training on the historical data. I will update once I figure out one.


Error using image: Error when checking input: expected dense_56_input to have 2 dimensions, but got array with shape

The error occurs due to mismatch in the input shape. In the model you have specified it as input_shape = (784,) but the actual images that the model is getting as input are of different size. Just put the input shape as input_shape = (32, 32, 3) and you are good to go. Take a look here for more details on how to specify the input_shape.

Also, you'll have to use convolutional layers to process images or you can use a flatten layer before using dense layers!


Is my data good for (DBSCAN) clustering?

You can look at several clustering metrics (e.g., Silhouette Coefficient, Calinski-Harabasz index, or Davies-Bouldin index) and see if they revealing patterns in your clusters.
You can look at relative performance by trying different hyperparameters for DBSCAN and comparing DBSCAN to other clustering techniques.


Which train test split performs better: 50:50 or 60:40?

It sounds like you have a lot of data, so probably a simple train-test split is enough. No need for cross validation.

I would just use something like 75-25. That is in fact the default value in sklearn.

I would use less data in the training only if your algorithm is too slow and cannot cope with the extra data. In that case, instead of throwing away the data, you might as well use it for validation or for testing hyperparameters.

All this being said, more important than all is how you split the data. You should make sure that customers in the testing data are not in the training data to make sure your algorithm is generalizing, and not merely memorizing customers. This is standard procedure in medical data mining, and it is very important. Make sure you do not have customer overlap.

You might also want to make sure that the distribution of the variable you want to predict is similar in the training and the testing data.


Fixed target values but more ID for clafssification

I believe all id's are unique(no repetitions/duplicates). You can delete this variable and proceed for analysis. The variables like this(ids) and the variables which have the same values (zero variance) are not at all able to draw patterns to predict the target variable. You can delete these type of variables without a doubt.


Training with multi-series of different length with stateful LSTM

To begin with, you can think of the batch size as a way to control the smoothness of the learning curve. With a huge batch size, you are taking the average of many errors for each update, and this average loss (on average), doesn't have great variance.

Using a batch size of 1, your cost on each iteration is solely dependent on the single sample that you fed the network. Each sample is hopefully a little different from the others, which will lead to a very noisy loss curve.

Assuming model parameters are well-suited and the model converges irrespective of batch size, you should reach similar results. However, there are works that analyse batch size and show other trade-off, which are usually training-times, memory consumption and the like.

This recent paper (Masters, Luschi) analyses the trade-offs of batch size on a standard dataset (CIFAR10). Here is Figure 15 from the paper that shows this pretty succinctly:

Warning: all models and datasets could behave differently, however, so don't take any results like these as fact!

I know that it is possible to train a model that has varying input shapes by using a technique called Adaptive Pooling (or adaptive average pooling, in PyTorch, but you would likely have to come up with your own function that is able to do such a thing within the constraints of a stateful LSTM.

So as the shape of your dataset must be divisible by the batch size, there are a few ways to make that a reality:

using the highest common multiple of your datasets: so with your example datasets having 4000 days (set A) and 500 days (set B), you would compute the

This of course limits the possible choices of sequence length, but for the two examples of 4000 and 500, you could choose from these: 1, 2, 4, 5, 10, 20, 25, 50, 100

Trimming the different datasets so that you get a nice sequence length that works for both

This means possible leaving out some of your data, which is undesireable, but it might not be much.

Which of those two possiblities you might go for will depend on your specific dataset sizes. It might be optimal to use both - so trimming a few datasets to allow the computation of a nice Highest Common Factor.

You can compute the HCF in Python using something like this:

def hcf(a, b):
    while b:
        a, b = b, b % a
    return a

One final idea from my comment below:

... in order to change the batch size between datasets, you could copy them model weights from a trained model (from your first dataset), then compile a new model with the batch size required for the next dataset, but set the weights equal to those from the first model. This is almost like a manual way of implementing stateful behaviour. Have a look here for a simple example.


What is the distribution of categories in imagenet training set (ILSVRC2012)

I couldn't find an URL text file for the ILSVRC2012 training set, but for complete imagenet you can download the URLs only as a text file: http://image-net.org/download

I wrote the following script to get a feeling for the data:

#!/usr/bin/env python

"""Analyze the distribution of classes in ImageNet."""

classes = {}
images = 0

with open("fall11_urls.txt") as f:
    for i, line in enumerate(f):
        label, _ = line.split("\t", 1)
        wnid, _ = label.split("_")
        if wnid in classes:
            classes[wnid] += 1
        else:
            classes[wnid] = 1
        images += 1

# Output
print("Classes: %i" % len(classes))
print("Images: %i" % images)

class_counts = [count for _, count in classes.items()]
import matplotlib.pyplot as plt
plt.hist(class_counts, bins=range(max(class_counts)))
plt.show()

which gave:

Classes: 21841
Images: 14197122

Classes which have less than 100 examples are pretty much useless. Lets remove them from the plot. Also increase the bin size to 25:

#!/usr/bin/env python

"""Analyze the distribution of classes in ImageNet."""

classes = {}
images = 0

with open("fall11_urls.txt") as f:
    for i, line in enumerate(f):
        label, _ = line.split("\t", 1)
        wnid, _ = label.split("_")
        if wnid in classes:
            classes[wnid] += 1
        else:
            classes[wnid] = 1
        images += 1

# Output
print("Classes: %i" % len(classes))
print("Images: %i" % images)

class_counts = [count for _, count in classes.items()]
import matplotlib.pyplot as plt
plt.title('ImageNet class distribution')
plt.xlabel('Amount of available images')
plt.ylabel('Number of classes')
min_examples = 100
bin_size = 25
plt.hist(class_counts, bins=range(min_examples, max(class_counts), bin_size))
plt.show()

Or with seaborn:

import seaborn as sns
sns.distplot(class_counts, kde=True, rug=False);
sns.plt.show()

Top 10

The top 10 classes with most data are:

top10 = sorted(classes.items(), key=lambda n: n[1], reverse=True)[:10]
for class_label, count in top10:
    print("%s:\t%i" % (class_label, count))

n02094433:    3047 (Yorkshire terrier)
n02086240:    2563 (Shih-Tzu)
n01882714:    2469 (koala bear, kangaroo bear, native bear, )
n02087394:    2449 (Rhodesian ridgeback)
n02100735:    2426 (English setter)
n00483313:    2410 (singles)
n02279972:    2386 (monarch butterfly, Danaus plexippus)
n09428293:    2382 (seashore)
n02138441:    2341 (meerkat)
n02100583:    2334 (vizsla, Hungarian pointer)

Using http://www.image-net.org/api/text/wordnet.synset.getwords?wnid=n02094433 you can look the names up.


Why do we have to divide by 2 in the ML squared error cost function?

It is simple. It is because when you take the derivative of the cost function, that is used in updating the parameters during gradient descent, that $2$ in the power get cancelled with the $\frac{1}{2}$ multiplier, thus the derivation is cleaner. These techniques are or somewhat similar are widely used in math in order "To make the derivations mathematically more convenient". You can simply remove the multiplier, see here for example, and expect the same result.


What approach for creating a multi-classification model based on all categorical features (1 with 5,000 levels)?

Personally, I think that building a predictor on such user Ids is not that efficient neither useful. Can you change your problem in a different manner?

When we do any analysis we drop those ids as they are of no use they are only the incidents in a large population and we try to group them in categories which describe their features. For example, users which have good eyesight will play basketball even if it rains.

So try to think like that.


How can I build a self-attention model with tf.keras.layers.Attention?

How about this? https://pypi.org/project/keras-self-attention/
I use it but I have not tried it without the LSTM layer yet. Also, you have to use keras directly, not tensorflow.keras. Here is a working model I am running right now.
    model = keras.models.Sequential()
    model.add(keras.layers.LSTM(cfg.LSTM, input_shape=(cfg.TIMESTEPS,
              cfg.FEATURES),
              return_sequences=True))
    model.add(SeqSelfAttention(attention_activation='sigmoid'))
    model.add(keras.layers.Dense(cfg.DENSE))
    model.add(keras.layers.Dense(OUTPUT, activation='sigmoid'))

Also, look here, there is quite a bit https://awesomeopensource.com/projects/attention-mechanism


Try 'arrange(Data.frame.name, ID)' function from package 'dplyr'


Classification model to classify large number of classifiers?

It looks like a very difficult problem, since there are many possible classes and very little information in the features to distinguish them. For the record, the reverse problem of estimating the travel time based on the route would probably be more feasible.

So you can't expect great performance on a problem like this, the goal will be to design the problem in a way which makes things as simple as possible for the classifier to do a decent enough job. Here are some suggestions:

Start with training a model specific to a pair BSTN,ASTN.
Discard the least likely routes, i.e. routes which are rarely used for the pair BSTN,ASTN (for instance routes with frequency lower than 10).
Inspect the data to see if the features allow a distinction between the (main) classes. For instance you can plot the distribution of BSEC, TFtime, Ttime for different routes: if the distributions are close there's little chance the classifier will succeed. You can also train a decision tree and inspect it manually so see what happens.


Question About Discriminator of CycleGan

This is explained in the original CycleGAN paper:

For the discriminator networks we use 70 × 70 PatchGANs, which aim to classify whether 70 × 70 overlapping image patches are real or fake. Such a patch-level discriminator architecture has fewer parameters than a full-image discriminator and can work on arbitrarily sized images in a fully convolutional fashion.

Therefore, the arguments for using patch-based discriminators are that:

It has fewer parameters.
It enables using them in arbitrarily sized images.


Performing a linear regression with Perceptron

If you use a linear activation function on your perceptron you are essentially creating a linear regression where the weights connecting to your perceptron are analogous to the the coefficients on your linear regression model. The only difference in this case is you would normally fit your perceptron using backpropagation, but for the linear regression model you would likely use OLS. In either case - you will end up with a series of coefficients multiplying on your input variables in a linear fashion with some bias term.
This of course is the most basic use of a perceptron. The advantage of the perceptron is you can use non-linear activation functions to fit non-linearities in your data, and your ability to stack them together to form a larger neural network.


RVM is identical to SVM, but provides a probability distribution of scores.

RVM:

Pros:

RVM is better than SVM in terms of accuracy.(depends on size of training data), more training data implies more accuracy in RVM. (argued and experimentally shown in the paper [1])
Since it provides the probability distribution of score, not a point estimate. It opens up a variety of options to get point estimate out of Bayesian inference, like percentile, mean, median, average.

Cons:

Since it uses EM algorithm for learning, it might get stuck at local minima.
Large data is required for learning

SVM:

Pros:

Standard SVM is a convex optimization problem(both hard and soft), therefore they always produce unique solution i.e. converge to global optima.

There is one very good paper about this. You can get more details here:

[1] (PDF) A comparative study of relevant vector machine and support vector machine in uncertainty analysis., accessed Jan 08 2019.


How to implement single Imputation from conditional distribution?

This is just an intuitive explanation of a group of a strategy for imputing missing data.

In practice, the distribution $P(x_{mis}|x_{obs};\mathbf{\theta})$ is unknown and can be estimated at best. The best way to estimate this probability is use-case specific. Understanding how the training data was collected can help you in estimating/defining this conditional distribution.

In practice, we often do not try to get a good estimation. Keeping things simple and assuming all features are sampled from a normal distribution might get you started.

This is $x_{mis}$ follows $N(\mu, \sigma)$ where

$\mu = \sum_i^N \frac{x_{obs,i}}{N}$
$\sigma = \sum_i^N \frac{(x_{obs,i}-\mu)^2}{N-1}$

. However, such assumptions are rarely realistic and will do guarantee good models.
See this.


Estimating Propensity Score via Regression Trees (in R Using rpart)

You can use logistic regression itself, since your target variable is binary.

While predicting, use type = "prob"
This will provide the propensities which is basically just the probability of being a 1 or 0.


What is the difference? "Adding more LSTM layers" or "Increasing epochs"?

Increasing the number of epochs :
This is the best when you have a large and diverse dataset. A large dataset refers to a number of diverse samples in equal amounts ( no bias ). Also, you make your model generalise better, you might increase the dropout rates and decrease the learning rate. At such a time increasing the number of epochs would help the model generalise better. Also, since it has generalised on a diversed dataset means it can perform well in the real world. For LSTMs, they can make correct predictions based on the sequences.
Increasing the number of layers of LSTM :
If we treat each LSTM as a memory unit then we are increasing the number of memory units and thus the overall memory. The first sequences would be feed to the next layer hence a model could create a hierarchical representation of the data. This is better if you have sequences which have such a context like an image.

On considering both the scenarios, if you have abundant data then the first option is optimum. If you have complex data which has less diversity then the second and the first points need to be considered.


Ok. I think I got it now. It should be like this:

Am I right? If not: whats wrong?


There are a couple of design patterns that are contributing to slow code:

Pandas is not designed for large-scale, fast data processing.

Your code is using a for loop which can be slow.

You are manifesting the sliding window before the program needs it. It might be better to create a view on the data and then manifest the data in memory only when the training needs it.

It would be better to use NumPy with numpy.lib.stride_tricks.sliding_window_view


As the error says: there is no categorical_values parameter for OneHotEncoder.  It was removed at the same time that OneHotEncoder was extended to deal with strings directly, and you may want to use ColumnTransformer for selecting out the categorical column(s).  For example, https://datascience.stackexchange.com/a/57383/55122


SMOTE train test split with validation data

Problem in applying smote on data and than applying the split (test/or Validation does not matter) is that you could suffer from data leakage. Meaning that some Information from the Train could Spill over to the future and falsly give good predictions. I would advise seperating smote data Generation process for all 3 data sets. So do the splits, than do the data Generation.


In what way is recurrent neural network state "hidden"?

The "hidden" layers and state in a neural network are those that do not have direct input or output with respect to the function being learned.
I don't think you need to read anything deep into the analogy. The term "hidden" was, according to Geoffrey Hinton, coined by him partly because he liked the mysterious sound of it (from hidden markov models). He freely admits that it is partly an awkward name since in the case of deep neural networks, the data in the hidden layers is not inaccessible or unknown.
I heard him state this in one of the lectures in a now closed Coursera course.
Sadly I am not able to find a current reference to this in his own words. I have found a reference to the same story in the book Backpropagation: Theory, Architectures, and Applications.


What exactly is bootstrapping in reinforcement learning?

Bootstrapping in RL can be read as "using one or more estimated values in the update step for the same kind of estimated value".

In most TD update rules, you will see something like this SARSA(0) update:

$$Q(s,a) \leftarrow Q(s,a) + \alpha(R_{t+1} + \gamma Q(s',a') - Q(s,a))$$

The value $R_{t+1} + \gamma Q(s',a')$ is an estimate for the true value of $Q(s,a)$, and also called the TD target. It is a bootstrap method because we are in part using a Q value to update another Q value. There is a small amount of real observed data in the form of $R_{t+1}$, the immediate reward for the step, and also in the state transition $s \rightarrow s'$.

Contrast with Monte Carlo where the equivalent update rule might be:

$$Q(s,a) \leftarrow Q(s,a) + \alpha(G_{t} - Q(s,a))$$

Where $G_{t}$ was the total discounted reward  at time $t$, assuming in this update, that it started in state $s$, taking action $a$, then followed the current policy until the end of the episode. Technically, $G_t = \sum_{k=0}^{T-t-1} \gamma^k R_{t+k+1}$ where $T$ is the time step for the terminal reward and state. Notably, this target value does not use any existing estimates (from other Q values) at all, it only uses a set of observations (i.e., rewards) from the environment. As such, it is guaranteed to be unbiased estimate of the true value of $Q(s,a)$, as it is technically a sample of $Q(s,a)$.

The main disadvantage of bootstrapping is that it is biased towards whatever your starting values of $Q(s',a')$ (or $V(s')$) are. Those are are most likely wrong, and the update system can be unstable as a whole because of too much self-reference and not enough real data - this is a problem with off-policy learning (e.g. Q-learning) using neural networks.

Without bootstrapping, using longer trajectories, there is often high variance instead, which, in practice, means you need more samples before the estimates converge. So, despite the problems with bootstrapping, if it can be made to work, it may learn significantly faster, and is often preferred over Monte Carlo approaches.

You can compromise between Monte Carlo sample based methods and single-step TD methods that bootstrap by using a mix of results from different length trajectories. This is called TD($\lambda$) learning, and there are a variety of specific methods such as SARSA($\lambda$) or Q($\lambda$).


Intuition for the regularization parameter in SVM

The regularization parameter (lambda) serves as a degree of importance that is given to misclassifications. SVM pose a quadratic optimization problem that looks for maximizing the margin between both classes and minimizing the amount of misclassifications. However, for non-separable problems, in order to find a solution, the miclassification constraint must be relaxed, and this is done by setting the mentioned "regularization".
So, intuitively, as lambda grows larger the less the wrongly classified examples are allowed (or the highest the price the pay in the loss function). Then when lambda tends to infinite the solution tends to the hard-margin (allow no miss-classification). When lambda tends to 0 (without being 0) the more the miss-classifications are allowed.
There is definitely a tradeoff between these two and normally smaller lambdas, but not too small, generalize well. Below are three examples for linear SVM classification (binary).

For non-linear-kernel SVM the idea is the similar.
Given this, for higher values of lambda there is a higher possibility of overfitting, while for lower values of lambda there is higher possibilities of underfitting.
The images below show the behavior for RBF Kernel, letting the sigma parameter fixed on 1 and trying lambda = 0.01 and lambda = 10

You can say the first figure where lambda is lower is more "relaxed" than the second figure where data is intended to be fitted more precisely.
(Slides from Prof. Oriol Pujol. Universitat de Barcelona)


Cosine similarity with arrays contaning NaN

I think it's rarely meaningful to consider cosine similarity on sparse data like this, not just because of sparsity (because it's only defined for dense data), but because it's not obvious the cosine similarity is meaningful. For example a user that rates 10 movies all 5s has perfect similarity with a user that rates those 10 all as 1. Magnitude doesn't matter in cosine similarity, but it matters in your domain.

It's much more likely that it's meaningful on some dense embedding of users and items, such as what you get from ALS.

To answer the question, you either need to impute the missing ratings (don't assume 0, but a mean value or similar), or ignore dimensions that aren't defined in both.


Can HDF5 be reliably written to and read from simultaneously by separate python processes?

HDF5 parallel I/O will not solve this problem.  That technology is primarily intended for performance, not for collision avoidance.
What you want is know as SWMR (single-writer/multiple-reader):

Data acquisition and computer modeling systems often need to analyze and visualize data while it is being written. It is not unusual, for example, for an application to produce results in the middle of a run that suggest some basic parameters be changed, sensors be adjusted, or the run be scrapped entirely.
To enable users to check on such systems, we have been developing a concurrent read/write file access pattern we call SWMR (pronounced swimmer). SWMR is short for single-writer/multiple-reader. SWMR functionality allows a writer process to add data to a file while multiple reader processes read from the file.

SWMR was first included in HDF5 version 1.10.0 released on 2016-03-30

Concurrent Access to HDF5 Files - Single Writer/ Multple Reader (SWMR)
The Single Writer/ Multiple Reader or SWMR feature enables users to
read data concurrently while writing it. Communications between the
processes and file locking are not required. The processes can run
on the same or on different platforms as long as they share a common
file system that is POSIX compliant.


Python: ggplot geom_histogram() binwidth or bin parameter missing?

bins=10 parameter is actually added in latest version (ggplot 0.11) https://github.com/yhat/ggplot/issues/513


I don't think any of the clustering techniques "just" work at such scale. The most scalable supposedly is k-means (just do not use Spark/Mahout, they are really bad) and DBSCAN (there are some good distributed versions available).

But you will be facing many other challenges besides scale because clustering is difficult. It's not as if it's just enough to run the algorithm and then you have clusters. Clustering is an explorative technique. There is no "correct" clustering. But rather you will need to run clustering again and again, and look at every cluster. Because there will not be a single parameter setting that gets everything right. Instead, different clusters may appear only at different parameters.

But the main challenge in your case will likely be the distance function. Except for idealized settings like MNIST, Euclidean distance will not work at all. Nor will anything working on the raw pixels. So first you need to do feature extraction, then define a similarity function.

When it comes to clustering, work with a sample. Cluster the sample, identify interesting clusters, then think of a way to generalize the label to your entire data set. For example by classification (your labeled data points are your training set, predict the labels of unlabeled points).


Is it feasible to use decision tree algorithms for sensor fault detection?

You need to determine how to formulate your problem. I see it as having two aspects:
1. Detect an abnormal value (in temperature)
2. Determine whether abnormal value is due to sensor or system problem

The first is an anomaly detection problem and there is lots of literature on the topic, including tree-based methods on sensor data. One prominent approach is Isolation Forests, which is implemented in scikit-learn. http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html

The second is fault localization. This is an area also well described in literature, and tree-based models were used even before machine learning became a thing.
A key here is to only train on anomalous data, then this should become a relatively simple classification problem assuming your have relevant features.


Why cant I overfit this dataset with my neural network?

It is usually good to start with a small model because you can then evaluate the contribution of adding layers, etc.
Also, Boston dataset is a popular dataset so there are several tutorials showing good neural network architectures, like this one.
Concerning your model, here are some notes.

The use of sigmoid activation is likely to worsen results, since sigmoid function compresses values between 0 and 1, while you are trying to predict outputs between 5 and 50.
Instead of sigmoid, you can use ReLU activation, which has better convergence properties for inner layers
You can standardize your target data to reduce the variance of your data and control the mean, this usually improve a lot regression models. In this case, a sigmoid activation on your final layer would be a good choice


Is there a text on Apache Spark that attempts to be as comprehensive as White's Hadoop: The Definitive Guide'?

Learning Spark: Lightning Fast Big Data Analytics is a fairly comprehensive book covering the core concepts as well as the higher level components involved in the Spark stack. This is the book recommended by Databricks for their Spark Developer Certification as well.

If you are interested more about the use cases built using Spark, I suggest Advanced Analytics with Spark


You have to normalize your data to accelerate learning process but based on experience its better to normalize your data in the standard manner, mean zero and standard deviation one. Although mapping to other small intervals near to zero may also be fine but the latter case usually takes more time than the other. If you use ReLU, again based on experience, you have to normalize your data and use standard initialization techniques for your weights, like He or Glorot methods. The reason is that your should avoid each activation to be so large, because your net would be so much dependent to that activation, and you may have overfitting problem. When you use ReLU because there is no limit for its output, you have to normalize the input data and also use initialization techniques that avoid having large values for weights. For more information I encourage you taking a look at here and here.


Distinguising features of linear vs, non-linear machine learning models (algorithms)

Actually, there are many linear and non-linear machine learning algorithms. Selecting a right algorithm highly depends on your data-set and the nature of your data. for instance, in machine-learning era, it was customary to estimate functions by assigning a typical model to the problem and reducing the error by predicting the appropriate coefficients using a cost function for regression tasks. In such cases, you should have bias-variance trade-off. That means you should not fit the data nor miss it. you should find a good estimate, a good model that has generalization capability. For doing all of these you have to choose features which help you describe the problem better for making a model. In deep-learning, we usually do not have this trade-off. If you increase the size of your training set, you can almost be sure that you can have better results. In machine-learning, you can always be sure that by making complex non-linear models, you overfit your data while using complex deep-learning models does not necessarily mean that if you employ generalization techniques which avoid overfitting. For choosing a right model, it is customary to use a simple linear model and make it complicated step by step. Problems which their inputs have numerous features, you can not see and visualize the data-set to check whether they are linearly separable in that space or not. Consequently, beginning with a simple model and making it complex step by step is a logical solution. For measuring performance there are different solutions that all depend on your goal. Take a look at here.


I would go for dimensionality reduction. You can start with SVD (should be available in Weka). If SVD is too slow / too memory consuming, then there are still some options:

CUR-decomposition: a variant of singular-value decomposition that keeps the matrices of the decomposition sparse if the original matrix is sparse (see: this chapter of Mining Massive Datasets book)
Random projections: projecting the data onto a random lower-dimensional subspace (see: the Random projection in dimensionality reduction: Applications to image and text data paper)
Coresets: Given a matrix A, a coreset C is defined as a weighted subset of rows of A such that the sum of squared distances from any given k-dimensional subspace to the rows of A is approximately the same as the sum of squared weighted distances to the rows in C see the Dimensionality Reduction of Massive Sparse Datasets Using Coresets paper)

That's the tip of an iceberg. More approaches are there in the wild. The problem is that I doubt any of these solutions come with Weka (please, correct me if I am wrong on this). I would search for a usable Java implementation of any of these algorithms and try to port it to work with Weka's arff files.


What to choose for null hypothesis and alternate hypothesis?

You always choose the statement that you want to disprove as Null hypothesis. You can either reject it in favour of alternative hypothesis or not reject it. We don't say we accept null hypothesis. Because we don't have enough evidence for the null hypothesis to be true. It might be true, might be not. But with significant t/p value we can reject null hypothesis in favour of alternative hypothesis
Like in this example, if you don't get any significant value after the experiment you do not reject that the average screen size of the sample does not differ from 10 cm.But you don't have any proof to accept it either.
Always choose the statement you want to disprove as null hypothesis.


Export pandas dataframe to a nested dictionary from multiple columns

Using dict comprehension with nested groupby:

d = {k: f.groupby('subgroup')['selectedCol'].apply(list).to_dict()
     for k, f in df.groupby('maingroup')}

Output:

{'A': {123: [1, 9], 321: [3]},
 'B': {345: [5], 768: [2, 3]},
 'C': {178: [6, 5], 712: [4]}}


How to choose the best optimiser in Deep Learning algorithms?

I found this helpful.

I usually stick with Adam, and rarely attempt SGD. Simple words: try to go with the recent optimiser implemented on popular datasets and that has been implemented by the framework which claims to be better than others.

To specifically say that this optimiser is best for this kind of data is subjective, and I haven't read any relevant material related to that. In future, If anyone finds it, please do comment or edit this post.

Hope this helps.


How to plot multiple variables with Pandas and Bokeh

This changed in the latest version of Bokeh (I guess 0.12.7). This is the new way of doing it.
from bokeh.io import show, output_file
from bokeh.models import ColumnDataSource, FactorRange
from bokeh.plotting import figure

output_file("bars.html")

fruits = ['Apples', 'Pears', 'Nectarines', 'Plums', 'Grapes', 'Strawberries']
years = ['2015', '2016', '2017']

data = {'fruits' : fruits,
        '2015'   : [2, 1, 4, 3, 2, 4],
        '2016'   : [5, 3, 3, 2, 4, 6],
        '2017'   : [3, 2, 4, 4, 5, 3]}

# this creates [ ("Apples", "2015"), ("Apples", "2016"), ("Apples", "2017"), ("Pears", "2015), ... ]
x = [ (fruit, year) for fruit in fruits for year in years ]
counts = sum(zip(data['2015'], data['2016'], data['2017']), ()) # like an hstack

source = ColumnDataSource(data=dict(x=x, counts=counts))

p = figure(x_range=FactorRange(*x), plot_height=250, title="Fruit Counts by Year",
           toolbar_location=None, tools="")

p.vbar(x='x', top='counts', width=0.9, source=source)

p.y_range.start = 0
p.x_range.range_padding = 0.1
p.xaxis.major_label_orientation = 1
p.xgrid.grid_line_color = None

show(p)


I suggest chapters „3 Linear Regression“ and „8 Tree-Based Methods“ in An introduction to statistical learning


Predicting credit applications with timeseries

Intuitively I think that the model would need some additional features based on the economic context in order to be more accurate, and that would also be a part where the evolution across time really matters.

  Would it now be a good idea to create train/test folds based on theclient_nr? Or should I make splits by month?

If possible you need to have a full time series for a client as an instance, so I'd say splitting on the client number is much better. But select the client numbers randomly from the full set of client ids, because it's possible that the clients number are assigned following a particular order in time.

  Should I then select the first $n$ months as features, and create one label from $k$ months after $n$ that indicates whether the client has applied for a credit in those $k$ months? Or is there a better way?

That depends on the exact goal and on the kind of algorithm being used, but afaik usually with a time series the label is predicted at any given time given the past and current features and the past labels (or past predicted labels).

  Would it be better to use regression using nr_credit_applications, or classification on credit_application?

This is more a matter of convenience for your application or for the algorithm being used, as the accuracy should be very similar (assuming you use the same kind of method of course).


Is it possible to implement a vectorized version of a Maxout activation function?

If you could combine all the weight vectors into a matrix W and all b's into a vector b, then you could do
np.maximum(np.dot(W.T,x)  + b)


I have a few suggestions to improve your accuracy (these are hardly original with me):

Use an ImageDataGenerator from keras.preprocessing.image to augment your data.
Use k-fold validation to use your training data more thoroughly.
Try the keras.optimizers.Adam() optimizer.


JupyterHub does not provide version control system nor facilitates sharing of Notebooks. You mentioned yourself limitation of Binder.

Try Zeppelin. Version 0.7 should be released within a few next days.

As you can see from the roadmap, this version delivers "enterprise" features which are exactly about collaboration.
Version control system (git) is integrated.
It's self-hosted.

In essence, I think it meets all requirements you posted. On top of that it delivers richer visualisation capabilities and plethora of other features (works with Shiro, Knox, Kerberos - secure Spark anyone?).


Supervised or unsupervised learning for predicting energy consumption for new buildings

Building on The Lyrist answer, I would try to organize all that data and create as many instances as possible of Input-Output. If I understand your case correctly, the input would be the multiple data point that you feel have an impact on the load (if you're not sure about it's impact, chuck it in for now) that you would set up as a vector of values. Then the output would be a single value of load, such as Kwh/day (that doesn't sound right), but just get one 'value' you're interested in predicting. Don't forget to keep training, validation and test sets.

Now you can frame your problem both as a classification problem by creating categories, such as from 0 to 10, 11 to 20, etc, or go for a regression problem, predicting a continuous value. Do lookup past application of similar problem, as they are, to some extent, application specific. Note that the some algorithm might work better with regression, other with classification. Personally, I think that creating buckets of classification might be easier, unless predicting an exact value is important to you.

Now regarding the specific model to go for, it usually is recommended to start with simple model (SVM's is a popular one) and build to more complicated model (kerneled SVM), if you enough data! Resist the temptation of going straight for a 12 layers MLP, as those old SVM's.


What is the intuition behind larger number of samples are better for statistics?

Intuitively we can imagine statistics as trying to see an image using only finitely many pixels/samples. If you have more pixels/samples and they're well-distributed, then the image is more clear.


There could be a problem with the linear layer after the attention inside a transformer?

No, this is not a problem.
If we zoom into the scaled dot product attention blocks, which happen before the projection with $W^O$ we see this:

There, you can see how the masking of the current and future positions happens inside the scaled dot product attention, which happens before the multiplication by $W^O$. Therefore, the values learned for $W^O$ are trained only with the information from the previous tokens in the decoder.


Generate new images based on a dataset of images

Yes, there are plenty ways to do this, what you basically want is to interpolate between the two.

This is current state-of-the-art for solving problems like yours: https://github.com/carpedm20/BEGAN-tensorflow

Also keep an eye out for my question here: 2D Image-based Avatar Animation With Expressions / Image-To-Image Translation

If I end up using the ML approach, you could very well also use it with tiles instead of faces and be able to adjust the likeness. I will link to my project in my question.


If you have only one example for certain classes SMOTE won't work. Most of the Machine Learning algorithms won't work either.

There is a technique called One Shot Learning (it is normally used in computer vision) that  "Whereas most machine learning-based object categorization algorithms require training on hundreds or thousands of samples/images and very large datasets, one-shot learning aims to learn information about object categories from one, or only a few, training samples/images."

Maybe you could try with one OSL to help you with the classification but normal ML algorithms require more samples to be able to generalize.


how to apply feature selection on LSTM-RNN?

Considering that Neural Networks (LSTM-RNN in this case), like the rest of deep learning methods, are like black boxes.

The transformation applied are too complex to determine how much a value, variable, weight affects the solution.

Although is a hot topic, there's no current method viable or wide-known used to understand which variables are better or have higher influence in the model.

However, what you can do is an EDA (Exploratory Data Analysis) between your explanatory variables and response variables. Their relationships can give you some skyview of which variables are more correlated or influenced by others.


Recurrent Neural Network (RNN) Vanishing gradient problem - Why does it affect earlier timesteps more?

Of course that all weights are the same, but the update applied to the weights has a contribution from each of the timesteps, and the contribution associated with the first timesteps is what is more affected by the vanishing gradient problem.


Unable to make CNN model with same output and input dimensions

The error indicates that you are trying to add tensors of incompatible dimensions, as x has 128 channels and inputs has 3 channels.
The reason why x has 128 channels is just in the line above, where you pass feature_size with value 128 as the number of output channels. Change the number of output channels in that line to 3 and it should work, like this:
x = Conv2D(3, (3, 3), kernel_initializer='he_uniform', padding='same')(x)


Looking for research for separating conversational audio files

The task of separating a conversation with multiple speakers taking turns is called Speaker Diarization. This is different from Speech Separation, which is a special case of Source Separation, refers to separating out multiple overlapping/concurrent sources from an audio stream.
Some free datasets include

ICSI Meeting Corpus
AMI Meeting Corpus
VoxConverse

More can be found in the Github project wq2012/awesome-diarization.


Terminology: "flat geometry" in the context of clustering

Are flat and non-flat geometry a legit terminology in machine learning
  and statistics?

These are terminologies from Mathematics, they are valid in any field.

  What is the mathematical definition?

(Non) flat manifold

In mathematics, a (Riemannian) manifold is said to be flat if its curvature is everywhere zero; otherwise non-flat. This is very different than the definition of flat object in geometry. According to that definition, only points, lines, and hyper-planes are flat (not for example hyperspheres or segments).

For example, set $\left\{(t,t):t\in(-1,1)\right\}$ is a 1D flat manifold in ${\Bbb R}^2$, set $\left\{(t,t^2):t\in(-1,1)\right\}$ is a 1D non-flat (positively curved) manifold in ${\Bbb R}^2$, and a hypersphere is an $n$D non-flat (positively curved) manifold in ${\Bbb R}^{n+1}$.

Accordingly, a cluster with a (non) flat shape corresponds to a (non) flat manifold.

Here are some examples from the document.

Points are concentrated around (A) two 1D non-flat manifolds (circles) which are non-convex, (B) two 1D non-flat manifolds (arcs) which are non-convex, (C) three 1D flat manifolds (segments) which are convex, (D) three 0D flat manifolds (centers as points) which are convex.

(Non) flat geometry vs a Flat

By "flat geometry" the document means "Euclidean geometry" not "a flat object defined in geometry". If we measure distances (consequently lengths, areas, volumes, etc.) via Euclidean distance we are obeying the Euclidean geometry, otherwise, we are obeying a non-Euclidean geometry. For example, if we measure a distance between two points by following a non-flat manifold (a geodesic), we are obeying a non-Euclidean geometry. As an illustration,

In (A), the red line measures a distance obeying a flat geometry, the blue line measures the distance obeying a non-flat geometry (by moving along the non-flat manifold). If an appropriate map of manifold to a lower dimension is possible (B), obeying a flat geometry would be equivalent to obeying the non-flat geometry before mapping.


Is there a site like fast.ai or deeplearning.ai but for machine learning instead of deep learning?

Can't comment due to low reputation, but as a comprehensive summary I'd recommend Elements of Statistical Learning by Hastie et. al. It reviews most current state-of-the-art ML methods, together with the classics, is free of charge (big thanks to the authors!), and even has a nice accompanying R package.

It's probably not the source you've expected, but is really a nice place to start, especially for practitioners.


How to get the probability of belonging to clusters for k-means?

Let us briefly talk about a probabilistic generalisation of k-means: the Gaussian Mixture Model (GMM).

In k-means, you carry out the following procedure:
- specify k centroids, initialising their coordinates randomly
- calculate the distance of each data point to each centroid
- assign each data point to its nearest centroid
- update the coordinates of the centroid to the mean of all points   assigned to it
- iterate until convergence.

In a GMM, you carry out the following procedure:
- specify k multivariate Gaussians (termed components), initialising their mean and variance randomly
- calculate the probability of each data point being produced by each component (sometimes termed the responsibility each component takes for the data point)
- assign each data point to the component it belongs to with the highest probability
- update the mean and variance of the component to the mean and variance of all data points assigned to it
- iterate until convergence

You may notice the similarity between these two procedures. In fact, k-means is a GMM with fixed-variance components. Under a GMM, the probabilities (I think) you're looking for are the responsibilities each component takes for each data point.

There is a scikit-learn implementation of GMM available if you wanted to look into that, but I'm guessing you just want a quick way to amend your existing code, in which case, if you're happy to assume your clusters are fixed-variance Gaussians, you could transform your distance matrix element-wise as $y = e^{-x}$ (giving you an exponential fall-off), and then calculating the softmax over your columns (normalising your distribution so $P(Y=1) + P(Y=2) + ... + P(Y=k) = 1$).

It's worth pointing out that the assumption your clusters are fixed-variance Gaussians isn't necessarily valid. If your dimensions have wildly different scales, this may produce strange results, as dimensions with smaller-magnitude units will appear more "probable". Standardising your data before running your clustering procedure should remedy this.


What is the outcome of a Cox regression in xgboost?

In the documentation, you can find that the predictions are returned on the hazard ratio scale:

  survival:cox Cox regression for right censored survival time data (negative values are considered right censored). Note that predictions are returned on the hazard ratio scale (i.e., as HR = exp(marginal_prediction) in the proportional hazard function h(t) = h0(t) * HR).

In other words, in a Cox proportional hazards rate model we have:

$h(t)=h_0(t)×X$

where $X$ in a traditional linear model is of the form $exp(b_1x_1+b_2x_2+...+b_px_p)$, and $h_0(t) = $ baseline hazard rate function. In the xgboost case with tree base learners, $X$ is the exponentiated prediction generated by the weighted sum of individual trees trained on the negative gradient of a loss function designed for survival data in particular.

In other words, the predicted values are not times of failure.

I find the xgboost documentation on the survival:cox setting to be extremely sparse and not well described. To my knowledge, I do not believe there exists an internal method to derive $h_0(t)$ for example.


Multi-Head attention mechanism in transformer and need of feed forward neural network

The basic reasoning, I think, is just to increase capacity. While it is possible in theory for a single head, using multiple simply makes it easier. More specifically though, the paper says (pg 4):

  Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.

In other words, for a given layer, with only one attention head, the weighted averaging performed by the mechanism prevents you from being able access (differently transformed) information from multiple areas of the input within that single layer.
I think this can be mitigated by using multiple layers, but adding this capability simply increases the capacity of a single layer, and therefore you can perform more useful calculations with less layers. Often this is good because stacking layers (function composition) can lead to more issues than parallelizing layers (as the multi-head does), especially when it comes to gradient computations. (This is partly why residual/skip connections are so useful).

In general,  using multi-head is a common way to increase representational power in attention-based models, for instance see Graph Attention Networks by Velickovic et al. The basic idea is that, although attention allows you to "focus" on more relevant information and ignore useless noise, it can also eliminate useful information since usually the amount of information that can make it through the attention mechanism is quite limited. Using multiple heads gives you space to let more through.
Presumably you are asking about the "Position-wise Feed-Forward Networks". Again, I think this is merely a question of model capacity: the model could function without these layers, but presumably not as well. These layers are interesting because they are essentially 1D convolutions, which usually appear near the end of a network or block in computer vision architectures (which I am more familiar with). In other words, they are localized transforms that operate identically across the input (spatially). The reasoning is often that the previous layers (here, the attention layers; in vision, the conv layers with larger kernel sizes) were reasonable for passing or mixing information spatially across the input. E.g., after an attention layer, the latent representation at each position contains information from other positions. After this, however, we want to consolidate a "unique" representation for each position (which has been informed of the other positions, of course). We do this via processing by a localized layer, which does not consider neighbours or other positions, and simply transforms the local representation on its own. One thing to keep in mind is that, while we want information to mix across "space" (e.g., across an image or sentence), for many tasks we often still need each position to maintain some resemblance/connection to its original identity (rather than say just averaging all information over all positions). So not every layer need be spatially aware. In vision models, 1D convolutions (i.e., kernel width 1) are often used to perform dimensionality reductions; in this case, the dimensionality does not appear to change however.


The problem is using predict_classes in:

model.predict_classes(X_valid)

this is designed to select the argmax (index of the maximum output) and choose it as the predicted class, for a classifier.

You have a regression problem, and just want the raw output from the network. So instead you should call:

y_pred = model.predict(X_valid)


Try changing your kernel regularizer. Either reduce its effect or shut it off entirely, then retrain and see what happens.

EDIT: I expect that the kernel regularizer is your best bet. You could also experiment with changing the depth of the network (number of layers) and how many units you give each layer. I would expect if you put more units into the layer that currently has 10, you'd see more overfitting.


XGBoost - feature importance just depends on the location of the feature in the data

You can use this chunk of code to plot the feature importance of your data. It is also possible that your data is columned with decreasing importance.
from xgboost import XGBClassifier
from xgboost import plot_importance
from matplotlib import pyplot
# load data
X =
y =
model = XGBClassifier()
model.fit(X, y)
plot_importance(model)
pyplot.show()


If your class imbalance is roughly 4:1, then a network that always predicts the more frequent class would have an accuracy of 80%. (That's why accuracy alone is an insufficient metric: for binary classification, you should use ROC and PRC curves. There are also more holistic scalar metrics such as MCC)
Let's say your positive samples are the frequent class. Without training, a random network might always output positive (80% accuracy). In this case, you'd only have true positives and false positives. After training, the network might start to also predict negatives; then you'll start getting true and false positives in addition to true and false negatives, which could yield a lower accuracy.


Need of maxpooling layer in CNN and confusion regarding output size & number of parameters

Question 1) Is my calculation for each case above correct?

No, it is not correct. The formula to calculate the spatial dimensions (height and width) of a (square shaped) convolutional layer is

$$O = \frac{I - K + 2P}S + 1$$ with $I$ being the spatial input size, $K$ being the kernel size, $P$ the padding and $S$ the stride. For your two cases that is (assuming padding of $1$ since otherwise numbers won't fit later with pooling):

Case 1:

Layer 1: The spatial dimensions are $O_1 = \frac{28 - 3 + 2\cdot1}1 + 1 = 28$. So the total layer has size $height \cdot width \cdot depth = 28 \cdot 28 \cdot 20$.
Layer 2: The second layer has $O_2 = \frac{28 - 3 + 2\cdot1}1 + 1 = 28$ with a total layer size of $height \cdot width \cdot depth = 28 \cdot 28 \cdot 40$.

With regards to parameters please note that does not refer to the size of a layer. Parameters are the variables which a neural net learns, i.e. weights and biases. There are $1 \cdot 20  \cdot 3  \cdot 3$ weights and $20$ biases for the first layer. And $20\cdot 40  \cdot 3  \cdot 3$ weights and $40$ biases for the second layer.

Case 2:

The sizes of the convolutional layer do not depend on the number of input channels. However, the number of parameters for the first layer increases to $2 \cdot 20  \cdot 3 \cdot 3$ while the number of biases remains unchanged. The second convolutional layer is not affected at all.

  Question2) I have read that the purpose of maxpooling is to reduce the dimensionality of the feature map. In my case each maxpooling is of size 3 and stride 2. What does it mean by reducing the dimensionality and then what will be the size of the output from each layer upon considering that there is a maxpooling operation after each convolutional layer?

Typically convolutional layers do not change the spatial dimensions of the input. Instead pooling layers are used for that. Almost always pooling layers use a stride of 2 and have size 2x2 (i.e. the pooling does not overlap). So your example is quite uncommon since you use size 3x3.

You can apply the same formula as above (assuming padding again - see footnote (1) for an explanation)

$O_{pool\text{ }1} = \frac{28 - 3 + 2\cdot0.5}2 + 1 = 14$

Since now the input to the second conv. layer has different spatial dimensions you would get:

$O_{CNN\text{ }2} = \frac{14 - 3 + 2\cdot1}1 + 1 = 14$

And finally:

$O_{pool\text{ }2} = \frac{14 - 3 + 2\cdot0.5}2 + 1 = 7$

Footnotes:

(1) The padding of $0.5$ is the average padding required in each spatial dimension. As an example, have a look at how the second maxpooling kernel scans an input (I have chosen the second one since the input is smaller and, therefore, easier to visualize). The input has width and height of $14$:

As you can see from the drawing the actual padding is only required on one side for height and width respectively. Therefore, the actually performed padding is
$$padding_{width} = \frac{padding_{left} + padding_{right}}{2}=\frac{0 + 1}{2}=0.5$$

And the same numbers apply to the height. Accordingly, the actual padding is $(0.5, 0.5)$.

Moreover, the drawing shows 7x7 dark blue squares (kernel centers). Which visualizes the calculations leading to an output of $pool_2$ with spatial dimensions of 7x7 ($O_{pool\text{ }2}=7$).


Should I pad zero or mean value in a convolution neural network? What about reflective padding?

Mean, zero, duplication (i.e., constant extrapolation), reflection, and symmetric padding are all valid and widely used methods of padding for conv layers. To my knowledge, there is no systemic study that says one is better than the other in all cases. In other words, one is not always better than any other. (Ideally, one would vary the padding type as a hyper-parameter I suppose.) I think intuitively that reflection and symmetric padding alters the local image structure and global statistics the least. (Clearly, the zeros from zero padding on the boundary are not truly part of the image, and the network has to learn this). However, these have different problems with "realism": for instance, reflection padding will "duplicate" a chair leg on the boundary if the padding is large enough, which may present its own problems depending on the task.

Overall, I think it doesn't matter too much, but it depends on the task and setup. For instance, in generative modelling and image-to-image translation, reflection padding avoid some artifacts on the boundary, as noted in the CycleGAN paper, for example.

An interesting recent paper is Partial Convolution based Padding  by Liu et al, where they sidestep this issue by essentially having the convolution completely ignore the boundary. This is a partial convolution in the sense that it ignores the part of the filter that reaches outside of the image. The overall improvements seem to be relatively marginal, except that it does seem to help significantly on the boundary, as one would expect. See also this earlier work doing a similar thing in the segmentation context and this work which applies partial convolutions to inpainting. This approach to avoiding padding is seemingly well-principled.

TL;DR: test different paddings and see which is best for your architecture/task (also check the literature). If that's too much work, I default to reflect padding. If you want to get fancy or worry about boundary effects, use partial convolutions.


From a credit scoring point of view : a $F_1$ score of $0.1$ seems pretty bad but not impossible with an unbalanced data-set. It might be enough for your needs (once you weight your errors by the cost). And it might not be possible to go higher (not enough data to predict an event that appears random). In credit scoring there is always a 'random' part in the target (sudden death, divorce ...) depending on the population and the goal of the loans.

You might want to investigate your features and your target. Basically : statistically, on an univariate approach, do you have features that appears predictive of the target ? (Age of the person ? revenue ? purpose of the loan ?). You might also need to investigate the target : do you have some questionnaire that would allow to get an insight on why the person defaulted ? (If the majority of default come from random event, you might not be able to modelise it).

The main problem with $F_1$ score in credit scoring is not data imbalance, but cost imbalance. Type I and Type II errors have far differents consequences. Given that you already gave the loans I am not even sure there is a cost associated with false positive (saying someone will default when it won't). It might be interesting to weight precision and recall (i.e. use $F_\beta$ as defined here). Another problem is that it is usually good for a binary decision. Depending on what you want to use the model for (measuring risk of already granted loans ? granting new loans ? pricing new loans ?) there might be alternatives that better capture model discrimination (AUC - see its statistical interpretation) or individual % chance of default (Brier Score).

Assuming that there is no specific problem with your current modelling (Feature engineering, imbalance treatment, 'power' of your model). There are some credit-scoring specific things you can do. Work on your target definition (what if you do 90+ days delinquant in the 5 years after origination ?). Try to collect more data about your clients and their behavior (purpose of the loan, others products they use at your bank... etc.).