Regression and Classification in one Neural network

Yes, these type of loss functions can be optimized using backpropagation, also in Tensorflow.
The value of the loss is a scalar (same as just the cross entropy, or the MSE, otherwise you wouldn't be able to add them), which means that it doesn't really work any different from just optimizing for any other scalar loss function. As long as the operations involved in calculating the loss function are differentiable ("smooth") enough, Tensorflow (or any other framework that does automatic differentation) doesn't care.
Think of it this way: calculating any interesting loss function involves summing up a bunch of terms living in a higher-dimensional vector space. In this case, for some of the directions in this vector space, you apply a different function on them than for others in that vector space. Doesn't really matter to Tensorflow.
What does matter though is how you sum them: since the cross entropy and the MSE are not working on the same type of units (think dimensional analysis), you have to determine some sort of scale between them. You can view this as a hyperparameter, where you can chose how important it is to have the classification correct vs how correct your bounding boxes are.


Find a low-dimensional emdedding of the images using a nonlinear dimensionality reduction method like autoencoding, then select the image closest to the centroid in the embedding space.


Can I get numeric data from a color map?

Maybe You can try some workaround (first manually then programatically):

Convert to grey-scale
Make density map (digital)
Trace=vectorise image with any Inkscape / Illustrator / etc...
Map vector parts with density map values.

You can write js script for Adobe Illustrator (Gimp /  probably also) to automate some things. I'm not suggesting to spend money, just play around with demo until you wiil find Ideal Solution.

Also probably You can try any online API from Big4 or new emergers:

tensorflow
https://cloud.google.com/vision/
https://aws.amazon.com/rekognition/
https://www.ibm.com/watson/services/visual-recognition/
http://opentracing.io/

Also there are some commercial with free options
i.e. imagga - "Sign up for Free: Tag up to 2000 images per month"

You can try DEMO Vector Magic's Desktop edition $295.00 / license One-time charge allows Group shapes by color (unfortunately do not offer special pricing for Academic Institutions or Non-profits)

Also have you tried FreeMat COLORMAP - Image Colormap Function?


Is Label Encoding with arbitrary numbers ever useful at all?

First, I generally agree that encoding unordered categories as consecutive integers is not a great approach: you are adding a ton of additional relationships that aren't present in the data.
CART
First, let me point out (because I nearly forgot) that there are two main types of decision tree: CART and the Quinlan family.  For the Quinlan family, categorical variables are dealt with by using higher arity splits, so no encoding is needed and this is mostly moot.
Q1, yes, ordinally encoding will be treated by the model as numeric (unless some other parameter controls that, e.g. LightGBM).  But for (most) trees, only the order is actually relevant: the scale is irrelevant, and e.g. the relationship "10 is twice as much as 5" is completely invisible to the tree.
As you point out, one-hot encoding for a CART model can be detrimental, especially when there are many levels in a categorical feature: will the tree ever actually decide to split on one of the dummy variables, if it is only 1 for a small subset of the data?
(Q2)But when you encode ordinally, there will just by chance be some splits that are useful and split many levels in each direction.  (You may even try more than one random ordering of the levels as different features!)
And yes, presumably the best approach is to use an implementation that can take advantage of the raw categoricals, using the average-response trick.  (There's even some debate on how much that helps: some studies have been done, but generally the datasets are synthetic or too small to be representative.)
Other models
In other models, very often one-hot encoding is just fine, and doesn't suffer from the same problem as trees.  If there are too many levels, and especially if some of them are too small, you may consider smoothing techniques to avoid overfitting.  (Q2)I'd be surprised if ordinally encoding is ever worth it for most models, but one would need to consider each model type individually, and probably do some testing.
Names
As for naming, things are a bit muddy, but I don't think this is sklearn's fault.  The "label" in LabelEncoder means it is supposed to be used on the labels, a.k.a. the dependent variable.  And for that usage, there is no debate about whether it's appropriate: sklearn just requires consecutive integer labels for its multiclass classification; it doesn't use the numeric values as though they were mathematically meaningful.
As for OrdinalEncoder, it is meant to be used with input ordering of the categories.  See sklearn Issue#13488 below.  But one could argue that you are encoding the categorical variable in an ordinal way, so even with unordered categories this isn't necessarily a misnomer.
See also
See Issue#13488 for some related discussion.


Neural Networks basically act as a high memory-based machine learning algorithm. So for a given dataset the chance of it perfectly aligned with all the data at a given instance is high, as it most likely just ends up remembering every data point you give.

Overfitting occurs precisely because of this, when a new expansive data set is introduced there is no way it can adjust its fit to the new data, graphically it ends up missing more of the values than its supposed to fit.

In conclusion, it does not work well in the case where the scoring population is significantly different compared to training sample.


The first groupby - counts the number of persons per day per meal

The second groupby - counts the number of unique persons in each day

The inner merge between the 2 - matches the number of persons per day+meal to be equal to the number of persons on that day

We've left with a df that contains day-meal couples in each row, with the number of persons that had a meal which matches the total number of persons on that day:

Now we only need to groupby day and create the set of the meals

df.groupby(['day','meal']).count().reset_index()\
  .merge(df.groupby('day')['person'].apply(lambda x: len(set(x))).reset_index(),
         on=['day','person'])\
  .groupby(['day'])['meal'].apply(set)


How to perform feature selection on dataset with categorical and numerical features?

It is fine to apply feature selection technique on one hot encoded variables. Because if one particular segment of that variable is correlated with your target, then it is a good news. Your model will understand the scenario better.
Or, You can label encode your categorical variable first so that you still have 30 variables (29 numerical + 1 label-encoded categorical variable). Now try to find the importance value of each variable, and take the relevant ones (Use any method for it: be it RFE, random forest feature selection, pearson's correlation etc). Once you have the final list of variables, and the label-encoded variable is also coming as relevant, it is ok to put it into the model.


Neural Networks: How to prepare real world data to detect low probability events?

You bring up a number of good questions here Ans. I will do my best to cover each of them in turn.  It isn't an exhaustive treatment but hopefully it helps...

1. How to normalize the categories.

First, assess whether your categorical variables can be considered zero variance (e.g. all records possessing one category only) or near zero variance (vast majority of records belonging to very few categories). Create a basic frequency distribution to identify this.

While it doesn't matter as much in Neural contexts per se, it is a good idea to consider filtering low variance variables from your model. Just be careful as eliminating near zero variance variables may have you throwing out the baby with the bath water.

You've worked with Neural Nets before so you know that you need to convert categories to numeric values.  A good question to ask is whether a given categorical value is ordinal in nature (e.g. on a Likert scale of 1 to 5) and whether you want to preserve ordinality. This pushes you into an area such as that described by Pinto da Costa and Cardoso (https://www.researchgate.net/publication/221112186_Classification_of_Ordinal_Data_Using_Neural_Networks).

2. How to deal with missing data.

Assuming that you are talking about missing continuous values, you will want to impute these numeric values based collectively on the values that are present across the entire attribute.  There are a number of approaches to use here but something important to keep in mind is dispersion.  In a basic sense, if your attribute is skewed by outliers, you will want to steer clear of a mean-based calculation and go with a median-based approach.

3. How to deal with the fact that 'Defaults' are ~5% of the data.

This isn't an immediate concern given that you have 2500 examples of what denotes a 'Default' (in a 50,000 dataset). A sane approach here would be to ensure that you are using a K-fold cross validation scheme (say with 10 folds) to ensure that you are truly randomizing training vs test.  This will help protect you against overfitting.  Again, this is pretty high level guidance, but it is prudent.

Extra Credit:
I haven't gotten into other standard tactics such as normalization of your continuous attributes, but you may want to get up to speed there to better generalize your model (while also developing a deeper understanding of the dynamics at play in the data). This could point you to trying out other algos, etc.


Why does a machine learning algorithm need a bias?

Inductive bias means all assumptions your learning approach assumes for generalising to unseen data. What we do in machine learning is induction which means we don't have rules. By seeing data, we make rules. For example, in linear regression you may consider that the output is linear with respect to inputs, or it is polynomial. Another example can be this that in women gesture recognition, you don't care about moustaches because you know they do not have that. You can also take a look at here.


splitting into train test by train_test_split of float values?

The issue is your data violates the requirements of StratifiedShuffleSplit, specifically it's not possible to do a 70:30 split of the data and maintain the same number of distinct y values in the test and train set - most likely because you have a y value which only occurs 1 time?

Perhaps just use ShuffleSplit?


Sensorfusion: Generate virtual sensor based on analysis of sensorsdata

The first major point here is that you do not have historical water level values. These are the labels that you'd use to teach/train any supervised model. Without labels, you are limited to using unsupervised methods. Have a look at the Wikipedia overview

You could try looking into something like clustering, with k-Means for example, which would be quite fast. Other clustering algorithms like DBSCAN are also very good, but your variables (temperature and pressure) might not necessarily cluster well on their density in the input space (the D in DBSCAN).

There are unsupervised neural network approaches, but these are a bit more involved. Autoencoders might be a good place to start.

Have a look here for some more ideas on specific models to try out, such as Denoising AutoEncoders, which can use recurrent components (such as LSTMs), which can perform well on time series data.


Which model to chose based on learning curve

You should go for LightGBM which has the lowest training and cross validation RMSE. By the way, score is a bad name for RMSE because you would expect a higher score to mean that your model is better but here the opposite is true.


Causation vs correlation problem in case of fixing other parameters and changing only one

The concept of causality is a little tricky. You can (usually) not test for that. You need to come up with a theory-based idea of how things work, such as if the price for a good goes up, demand goes down. And in empirical applications you need to think about, what other things influence demand, so that you don't omit important aspects of the data generating process.

When I read your description, it seems that you might have a good chance to isolate the causal effect of the change in difficulty. This seems to be a classical example for "difference-in-difference" (DiD). The only problem may be that you changed the diffculty for all players, so you need to see how to handle this.

Another thing that comes to my mind is, that you essentially look at survival rates (after X days), this would be a task for "survival analysis.

As a (more or less) naive approach, I would say that your approach goes in the right direction. However, make sure that you control for anything you can, like player characteristics (because it is possible that now different people play your game compared to the past).

So when you have

$$y = \beta X + \gamma d + u$$,

where $y$ is D7, $X$ is anything you know about players, and $d$ is difficulty (0 diffcult, 1 less difficult), you should get a more or less good idea, what changing $d=0$ to $d=1$ does to y. When you use OLS (linear regression), the coefficient $\gamma$ should directly give you the change. But remember, this is a more or less naive approach (which should give "okay" results).


SMAC has a parallel version available.

I'm not sure it handles the single runs as you want, but it sounds strange to me if it don't. Give a try!


Creating new column in dataframe based on conditions in 2 other columns

-the problem with an inaccurate filling of column group_gender is that in df['group_gender'] = 'dp_m' in the following code, if i == 'M' you are filling the whole column with dp_m, instead you should use methods like iloc but it is not really an efficient way specifically when having a large dataset. In following, I have provided a better way.

for i in df['gender']:
    if i == 'M':
        df['group_gender'] = 'dp_m'
            else:
        df['group_gender'] = 'dp_f'

The more appropriate way

#creating an instant dataframe
import pandas as pd
d = {'gender':['M','M','F', 'M', 'F','M']}
df = pd.DataFrame(d)

#filling the 'group_gender' column
df.loc[df['gender'] == 'M', 'group_gender'] = 'dp_m'
df.loc[df['gender'] == 'F', 'group_gender'] = 'dp_f'

-You are not filling in experimental_gropuping and returning the column:

group(df['experimental_grouping'])

-What is this group? (Can you provide more details about the first part of your code?)


I only have microsoft office, how can i write code to manipulate relational data sets?

It should be possible to install Python without admin privileges. This has been discussed on Stackoverflow here.

The easiest method appears to be installing Anaconda.

And according to this answer you should also be able to use WSL without admin rights, which will get you bash.

For handling relational data you can use sqlite, which is built-in to Python. For example:

import sqlite3

conn = sqlite3.connect('example.db')
cur = conn.cursor()
cur.execute('''CREATE TABLE users(id INTEGER PRIMARY KEY, name TEXT''')
conn.commit()
conn.close()


Slow keras fit method with 100x100 array, how can I make it faster?

using a 50 x 50 matrix I get 2500 cells, and in the construction of the neural network I have a 2500x2500 parameters + 2500 for a total of 6252500.
I think that slows down operations.
and there are only 3 Danse layers, the last one is size 4 because of the possible actions that are 4.
Is it possible to reduce the time of operation by adding more Danse layers?
this is my model
def build_model(env, **opt):
loss = opt.get('loss', 'mse')
a = opt.get('alpha', 0.24)
model = Sequential()
esize = env.maze.size
model.add(Dense(esize, input_shape=(esize,)))
model.add(LeakyReLU(alpha=a))
model.add(Dense(esize))
model.add(LeakyReLU(alpha=a))
model.add(Dense(num_actions))
model.compile(optimizer='adam', loss='mse')
return model


What should be the training frequency of a rnn model for timeseries prediction?

It depends on any number of factors. What kind of accuracy are you currently getting? How often are you getting new predictions from your model? Do you have human intervention at some point to check your errors, relabel, and improve your training sets?

It's also possible that you don't need a schedule, per-se. Depending on your data, the pipeline and the programming of your model, you could also just have continuous learning built into your model with a new release of said model after every epoch. So, really, a lot of the answers you are seeking are very specific to your implementation.


Help getting corresponding dataframe values

I would recommend merging your two dataframes to get the exercise names:

exercise_name_df = thisSession_df.merge(exercises_df, left_on='exerciseId', right_on='id')

This will give you a dataframe like

exercise_name_df:
    id   exerciseId  sets  exerciseName
    1       1         12      Squat
    2       1         14      Squat
    2       2         15     Pullup
    2       2         15     Pullup

Then you can replace exerciseId with the name if you really want to:

# reassign id to name
exercise_name_df['exerciseId'] = exercise_name_df['exerciseName']
# drop redundant column
exercise_name_df.drop(columns=['exerciseName'], inplace=True)

I would argue that the above method is more readable than the map() solution. But I think the map() solution should also work with a small tweak:

thisSession_df['exerciseId'] = thisSession_df['exerciseId'].map(exercises_df.set_index('id')['exerciseName'])


How would I map categories between two similar but different sets of categories

This sounds like a pretty standard supervised learning problem. In this case, your records would be businesses on site X and their actual category on site Z. Your predictors would be tags/categories for a particular business on site X, and your target variable, y (i.e., what you're trying to predict), would be the category on the other website. As far as the code goes, you have a variety of options depending on your preferred language. You could use the caret package in R, the scikit-learn library in Python, or the Weka library (maybe even Spark's ML lib because of its simplicity) in Java/Scala.

Side note, in your question I think you meant to say "logistic regression" instead of "logical regression". You don't need to use logistic regression (although it wouldn't hurt). You could also try algorithms like Random Forests or Naive Bayes.

Also worth noting: your target variable will have many classes (ie every possible category for the site you're trying to predict), so don't get alarmed if it seems like there are a lot of classes. That's normal for a problem like the one you've described.


Your understanding is correct.

This is known as the indicator function.

The indicator function of a subset $A$ of a set $X$ is a function

$$1_A(x)= \begin{cases}1, & x \in A \\ 0, & x \notin A \end{cases}$$


ValueError: not enough values to unpack (expected 4, got 2)

Its because you have not looked how the values are packed in plt.subplot function.

>>> plt.subplots(2,2,figsize=(10,4))
(<matplotlib.figure.Figure at 0xa3918d0>, array([[<matplotlib.axes._subplots.AxesSubplot object at 0x000000000A389470>,
     <matplotlib.axes._subplots.AxesSubplot object at 0x000000000A41AD30>],
    [<matplotlib.axes._subplots.AxesSubplot object at 0x000000000A6F7EB8>,
     <matplotlib.axes._subplots.AxesSubplot object at 0x000000000BC232E8>]], dtype=object))

Instead of unpacking all values at once, unpack in steps. You will get a better idea then.

For your solution, to unpack -

>>> fig, [[axis1, axis2],[axis3, axis4]] = plt.subplots(2,2,figsize=(10,4))


How to handle a zero factor in Naive Bayes Classifier calculation?

An approach to overcome this 'zero frequency problem' in a Bayesian setting is to add one to the count for every attribute value-class combination when an attribute value doesn’t occur with every class value. So, for example, say your training data looked like this:

$$\begin{array}{c|c|c|}
 & \text{Spam} = yes & \text{Spam} = no \\ \hline
\text{TimeZone} = US & 10 & 5 \\ \hline
\text{TimeZone} = EU & 0 & 0 \\ \hline
\end{array}$$

$ P(\text{TimeZone} = US | \text{Spam} = yes) = \frac{10}{10} = 1$

$P(\text{TimeZone} = EU | \text{Spam} = yes) = \frac{0}{10} = 0$

Then you should add one to every value in this table when you're using it to calculate probabilities:

$$\begin{array}{c|c|c|}
 & \text{Spam} = yes & \text{Spam} = no \\ \hline
\text{TimeZone} = US & 11 & 6 \\ \hline
\text{TimeZone} = EU & 1 & 1 \\ \hline
\end{array}$$

$ P(\text{TimeZone} = US | \text{Spam} = yes) = \frac{11}{12}$

$P(\text{TimeZone} = EU | \text{Spam} = yes) = \frac{1}{12}$


Can an ordinal variable with a wide range be considered as a continuous integer variable in regression?

Think carefully before you do this. You have no idea what the underlying height distribution is. Here are four possibilities.

If you were building a regression model, each of these sets of height data would be interpreted differently. However, if replaced by your ordinal variable, they all look numerically equivalent. If you use this variable as a continuous variable, you are assuming a level of precision and information not present in the data.

Keep your ordinal variable as an ordinal variable. It may even be advantageous to bin this variable into fewer categories (for example "low", "low-middle", "middle", "middle-high", "high"). There are two reasons to do this.

This is a variable that has a unique value for each observation, which may lead to it looking disproportionately important. When you bin it like this, you remove that problem.
Think about future data that your model will be used to evaluate. Will those height classes in the new data reference the original classes? What if the new set has 100 observations? Now the new data has a range larger than what the model originally considered. Binning this variable is equivalent to normalizing a continuous variable.

Ultimately, your choice should be based on the performance of your regression model. If it behaves better with the variable as a continuous numerical variable, then do that. If the model behaves better with the variable as an ordinal variable in bins, do that.


In your confusion matrix kernel , you passed two arguments. Y_pred and y_true. They are supposed to have same dimensionality. But they are different. That's why the error occured.


The model is that process or algorithm which you are using to train it with training or labeled data. After the training of that model, it will be dubbed trained model.


how does gradient descent update weigths in neural network

In short it is "Added" to previous value of the weight.

Here is the algo from tom mitchell's, in your case it shall be W_1 = w_1+delta,W_2 = w_2+delta, W_3 = w_3+delta.


How do you predict a continuous variable when all your independent variables are categorical

It depends on the problem statement, have you got some background about the source of this dataset.. if not then predicting sales in the most common type of analysis you can do here.

And regression doesn't really only depend on input values, it is also considering the past output values so do not worry if the input values are all categorical. You can also try gradient boosting regression trees for predicting sales in partner stores.

Aprt from predictive models, its also a good idea to do some data analysis and  find trends and patterns within your data by utilizing graphs


Analyzing A/B test results which are not normally distributed, using independent t-test

The distribution of your data doesn't need to be normal, it's the Sampling Distribution that has to be nearly normal. If your sample size is big enough, then the sampling distribution of means from Landau Distribution should to be nearly normal, due to the Central Limit Theorem.

So it means you should be able to safely use t-test with your data.

Example

Let's consider this example: suppose we have a population with Lognormal distribution with mu=0 and sd=0.5 (it looks a bit similar to Landau)

So we sample 30 observations 5000 times from this distribution each time calculating the mean of the sample

And this is what we get

Looks quite normal, doesn't it? If we increase the sample size, it's even more apparent

R code

x = seq(0, 4, 0.05)
y = dlnorm(x, mean=0, sd=0.5)
plot(x, y, type='l', bty='n')

n = 30
m = 1000

set.seed(0)
samp = rep(NA, m)

for (i in 1:m) {
  samp[i] = mean(rlnorm(n, mean=0, sd=0.5))
}

hist(samp, col='orange', probability=T, breaks=25, main='sample size = 30')
x = seq(0.5, 1.5, 0.01)
lines(x, dnorm(x, mean=mean(samp), sd=sd(samp)))

n = 300
samp = rep(NA, m)

for (i in 1:m) {
  samp[i] = mean(rlnorm(n, mean=0, sd=0.5))
}

hist(samp, col='orange', probability=T, breaks=25, main='sample size = 300')
x = seq(1, 1.25, 0.005)
lines(x, dnorm(x, mean=mean(samp), sd=sd(samp)))


ValueError: Found array with 0 sample(s) (shape=(0, 13)) while a minimum of 1 is required

It looks like there are no rows in your X_train array or y_train array. Check the shape and contents of X_train using
np.shape(X_train) if X_train is an array or X_train.shape if X_train is a pandas dataframe.
Also, if you are using a pandas dataframe for X_train, you should change tmp_model = lr_model(X_train[0:12], y_train[0:12] ) to tmp_model = lr_model(X_train.iloc[0:12,:], y_train[0:12] )  as this way you are correctly accessing the rows and columns of the pandas dataframe.


It will not
It's about using a model which was trained on thousands of Classes on Millions of images of ImageNet.
 Chances are very high that most of the classes you have in your dataset is already there.
In general, if you trained a model on a super-class (e.g. vehicle), then you may reuse it to classify the Car variant(Utilizing its initial layers).
Point is to reuse the characteristics already learned by another model. It can reduce training time drastically.


What are the points called in a dataset where the points have the same features but different labels

I'm not aware of any specific term for this but in general I would call this either:

noisy data, typically in the case where these inconsistencies are due to mistakes in the annotation process (normally the proportion of mistakes is much lower than the proportion of correct instances).
poor data design, if the way instances/features are defined doesn't match the goal of the ML process (the example provided would fall into this category).


Multioutput regression with MLPRegressor - Does it work?

Not sure if this makes a difference, but I think you need to call fit from the initialized object. So for example, your fit line should be:

regressor_ANN.fit(X,y) instead of MLPRegressor.fit(X, y)

then predicting a new set will be:

regressor_ANN.predict(X_test)


How is the fit function in SimpleImputer working to find the mean in the Salary column as well when just the Age column is given as its argument?

imputer = imputer.fit(df[['Age']])
X[:, 1:3] = imputer.fit_transform(X[:, 1:3])

You are again applying "fit" with fit_trasnform. Try with "transform" only
You will get a value error

ValueError: X has 3 features per sample, expected 2


Why is the learning rate for the bias usually twice as large as the the LR for the weights?

I can't find any literature discussing setting a fixed learning rate differently for the bias specifically than for the other weights, but if you asked the researcher responsible for that model their answer would probably be something like:

  "The network wasn't training properly and I noticed the error for the bias term wasn't converging as fast as I'd like. When I tried doubling the learning rate for just that term it seemed to fix the problem. I dunno man, it just worked."

As far as I can tell, training neural networks is still more art than science and the amount of skill, patience, and care taken in training a neural network (depending on the architecture) can have as much impact on the model's performance as the network topology. Consider issues like mode collapse in GANs, vanishing gradient with ReLUs, or just getting stuck in local optima. Neural networks often need to be "babysat" during training, and consequently trial and error may contribute significantly to certain training process decisions. I'm sorry if that's not a satisfactory answer, but I have a strong suspicion that's what's going on here. If you post some specific examples, it might be easier to figure out concretely what's going on, but otherwise my guess is that this was just the result of trial and error.

Regarding using parameter-specific learning rates generally, most modern optimizers do this, including AdaGrad, RMSProp and Adam.


Found the answer hiding in lines 93-100 in the bagging.py file.
Here is what I understand - the bootstrapping process works in three steps:

Calculate number of samples to train each estimator on (the max_samples variable in the bagging.py code). In your case its $1.0 * x.shape[0] = 150$.
Select with repetition the needed max samples(as calculated in the previous step).
The selection is done by using randint function, and it generates an array of the x series indices. A given index can appear in this array more than once.
In order to account for indices(samples) that were selected more than one, a weights vector is passed into the base estimator fit function. So, for example, a sample that was selected twice will have $weight * 2$ and it will have the desired impact on the fitting algorithm.

As to my understanding, one can use the estimators_samples_ only to find out which samples were included and not how many times each of them did.


Why does measuring for Asymmetric Binary Attributes exclude t (Values which are 0 for both)?

I know it is the old post, but I also search for the answer to this question. and here what I got:

first of all, binary asymmetric attributes only care the 1 values. for example in measuring the distance of the patient based on their symptom, 1 means the symptom exists and 0 mean it does not exist. Or another example in measuring the distance between 2 documents based on the occurrence of words/term (most of these cases not using binary, but still asymmetric), 0 means the term does not exist in the document.

in those 2 cases, the number of symptom or term can be very huge and may cause a lot of 0 values appear in 1 records. This is because one disease may only have a small number of symptoms or one document may only contain a small number of words/term. If all of these 0 values are used to calculate the distance between two records, the similarity can be huge. example

record 1 :  1 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

record 2 :  0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

assymetric:
d(1,2) = 3/4  --> similarity(1,2) = 1-3/4 = 1/4

IF we consider it as symmertic:
d(1,2) = 3/20 --> similarity(1,2) = 1-3/20 = 17/20 (large similarity)

that is why the attributes with the same 0 values between two records are not included in the distance calculation.


Application of Deep Reinforcement Learning

This honestly sounds more like a supervised learning problem.
For reinforcement learning to work, you would need a model that can constantly return values for a given input.

With social media, that would mean

a) posting the same image with all kinds of different hashtags and expecting people to give the most appropiate hashtags likes in a short amount of time. This will not happen.
b) searching for all occurrences of an image with different hashtags. This is basically supervised learning.

I recommend finding a social media dataset first. Try to group all occurrences of an image and find the median of likes for each hashtag. Don't forget to compensate for follower count, as more popular posters will get more likes on average regardless of hashtag. Store the best hashtag as label for the given image.

You now have Y different hashtags for X different images.

From now, you can treat it as classification problem. X is your sample count, Y is the amount of possible outputs. If you don't want to write your own loss function and only want to predict the single best hashtag, use cross entropy as loss function.

Of course you still have to choose your social media dataset, appropriate images for training and NN structure, but I hope this helps as a general approach.


The ratio doesn't take into account the fact that the last layer may have negatives results, in which case the ratio doesn't work, while the softmax does. Another corner case is if the denominator of the ratio is zero.

Also, softmax has two interesting properties:

It will accentuate the differences between the input and the output, by squashing the lower values and increasing the highest value, which makes the output of the network closert to a max instead of a softmax (note that max is not differentiable and hence cannot be used).
In a classification problem, where a softmax layer is used, the loss function is the cross entropy, which has the form - y * log(y_hat). Turns out that log of softmax is very easy to compute, as log(a/b) is log(a) - log(b) and log(exp(x)) is simply x. That makes softmax more desired.


Does the Koalas library allow to use all Pandas machine learning libraries like Scikit-Learn, XGBoost, and TensorFlow?

I think you have misunderstood the koalas library. You can say its Pandas on Distributed System. You can use Koalas similar to pandas. There are few drawbacks with respect to APIs which is documented in their docs and few articles already written on medium.

You can do your EDA and straight away use them in all the libraries you have mentioned.

Recent pandas 1.0 is faster compared to older versions. It also uses Numba Behind the scene.

Vaex is another library which is available and you can use for EDA but the api names are different from pandas.

Also you can use Modin and dask and use it like pandas with few limitations again.

Clearly these libraries have no dependencies on Sklearn, XGBOOST or TF. You can split train and run your model.


How to overcome training example's different lengths when working with Word Embeddings (word2vec)

Let me suggest three simple options:

average the vectors (component-wise), i.e., compute the word embedding vector for each word in the text, and average them.  (as suggested by others).
take the (component-wise) maximum of the vectors.  (max, instead of average)
take the (component-wise) minimum of the vectors.  (min, instead of average)

Each of these yields a feature vector that is independent of the length of the text.

There is some research suggesting that concatenating the max and the min yields a pretty effective feature space: it's not the absolute optimal, but it's close to optimal, and is simple and easy to implement.  See this question on Statistics.SE for details.

Here is an alternative idea, inspired by cubone's answer, that as far as I know hasn't been tested before.  The idea is to tag the text using a part-of-speech tagger and then use those tags to inform the featurization process.

In particular, write down a list of all possible POS tags that could be emitted by the POS tagger.  Suppose there are 20 possible tags (CC, DT, JJS, MD, NNP, ...).  Then the feature vector will be 20*300 = 6000 elements long: it will have one 300-vector per POS tag, concatenated in some canonical order.  The 300-vector for each tag could be computed by averaging the word embedding vectors of all words that are tagged by the POS-tagger with that tag.  Or, you could get one 600-vector per POS tag, obtained by computing the min and max over all vectors of words with that tag.

This might yield a richer feature space, I don't know if it would yield any improvement, but it's something you could try if you wanted to experiment with different ideas.


You could calculate these metrics and then attempt to clarify how your model is failing during validation, but then you're effectively turning your simulated test environment into a second round of training. Any further tuning decisions made from calculating these metrics defeat the purpose of using the validation set as a chance to prove your model's ability to generalize.


How to interpret feature importance (XGBoost) in this case?

I think you can find the correlation matrix for the feature which could provide you with evidence to justify your hypothesis.


How to interpret the loading values of a pca?

The columns of your loadings matrix are a basis of orthonormal eigenvectors.  This is an important concept from linear algebra, and well worth learning about in detail if you're not familiar.  But for the purposes of this answer it can be understood as defining a system of coordinates.

For each student, we can define a point in a four-dimensional space (specifically, in $\mathbb{R}^4$) which represents their grades (after centering and normalization).  Or to put it another way, you can imagine the set of all students' grades as a scatterplot in four dimensions, with four perpendicular axes.  We can orient these axes in various directions (just as we can in two or three dimensions).  The most obvious choice is to have one axis for each subject, so the axis which is collinear with the unit vector pointing from the origin to the point $(1,0,0,0)$ represents their grade in German, and likewise the axis which is collinear with the vector $(0,1,0,0)$ represents their grade in Philosophy, the axis which is collinear with the vector $(0,0,1,0)$ represents their grade in Math, and the axis which is collinear with the vector $(0,0,0,1)$ represents their grade in Physics.

However, there's no reason to expect that the direction in which our scatterplot is most spread out (the direction of greatest variance in the data) will align with one of these axes.  PCA picks out a new set of axes so that one axis aligns with the direction of greatest variance, and another aligns with the direction of the greatest remaining variance after the first direction is projected out, and so forth.  The unit vectors (expressed in the original coordinate system) which point along these new axes are the columns in your loadings matrix.

In the case of this particular example, the loading vector for the first principal component is along an axis that basically expresses whether they're better at Math and Physics, or better at German and Philosophy.  The loading vector for the second principal component is along an axis that basically expresses how good or bad a student they are over all (hence all the components of the vector have the same sign and similar magnitude).  You wondered about the negative sign on all four components - if you're familiar with eigenvectors you'll know that changing all components of the vector by an overall sign is irrelevant.  Basically, it's the same as just swapping which end of the axis we call positive and which we call negative.

So in this case the first two loading vectors are fairly close to what many of us might have expected to see.  But even in this fairly intuitive example, you shouldn't be surprised that the loading vectors for the later principal components don't seem as obvious to you.  That's because these are only addressing the variance that remains after we project out the variance that's explained by the first two factors.  We all probably know that students who are good at Physics tend to be good at Math, but how many of us know (for example) if, after controlling for how good they are at Physics, the ones who are also better at Philosophy than German will be better at Math?  These subtler effects will be less obvious to a causal observer than the dominant effects.

Once you get to the loading vector for the fourth principal component (out of four), you really don't need to wonder at all about why it has the particular value that it has.  In fact, this vector was entirely determined by the previous three (up to the irrelevant overall sign) .  This can be understood by remembering that PCA picked out four perpendicular axes in a four-dimensional space - once the first three are specified, there's only one remaining possible choice that's perpendicular to all of them.


This shouldn't be terribly complicated.

Big picture

Assuming you have Linux on your server: SSH to your server from your work laptop, train the network, receive results (the trained network).

Details

Train the network

If you want to stick with Python, there are two basic options that I am familiar with.

The first is PyBrain, a library built specifically for training neural nets. The syntax is fairly straightforward. The dataset structure is a bit unusual (especially if you are used to just using numpy arrays in scikit-learn), but otherwise, it works rather well. However, it doesn't support GPUs AFAIK, and I don't know how optimized/fast it is.

The second is Google's Tensor Flow. A bit of a heavy-weight if you just want to run vanilla neural nets, but the syntax is also pretty easy to get around (although very different from PyBrain). It's also probably faster than anything else you could find in Python. It also supports GPU training.

Get the results back

This depends on the library you choose (or it could be definitely up to you write your own implementation). But both libraries mentioned above will let you return the trained weights, which you can save as csv files and download upon training. Or you could pickle them and get back that way. Also, Tensor Flow has this Tensor Board functionality which should let you visualize the training and the network structure, but I haven't tried it yet, so can't help on that front.


Cross validation while preserving a column (not the target ) distribution

One idea would be to combine the two columns (one predictor and the target) and then stratify using the combined column.

Example: say for some observations, the target and the column take the following values: target = [0,1,0] and column = [A,A,B]. Then the combined column could look something like [A0,A1,B0] and could be used for stratification.

I'm obviously assuming that the predictor column is categorical - you might have to take a different approach for a continuous variable.


Validation showing huge fluctuations. What could be the cause?

From simple inspection of your plot, I could make a few conclusions and list things to try. (This is without knowing any more about your setup: training parameters and model hyperparameters).

It looks like the loss is decreasing (put a line of best fit through the validation loss). It also looks like you might be able to train for longer to improve results, as the curve is still headed downwards.

First I will try answer your title question:

  what is the cause of the fluctation in the validation loss?

I can think of three possibilities:

Regularisation - to help smoothing the learning process and make the model weights more robust. Adding/increasing your regularisation will prevent large updates to weights being introduced.
Batch size - is it relatively small (e.g. < 20?). This would mean that the measured mean error at the end of the network is computed using only a few samples. With a batch size of, say 8, then getting 4/8 correct and compared to getting 6/8 correct has a large relative difference when looking at the loss. Taking the mean of the errors with such small batches will lead to a not-so-smooth loss curve. If you have enough GPU memory/RAM, try increasing batch size.
Learning Rate - might be too large. This is similar to the first point regarding regularisation. To make smoother improvements, you might need to slow down the pace of learning as you approach a loss-minimum. You can make this perhaps run on a schedule, whereby is is reduce by some factor (e.g. multiply it by 0.5) every time the validation loss has not improved after, say 6 epochs. This will prevent you from taking big steps and then maybe overshooting a minumum and just bouncing around it.

Specific to your task, I would also suggest trying to perhaps unfreeze another layer, to increase the scope of your fine-tuning. This will give the Resnet-18 a little more freedom to learn, based on your data.

Regarding your last question:

  Is this something that I should be worried about, or should I just pick the model which scores the best on my performance measure (accuracy)?

Should you be worried? In short, no. A validation loss curve like yours can be perfectly fine and deliver reasonable results; however, I would try some of the steps I mentioned above before settling for it.

Should you just pick the best performing model? If you mean taking the model at its point with best validation loss (validation accuracy), then I would say to be more careful. On your plot above, this might equate to around epoch 30, but I would personally take a point that has trained a little more, where the curve gets a little less volatile. Again, after having tried some of the steps outlined above.


UniLM - Unified Language Model for summarization

Here's what you should do

Prepare your dataset: Follow similar instructions as described in the paper and preprocess your dataset. This will be your major task as after this you will only have to fine-tune the model. If you don't have a dataset, you can use the dataset used in this research paper, which can be downloaded from here.
Download the pre-trained model. Or you can choose to start from the provided fine-tuned model checkpoint (from the link). You will have to check which version of the model works best for your dataset. If you select model fine-tuned for summarization task and your dataset is similar to CNN/DailyMail dataset [37] and Gigaword [36] you can skip fine-tuning.
Fine-tune model: In this step, you will be using the command mentioned in the readme of the Github repository. Note that, there are some parameters that should be according to the language model you have downloaded in the previous step. Based on the size of your dataset you can change the number of epochs in the following command. You should also note that this will require GPU. The repository readme recommends 2 or 4 v100-32G GPU cards for finetuning the model.

    OUTPUT_DIR=/{path_of_fine-tuned_model}/
    MODEL_RECOVER_PATH=/{path_of_pre-trained_model}/unilmv1-large-cased.bin
    export PYTORCH_PRETRAINED_BERT_CACHE=/{tmp_folder}/bert-cased-pretrained-cache
    export CUDA_VISIBLE_DEVICES=0,1,2,3
    python biunilm/run_seq2seq.py --do_train --fp16 --amp --num_workers 0 \
      --bert_model bert-large-cased --new_segment_ids --tokenized_input \
      --data_dir ${DATA_DIR} \
      --output_dir ${OUTPUT_DIR}/bert_save \
      --log_dir ${OUTPUT_DIR}/bert_log \
      --model_recover_path ${MODEL_RECOVER_PATH} \
      --max_seq_length 192 --max_position_embeddings 192 \
      --trunc_seg a --always_truncate_tail --max_len_a 0 --max_len_b 64 \
      --mask_prob 0.7 --max_pred 48 \
      --train_batch_size 128 --gradient_accumulation_steps 1 \
      --learning_rate 0.00003 --warmup_proportion 0.1 --label_smoothing 0.1 \
      --num_train_epochs 30

Evaluate your model: Use biunilm/decode_seq2seq.py to decode (predict the output of evaluation dataset) and use the provided evaluation script to evaluate the trained model.
Use the trained model: In order to use this model to make a prediction, you can simply write your own python code to:

load Pytorch pre-trained model using pytorch_pretrained_bert library as used in decode_seq2seq.py file
Tokenize your input
predict the output and detokenize the output.

Here is the logic which you can use:

model = BertForSeq2SeqDecoder.from_pretrained(long_list_of_arguments)
batch = seq2seq_loader.batch_list_to_batch_tensors(input_batch)
input_ids, token_type_ids, position_ids, input_mask, mask_qkv, task_idx = batch
traces = model(input_ids, token_type_ids,position_ids, input_mask, task_idx=task_idx, mask_qkv=mask_qkv)

Note that this is not the complete logic. This code just show how github repository code has handled the saved model and used it to make predictions. Use traces to convert ids to tokens and detokenize the output tokens (as used in the code here). The detokenization step is necessary as the input sequence is tokenized to subword units by WordPiece.

For reference here is the code which loads the pre-trained model. You can go through the loop to understand the logic and implement it in your case. I hope this helps.


Maybe this article will help you How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings.
Talks about contextual word embeddings like BERT and GPT how they can capture various polysemous concepts rather than the static word embeddings which create a single representation for each word, such as GloVe.


Text Similarities: which nlp methods to use?

There are many ways to see how texts are similar, but this will depend on your use case.

Semantics

Nowadays, word embeddings are getting popular. Like it was suggested in the comments, you could use Doc2Vec to transform your sentence into a vector and calculate the cosine distance from each sentence.

The idea with these learned embeddings is that you kind of encode the meaning of the sentence by looking how words are used in context. So, to sentences would be similar if words are used in the same manner in a context.

However, this could potentially be difficult to interpret.

For example:

Good tattoo shop. Clean space.
Good pizza restaurant. Large space.
Terrible tattoo shop. Dirty space.

Which of these are semantically close? It will depend a lot on your training and your own judgment to tell if the results are useful to you.

Sentiment

Maybe you are interested in saying that sentences are similar if their sentiment is positive or negative (or on some other scale).

For example:

Suppose you can have sentiment on a scale $[-1,1]$, where $-1$ is negative, $0$ is neutral and $1$ is positive.

Good tattoo shop. Clean space.
    Sentiment = 0.7

Good pizza restaurant. Large space.
    Sentiment = 0.75

Terrible tattoo shop. Dirty space.
    Sentiment = -0.9

There are many resources online on how to do this. Here is an answer with some links.

BoW

The bag-of-words is a very simple approach, but depending on what you are doing, it could do the trick. Suppose you want to say that sentences are similar if they are about a business in the same industry.

You can make a simple dictionary for every industry:

word_per_industry = {
    'restaurants' = ['restaurant', 'food', 'chef', 'dish', 'salad', 'lunch'],
    'tatto_shop' = ['clean', 'dirty', 'art', 'sterlized']
    .
    .
    .
}

Then, per sentences, you can count which business has more words in the sentence, and if sentences are from the same industry they are similar.
Of course, you could also make lists based on another characteristic, not necessarily on industries.


Do i need to use hyperparamters from Gridsearch to train on WHOLE training set to get final model?

As far as I understand (disclaimer: I'm not very familiar with Python) your approach is correct: the selected hyper-parameters are tested on the hold out test set which is different from the training set, this way there's no data leakage and you can evaluate the true performance of your model before applying it to the test set.

For analysis purposes it could be useful to compare the performance of the best model on X (training set) and X_test (hold out) in order to check for overfitting.

Note that in a case like this where you directly select the best hyper-parameters I would consider it acceptable to skip the testing on the hold out set, however in this case you wouldn't know the true performance of your model (so for instance you wouldn't be able to check if it's overfit). To be clear: I don't think you should do this, it's just a remark to show the difference with/without hold out set.


[R]Fuzzy C-Means, different between ppclust vs e1071?

Look at the source code. The guts of e1071::cmeans are written in C, but ppclust::fcm looks like all R and has some ugly looking eval stuff which might make it dead slow.


What is the efficient way to generate a similarity score when comparing two face images?

A common method is defining "landmarks", a set of specific points that exist on every face . For example, the outside edge of each eye or the bottom of the chin are landmarks on most faces. Train the deep learning network to learn the relationship between these landmarks which will form the embedding space. Then define a distance metric in this embedding space. Finally, create a threshold for similarity between sets of points in this embedding space. That threshold can be learned through "one-class" classification techniques.

One gotcha is pose. Faces turned different directions look totally different to a computer. Assuming you have isolated the faces in an image, it is best practice to warp each face so that the eyes and lips are always in the place in the image. This (and other normalization steps) should be done early in the processing pipeline to make subsequent analysis much easier.


Designing CNN that does one column convolution across the x-axis
So the cnn somehow has to use a kernel that has to be 276 rows and 1 column wide, but is it possible in keras to make a 2d kernel and perform 1d convolution.

I'm not sure I understand your question correctly, but if you have input images that are 276 high, X=500 wide and have 3 color channels, then the following performs column-wise convolutions across the X-axis:
rows, cols, chans = 276, 500, 3

...
model.add(Convolution2D(64, rows, 1, input_shape=(rows, cols, chans)))
...

The output shape of this layer will be (1, 500, 64).
I inserted "64" as the number of filters.. you can set that to whatever you need / whatever works best.
Note: shapes are specified in tensorflow mode (i.e. the order is (rows, cols, channels)). Depending on your configuration you might have to change those shapes to theano mode: (channels, rows, cols).

So the cnn somehow has to use a kernel that has to be 276 rows and 1 column wide,

It actually has to use a kernel that is 276 rows, 1 column and 3 channels wide since your input images have 3 channels.. it would be 276 rows and 1 column wide if you used grayscale images.. but that shouldn't matter.


How to get similar visualization to R fviz_cluster function in Python?

In Wolfram Language you may use FindClusters to identify the clusters, ConvexHullMesh to calculate the convex hull of the clusters, and ListPlot with the Prolog option and Labeled to display.

First we will collect some example data from the USA "AdministrativeDivision" states.  The "Name" property includes the county so this is removed with StringDelete.

data = Select[FreeQ[_Missing]]@EntityValue[
    EntityClass["AdministrativeDivision", {"ParentRegion" -> Entity["Country", "UnitedStates"]}],
    {"HomeOwnershipRate", "HealthInsuranceCoverageRate", "Name"}];
data = MapAt[StringDelete[", United States"], {All, -1}]@data;
First@data

{70.7% , 85.2%, "Alabama"}

For each state we get two "Percent" Quantity objects and the state name.

We can FindClusters on this list by using all but the last item in each list (Most) for the cluster algorithm but returning the entire list (Identity), which includes the name, as the cluster item.

clusters = FindClusters[Rule @@ Through[{Most, Identity}@#] & /@ data, Method -> "MeanShift"];

The hulls of clusters with ConvexHullMesh.

hulls = ConvexHullMesh /@ Map[Most] /@ clusters;

Then ListPlot clusters with each point Labeled while using Prolog to plot the hulls beneath the points.

ListPlot[
 Map[Labeled[Most@#, Last@#] &] /@ clusters,
 PlotStyle -> ColorData[110],
 PlotTheme -> "Detailed",
 FrameLabel -> {"HomeOwnershipRate", "HealthInsuranceCoverageRate"},
 Prolog -> {
   Opacity[.1],
   MapIndexed[
    {ColorData[110] @@ #2,
      EdgeForm[{Thin, ColorData[110] @@ #2}], #1} &,
    hulls
    ]}
 ]

The default behaviour is to automatically use Tooltip in places where labels would overcrowd.

Hope this helps.


What is the intuition behind using 2 consecutive convolutional filters in a Convolutional Neural Network?

Could somebody kindly explain to me the intuition behind stacking 2 or more consecutive convolution filters. Why couldn't the 2 filters be merged into 1?

No, when having two consecutive convolution layers can't be combined into one. The subsequent filter's inputs are the features extracted from the previous one. This results in the second layer's features are of higher-level than the previous.

This is the basis of the whole CNN. Having multiple convolutional layers stacked along the depth of the network, allows the network to extract high-level features (not just edges and corners) from the input images.

Edit.

The first convolutional layer of a CNN is essentially a standard image filter (+ a ReLU). Its goal is to take a raw image and extract basic features from it (e.g. edges, corners). These are referred to as low-level features.

The second convolutional layer, instead of the raw image, accepts the features extracted by the first as its input. This allows it to combine these basic shapes into more complex features.

The features extracted become more and more complex as we go further down the network. Layers near the middle of the network extract the so called mid-level features, while the final layers extract high-level features.

CNNs are powerful tools because it is trained to extract the best features for each task. This results in the network extracting different features for different tasks.

For example, take the image below:

While the low-level features are the same for each task, mid and high-level features differ greatly. For instance, for the first CNN that recognizes faces:

The low-level features are basic geometric shapes (edges, corners, circles, etc.).
The mid-level features combine the previously extracted shapes to extract more complex features (eyes, noses, lips), specific to the task of facial recognition.
Finally, the high-level features combine the previous to extract even more complex features (in this case different faces).


Which Format is Faster, Matlab (.mat) or NumPy (.npy)?

From my research:

both can store data in binary format
both store the data type of the data
I am unsure about compression ratios and load time, which seems to be the subtext of your question.

One thing you don't seem to address is what you are loading the data into, or whether you are considering moving from a MATLAB environment to Python environment or visa-versa.

That said, I found this post useful and thought it may be helpful to you if you have not seen it already.  https://stackoverflow.com/a/10997335/3259054 Perhaps you could write a small script to sample some files and see the difference.

Have you considered the HDF5 format?  If you are looking to make a change, you might as well test other options too and HDF5 has a lot of momentum towards becoming the de facto standard for scientific computing.

Finally, purely out of a desire to learn, why are you concerned about the file format if you have memory constraints?


Comparing text corpora sentence-wise in Python

There are several ways to do this, but, assuming the lists are the same length and the sentences you wish to compare are in the same indices int their respective lists, you can handle this with a simple list comprehension:

diffs = [(s1[i], s2[i]) for i in range(len(s1)) if s1[i] != s2[i]]


Is there any reason why providing symbolic features into an MLP wouldn't outperform feeding raw pixels to a CNN in a RL task?

If the features are able to fully define the image, there would be no reason to use a pixel-based approach.  You are essentially performing lossless compression and (nearly) instantly acquiring a feature set, so you would see a dramatic performance difference between the CNN and MLP.

If you needed a quick solution, CNNs are great out-of-the-box algorithms, but if you are willing to put in the time to develop an MLP, it would provide a performance boost without losing accuracy.


PCA on Neural Networks dimensions reduction?

The approach you use to do dimensions reduction is agnostic to the method you use for classification. You can use PCA to preprocess your data before to train any type of classifier, including artificial neural networks if that's what you want to use.


I need to implement recommender which, for a set of items, recommends a new set of items.
Is this a good idea?

Have you looked into Association Rules Mining? If you're open to other procedures this one is first one that came to my mind for recommenders based on sets of items. For those not familiar, this is simple method for retail shops to determine "75% of customers who bought A, B also bought C". Within these algos, the Apriori algo is straight-forward, easy to implement, and may get you what you need.


How much of data wrangling is a data scientist's job?

This is a situation that many blogs, companies and papers acknowledge as something real in many cases.
In this paper Data Wrangling for Big Data: Challenges and Opportunities, there is a quote about it

data scientists spend from 50 percent to 80 percent of their time
collecting and preparing unruly digital data.

Also, you can read the source of that quote in this article from The New York Times, For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights
Unfortunately, the real world is not like Kaggle. You don't get a CSV or Excel file that you can just start the Data Exploration with a little bit of cleaning. You need to find the data in a format that is not suitable for your needs.
What you can do is make use of the old data as much as you can and try to adapt the storing of new data in a process that will be easier for you (or a future colleague) to work with.


cross_val_score does not return the history of the training. You can use fit instead:

history = model.fit( ...

See this example.

As you mentioned, the history object holds the results of the training for each epoch.

Here is the relevant bit:

history = model.fit(X, X, epochs=500, batch_size=len(X), verbose=2)
pyplot.plot(history.history['mean_squared_error'])
pyplot.plot(history.history['mean_absolute_error'])
pyplot.plot(history.history['mean_absolute_percentage_error'])
pyplot.plot(history.history['cosine_proximity'])
pyplot.show()

I was actually working on the same example that you referenced yesterday. I think it is hard to understand because it introduces many functions and concepts: estimators, StandardScaler, KerasRegressor, Pipeline, KFold and cross_val_score.

However, I did like the approach to creating and testing models, and Cross Validation would produce more robust models.


The basic ways to make a recommendation engine are :

Link a profile to similar ones
Extract characteristics of a profile, and match them with characteristics of, here, jobs
A mix of the two solutions above

Either way, you need data, labels, characteristics extractors...


Generating the right target for an LSTM model

The way you framed it is not a regression problem since regression predictions are unbounded. Your actual scores are strictly bounded between zero and one which makes it more similar to predicting a probability.
To get the best predictions, it would best to apply a sigmoid to the final output node activation to bound predictions between zero and one.
For a scoring, cross-entropy would work well as a loss function:
$$H(p,q) = - \sum_{i}p_i logq_i$$
The true label for a datapoint would be modeled as $p_i$ and the predicted value of the current model would be modeled as $q_i$.


Can you elaborate what you mean by 'methodologies'?

In the meantime, take a look at The Field Guide To Data Science by Booz Allen Hamilton. This guide talks about data science processes and frameworks.

Data Science Design Patterns by Mosaic talks about, you guessed it, data science design patterns. This is quite useful to get a sense of common design patterns. They are also working on releasing a book on the same subject.

Then there are several resources out there that will come up as results to more targeted searches, such as machine learning paradigms, recommender systems paradigms, etc. Data Science is a large and varied field, and you'll find many resources out there for each subsection of it. As far as I know, there isn't one book that covers it all.


Why is word prediction an obsession in Natural Language Processing?

The first line of the BERT abstract is

  We introduce a new language representation model called BERT.

The key phrase here is "language representation model". The purpose of BERT and other natural language processing models like Word2Vec is to provide a vector representation of words, so that the vectors can be used as input to neural networks for other tasks.

There are two concepts to grasp about this field; vector representations of words and transfer learning. You can find a wealth of information about either of these topics online, but I will give a short summary.

  How can BERT be expected to get this right, and how can humans or another algorithm be expected to evaluate whether "soup" is a better answer than "coffee"?

This ambiguity is the strength of word prediction, not a weakness. In order for language to be fed into a neural network, words somehow have to be converted into numbers. One way would be a simple categorical embedding, where the first word 'a' gets mapped to 1, the second word 'aardvark' gets mapped to 2, and so on. But in this representation, words of similar meaning will not be mapped to similar numbers. As you've said, "soup" and "coffee" have similar meanings compared to all English words (they are both nouns, liquids, types of food/drink normally served hot, and therefore can both me valid predictions for the missing word), so wouldn't it be nice if their numerical representations were also similar to each other?

This is the idea of vector representation. Instead of mapping each word to a single number, each word is mapped to a vector of hundreds of numbers, and words of similar meanings will be mapped to similar vectors.

The second concept is transfer learning. In many situations, there is only a small amount of data for the task that you want to perform, but there is a large amount of data for a related, but less important task. The idea of transfer learning is to train a neural network on the less important task, and to apply the information that was learned to the other task that you actually care about.

As stated in the second half of the BERT abstract,

  ...the pre-trained BERT model can be finetuned with just one additional output layer
  to create state-of-the-art models for a wide
  range of tasks, such as question answering and
  language inference, without substantial taskspecific architecture modifications.

To summarize, the answer to your question is that people DON'T care about the masked word prediction task on its own. The strength of this task is that there is a huge amount of data readily and freely available to train on (BERT used the entirety of wikipedia, with randomly chosen masks), and the task is related to other natural language processing tasks that require interpretations of the meaning of words. BERT and other language representation models learn vector embeddings of words, and through transfer learning this information is passed to whatever other downstream task that you actually care about.


How to do multitask learning using Caffe?

Every Blob in caffe can be assigned a nonzero loss weight.

And you can have an arbitrary number of outputs.

This means you can just learn n different networks on the same data with different targets and assign every loss function it's own weight. Caffe takes care of adding up all the loss.


Does a Convolutional Layer in a Neural Network learn the correlation between its input signals via its kernel?

In a convolutional neural network (CNN), a convolutional layer has several channels, each of which has one convolution kernel, often written down as a matrix. This convolution kernel is nothing more than a collection of weights used to compute a linear combination of elements of the input.
While both "traditional" dense layers and convolutional layers compute a linear combination of their inputs, convolutional layers have the added structure of preserving spatial information. Additionally, the output of a convolutional layer, called a feature map, can be understood as the result of sliding the kernel along the input.
For instance, take a simple kernel $K = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{bmatrix}$ and a small input "image" $I = \begin{bmatrix} 9 & 8 & 7 & 6 \\ 5 & 4 & 3 & 2 \\ 1 & 2 & 3 & 4 \\ 5 & 6 & 7 & 8 \end{bmatrix}$. We take an input with only one channel for convenience: if it's the first layer of the CNN, we can assume the values of $I$ represent a grayscale image. Let us look at how the convolution is performed if no padding is added to the image, in which case the output will be a 2 by 2 matrix.
Starting at the top left corner, we convolve $K = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{bmatrix}$ with $\begin{bmatrix} 9 & 8 & 7 \\ 5 & 4 & 3 \\ 1 & 2 & 3 \end{bmatrix}$. To do so, we multiply the two matrices element-wise, then sum up all the elements of the resulting matrix: $1 \cdot 9 + 2 \cdot 8 + 3 \cdot 7 + 4 \cdot 5 + 5 \cdot 4 + 6 \cdot 3 + 7 \cdot 1 + 8 \cdot 1 + 9 \cdot 3 = 146$.
Doing the same three more times yields the following feature map: $\begin{bmatrix} 146 & 157 \\ 200 & 233 \end{bmatrix}$.
Depending on the kernel of a layer, the feature map will look different, and the information extracted from the input has a different "meaning". The idea behind a CNN is to make the coefficients of the kernel learnable parameters. Often, multiple "meanings" must be extracted from a single input, so a convolutional layer has multiple channels.
However, CNNs didn't invent convolution, they merely made the kernels learnable. Indeed, such methods have been used in "traditional" computer vision (CV) for a long time. In traditional CV, the kernels are hand-crafted to fill specific roles, extract specific types of information.
For edge detection or line detection, you might use a Prewitt filter which is the discrete equivalent of a derivative operator or a slightly more complicated Sobel filter.
In fact, it is not uncommon for a CNN to learn approximations of well-known hand-crafted convolution kernels from traditional CV on the first layer. Also, by visualizing feature maps on deeper layers, as the training of a CNN progresses, gives you good insight into what the specific convolution channels are learning to detect.


any reason for this project to use hadoop/spark?

Especially when your main goal is learning, I would break it into several phases:

Get familiar with pandas dataframes and visualization using matplotlib. Try loading subsets of your datasets using pandas and visualize them using matplotlib, for example plot the time-series, or word count histograms. This will come in handy lateron to be able to understand the predictions/clusters you will create using machine learning methods. Pandas also provides functions to clean, aggregate and resample the data, which will be useful to align multiple time-series (e.g. weather and time-series) to the same sampling rate.
Familiarize with scikit-learn using the articles and examples in the excellent online documentation. Play with the feature extraction, dimensionality reduction, classification and regression methods relevant to your data, apply them on smaller subsets and visualize the results. Learn about cross-validation and find a good algorithm and parameters that work well on a subset of your data.
One you have a pipeline setup and you want to run it on the full dataset, Spark can come in handy for feature extraction and cross-validation, as those tasks can be often run very well independently and in parallel. Install the Spark libraries on your machine, and reproduce the Pyspark examples (e.g. calculating Pi) in local mode. No need to setup a Hadoop cluster for that, Spark can make full use of your machine in local mode, reading from local files on disk. Once you have that running, try to express the expensive steps of your code from step 2 as Spark RDD operations. As Spark can make use of all cores of your machine, you should already see a speedup - debugging becomes much harder though compared to working with pure Python scripts.
Play with advanced stuff like deep learning, using the word2vec feature extraction. Compare with traditional feature extraction (bag of words).
(optional) If you want to experience the speedup you can get from Spark in a distributed setting, get an account in AWS, put your data on S3 and fire your Spark scripts against a multi-node cluster instantiated using Amazon EMR.

Regarding data storage, from my experience CSV files or similar work best to load the data into both Pandas and Spark. There are Spark connectors for databases (e.g. Cassandra), but I am not sure about MySQL and Solr. For processed data and intermediate results (e.g. models) I would always use Python's serialization framework pickle, so you can directly get the object instances back into Python without the need to parse file formats and reinstantiate objects.


Multiple formulas for r squared eval metric for regression

Both are correct. R squared tells you how many variance of the dependent variable is explained by your model. This can be written in different ways.
It is like to say that the probability of being alive is 1 minus the probability of dying.

ESS=TSS-RSS

https://en.m.wikipedia.org/wiki/Coefficient_of_determination

[1]: https://i.stack.imgur.com/QlM6O.jpg


How can collaborative filtering be extended to include more features?

There's a lot of different ways to embed contextual user & item features into recommendations systems. These algorithms are often called "Hybrid Recommendation Systems".
The highest performers are usually big complicated neural nets, but where I would start with is Factorization Machines.
FMs are a staple in the recsys community today, and is probably the best thing to try first if you're trying to build a hybrid recommendation model.
Some characteristics of FMs to be aware of:

They perform well with sparse data
FMs have linear complexity, so they train fast and once deployed they predict fast.
Very flexible in terms of input data, it can take any real valued feature vector.
They're much more easily interpretable than neural networks, if you know how to interpret a regression you're probably capable of figuring out what the FM is doing.
They are capable of both regression (ratings) and binary classification. In my experience you can mix implicit and explicit feedback by weighting interactions differently.
They have a very unique data model. You will notice that in the FM data matrix, there are at least 4 sections, the user-index, user-features, item-index, and item-features. You will sometimes see a interaction-feature section as well. The fundamental idea here is that each row is a record of all the data involved in the interaction between a user and an item (or interactions between whatever). The user & item features section will look more familiar, they are the features of the respective users and items. What may be new to you is the user and item indices. It's basically two separate one-hot-encodings for both the users and items, but you should only ever have 1 user and item on for any interaction. This will make more sense once you see the matrix visuals, I suggest starting at it for awhile to wrap your head around it.

Some additional resources/further reading:

Jefkine Blog
Berwyn's Blog
FastFM Python library (This is an older library, it works well and is good for out of the box performance but if this is for a long term commercial project I would suggest looking for a more modern implementation)
LightFM Python Library (Another FM Library that I personally prefer over FastFM)
RankFM Python Library (Another FM Library specifically adapted for when you only have implicit feedback and want to rank/recommend items.)


Why doesn't the binary classification log loss formula make it explicit that natural log is being used?

This comes down to the change-of-base formula. For any two numbers $a$ and $b$, the following equation is true.

$$
\log_a(x) = \frac{\log_b(x)}{\log_b(a)}.
$$

What this means is that the errors are proportional. So if you wanted to change to using $\log_{10}$, you would simply end up multiplying by a constant factor, and model selection would be the same.

Explicitly,

$$logloss(N=1) = \frac{y \log_{10}(p) + (1 - y) \log_{10}(p)}{\log_{10}(e)}$$

Or, equivalently

$$
logloss(N=1) \cdot \log_{10}(e) = y \log_{10}(p) + (1 - y) \log_{10}(p)
$$

In other words: The base of the logarithm doesn't matter because everything ends up being proportional.


Transfer learning VGGish (AudioSet). Impact of zero padding to fit the input size

Will this have any negative consequences on the ability of the network to learn using my dataset?

That's really hard to say, but I suspect it does not make a difference for any sounds that aren't exclusively defined by higher frequencies.

As a rule of thumb: If a human can still correctly identify a sound sampled at 8kHz, so should a machine. If the machine cannot do it, because of the missing higher frequencies, it may not have learned the characteristics of the sound to identify in the first place, but something else.

  Do I also need to finetune the ConvLayers earlier on in the network?

I'd start with not re-training those. Having been trained on AudioSet, they should be fairly universal.

  Any other thoughts on how I can work around this (potential) issue?It is not feasible to increase the frequency of the recordings, as they are being streamed from low power devices.

Not really. You could upsample your audio to 22.05kHz and go from there—but you cannot create new information for those higher bands out of thin air. So, I don't recommend upsampling.

  Suggestions on other pretrained networks are also welcome!

Read Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings by Cramer, Wu, Salamon, and Bello. They offer an open, pre-trained embedding model for environmental sounds at https://github.com/marl/openl3 (which, IIRC, is better than VGGish).

You might be able use its output as input for a shallow dense network to train a classifier for your audio (unless of course you way too few samples).


Where can I find documentation or paper mentioning pre-trained distilbert-base-nli-mean-tokens model?

The distilbert-base-nli-mean-tokens model is the DistilBERT base model for natural language inference (NLI) using the MEAN pooling strategy for CLS tokens. For more information on the DistilBERT base model, have a look at the original paper. The pooling strategy is simply the way the model combines the different embeddings (and thus information) from the different layers.


What are some good error metrics for multi-label (not mutli-class) problem in industry?

A common example is the Jaccard similarity coefficient:

$J(Y, P) = \frac{|Y~\cap~P|}{|Y~\cup~P|}$

where $P$ is the set of predicted labels for an instance and $Y$ is the true set of labels. This gives a value between $0$ and $1$ for each instance, which you can average over the whole test set to give a score. If $P = Y$, then $J(Y, P) = 1$.

This is implemented in scikit-learn as sklearn.metrics.jaccard_similarity_score.


How to combine the features extracted from different CNN architectures?

You can tf.keras.layers.concatenate your extracted feature before flatten them in order to pass it to your fully connected layer. Note than you can actually use other operation such as multiply or add to merge your feature.


Brain Connectogram or Connectivity Visualization

You might find this link useful if you are R user.
https://www.r-graph-gallery.com/chord-diagram.html


Subtraction of Positive and Negative Frequencies in Sentiment Analysis

Let's take an example, consider two words A & B. A's positive/negative values are +1/0 and B's are +0.5/-0.5. Their difference would appear equivalent (diffAB = 1). When in fact they are quite different sentiments.
If you wanted to compute a single metric, you could do something like "polarity" i.e. take the squared sum of the values which would give you a positive value showing how "polar" the word is. In the case above, the word B would have a lower score because it's positive/negative values are less extreme.
When designing metrics, test a few scenarios to see if it matches your needs.


How to convert my tensorflow model to pytorch model?

You can build the same model in pytorch. Then extract weights from tensorflow and assign them manually to each layer in pytorch. Depending on the amount of layers it could be time consuming. Building the model depends on the model and I think not everything is possible in pytorch that is possible in tensorflow. Examples how to assign weights in pytorch and extract weights from tensorflow are given below.

Getting weights from tensorflow variables W and b:

weights = sess.run(W)
bias = sess.run(b)

where sess is a tf.Session.

Assigning weights to pytorch:

import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F

class Kernel_Emb(nn.Module):

    def __init__(self,D_in,H,D_out):
        super(Kernel_Emb, self).__init__()
        self.linear1 = nn.Linear(D_in,H)
        self.linear2 = nn.Linear(H,D_out)
        self.linear1.weight = torch.nn.Parameter(weights)
        self.linear1.bias = torch.nn.Parameter(bias)

If the variable is not defined in tensorflow (source: https://stackoverflow.com/questions/36193553/get-the-value-of-some-weights-in-a-model-trained-by-tensorflow):

  If you do not currently have a pointer to the tf.Variable, you can
  get a list of the trainable variables in the current graph by calling
  [tf.trainable_variables()][4]. This function returns a list of all
  trainable tf.Variable objects in the current graph, and you can
  select the one that you want by matching the v.name property. For
  example:

# Desired variable is called "tower_2/filter:0".
var = [v for v in tf.trainable_variables() if v.name == "tower_2/filter:0"][0]


conda-forge is just an alternative channel where you can upload to or download packages from. Usually it makes no difference where you download the package from but sometimes conda-forge has the latest version.


fastai - using 'untar_data' function in kaggle kernel

It goes to PosixPath('/root/.fastai/data/oxford-iiit-pet') so you won't see it on the data pane.


Classification problem where one attribute is a vector

1) If you want to build a model with:

Input: Items bought
Output: Win/Loss

then you will probably want to learn a non-linear combination of the inputs to represent a build.  For example item_X may have very different purpose when paired with item_Y than with item_Z.

For the input format, you may consider creating a binary vector from the item list.  For example if there were only ten items, a game in which the champion purchased items 1,4,5,9 (in any order) would look like row 1; a game where he also purchased item 2 and 7 would look like row 2:

item_ID   | 0  1  2  3  4  5  6  7  8  9
________________________________________
champion_1| 0  1  0  0  1  1  0  0  0  1
champion_1| 0  1  1  0  1  1  0  1  0  1

There are a variety of models that might suit this task.  You might use decision trees for interpretability.  A simple neural net or SVM would likely also do a good job.  These should all be found in most basic ML packages.

2) The win rates of various items are directly computable.  Simply count the number of times a champion used the items in question and won and divide by the total number of times a champion used that item combination.  You can do this for any given group size (1 to 6)


text classification : comparing classification reports

Okay so keeping it very short and precisely in context of your question-

Accuracy tells us, out of all the documents how many are classified correctly.

Precision tells us out of all documents which are predicted in a category, how often its correct.

Uni -gram- "nasa", "is" "space" , "agency"
bi-gram- "nasa is", "space agency"

Now lets go over the numbers, in both the cases accuracy and precision doesn't have significant difference.

But as  we can see bi-grams can give me much more information and hence can have better performance on unseen data. Try to test the model on unseen data/validation set and compare the difference.May be Try tri-grams etc also.


Creating Domain specific Question Answering Systems

This is broad question, but here is the general approach. There are several sub-problems involved in a typical QA system, and each may be solved using different approaches:

Question Classification - What type of question is the user asking? This can be posed as a classification problem, assuming you have labels.
Parsing the question - Multiple NLP techniques would be required here
Converting question to canonical form - Every question would be converted into a canonical or structured format (which can be used to query). This is usually the most difficult part of the process. This is usually referred to as Semantic Parsing. One approach is to get variants of phrases and then map it to a single canonical form using some similarity metric. You can start with this paper.

If you can get from a natural language text to a canonical form, you are essentially done. Storing data and retrieval are trivial aspects which could be done in numerous ways. Coming to the question of domains, it is always going to be a trade-off between accuracy for one domain vs a general QA system (which is way too difficult to crack). A single model for every domain will most likely give you better results at the cost of extra effort.


ROC-AUC loss for GRU Model: Cannot use tflearn's loss in keras

This worked for me when I changed the declaration of the function:

def roc_auc_score(y_true, y_pred)

instead:

def roc_auc_score(y_pred, y_true)

I defined it wrong and it worked, when I fixed it I got the same error as you do. I don't really understand your code logic, but you can if this works for you.


How to print x-axes labels in pandas.Series.plot()?

Having a look at the Pandas plot method (on the DataFrame object), we can see that it returns a matplotlib Axes object.

Try something like this:

ax = df.groupby('owner_team').inc_subj.count().plot.bar(ylim=0)

ax.set_xticklabels(df.owner_team)     # if they are still present as strings

If you removed that column, go back to your original processing and keep a copy of it somewhere then use that column above, instead.

Matplotlib will also generally be able to link to the current/latest plot (figure) that has been created. So using the Pandas plot method, you would need to intercept that.You can then try using standard matplotlib methods (e.g. plt.xlabels and so on).

Ther emight be a nice way using the pandas API directly, but I haven't come across that.


how can i sum first day value of each id together?

Just a suggestion:

Let the length of the most transaction over all ID be $n$.

Prepare a list of size $n$, $A$, initialize all entries to $0$.

Now go through each ID, suppose a particular ID has $m$ transaction.  We add the value of the $i$-th transaction to the $i$-th entry of $A$.

The list $A$ would have your desired information. The $i$-entry stores the sum of values of the $i$-th day.

Alternatively, you can also introduce a new feature that encode which transaction  is it for each ID and then use groupby to sum it up again.


How to create full graph from vertex data frame with igraph?

All full graphs of the same order and direction are the same, so just make a full graph of the right size and then put your vertex data on the vertices:

> d = data.frame(Id=c("01","02","03","04"),Name=c("Alpha","Beta","Delta","Epsilon"),stringsAsFactors=FALSE)
> d
  Id    Name
1 01   Alpha
2 02    Beta
3 03   Delta
4 04 Epsilon
> g = graph.full(nrow(d))
> V(g)$Name = d$Name
> V(g)$Id = d$Id


URL <- 'https://archive.ics.uci.edu/ml/machine-learning-databases/communities/communities.data'
data <- read.table(URL, sep = ',',  na.strings = '?')


ML algorithm where variable importance depends upon other variables - specifying conditionality

A simple place to get started is logistic regression for white outcomes first and then for black outcomes. Set cheatflag to 0 if there's no cheating and 1 if white cheated, then run the regression. Do it again, but set cheatflag to 0 if there's no cheating and 1 if black cheated.


Handling categorical variables in large df

No. If, as you said, the variables are categorical, performing a scaling does not make any sense. Plain LabelEncoder already does what you want.


How does the "skip" method work for upsampling? (fully convolutional NN)

So as you already know, the convolutions extract some local information from the image, but for some decisions you need more global info. This is what pooling is supposed to do, it basically erases some locality information. In plain classification this is obviously necessary, because the question, if there is a cat in a picture, is a question about the entire image, not about one or a few pixels. This is why in the end of the classifier you get rid of all locality by adding fully-connected layers.

For semantic segmentation it is not quite as simple, because the output is supposed to still have locality information in it. You actually are just doing a pixel-wise classification and for each pixel the broad context matters. To decide, if a pixel is part of the cat in an image requires information about many pixels within a certain distance from the pixel in question. So this is why pooling layers are used for semantic segmentation as well. But pooling also reduces the image size, that's why upsampling and transpose convolutions are used to get back to the original size.

But there is a downside. As we said, pooling erases some information about the locality, in other words, it reduces the resolution. Once this information is gone, no combination of upsampling and transpose convolutions is going to get it back. Transpose convolutions can only learn to do this the least bad, kind of like a pseudo-inverse matrix. This limits the resolution/detail you can expect on your segmentation mask.

This problem is solved by skip connections by providing the relevant locality information where it is needed. Whenever you do upsampling, you find the last layer before the pooling, where the image still had the same size and simply add it pixel-wise to the upsampled image. This pixel-wise connection is implemented by using a 1x1-convolution in the link you posted. This way, the now upsampled feature map has both the locality detail from before pooling, but also the broad context info from after pooling. This allows for much better detail in the segmentation mask.


How to interpret confusion matric for Extreme learning machine ? why it has decimal points in it?

The confusion matrix shows the percentage of observations that fall into a combination of the actual and the predicted value. These percentage add up to 100% row-wise (0.3419 + 0.6581 = 1). This means that of all observations that had the label 'Phishing', 34.19% were correctly classified as 'Phishing' and 65.81% were incorrectly classified as 'Legitimate'.


Probability Calibration : role of hidden layer in Neural Network

It makes sense that a network with ReLU activations produces worse probabilities than a sigmoid activation when you think about how these activation functions interact with the unit in the output layer of the network. Sigmoid scales the output to be in the range $[0, 1]$, so the values going into the final sigmoid classification unit will be fairly low. Meanwhile, the ReLU-based network has the potential for very high activation magnitudes, which will skew the estimated probabilities to either end of the spectrum.

Having said that, calibration of neural networks is not very well understood in general. A paper was published last year at ICML (https://arxiv.org/pdf/1706.04599.pdf) which showed that often neural nets are poorly calibrated, but it doesn't really explain why this is the case. However, they did investigate factors that affect calibration, and show that deeper, wider networks can often be less well calibrated than their shallower, narrower counterparts. They also show that applying $L_2$ regularisation can improve calibration.

Given this information, I'm surprised that your (fairly shallow and narrow) network is so poorly calibrated. It might be overfitted to NLL, so I would try applying heavier regularisation or training for fewer epochs. If this doesn't work, try implementing temperature scaling, which is described in the paper I linked above. They provide a PyTorch implementation here, so it shouldn't be too difficult hopefully to translate into Keras.


Overall AUC higher than all "stratified" AUCs

AUC can be defined as $P(X_1 > X_0)$ where $X_1$ is the score of a randomly chosen positive instance and $X_0$ is the score of a randomly chosen negative instance.
Like in Simpson's "paradox", what you see could happen because the group has a relatively large effect on your target. For example, one group could be mostly positive, another could be mostly negative. The classifier might not work at all within each of the groups, but if it is merely able to predict the group we are in (or as in your case it is simply one of the inputs), it can have a high AUC.


what are the top level subsets/domains of ML?

Google (in images!) on 'machine learning cheat sheet', to find an example like this:
https://docs.microsoft.com/en-us/azure/machine-learning/studio/algorithm-cheat-sheet

So you are well on your way in identifying all kinds of techniques, but you may want to turn your plot 'sideways', e.g. two-way / multi-class classification are problem (sub)domains and result in different recommended machine learning techniques.


Activation method and Loss function for multilabel multiclass classification

Assuming all the labels have the same importance you can have a sigmoid for every class at the output. For each of the classes it will ask, is this class part of this sentence or not? The loss is just the sum of the individual log losses for the outputs. If some labels are more important you can scale them accordingly in your loss function.


Certain ingredients are needed to give you the best chance of a successful PhD. One of the important ones is that you and your supervisor have mutual interests.

A second important ingredient, in my opinion, is that you immerse yourself in that environment. It's important to develop a network of colleagues. It helps to spread ideas, start collaborations, get help when needed, and to explore unthought of opportunities.

From what you have said, I think you will be missing out on these two important ingredients if you continue in the same place or if you work remotely.

What is also important is what you to do after the PhD. PhD is required for academic position. But I think you will be in a weak position to get to the next step (fellowships, faculty positions, etc.) if you do what you proposed. In certain industrial positions it can be looked on favourably, not necessarily for the topic you pursued, but because it says something about your personally. Basically that you can get things gone, rise to a challenge, work independently, work as a team, communicate difficult topics and can bring creativity to solving a problems.

My advice would be to find a machine learning research group and apply for PhDs. If this is not possible why not consider following the topic of your supervisor and keep machine learning as a hobby? You will become and expert in statistic and so you will find manny concepts will translate between the various statistical disciplines. But only do this if you get along with her/him, and you can see yourself studying this topic.

Finally you could try a compromise? Are there applications for "mixing statistics" in machine learning? Can you find one? Is there an unexplored opportunity to do something new?

As a side note I find it ridiculous that PhD supervisor ask the student for topics. This always leads to problems because the student doesn't really have a clue about the research field. There is room for flexibility but often this hides supervisor laziness.


Based on how much i got to know i propose the correct scheme would be third one i.e.

Take the first batch of data as first S (Number of time steps) (S1, S2, .., Ss) words from each article (for the sake of simplicity let us assume batch size = m)
Set the initial hidden state H0 = [0,0,..,0]
Calculate loss and gradient on this batch and update the parameters
We move s words forward to next non overlapping s words in each article and initialize H0 to Hs from last iteration
Do this to the end

Why we wont take any but Scheme 1 and Scheme 3 is because:
H0 should only be initialized with a vector of zero when there is no context available (if you have some additional information, why not use it?!). The objective is to maximize the probability of each article and the sentences are not independent. Also, there is no reason to not treat period "." as a step in itself. We do calculate its word embedding too

Why not Scheme 1 - Because of catastrophic interference ( As the network in Scheme 1 fed in same input again and again ) combined with slow learning it would result in


Even if input to a neural netwrk are scaled or normalised, the raw output values can still go outside of that range.

In your case, the output values are being interpreted as to make a binary YES/NO decision, but the raw values cannot necessarily be interpreted as raw probabilities! They are merely the final activations of the network.

To get what you expect, the final activations are usually passed through a softmax function, which essentially squashes the values you see in your table to sum to 1 on each row - this allows us to treat them as probabilities to make the final classification.

In practice, this means simply adding the softmax activation to your final Dense layer in Keras (activation="softmax") and then compile the model using:
 loss="categorical_crossentropy"