How does action get selected in a Policy Gradient Method?

You are on the right track. We no longer select an action that we think maximizes the score. Rather we predict what the best action to take is. This can be very effective in large or continuous state spaces where taking an argmax of the score over all possible actions would be prohibitively expensive.

Our policy network computes the probability of taking an action, and during training we sample actions from the network. This continues until we receive a reward signal, which we multiply by the predicted log probabilities to propagate the error back through the network over past actions leading up to that reward. This will tend to adjust the past actions' predicted probabilities down for more negative rewards, and up for more positive rewards.

Of course not all actions leading up to the reward necessary lead to that good or bad reward, but over many training iterations good actions will tend to lead to more positive reward and negative actions will lead to more negative reward.

For more discussions see Andre Karpathy's blog post Deep Reinforcement Learning: Pong from Pixels and RL Course by David Silver - Lecture 7: Policy Gradient Methods


Correlation with Multi-Dimensional Clustering

That is not clustering since the rows are not being grouped together.
It is a filtering problem where the threshold to keep a row is based on frequency in each column.


Using non-negative matrix factorization (NMF) for facial recognition

One major issue with this approach is that if you represent your image as a vector of pixel intensities, you have zero translation-invariance.  Two pictures that are exactly the same but shifted over 10 pixels will likely look like dissimilar vectors.  That is why people try to use other methods that handle translation, like convolutional neural networks or SIFT features.


Multiple Linear Regression with k-fold Cross Validation

As Peter said, you need to split your dataset into two subsets: Training, and Test sets. Generally, 80% of data is allocated for Training set (20% for the Test set).
Thereafter, depending on the language/package you use (caret in your case), you use 5- or 10-fold cross-validation to train your model, and finally, you check the prediction ability of the model using the Test set.

I quickly checked the Caret Package website and ripped the required code for you.

1- For Training-Test split:

4.1 Simple Splitting Based on the Outcome

library(caret)
set.seed(3456)
trainIndex <- createDataPartition(iris$Species, p = .8,
                                  list = FALSE,
                                  times = 1)

2- For training with 10-fold cross-validation:

5.3 Basic Parameter Tuning,
5.5.4 The trainControl Function

fitControl <- trainControl(## 10-fold CV
                       method = "repeatedcv",
                       number = 10,
                       ## repeated the CV ten times
                       repeats = 10)

Usually, we do the cross-validation only once (repeats = 1); but to check the consitency of the results, you may need more repeats.


How does sigmoid saturate with large weights?

The sigmoid function $$ \theta(z) = \frac{1}{1+e^{-z}}$$ looks like this :

where $$z=w_i a_i + bias$$
for activations $a_i$ from the previous layer, and weights $w_i$ of the current neuron.

When the weights $w_i$ are too large (positive or negative), $z$ tends to be large as well, driving the output of the sigmoid to the far left (value 0) or far right (value 1). These are saturation regions where the gradient/derivative is too small, slowing down learning.

Learning slows down when the gradient is small, because the weight upgrade of the network at each iteration is directly proportional to the gradient magnitude.


Isolation forest: how to deal with identical values?

No need to use all the features. That is the cool part of having an ensemble. If one tree who is using a small set of data with some set of features is unable to use the feature, some other tree will use it. If the point is a true outlier, the score that is calculated later will offer the due reflection.


kde plot for interpreting the correlation

CoreyLevinson on kaggle answered my question. I am simply adding the explaination.

The kde shows the density of the feature for each value of the target. There are usually 2 colored humps representing the 2 values of TARGET. If the humps are well-separated and non-overlapping, then there is a correlation with the TARGET. If the humps are overlapping a lot, then that means the feature is not well-correlated with the TARGET, because the TARGET is equally as common on those values of the feature.


How to consume single piece of text for classification in a model?

You have to repeat the same preprocessing steps for prediction as you did during training. Let's say you trained like this:

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=777)

vectorizer = TfidfVectorizer()
X_train = vectorizer.fit_transform(X_train)

mod = LinearSVC()
mod.fit(X_train, y_train)

Then you need to use that fit TfidfVectorizer to transform your test prediction as well (which you seem to have managed). Like this:

X_test = vectorizer.transform(X_test)
pr = mod.predict(X_test)

And the same goes if you just want to feed in a manual text:

X_manual = vectorizer.transform(['fresh fish'])
pr = mod.predict(X_manual)


Sns Heatmap save as png jpeg image with full labels visible via python

results_path = 'results.png'`
#print(results_path)`
plt.savefig(results_path)`


Using embedding layer output as input to .fit() call in Keras

I got it. You define a new model, which has an input, the shared embedding layer and a flattened output. Pass the output of .predict() from that model to y parameter of your main model's .fit() call, in like fashion:

NUMERIC_FEATURES = [
    # Define the subset of features that need passing to the numeric input layer
]

vocab_size = 10000 # number of items
n_dimensions = 32 # dimensions to embed down to

# Shared layer for Item ID embeddings.
itemEmbedding = Embedding(vocab_size, n_dimensions, name='Item-Embedding')

# "Main" model definition, which has ItemID feature plus a bunch of
# numeric features to input
ii = Input(shape=(1,), name='Item-Input')
ie = itemEmbedding(ii)
if = Flatten()(ie)

ni = Input(shape=(7,), name='Numeric-Inputs')

c = Concatenate()([ni, if])

d1 = Dense(512, activation='relu')(c)
d2 = Dense(256, activation='relu')(d1)
d3 = Dense(128, activation='relu')(d2)
o = Dense(n_dimensions, activation='relu')(d3)

model = Model(inputs=[ni, ii], output=o)
# Don't take my word for the loss here, this is a toy example of code that works but is not
# intended to be completely "correct"
model.compile(optimizer='adam', loss='mse')

# "Labels" model definition, which embeds the labels using the shared
# embedding layer
li = Input(shape=(1,), name='Label-Input')
le = itemEmbedding(li)
lf = Flatten(le)

labelModel = Model(inputs=li, output=lf)

# Train model, using output of labelModel.predict() as the y parameter
model.fit([df[NUMERIC_FEATURES], df.ItemID], labelModel.predict(df.ItemID))

That's how it's done, whether it's useful, I have not yet evaluated.


Performance of model in production varying greatly from train-test data

This happened to me in the past.

The first question I would ask myself will be, how am I splitting train, validation and test sets? The second one would be, are my production data in the same domain as of those in train/validation/test?

Sometimes a simple split of data might still result in testing data leaking into the validation and even test data, this is the case of domains where we want to study the behaviour of users, normally you will not want the same users in train/test/validation sets (that is, ideally a user should remain within the same data set).

Apart from that, looks like you might just be overfitting the data.


How does the forward method get called in this pyTorch conv net?

If you look at the Module implementation of pyTorch, you'll see that forward is a method called in the special method __call__ :

class Module(object):
   ...
   def __call__(self, *input, **kwargs):
      ...
      result = self.forward(*input, **kwargs)

As you construct a Net class by inheriting from the Module class and you override the default behavior of the __init__ constructor, you also need to explicitly call the parent's one with super(Net, self).__init__().


How to model segmentation of a sequence to similar parts?

The problem can be reframed as a binary classification by ignoring the order, and then a decision tree algorithm can learn to separate the groups with perfect accuracy.

import numpy as np
from sklearn.tree import DecisionTreeClassifier

# Define training data
training_data    = np.array([1, 2, 1, 1, 2, 1, 2, 5, 6, 5, 4, 5, 1, 1, 2, 1, 1, 2, 1]).reshape(-1, 1)
training_targets = np.array([1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1]).reshape(-1, 1)

# Train a descision tree with scikit-learn
clf = DecisionTreeClassifier(max_depth=1)
clf.fit(training_data, training_targets)

# Let's if the model can predict unseen data
test_data_low = np.array([0]).reshape(1, -1)
prediction = clf.predict(test_data_low)
assert prediction == 1

test_data_high = np.array([7]).reshape(1, -1)
prediction = clf.predict(test_data_high)
assert prediction == 2

The ability of a model to learn to generalize (i.e., predict total new values) can be increased by increasing regularization. In the above example, one method to increase regularization is limiting the depth of the decision tree. I have set the depth to be a single split, aka decision stump. A decision stump is the most regularized version of a decision tree.


The idea behind consecutive convolutional layers with no pooling is actually not to skip any pooling but to replace a single layer with a bigger receptive field. So think of it this way:

Two consecutive 3x3 layers actually lead to a receptive field of 5x5.
Three consecutive 3x3 layers (like in AlexNet) lead to an actual receptive field of 7x7.

Intuitively I like to think of it this way: For two 3x3 layers the actual receptive field gets extended by 1 in all directions as the information is "pulled in" from outside of the 3x3 field by applying two layers consecutively.

In the first layer the 3x3 receptive field is applied (here using A and B as examples):

Now, that means when the second 3x3 layer is being applied cell B contains information from beyond the dark grey area. And this information in B will be considered for A when you apply the second layer. So basically the receptive field of A has grown as it includes information from beyond its actual 3x3 field:

And since this happens in all directions as shown in the second picture you now end of with a sort of 5x5 receptive field. Finally, if you apply this a third time (like AlexNet does with its 3 consecutive 3x3 layers) then the receptive field will be extended a second time by 1 in all directions, i.e. you get a 7x7 receptive field.

Question is why would you actually do this and not just use a single layer with a larger receptive field?

The paper Very Deep Convolutional Networks for Large-Scale Image Recognition
 gives two reasons:

  First, we incorporate three non-linear
  rectification layers instead of a single one, which makes the decision function more discriminative.

  [...]

  Second, we decrease the number of parameters: assuming that both the input and the output of a three-layer 3 × 3 convolution stack has C channels, the stack is parametrised by $3(3^2C^2)=27C^2$ weights; at the same time, a single 7 × 7 conv. layer would require $7^2 C^2=49C^2$ parameters, i.e. 81% more. This can be seen as imposing a regularisation on the 7 × 7 conv. filters, forcing them to have a decomposition through the 3 × 3 filters (with non-linearity injected in between).


Criteria for saving best model during training neural network?

The loss is mostly a measure that helps the model learn and is not looked at too much when deciding which model to select. A more business oriented measure is often used for this, e.g. accuracy. Since in this case you are mostly interested in the dice score it would make most sense to select the model from epoch 8.


Interpreting the results based on Granger Causality test

(0.0053726552728412666, 0.9415686658133314, 1), that 0.005 represents the test statistic Yes,
0.94 represents the P-value - Yes.
and 1 the degrees of freedom? No.
And 1,2,3 and 4 are ordinary serial numbers and not, degree of freedom.
At 10% level of confidence, you need at least 2.71(value of Chi-square). You have computed the chi-square = 2.576 and therefore, the null-hypothesis is acceptable. Your assertion  - the null hypothesis can absolutely not be rejected- is valid in this particular case.


Is it ok to use photo collages as dataset instead of single images for training the object detection ssd model?

1.You can use Progressive Resizing in your dataset in order to improve your results.
Link here
This method can be helpful when the images only depict parts of the sample.
2.Also, in order for your model to be good at predicting smaller leaves that do not take the entire images, you should make your validation set balanced ie. a number of smaller leave images and leaves that take the whole image should be balanced in the validation set.
3.I don't know much about the collage-based approach, but you can check this
Hope it helps. Please upvote if it helped.


How can one use unstructured data for forecasting?

How you use data depends mostly on the domain problem you're trying to address.

In the case you mention, it would depend on what exactly you're trying to forecast. For example, if you're trying to forecast sales or renewalls, there are some immediate use cases where you can leverage "unstructured" data to increase the number and/or the quality of the signals (features) that you are feeding into your model: 1) process audio data from contact centre interactions to determine the overall sentiment of a customer (i.e. "is he up for a sales call?"); 2) process text from customer reviews and use sentiment/keywords as a preditor for a churn model; etc!

  it seems that they always end up reducing the unstructured data to
  some structured format first, then feed it to whatever forecasting
  method they are using.

Probably because their forecasting algorithm or method requires that data is inputted in a specific format, like most forecasting or classification algorithms do; different algorithms will have different tolerances to data issues like missing values or outliers. So you process a piece of data that is unstructured with the goal of using it, usually by adding it to a data model or inputting it as data for a report or as a feature for a predictive model.

PS: I'm curious if anyone knows a case where you just throw unstructured data at a predictive algorithm (e.g. a neural network) and get something meaningful or useful out of it.


Unsupervised Hierarchical Agglomerative Clustering

I think I've figured out how to implement the algorithm described in the paper I'm studying. I suspect they used scipy.cluster.hierarchy.
Anyway, my process is:

Generate a distance matrix y from my list of examples x.
Compute the linkage using scipy.cluster.hierarchy.linkage
Generate flat clusters using scipy.cluster.hierarchy.fcluster

The last step is where the threshold mentioned is applied. I still have a question around how to use fcluster to generate clusters based on heterogeneity
What I've found confusing is there are a lot of tutorials on how to determine the number of clusters for sklearn.cluster.AgglomerativeClustering which use scipy.cluster.hierarchy.linkage then scipy.cluster.hierarchy.dendrogram to plot a dendrogram and which is then used to visually identify how many clusters are required.


Caret and rpart - does caret automatically prune rpart trees

To give a proper background for rpart package and  rpart method with caret package:

1. If you use the rpart package directly, it will construct the complete tree by default.
If you want to prune the tree, you need to provide the optional parameter rpart.control which controls the fit of the tree. R documentation below, eg.:

    rpart(formula, data, method, control = prune.control)

    prune.control = rpart.control(minsplit = 20,
                    minbucket =  round(minsplit/3), cp = 0.01,
                    maxcompete = 4, maxsurrogate = 5, usesurrogate = 2,
                    xval = 10, surrogatestyle = 0, maxdepth = 30 )

these are the hyper parameters you can tune to obtain a pruned tree.

One followed way is to not provide the cp i.e complexity parameter and perform cross validation (xval), something like:

    rpart.control(minsplit = 20, minbucket = round(minsplit/3), xval = 10)

complexity parameter (cp) can be thought of as a measure of complexity/ no of splits of your model and you want to increase complexity until your model generalizes to new observations. i.e. regularization

Therefore evaluate the cross validated error vs cp and choose the cp that gives the good value (cp_good).

Finally, add it as your control parameter i.e. rpart.control(cp = cp_good) or use the prune function i.e. prune(fit, cp = cp_good) to get the desired tree.

2. caret package on the other hand already implements the rpart method with cp as the tuning parameter. caret by default will prune your tree based on a default run it makes on a default parameter grid (even if you don't supply any tuneGrid and trControl while training your model:

    model <- train(data,
                   labels,
                   method = "rpart")


How to compute f1_score for multiclass multilabel classification

There seems to be a confusion between multiclass and multilabel classification:

Multiclass is the regular case where the task consists in predicting among N possible classes. For example an image can be either a dog or a horse or a cat, but always exactly one among these three animals.
Multilabel is the when the task consists in predicting a set. For example an image can be any subset of {dog, horse, cat}: it could be {dog, cat}, it could be {horse}, it could be {dog, horse, cat}, it could even be the empty set (no animal at all).

Practically in the multilabel case you predict each possible animal independently as a binary problem, so for every image the system answers 3 questions:

does this image contain a dog? (y/n)
does this image contain a horse? (y/n)
does this image contain a cat? (y/n)

Because each question is predicted independently, it doesn't make sense to pick the class which has the max probability. In effect there are 3 independent binary classification problems and 3 corresponding confusion matrices.
Apparently you didn't mean to take into account the multilabel case and you don't have any image labelled with multiple animals, right? If so you should change the system to solve a regular multiclass problem.
It looks like the confusion might be due to the one-hot-encoding of the class: maybe you thought that the class being categorical, it would be a mistake to encode the class as a numerical value. This is true for categorical features but actually not for the target, you can perfectly use for instance LabelEncoder to represent the class as a single target. This is much simpler and probably more appropriate for your problem.
One difference you will notice is that the probabilities obtained as predictions sum to one, because the classifier doesn't consider the classes as independent (as opposed to what happens in your current experiment: the probs don't sum to 1).


Why is this Binning by Median code wrong?

The error is due to the well-known R gotcha that the : (colon operator, which calls seq()) takes higher precedence than arithmetic. Always parenthesize arguments to : if they involve arithmetic or are expressions: a[(i-3):i]

Your code a[i-3:i] doesn't do what you want it to do a[(i-3):i], it does a[i - (3:i)]). So the medians you are printing are for these slices:

4-3:1 # i.e. 1:3
8-3:1 # i.e. 5:7
12-3:1 # i.e. 9:11
16-4:1 # i.e. 13:15

PS some coding-style tips

You don't need to iterate over all possible values of i and check them modulo binsize, just do:

for(i in seq(binsize, length(a), by=binsize)) { ... }

so in your median case:

for(i in seq(binsize, length(a), by=binsize)) {
  print(rep.int(median(a[(i-3):i]),binsize))
}
[1] 36.25 36.25 36.25 36.25
[1] 48.9 48.9 48.9 48.9
[1] 65.25 65.25 65.25 65.25
[1] 79.25 79.25 79.25 79.25

But in fact you can replace even that with:

split(a, ceiling(seq_along(a)/binsize))

as per the "Split a vector into chunks in R"

To make it even clearer, you could define a helper function chunk <- function(x, binsize) { split(x, ceiling(seq_along(x)/binsize)) }
Then you can replace the for-loop with sapply:

.

sapply(split(a, ceiling(seq_along(a)/binsize)), mean)
sapply(chunk(a,binsize), mean)
     1      2      3      4
32.600 49.600 66.175 83.925

sapply(split(a, ceiling(seq_along(a)/binsize)), median)
sapply(chunk(a,binsize), median)
    1     2     3     4
36.25 48.90 65.25 79.25

Much cleaner, easier to read, and prevents errors, right?

The generalization of the colon operator is the seq() function, give it a read, it's pretty useful.


Python OneHotEncoder Using Many Dummy Variables or better practice?

If your categorical variables include variables that suggest some numerical values like ranks, you should consider just label encoding them. (for e.g. First, Second, Third, and so on can be encoded as 1, 2, 3 and so on).

Also, find out if all these categories are important. Plot some graphs(such as histograms, distribution plots) to visualize the dataset. Remove categories that don't seem necessary.(e.g. in the above example, if First occurs more than 80%, you should consider if that certain features really contributes to your model.)


Relationship between train and test error

First of all be very clear with the use of the Training set, Validation set and  Testing set. These play a crucial part in  tuning your DL model. Usually, a validation dataset is used for keeping a check over the model during the training. An intutive observations are noted with the training validation and testing accuracy during data fitting:

If the model is having high validation accuracy and low training accuracy, it is an underfit model.
If the model has higher training accuracy and low validation accuracy, it is overfit.

Bias-variance trade-off problem is a central problem of supervised machine learning algorithms.

The bias–variance decomposition is a way of analyzing a learning algorithm's expected generalization error with respect to a particular problem as a sum of three terms, the bias, variance, and a quantity called the irreducible error, resulting from noise in the problem itself.

Ref


How to make R or Python as fast as SAS for ODBC Oracle queries?

This answer some of your questions from a Python perspective:

  Is Python any faster?

This question is a bit tricky to answer, it will depend on your usage of Python, but Python is not a fast language per se. However, the pandas library in Python have been reported to handle tables of 33M-100M rows, see this. I myself have used to handle around 10M rows from a Postgres table. For a detailed experimentation using pandas, see this. In the link they apply some operations on datasets of 88M rows and 74 columns.

  Do we need Hadoop or parallel processing or something else to make R/Python as fast as SAS?

Before trying using Hadoop or Spark, I recommend you to follow some optimization tricks (tips):

[1] A Beginner’s Guide to Optimizing Pandas Code for Speed

[2] Using pandas with large data

This link (Don't use Hadoop - your data isn't that big) can be useful too.

Also for a comparison of SAS and pandas you can read the pandas documentation on such comparison or this.

  Is it because R and Python are matrix-based languages and SAS is more
  traditional-database-oriented?

Python is not a matrix-based language, as far as I know the language does not offer by default any capabilities for handling matrices. I think your referring to the numpy/scipy stack, this is a separated library.


Is it possible to find a model that minimises both false positive and false negative?

Yes and No! depending on what do you mean by minimization. When you say minimizing $f$ and $g$ according to something, you are actually looking for a point which minimizes both. It does not mean that this point necessarily finds the minimum of $f$ or $g$. So yes in this sense.

But in case you mean a point in which both of them are in their minimum, this case is not realistic in practice as usually increasing one of them means decreasing the other one. Have a look at the plot bellow. FRR means False Rejection Rate, FAR means False Acceptance Rate and EER means Equal Error Rate.This is the terminology used in Biometric, a ML-based field studying the person identification/recognition.

It illustrates what I explained above.

Note

You should consider the importance of each error. In some use-cases (including some biometric use-cases) you may sacrifice one error for the other one. In those cases you minimize the error you need to be minimum without really caring about the other one (of course trying to keep the other one as low as possible as well but not at top priority).

For example I have a Face Recognition system which opens the door of a highly confidential military site to permitted users. Should I try to keep the EER low? Sure not. If sometimes the door is not open for a permitted person it's ok (at most they complain a bit) but door MUST NOT be opened for someone who is not allowed so here you care about keeping FAR low.

Hope it helped


feature extraction from single word for classification into nouns and names

I suppose you could try to learn a mapping from characters (or character-level n-grams) to part-of-speech.  This would be analogous to document classification.  Instead of a document, you have a single word.  And instead of a sequence of words, you have a sequence of letters.

With this formulation, most of the tricks which extract features from text could also be applied to extract features from characters (although your mileage may vary).  At the simple end of the scale, you could try "old-school" techniques like bag-of-words and TF-IDF (except you would have a bag-of-characters, and it would be Character-Frequency Inverse-Word-Frequency).  On the complex end of the scale, you could try to learn embeddings of characters or character n-grams.

However, before you get started, there are a few things I think you should keep in mind:

Do you think there is enough information in sub-word features to classify parts of speech?  I think it's pretty unlikely (at least for English).  Your model might learn some of the easy cases (e.g. an "-ly" suffix often indicates that the word is an adverb), but I don't think it would perform well in general.
This is actually a multilabel classification problem, because there are many words which can serve as more than one part-of-speech.  Also, names (proper nouns) are a subclass of nouns.
What happens if you feed your model a non-existent word?  Should your model try to classify it as a part-of-speech, or do you want it to recognize non-existent words?
If you're new to machine learning, consider putting this aside for a little while.  Jumping headfirst into a difficult classification problem with deep learning is a hard, confusing way to learn.

Finally, are you doing this for experimental/fun reasons, or is this intended to be part of a production application?  If it's the former, then go for it!  Worst case: it doesn't work but you learn something.

But if this project is for anything more serious, then you should not train a ML model to solve this problem.  Your model will never outperform a dictionary :)


Financial Time Series data normalization

Z-score normalization, as you have already guessed, cannot deal well with non-stationary time series since the mean and standard deviation of the time series vary over time.

Min-max and another commonly used normalization in stationary data, the decimal scaling normalization depend on knowing the maximum values of a time series.

The most commonly used method for data normalization of non-stationary time series  is the sliding window approach (J. Lin and E. Keogh, 2004, Finding or not finding rules in time series). In short:

  The basic idea of this approach is that, instead of considering the
  complete time series for normalization, it divides the data
  into sliding windows of length ω, extracts statistical
  properties from it considering only a fraction of ω
  consecutive time series values (H. Li and S. Lee, 2009, Mining frequent itemsets over data streams using efficient window sliding techniques, Expert Syst. Appl., v. 36, n.
  2, p. 1466-1477. J.C. Hull, 2005), and normalizes each
  window considering only these statistical properties. The
  rationale behind this approach is that decisions are usually
  based on recent data.
  (...)
  The sliding window technique has the
  advantage of always normalizing data in the desired range.
  However, it has a drawback of assuming that the time series
  volatility is uniform, which is not true in many phenomena

Another more advanced and less used (so far) is Adaptive Normalization

  can be divided into three stages: (i)
  transforming the non-stationary time series into a stationary
  sequence, which creates a sequence of disjoint sliding
  windows (that do not overlap); (ii) outlier removal; (iii) data
  normalization itself.

Check the link on Adaptive Normalization and all its references, there is relevant information in there.


What NN architecture to predict fantasy character names based on description?

First off, I think that since the goal of your model will be to generate new names based on a description, your model should work at a character-level and not word-level.

You can think of the level at which your model is working as the building blocks you are providing for it (it needs to learn them during training). These building blocks are than used for generation of new constructs. So if you want to construct new words (names) than you need to teach the model to understand the connection between the individual characters and the input description. Your model can deal with the input at a word-level but its output needs to be at character-level.

You can read more about it at: Besides Word Embedding, why you need to know Character Embedding?


How to transform and cummulate data in python

I think the following lines should give the output you are looking for:

# Create pivot table
df = pd.pivot_table(df, values="value", index="country", columns="year")
# Calculate cumulative sum and forward fill NaN
df = df.cumsum(axis=1).fillna(method="ffill", axis=1)
# Reshape data back into long format
df = df.reset_index().melt("country")

I first create a pivot table with the countries in the rows and the years in the columns to get all the years that may be missing in the long format data. I then calculate the cumulative sum, after which I forward fill the values over the columns. To finally reshape the data back into the original long format I reset the index and melt the dataframe with the country as the id column. Given your input, this returns the following dataframe:

   country  year  value
0    JAPAN  1999   10.0
1    KOREA  1999    NaN
2      USA  1999    NaN
3    JAPAN  2000   10.0
4    KOREA  2000   15.0
5      USA  2000   20.0
6    JAPAN  2001   10.0
7    KOREA  2001   15.0
8      USA  2001   33.0
9    JAPAN  2002   40.0
10   KOREA  2002   15.0
11     USA  2002   33.0


Number of features vs. number of samples : if small sample size is sufficient, why take large number of samples?

Bounds on the needed amount of samples are very common in PAC learning.
When you define a concept class you can compute a minimal size set of sample that will enable learning.
However,

More samples will allow improving accuracy
More samples will enable learning more complex concepts, that might fit your data better.
As @Emre wrote, real life data sets usually are not clean as in PAC learning. The concept class is not given to you, the data has noise and a given distribution is not guaranteed.

Showing that a classifier can be learnt with a small amount of data is great.
It is a big advantage of the learner.
However, more data usually helps and if it helps more than expected for such a classifier, it is possible that the classifier requirements doesn't hold.


how to reconstruct image from feature space of convolutional neural network?

One choice is to train a neural network model to take these values and output original images.
Notice that usually some data is loss in this process so it might be impossible to reconstruct the image with perfection.
You could try inverting the functional form but:

CNNs usually use ReLu activation which is not bijective.
Pooling layers throws information away (it still could be fixed if we had redundant information)
The weight matrix might not be inversable


I got the answer by taking reverse of the above list and then using inner join of rdd to get all corresponding transitive pairs. After removing duplicates i.e. (b,a) and same edges (a,a) or (b,b) got the resulting rdd.

Thanks for the help.


What does the one function $\mathbf{1}_{i,y^{(t)}}$ exactly mean in backward propagation of RNN in the book "Deep learning" of Bengio

The authors or the publisher may have corrected the inconsistent notation. The current version online looks like below:

Here the $\mathbf{1}_{\text{condition}}$ is the indicator function, as you mentioned.

Regarding the two version of softmax, use the first one for calculating gradient. The second version is for improving numerical stability, and mathmatically it should yield the exact same value as the first version.


How to calculate NDCG in recommendation system

IMHO,
The fundamental definition of DCG is that it is a measure of ranking quality. This assumes that you have computed the utilities of each document/item and ranked them in a certain order.
With this definition in mind, if you have n-items with same utility (which is 0 in your case), computing NDCG to measure the ranking quality within this subset of items (since you are only looking at items 5, 4, 3, 2 and 1, all of which are not recommended), will yield you a NDCG score of 1 - since your ranking is perfect if you are only looking at these items.
NDCG is merely a way to quantify the quality of ordering, i.e., current order Vs perfect order (items sorted w.r.to their utilities). This is meaningless if you are looking ONLY at items with same utility score.
I hope this answers your question.


Classification vs Regression Model what should I choose?

The segmentation sounds rather arbitrary if it is simply a reflection of the revenue you are trying to predict rather than having been derived from other bjective data that then predicts revenue it is unlikely to add very much (though obviously past revenue is a good predict of future revenue - the question is does the sementation improve the prediction vs just using the past data.

So if it were me I would be inclined to do the following:

(1)  try some dimension reduction and/or clustering to explore whether or not you really have meaningful customer segments rather than just an arbitrary cutoffs by revenue.

(2) if you have a good segmentation try creating a customer classifer to predict segment memberhip

(3) if your classifier has some value use the (non-linear) segment prediction as a additional input to a regression to predict revenue.

(4) validate and/or test the model against some reserved data not just the main data set.

With any regression though correct model specification is vital. So the first thing you need to do is explore the dataset (plot lots of stuff against lots of other stuff, look at some distributions, etc) to try and understand its structure before you specify a model.

Particulary, if your primary data source is the time series of past sales you probably need to look at including mean-reversion and momentum terms in the specification (see ARM, ARIMA etc models ).

(5) It would also be fun to try a Recurrent Neural Network, though I have never done this myself.


ML Models: How to handle categorical feature with over 1000 unique values

If necessary, there are other methods of encoding categorical features:

Label encoding (might need some judgement regarding implied ordering)
Target encoding
Hashing trick

A handy python package is Category Encoders: link

I would suggest first investigating if your model needs categorical variables to be encoded - lightgbm and catboost are examples of ML algorithms which support categorical variables.


Need help on Deep learning model on sentiment analysis

Most of the code/references available out there use test_data during training. test_data wasn't part of my training.

While this is the way we should do it but stuff like Encoding must be done holistically.
 In your case, you have called the pre_process separately for Test and Train.
 So, the words are converted to Numbers independently. This should not happen.
tokenizer.texts_to_sequences(test)
Above Tokenizer should be the one that was fit on train data.
If I randomly print token with key 101 for train, test. This is the result

print(train_tokn.index_word[101])
print(test_tokn.index_word[101])

think
characters

I think you should use the train_tokn for the test data and it should improve. I believe a very simple LSTM can achieve 85% on this dataset
Or, manually embed both Train, Test using the GloVe embedding.
 A simple example for the issue
from keras.preprocessing.text import Tokenizer

train = ['I am sorry']
test = ['I am very sorry']
max_words = 10

# Train
tokenizer = Tokenizer(num_words=max_words)
tokenizer.fit_on_texts(train)
tokenizer.index_word # {1: 'i', 2: 'am', 3: 'sorry'}
# Test
tokenizer = Tokenizer(num_words=max_words)
tokenizer.fit_on_texts(test)
tokenizer.index_word  # {1: 'i', 2: 'am', 3: 'very', 4: 'sorry'}


Welcome to the Data Science forum.

Yes, data preprocessing is an important aspect of sentiment analysis for better results. What sort of preprocessing to be done largely depends on the quality of your data. You'll have to explore your corpus to understand the types of variables, their functions, permissible values, and so on. Some formats including html and xml contain tags and other data structures that provide more metadata.

At a high level the sentiment analysis (using bag of words) will involve 4 steps:

Step 1: Data Assembly
Step 2: Data Processing
Step 3: Data Exploration or Visualization
Step 4: Model Building & Validation (train & test)

Lets understand different possible data preprocessing activities:

Convert text to lowercase – This is to avoid distinguish between words simply on case.
Remove Number – Numbers may or may not be relevant to our analyses. Usually it does not carry any importance in sentiment analysis
Remove Punctuation – Punctuation can provide grammatical context which supports understanding. For bag of words based sentiment analysis punctuation does not add value.
Remove English stop words – Stop words are common words found in a language. Words like for, of, are etc are common stop words.
Remove Own stop words(if required) – Along with English stop words, we could instead or in addition remove our own stop words. The choice of own stop word might depend on the domain of discourse, and might not become apparent until we’ve done some analysis.
Strip white space – Eliminate extra white spaces.
Stemming – Transforms to root word. Stemming uses an algorithm that removes common word endings for English words, such as “es”, “ed” and “’s”. For example i.e., 1) “computer” & “computers” become “comput”
Lemmatisation – transform to dictionary base form i.e., “produce” & “produced” become “produce”
Sparse terms – We are often not interested in infrequent terms in our documents. Such “sparse” terms should be removed from the document term matrix.

To give you more insight onto the steps involved, here are some example sentiment analysis using logistic regressions codes https://github.com/srom/sentiment

https://github.com/jadianes/data-science-your-way/blob/master/04-sentiment-analysis/README.md

Hope this helps.


Similarly to BERT, they sample negative (i.e., non-adjacent) examples and train a classifier telling whether the sentence are consecutive or not.


How can I use Data Science to profoundly contribute to Humanity?

Since I have already answered a similar question on Data Science StackExchange site, plus some related ones, I will mention all of them here and let you decide, if you find them helpful:

on data science project ideas, including those with social impact;
on working as a data scientist for a non-profit company;
on using data science in political campaigns.


Difference between normalization and zero centering

what is the difference between these two steps?

In that specific Notebook that you linked, normalization means: shrink a numerical distribution in the [0,1] interval. It is commonly referred to as Min-Max Scaling. Shrinking the distribution in the [0,1] interval moves its mean somewhere between 0 and 1.

Zero-centering instead means: to "shift" the values of the distribution so that its mean is equal to 0.

--

  if one normalize the image pixel values to range [0, 1], is it have a
  benefit to also do zero centering?

Zero centering variables improves ML models performance. Activation functions tend to be very responsive to weights' changes especially around zero. I found some good explanations here.

--

  Is there a difference if I do normalization first or zero centering
  first?

The order matters, since "normalization" would move the mean away from zero.


Why is hierarchical clustering quadratic and k-means linear?

I think these time complexities are optimistic cases, but apart from that I think the reason is that in hierarchical clustering you consider the distances between many, if not all pairs of data points. The number of pairs scales quadratically with the number of points.

For k-means you somewhat cheat your way around considering all pairs by looking at the distances between each data point and the k means only. This scales linearly in both k and the number of data points.

So I think the speed up is caused by using the distance to the means as proxy for the distances between all points.


What is the best practice to test a ETL pipeline?

I have been writing and testing ETL pipelines for a few years now and there are typically two types of pipelines.

Code only pipelines written in python or whatever
GUI pipelines using tools like SSIS or Informatica

The first set you can test like any code, unit and integration tests and the second you can only really test using integration tests I.e. you deploy the code and run it in a test environment.

I expand on how to unit test code pipelines here: https://the.agilesql.club/2019/07/how-do-we-test-etl-pipelines-part-one-unit-tests/

Essentially you will want unit and integration tests and also monitoring which is a form of continuous testing in production :)


AttributeError: 'DataFrame' object has no attribute 'Values'

You tried to use .Values with a capital v instead of .values. Changing the capital v to a lowercase v should solve fix the error you're getting.


1st order Taylor Series derivative calculation for autoregressive model


CNN training data size for determing the winner of tic-tac-toe

I experimented with some other approaches as well and discovered that MLPs produce equivalent or better results for this problem. Thanks to the commenters Neil Slater and Student T for pointing me in the right direction.

For anyone interested, this is the model that eventually worked for me

boardSize = 3*3
featureSize = #I experimented with 500-1500 with varying results here
output = 3
x = tf.placeholder(tf.float32, [None, boardSize])
y_ = tf.placeholder(tf.float32, [None, featureSize])
learning_rate = 0.01
seed = 128
weights = {
    'hidden': tf.Variable(tf.random_normal([boardSize, featureSize], seed=seed)),
    'output': tf.Variable(tf.random_normal([featureSize, output], seed=seed))
}
biases = {
    'hidden': tf.Variable(tf.random_normal([featureSize], seed=seed)),
    'output': tf.Variable(tf.random_normal([output], seed=seed))
}
keep_prob = tf.placeholder(tf.float32)
hidden_layer = tf.add(tf.matmul(x, weights['hidden']), biases['hidden'])
hidden_layer = tf.nn.relu(hidden_layer)
hidden_layer2 = tf.add(tf.matmul(hidden_layer, tf.Variable(tf.random_normal([featureSize, featureSize], seed=seed))),
                tf.Variable(tf.random_normal([hidden_num_units], seed=seed)))
hidden_layer2 = tf.nn.relu(hidden_layer2)

y_conv = tf.matmul(hidden_layer2, weights['output']) + biases['output']
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_conv, labels=y_))
train_step = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


Columntransformer multiple columns with vector inputs

See the section 6.1.4 in the documentation.

As per the documentation, whenever the transformer expects a 1D array as input, the columns were specified as a string ("title"). For the transformers which expects 2D data, we need to specify the column as a list of strings (["title"]).


Can we use two independently measured features in a same ML model?

From the example I assume that an instance corresponds to a user, and you have both full sequences of the mouse and keyboard as features for predicting the user. I can think of two options for using these features in the same model:

With feature engineering, find a way to represent both sequences as a fixed array of features. For example you might have features such as average typing speed average mouse speed, number of mouse movements, number of times each key is pressed, etc.
Similar idea but in a more DL approach: find a way to represent both sequences as embeddings (there are methods for word embeddings, sentence embeddings, graph embeddings...)

In my opinion the main issue is the variable length of the sequences, not the fact that they are not aligned (the alignment would matter if the target variable was for one element in the sequence).


CIFAR-100: What is the difference between vehicles 1 and vehicles 2?

The difference between vehicles 1 and vehicles 2 is that vehicles 1 are more common and vehicles 2 are less common.

CIFAR-100 dataset super/coarse labels are not very rigorous. The CIFAR-100 official documentation points out flaws in their taxonomy:

  fruit and vegetables: apples, mushrooms, oranges, pears, sweet peppers
  large carnivores: bear, leopard, lion, tiger, wolf
  …
  Yes, I know mushrooms aren't really fruit or vegetables and bears aren't really carnivores.


By scaling X but not y, and having small weights at initialization, I think you are making it hard for the NN to bring the weights up to the scale they need to be.  Possibly if you let it train for more epochs it would find the right scale, but quick experimentation suggests it gets stuck into an oscillating loss that is quite large.
I had success in reaching a parabolic curve by either (1) scaling y as well, to land roughly in $[0,1]$, or (2) setting the initial weights to be larger, e.g. from keras import initializers; nn_initializer=initializers.RandomNormal(std=0.5) and using kernel_initializer=nn_initializer throughout.  (The default for std in RandomNormal is 0.05).
For the indicator function target, obviously scaling y isn't the issue.  But interestingly, scaling up the initial weights does work.  Maybe because you need to get those two large-slope bits at 40 and 60?
One other thing I'd suggest trying (I haven't here) is varying the learning rate, dependent on the scale of things.  Especially when the initial weights are far below what we know they'll need to end up as, maybe a large initial learning rate with some decay would help.


Can someone explain the Tensorflow code below?

gaussian is different from sigma, because sigma is trainable while gaussian is not. It implies that the value of sigma can be optimized but not of gaussian.
optimal is different from mse as

in the case of mse, the final loss is computed as MSE.
but in the case of optimal, we use gaussian as the loss function, with MSE acting as the variance of the gaussian kernel.


(Feature Selection) different results from L2-based and Tree-based

Negative coefficient in logistic regression means negative relationship between predictor variable and the response variable.

Example- in a model price can be a predictor and will have negative relationship with binary response variable product purchased or not.

And neagtive coefficient  in logistic regression does not mean relationship has low strength, it only means changes in predictor has a reverse effect on response variable and if coefficient is highly negative it means feature is very important and small changes in it impacts response but in reverse direction.

Feature importance does not tell you nature or direction of relationship but only tells you strength of relationship so they are never negative. Coefficients in logistic regression tells you both strength and direction or nature (positive or negative) of relationship. Also high importance in random forest means strong relationship between predictor and response but importance column derived from tree based models is silent on nature or direction.

Hope this helps.


How to improve the accuracy of a Doc2Vec model (Gensim) in case of a toy-sized data set?

20 is a very very small number, even performing the following might not help.
However, you can try changing the parameters such as the window size, and the number of iterations. However, increasing the number of iterations a lot can cause the model to over fit.
In my case, decreasing the window size helped.


Help with type of ML problem: when training data is spread across different subgroups/categories

I'll try to rephrase your question: How does one use the information contained in some categorical feature x to predict y in the presence of unseen category values in the test set?
Assuming the train set is representative of the test set distribution, you'd expect the large categories to also be present in the test set.
We're thus mostly concerned with the small categories which might be present in the train set but absent from the test set and/or present in the test set but absent from the train set.
One way of dealing with such situation is merging small categories (e.g. under 2% of all observations) into a single category. That way you treat any new category level as part of the merged category.
Below I share how I implement the above in python, in a way that can be combined in scikit learn pipelines:
from collections import Counter
import pandas as pd

class mergeSmallCategoryLevels():
    def __init__(self, min_frac):
        self.min_frac = min_frac

    def fit(self, X, y=None, **fit_params):
        category_counts = pd.DataFrame.from_dict(Counter(X), orient = "index", columns = ["count"])
        min_category_count = len(X)*self.min_frac
        large_categories = category_counts[category_counts["count"] >= min_category_count]
        self.large_categories = list(large_categories.index.values)
        return self

    def transform(self, X, **transform_params):
        ans = [val if val in self.large_categories else ".merged" for val in X]
        ans = pd.DataFrame({"category_feature":ans})
        return ans.to_numpy().reshape(-1, 1)


How to handle features with very broad range

I suggest to try a log transformation. This has two potential benefits:

The range of x values becomes smaller
Your transformed data might be closer to resemble a normal distribution (only relevant for some models, e.g. not for trees)

Here are two toy examples to illustrate:

Toy example 1

s = np.random.lognormal(3, 1, 1000)
plt.hist(s, 100)
plt.show()

plt.hist(np.log(s), 100)
plt.show()

As you can see from the second plot the range of x values has become smaller and the transformed distribution resembles a normal distribution. Of course this is a highly artificial example since the non-transformed distribution is log_normal.

Toy example 2

s = np.random.geometric(0.2,100000)
plt.hist(s, 100)
plt.show()

plt.hist(np.log(s), 100)
plt.show()

This one does not look as nice (i.e. normally distributed) anymore as example 1 but still your range of x values has become more compact.

In case your data includes x values of $0$ you can use np.log(x+1). If your data included negative values it would become trickier (approaches for this case include signed log as described here or simply adding a constant value, like the minimum, to your data).


when will the mlp give constant prediction?

This means that your features do  ot have predictive power to estimayevthe target. Hence the MLP has learnt to predict the mean of the target distribution so as to minimize the loss.
Try checking if the constant value is indeed the mean of the target variable.


As a simple measure, you could simply take the point biserial correlation between duration and rating for a given user.  If they aren't correlated (positively or negatively), this variable is likely not important to that user.

Another way to determine the impact of a variable is by training another model excluding that variable and determining how that affects the quality of the predictions.


where does the database created in spark go?

Where is this database created?

A powerful paradigm in modern data storage and processing is the separation of compute and storage. Building systems with decoupled compute and storage has benefits associated with scalability, availability, and cost.
Apache Spark loads and performs computation on the data - it is a distributed data processing engine. It does not handle permanent storage. In Databricks (you are using databricks documentation) data is often stored in Delta Lake, which is specifically designed to work with Spark. However, Spark can work with data stored in many other ways, such as other cloud storage (eg Amazon S3, Azure Blob), traditional SQL databases, NoSQL databases, HDFS and many more.


Very interesting question, it's always hard to prove a negative.
I have a vague idea but I really don't know if it's worth anything or even applicable to this problem, so please take it with a grain of salt!

The idea is to use randomness and multiple samples in order to compare the result of predicting from random noise vs. actual data X: if the result from X isn't significantly better than from random noise, then you have proved that X doesn't have any predictive power. Of course this relies on the assumption that the model used to predict is sensible enough.

I have seen methods using this idea but I don't remember the details unfortunately. The only directions I can give are these:

the impostor method for authorship verification, which is based on a somewhat similar idea
I've seen some works using the binomial test to formally test whether the outcome from a method X is significantly different from a random baseline.

Sorry for the lack of details, I hope this helps.


I suggest calculating R-squared and VIF for each permutation of variable combinations.

Also consider the fact that variable interactions may influence your R-squared and VIF values. Depending on what programming language you are using there are many ways to build interactions into your regression model, and you can iteratively test for effects from there.


You should use sklearn MultiLabelBinarizer

from sklearn.preprocessing import MultiLabelBinarizer
lb = MultiLabelBinarizer()

lb.fit_transform([['A', 'B', 'C'],[ 'A', 'D', 'E', 'B']])

array([[1, 1, 1, 0, 0],
$\hspace{1cm}$ [1, 1, 0, 1, 1]])

If required, remove the columns below a threshold value (sum of the column). This will reduce the Features count by removing the low variance Features


Overfitting with text classification using Transformers

Your model is overfitting. You should try standard methods people use to prevent overfitting:

Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) also sometimes help (0.1-0.3 might be reasonable values).
If you have many input classes, label smoothing can help.
You can try a smaller model dimension.

If you use a pre-trained Transformer (such as BERT), you, of course, cannot change the model dimension. In that case, you can try to set a much smaller learning rate for fine-tuning BERT than you use for training the actual classifier.


Gradient descent with vector-valued loss

I see clearly that this works for $l(w) \in \mathbb{R}$, but am wondering how it generalizes to vector-valued loss functions, i.e. $l(w) \in \mathbb{R}^n$ for $n > 1$.

Generally in neural network optimisers it does not*, because it is not possible to define what optimising a multi-value function means whilst keeping the values separate. If you have a multi-valued loss function, you will need to reduce it to a single value in order to optimise.

When a neural network has multiple outputs, then typically the loss function that is optimised is a (possibly weighted) sum of the individual loss functions calculated from each prediction/ground truth pair in the output vector.

If your loss function is naturally a vector, then you must choose some reduction of it to scalar value e.g. you can minimise the magnitude or maximise some dot-product of a vector, but you cannot "minimise a vector".

* There is a useful definition of multi-objective optimisation, which effectively finds multiple sets of parameters that cannot be improved upon (for a very specific definition of optimality called Pareto optimality). I do not think it is commonly used in neural network frameworks such as TensorFlow. Instead I suspect that passing a vector loss function into TensorFlow optimiser will cause it to optimise a simple sum of vector components.


One solution to this problem can be found in Grammar as a Foreign Language. The paper outlines a system that "translates" English sentences into syntactic constituency parse trees using a sequence-to-sequence LSTM (Long Short-Term Memory model). A TensorFlow implementation can be found here.


Generating synonyms or similar words from multiples word embeddings

An automatic system for finding synonyms using word embeddings is not possible. Word embeddings find co-occurrence. For example, "good" and "bad" co-occur together in a corpus, thus are near each other in the embedding space. However, "good" and "bad" are antonyms.

A copilot system could work. Word embeddings can find a set of candidate synonyms by finding the nearest words as measured by a distance metric. Then other methods (e.g., a person) can select amongst the candidates.


Logistic Regression implementation does not converge

This has been asked, and answered, on CrossValidated. See https://stats.stackexchange.com/questions/11109/how-to-deal-with-perfect-separation-in-logistic-regression and https://stats.stackexchange.com/questions/45803/logistic-regression-in-r-resulted-in-hauck-donner-phenomenon-now-what for two related answers (and there are other related questions that you can explore there). That logistic regression can blow up is a known effect in computational statistics.

Also, in situations where exp(-30) gives you roughly the relative accuracy of the double type, you would want to be extremely careful with accumulation of the round-off errors, as 1+exp(-30)=1: summing the likelihood contributions may run into numerical problems, especially when you start computing numerical derivatives and gradients. For a brief introduction into the issue in application to the typical problems in statistical computing and the specific problems it tends to encounter, see http://www.stata.com/meeting/nordic-and-baltic14/abstracts/materials/dk14_gould.pdf.


Name of ML discipline for trainable clusterer - supervised clustering, or group ID assignment

Perhaps you are looking for some combination of Meta clustering and Ensemble clustering?

Meta clustering

  A sampling-based approach that searches for distance
  metrics that yield the clusterings most useful to the user

...

  Generate many good, yet qualitatively different, baselevel
  clusterings of the same data.
  Measure the similarity between the base-level clusterings
  generated in the first step so that similar clusterings
  can be grouped together.
  Organize the base-level clusterings at a meta level (either
  by clustering or by low-dimension projection) and
  present them to users.

Ensemble Clustering

  In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.


how to prepare data for cross validation in mnist dataset?

from sklearn.model_selection import cross_val_score
clf = svm.SVC(kernel='linear', C=1)
scores = cross_val_score(clf, iris.data, iris.target, cv=5)
scores

They are not using the whole data for cross validation as such ( it's just an illusion)

When the cv argument is an integer, cross_val_score uses the KFold or StratifiedKFold strategies by default, the latter being used if the estimator derives from ClassifierMixin..

So it's kind of automated inside the call..

Check this kaggle kernel link


categorical_crossentropy should never be negative. The most common reason it becomes negative is incorrect data encoding. It is best if all features and targets are scaled between 0 and 1.


How do I download a Google Colab notebook?

You do not need to download Google Colab to send a file through email.
To send directly:

On Upper right-hand side there is a share button, Click on it.
A pop-up appears, Enter email addresses or usernames in the field 'Add People and groups'.
(Optional) Go to the setting gear button upper right corner and adjust permissions.
Click on 'Done'.

Download then send to email:

Click on the "File" menu in the upper left corner.
A popup appears, find 'Download .ipynb' and click on it.


How are Decision Trees averaged in Random Forest?

It depends on the type of variable that the Random Forest is predicting, and perhaps on the specific implementation of Random Forest.  Following is an overview of the simplest techniques.

Continuous Target

In the case of a continuous target variable $y$, each of the trees in an ensemble will generate a prediction $\hat{y}_i$.  The most naive way to combine the results from trees in an ensemble is to take the mean of all predictions.

Suppose you have an ensemble of 10 trees. Then the combined prediction $\hat{y}$ would be computed as
 $$\hat{y} = \frac{1}{10} \sum^{10}_{i=1} \hat{y}_i$$

This is the approach taken by forest regressors in scikit-learn.  Notice the lines

    # Parallel loop
    lock = threading.Lock()
    Parallel(n_jobs=n_jobs, verbose=self.verbose,
             **_joblib_parallel_args(require="sharedmem"))(
        delayed(_accumulate_prediction)(e.predict, X, [y_hat], lock)
        for e in self.estimators_)

    y_hat /= len(self.estimators_)

The "Parallel Loop" part sums up the predictions made by each tree in the forest, and the last line divides by the total number of trees so that y_hat is the mean of the predictions from each tree.

An alternative approach is to take a weighted average of the predictions
$$\hat{y} = \sum^{10}_{i=1} \omega_i \hat{y}_i$$

Where each tree has a weight $\omega_i$.  The weight could be determined, for example, by the tree's performance.  This approach might slightly improve accuracy in some cases, but it's easier to overfit.

Categorical Target

In the case where your random forest is predicting the value of a categorical variable, you can allow each tree to act like a member of a "committee" and cast a vote.

Suppose a categorical variable has three possible values, $A$, $B$, or $C$, and your random forest has 10 trees.  You generate a prediction with each tree, and that prediction counts as one vote.

For example, suppose 5 of your trees predict $A$, 3 of them predict $B$, and 2 of them predict $C$.  The combined prediction will be $A$.

Notice that you may run into cases where there is a tie.  Suppose 5 of your trees predict $A$, and the other five predict $B$.  Which prediction do you go with?

One option is to predict the majority class in the case of a tie.  In the example above, if the training data contains more $B$ examples than $A$ examples, then we would predict $B$.

Another option for resolving ties is to weight votes by the accuracy of the tree.  Suppose again that you have five trees that vote for $A$ and five trees that vote for $B$.  If you observe that, on average, the trees voting for $B$ are more reliable, you may give their votes extra weight and predict $B$.

Hopefully that helps!


How to calculate the mean of a dataframe column and find the top 10%

This is the import you need, and how to get the mean for a column named "RBIs":

import org.apache.spark.sql.functions._
df.select(avg($"RBIs")).show()

For the standard deviation, see
scala - Calculate the standard deviation of grouped data in a Spark DataFrame - Stack Overflow

For grouping by percentiles, I suggest defining a new column via a user-defined function (UDF), and using groupBy on that column.  See

Spark SQL and DataFrames - Spark 1.5.1 Documentation - udf registration


First, some caveats

I'm not sure why you can't use your preferred programming (sub-)paradigm*, Inductive Logic Programming (ILP), or what it is that you're trying to classify. Giving more detail would probably lead to a much better answer; especially as it's a little unusual to approach selection of classification algorithms on the basis of the programming paradigm with which they're associated. If your real world example is confidential, then simply make up a fictional-but-analogous example.

Big Data Classification without ILP

Having said that, after ruling out ILP we have 4 other logic programming paradigms in our consideration set:

Abductive
Answer Set
Constraint
Functional

in addition to the dozens of paradigms and sub-paradigms outside of logic programming.

Within Functional Logic Programming for instance,  there exists extensions of ILP called Inductive Functional Logic Programming, which is based on inversion narrowing (i.e. inversion of the narrowing mechanism). This approach overcomes several limitations of ILP and (according to some scholars, at least) is as suitable for application in terms of representation and has the benefit of allowing problems to be expressed in a more natural way.

Without knowing more about the specifics of your database and the barriers you face to using ILP, I can't know if this solves your problem or suffers from the same problems. As such, I'll throw out a completely different approach as well.

ILP is contrasted with "classical" or "propositional" approaches to data mining. Those approaches include the meat and bones of Machine Learning like decision trees, neural networks, regression, bagging and other statistical methods. Rather than give up on these approaches due to the size of your data, you can join the ranks of many Data Scientists, Big Data engineers and statisticians who utilize High Performance Computing (HPC) to employ these methods on with massive data sets (there are also sampling and other statistical techniques you may choose to utilize to reduce the computational resources and time required to analyze the Big Data in your relational database).

HPC includes things like utilizing multiple CPU cores, scaling up your analysis with elastic use of servers with high memory and large numbers of fast CPU cores, using high-performance data warehouse appliances, employing clusters or other forms of parallel computing, etc. I'm not sure what language or statistical suite you're analyzing your data with, but as an example this CRAN Task View lists many HPC resources for the R language which would allow you to scale up a propositional algorithm.


Part of the problem with answering this question is there are actually two questions.  The first:

Are there any image classification algorithms which are not neural networks?

Yes, lots. But now the actually question:

Is there any paper which tries an approach which does not use neural networks in LSVRC?

In your question, you rule out methods such as kNN and SVM because of speed. Bag of Words is one method used to solve this problem.  MATLAB has a good demonstration (http://www.mathworks.com/help/vision/examples/image-category-classification-using-bag-of-features.html).  But BoW incorporates k-means clustering, so that may not fit your needs.

There are some other interesting image classification methods such as texture analysis. TA is being researched as a way to classify malevolence of disease in medical images (such as tumors).  Here is a commonly referenced paper:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2701316/

Here's an overview of image classification:
http://www.tandfonline.com/doi/full/10.1080/01431160600746456


Are you familiar with multi-view geometry? There's a classic book on it: Multiple View Geometry in Computer Vision by Hartley and Zisserman. A good starting point is the wiki article on 3D reconstruction from multiple images.

Essentially, yes, the idea is to solve an optimization problem. Let the object be a point set $P$ in 3D. In the general case, we have $N_C$ cameras $\{C_i\}_{i=1}^{N_C}$, such that
$$
q_i = \mathcal{I}_k\mathcal{E}_k p_i
$$
$$ (x_i,y_i)=(q_{i1}/q_{i3},q_{i2}/q_{i3}) $$
where $p_i\in P$ is the $i$th point in homogeneous coordinates, $\mathcal{I}$ is the intrinsic camera parameters matrix, $\mathcal{E}$ is the extrinsic camera parameters matrix, and $q_i$ is the $i$th projected point with image coordinates $(x_i,y_i)$.
(Check out the camera matrix and camera calibration for more.)

To solve this problem depends on how much information you have (especially regarding the camera parameters) and what assumptions you make (e.g. perspective vs orthographic projection equations). Note there can be some inevitable ambiguity, depending on the number of cameras/images and their view-points, as well as the number/location of corresponding points. If you have some prior knowledge of the 3D structure itself, that can also be used.

In the two-view case, the
 (grandly named) fundamental matrix and the essential matrix, allow estimation of the 3D point positions (up to translational scaling) based on the epipolar constraints. With more views, one can for example do triangulation via direct linear transformation estimation.

A nice place to start might be the PhD thesis of Daniel Martinec on Robust Multiview Reconstruction.

Two libraries of potential interest are OpenCV and OpenMVG.

Now, entirely separate from this view as a classical computer vision problem is the machine learning approach. The idea is to complement (or sometimes replace) the geometric relations from multiview geometry (which can be brittle due to noise, measurement error, etc...) with data-driven estimations. Some starting points might be:

SurfaceNet: An End-to-end 3D Neural Network for Multiview Stereopsis
DeepMVS: Learning Multi-view Stereopsis
MVSNet: Depth Inference for Unstructured Multi-view Stereo
Learning a Multi-View Stereo Machine
Learned Multi-Patch Similarity

Such approaches may be more in line, I suppose, with the average users of a "Data Science" site :)

Note that these approaches tend to assume that no point-wise correspondence is known. This makes the problem significantly harder. In your case, however, it seems this is not the case, though you haven't given an explicit formalization. Just keep this mind when looking at these various methods, as an easier approach may be preferable.


Constant validation loss & accuracy, training accuracy fluctuates

Did the loss decrease up to a certain point and then start fluctuating between .02 and .09? If so you may have just reached your convergence and now it's oscillating due to the size of the learning rate being too large in relation to the size of the minima you have hit. Reduce your learning rate and see where it ends up, my guess is that you will get close to .02 this time around, unless you have truly hit a local minima, in which case you may go lower.


Output data from scikit learn logistic regression

Each line is treated independently at prediction, so you can be sure that the data is kept in the same order.

For simplicity, you can keep your data in a pandas dataframe. Here's a short working example:

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=100, n_features=10)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

clf = LogisticRegression()
clf.fit(X_train, y_train)

data_test = pd.DataFrame(data=X_test, columns=['f{}'.format(i) for i in range(1, 11)])
data_test['y_test'] = y_test
data_test['y_pred'] = clf.predict(X_test)


percentage of confidance on desion trees results

Some classification algorithms can indeed return a probability distribution over the considered classes (see Wikipedia on probabilistic classification).

In the topic of your question you're asking about Decision Trees. Well, these have their limitations in terms of providing probability estimates (see this paper on probability estimates from decision trees).

In case you would like to play around with this, it's very easy to start with scikit-learn:

import numpy as np
from sklearn.tree import DecisionTreeClassifier

dtc = DecisionTreeClassifier()

# training samples
X = np.array([[1, 1, 0],
              [1, 1, 0],
              [0, 0, 1],
              [0, 0, 1]])

# target values for training samples
y = np.array([0, 0, 0, 1])

dtc.fit(X, y)

print 'Class probabilities for training samples:'
print dtc.predict_proba(X)
print 'Probabilities for previously unseen samples:'
for sample in ((1, 0, 1), (0, 0, 1), (1, 1, 1), (0, 0, 0)):
     print 'Sample {}. Result: {}'.format(sample, dtc.predict_proba(sample))

This code returns following results:

Class probabilities for training samples:
[[ 1.   0. ]
 [ 1.   0. ]
 [ 0.5  0.5]
 [ 0.5  0.5]]
Probabilities for previously unseen samples:
Sample (1, 0, 1). Result: [[ 0.5  0.5]]
Sample (0, 0, 1). Result: [[ 0.5  0.5]]
Sample (1, 1, 1). Result: [[ 0.5  0.5]]
Sample (0, 0, 0). Result: [[ 1.  0.]]

At this scale the results are easily interpretable:

a sample of features (1, 1, 0) is classified as 0 with 100% probability
a sample of features (0, 0, 1) is classified 50/50 as 0 or 1. etc.

This brings us to an important question of how accurate is your model?


When I include validation_data=(x_val, y_val) in model.fit_generator, should I create another test dataset for accuracy measures?

You are correct in saying that it would be unfair - and if avoidable, you shouldn't do it.

In order to truly be able to claim (in a statistical sense) that a model achieves e.g. 90% accuracy, the test must be performed on unseen data. That is where your test data should be used.

Training a neural network requires the validation data (as you mentioned, within the fit_generator method of a Keras model), in order to compute errors and steer the weights in the right direction. The final accuracy you report needs to be on data to which the the training pipeline has never been exposed.

Creating a training/validation/test split is advised where possible; however, it can sometimes be a challenge, due to things like lack of data and imbalanced datasets. You can try things such as cross-validation - here is an example using Keras. Her is another example, in a similar question.


How to use multiple encoders(one-hot and numerical) together for PCA

There is a method that can handle multiple types of data simulataneously called Generalized Low Rank Models - actually one paper that deals with it is called PCA on a Data Frame. GLRMs in Python (and not only Python) are implemented in H2O.

Other than that you could try encoding your categorical data as numeric. There are multiple approaches to this. One example is mean encoding - see this answer for details. For implementation see Category Encoders. BTW if your task is totally unsupervised (you don't have any target) you can choose any continuous feature you have for mean encoding (so you can produce many columns from each categorical column).


Calculating possible number of configuration

The total number is:

$$5 \times 5 \times 6 \times 4 \times 4 \times 8$$

which is equal to $19200$. Here, we just count the number of possible values for each parameter.


Can somebody explain me this method? CNN Keras - starter

def ReadImages(Path):  #path to dir with all images
    LabelList = list() #Initialized empty labels list
    ImageCV = list()   #Initialized empty ImageCV list initialized
    classes = ["nonPdr", "pdr"]  #classes labels all image files startwith
    #return list of image folders in main dir path/
    FolderList = [f for f in os.listdir(Path) if not f.startswith('.')]

    #loop over dir of image folders
    for File in FolderList:
        #for loop returns index of each image in a image folder
        #and the image name
        for index, Image in enumerate(os.listdir(os.path.join(Path, File))):
            #Read image file resize it and append it to ImageCv list
            ImageCV.append(cv2.resize(cv2.imread(os.path.join(Path, File) + os.path.sep + Image), (224,224)))
            #return string of characters before dot 1 in file name
            #return index from classes !!!dangerous line assumes all files
            #of form nonPdr.* or pdr.*
            #append that index value either 1 or 0 to libelist
            LabelList.append(classes.index(os.path.splitext(File)[0]))
            print(FolderList) # print the folder just parsed
    return ImageCV, LabelList # return the two list created at the start
    #now containing images and labels based on names of image files


Deciding about dimensionality reduction, classification and clustering?

Should I use Self Organising Map to reduce the dimensionality?

Not the first choice. For dimensionality reduction you have more handy and basic methods like PCA and NMF (if you don't have negative values) or Archetypal Analysis which are recommended to be used first.

and then K-means to classify them into normal and abnormal?

Red alarm! Kmeans is not for classification but clustering. Be careful about the fundamental understanding of what you are about to do otherwise you get confused. If you have labels K-NN for instance is a supervised (thus proper) method.

Or I can use just K-NN to classify them into normal - abnormal without any dimensionality reduction?

Impossible. As you already know you have the problem called Curse of Dimensionality where the euclidean distances are highly distorted. You need to reduce the dimenstionality before K-NN. The only class of methods I know which may not need dimentionality reduction are Kernel Methods e.g. SVM.

How many features can I use with K-NN?

Once I examined the Curse of Dimentionality with K-Means in MATLAB and I saw after 7 dimensions everything was distorted. It of course depends on the nature of the data (distribution,etc) but this is what I got from my experiment.

And if later I'd like to find why this person has included into the abnormal class how can I find that this happened because of these two features, his weight according to his height?

This question is so fuzzy and about the goal of Machine Learning. In Machine Learning you get a set of answers (called labels) and try to learn them. You are not supposed to say if they are right or not. So if you are said a specific guy is abnormal you should ask the one who provided labels (in the terminology we call him expert) about the reason. If a guy with a certain weight and hight is called abnormal you only know this weight and hight indicate abnormal people. Why? Well, who knows?

If I did not get the last question correctly please drop me a line in comments. Would be glad to help.

My Suggestion

I recommend you to apply PCA to your data and take the first 5 PCs and feed them to K-NN or LDA. You can plot some of these PCs against each other and see how they discriminate your classes. It gives you a good impression of what is happening. You can also plot the variance or entropy of all features beforehand and drop those features whose variance or entropy is lower than a threshold.

Good Luck :)


Correcting log-bias in the output of an XGB

From this perspective, modern ML regression techniques are not different to classic ones. A mean unbiased model on the log scale will automatically be mean biased after retransformation (by Jensen's inequality).

Depending on the aim of your analysis, applying some retransformation bias correction factor might be useful as well for algorithms like XGBoost. (Be careful with models that impose L1/L2 penalties on the intercept though - a rare occasion.)

The specific correction factor can be calculated directly from the residuals in the same way as for OLS.


Is it bad to have a large gap between training loss and validation loss?

A model is said to be overfitting if there exist another model or hypothesis that has a higher error on the training set but smaller error overall. A model that overfits fails to generalize on out of sample examples.

In your case, having a much higher error loss on test set could not be a good sign. If you were to chose between the two models I think you should go for the second one, because the first one will probably fail to generalize.

Make sure your training and testing data are picked randomly and represent as accurately as possible the same distribution and the real distribution.

You could do more things other than regularization if want to reduce overfitting, you could try cross validation, or may be reducing the complexity of your model if possible (it depends on the model you training and the data you have)


Using Decision Tree methodology to identify Independent Variables for Multiple Regression

Decision trees are useful for determining nested/interactive relationships between combinations of IVs and a DV.

The model you specified, a multiple regression, presupposes a relationship between the IVs and the DV (e.g. linear).

As you are aware, these models are different. So using a decision tree coupled with some importance measure to find predictive variables won't necessarily provide you with an optimal set of IVs in a regression model.

That being said, it can be a helpful exercise to inform you of non-linear relationships or interaction terms that could be predictive, and which may not be captured by specifying a model such as a multiple regression.

If I were you, I wouldn't solely rely on using decision trees to determine a  set of IVs for a regression model. I would investigate penalized regression methods such as LASSO or ridge regression to help take you from a reduced candidate set of IVs to your final IVs. In addition, you might want to explore associative metrics related to your model specification that might be useful in exploring the relationships in your data, such as information values, chi-square tests, correlations, etc.

This may be helpful:
https://stats.stackexchange.com/questions/47367/decision-tree-as-variable-selection-for-logistic-regression


Why is Regularization after PCA or Factor Analysis a bad idea?

In principle, PCA is unsupervised and therefore label agnostic. That means the down projections forced into the PCs may as well not be related to what the model is trying to predict. That may be able to measure with the amount of variance your PCs are capturing.
In essence, PCA shall never be used as a means for regularisation but rather for dimensionality reduction. You could alternatively try a VIF approach, although not sure what your exact goal is.


After normalization, cast the numbers to float16 or bfloat16. Those are the smallest floats available in Keras.


How to retrain Glove Vectors on top of my own data?

What you should do is:

Create a new instance of a GloVe model with the old_words and new_words as vocabulary.
Replace the initial vectors/biases of the old_words with the ones you have already.
Train this model on mydata.txt.

The new old_words representations won't be the same but will be highly influenced by the old ones.


Is it necessary to convert labels in string to integer for scikit_learn and xgboost?

TL;DR
If your labels are part of your feature matrix, you need to convert them to numerals using OneHotEncoding (docs). However, if your labels are just your targets, you can leave them as is.
1) Is it necessary to convert the strings to integers?
You did not specify if your string labels are one of your features in your feature matrix X, or if they are your target y. So, here is some data to cover both cases:
X = [
    ['sunny'],
    ['rainy'],
    ['rainy'],
    ['cloudy'],
    ['very rainy'],
    ['sunny'],
    ['partially cloudy']
]

y = [
    'no umbrella',
    'umbrella',
    'umbrella',
    'no umbrella',
    'umbrella',
    'no umbrella',
    'no umbrella'
]

If you try to fit a DecisionTreeClassifier with this, like this:
from sklearn.tree import DecisionTreeClassifier
clf = DecisionTreeClassifier()
clf.fit(X, y)

You get an error:

ValueError: could not convert string to float: 'sunny'

So definitely, you need to convert them to integers, for example:
from sklearn.preprocessing import OrdinalEncoder
encoder = OrdinalEncoder()
encoder.fit(X)
X_encoded = encoder.transform(X)

>> [[3.]   # sunny
>>  [2.]   # rainy
>>  [2.]   # rainy
>>  [0.]   # cloudy
>>  [4.]   # very rainy
>>  [3.]   # sunny
>>  [1.]]  # partially cloudy

This transforms your labels into integers. And now you are able to .fit() your model.
This means that features in X must be transformed to integers, however, target labels in y can remain as strings.
2) Will the algorithms work less effectively?
If you don't convert your targets y into integers, there will be no decrease in your algorithms performance.
Now, given that you need to convert your string features X into numerals, the way that you convert will affect the algorithm.
In Sklearn you can use the OrdinalEncoder (docs) or the OneHotEncoder (docs). The way in which they encode X is different, observe:
OrdinalEncoder
from sklearn.preprocessing import OrdinalEncoder
encoder = OrdinalEncoder()
encoder.fit(X)
X_encoded = encoder.transform(X)

>> [[3.]   # sunny
>>  [2.]   # rainy
>>  [2.]   # rainy
>>  [0.]   # cloudy
>>  [4.]   # very rainy
>>  [3.]   # sunny
>>  [1.]]  # partially cloudy

OneHotEncoder
from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder()
encoder.fit(X)
X_encoded = encoder.transform(X)

[[0. 0. 0. 1. 0.]  # sunny
 [0. 0. 1. 0. 0.]  # rainy
 [0. 0. 1. 0. 0.]  # rainy
 [1. 0. 0. 0. 0.]  # cloudy
 [0. 0. 0. 0. 1.]  # very rainy
 [0. 0. 0. 1. 0.]  # sunny
 [0. 1. 0. 0. 0.]] # partially cloudy

For algorithms like DecisionTreeClassifiers, the second option, namely OneHotEncoder is better because there are more dimensions to finding boundary lines. So you can have a shallower trees. Where as, if you encode with a simple LabelEncoder, you will need to have deeper tree.
Have a look at the tree generated given the two types of input:
LabelEncoder, notice how 3 decisions nodes are needed.

OneHotEncoder, notice how only 2 decisions nodes are needed.


Theoretical Question: Data.table vs Data.frame with Big Data

The tidyverse also encompasses readr with faster functions for reading text files, e.g. read_csv (https://cran.r-project.org/web/packages/readr/README.html).

You are confounding two issues here, speed of operations of data.table vs. data.frame, and reading in such data.

I would venture a guess that read.csv and read.table is slow, because it does a lot of inefficient guessing and (by default) tries to convert strings to factors. When dealing with large data sets, you should always prefer to tell the routine what it is reading, rather than letting it guess.

Lastly, if you are dealing with very large data sets, specialized subroutines that are implemented in Rcpp or the like might be preferable, assuming your data is structured.


Predicting future airfare using past data

I liked the way you put across your question!

I think we cannot cannot say in specific will work well with data, it is most likely trial & error method, If ARIMA is not performing well and assuming that there is no trend in data then you can use AR, Exponential Smoothening. These are basic techniques but as you know in many scenarios basic models can explain better than complex models.

These both works well in such scenarios. Give a try.

It would be nice if you can share some graphs which can explain us a bit more about the how the data is, noise etc.
I mean if you have anything from your exploratory analysis.

Hope my answer is helpful!


Do you have some kind of outcome associated with the dataset? By 'top 10 most important features', do you mean most important in achieving a specific goal? If you have some kind of outcome column (a score or pass/fail) you can solve this through a good old fashioned regression technique. If you don't, you might have to redefine what importance means and use a technique like PCA to find the most variable features.

A worked example: What is most important for students to pass a test?

For this made up example, we are tracking 30 students and seeing what is more important for passing a test, intelligence (IQ) or study time (in hours).
The code below is in R, but I'm just using it to illustrate the process.

# Generating some sample data
set.seed(300)
IQ <- as.integer(rnorm(30,mean=100,sd=10))
study_hours <- as.integer(rlnorm(30,mean=1.5,sd=0.75))
score <- as.integer(100*tanh( 0.0012*(6*IQ + 20*study_hours
+rnorm(30,mean=150,sd=10))))

    IQ study_hours score
1  113           3    78
2  108           5    80
3  104           2    75
4  107           3    76
5   99           8    80
6  115           5    80
7  108           2    77
8  103           4    76
9  112           3    78
10 103           1    73
11 122           8    84
12  99           2    72
13  86           3    69
14 100           6    77
15 105          12    84
16 100          12    83
17 114           9    84
18  87           7    74
19  97           6    76
20 102           5    77
21  96           9    79
22 106          15    86
23  96           1    71
24 101           4    76
25  88           4    71
26  96           4    75
27 102           2    74
28 113           4    79
29  99           2    73
30 110           2    77

Both IQ and study time seem to have a positive effect on exam score. But how do you tell which one is more important?
Linear regression will give you an equation that fits a straight line to the data, and the slope of the line in each direction gives the relative importance of that feature.

But first you have to standardise the numbers, otherwise the bigger set of numbers (IQ) is going to throw things out of whack.

# Standardising the variables.
# (You take the number, subtract the mean and divide by the standard deviation)

standardisedIQ <- (IQ-mean(IQ))/sd(IQ)
standardised_study_time <- (study_hours-mean(study_hours))/sd(study_hours)

# Perform linear regression. Output variable is score and input variables are
standardised IQ and standardised study time

L <- lm(score ~ standardisedIQ + standardised_study_time)
L

The output is:

Coefficients:
            (Intercept)           standardisedIQ  standardised_study_time
                 77.133                    2.507                    3.182

which can be read as: Everything else being equal, the average score is 77.1, and IQ has an 'effect' of 2.5 and study time an 'effect' of 3.2. Since the study time number is larger, study time has more of an effect than IQ in this particular exam.

How does this translate to your problem?
In your case, you have 100 variables. Just standardise them, set up the appropriate regression on those 100 variables and the outcome variable, and choose the top ten variables with the highest (absolute value) scores.

The specific type of you use will depend on what outcome variable you have. If it's a pass/fail rather than a score, you want to use logistic regression.

As far as I can tell, doing things this way isn't completely statistically sound. However, if your client is non-mathematical and only wants 'an idea' of what the different features are doing, that shouldn't matter.

If you do not have any usable outcome variables, using PCA (principal component analysis) may help. It will highlight the 'variability' or 'salience' of the various features, but not necessary their importance towards any goal. Keep in mind that there are a bunch of caveats with using PCA, and doing some sort of data standardisation is necessary if you're using units of measurement (i.e. one feature is measured in metres, another in seconds, etc.).


are explanatory variables in multiple linear regression slopes?

They are the coefficients.

Suppose you keep the other variable fixed besides $x_1$. How much does increasing a single unit of $x_1$ would affect $y$? According to the formula, it would change by $b_1$. Similarly for the other variables.


Converting continuous to categorical variable

Whenever I want to convert a continuous feature into categorical with bins, I use one of the following two ways.

Freedman–Diaconis rule (wikipedia source)

Bin width = 2 x IQR(x) x n^(-1/3)

Where IQR(x) is max-min and n is the number of your observations in your sample.

Frequency Bins

You get such a bin width that each of them will have the same number of observations.

I don't have a rule of thumb, where to use each of them. I usually try both and measure the impact on accuracy. Then decide which I will use in the final model.


How should a neural network for unbound function approximation be structured?

This does not answere your question directly, but might include some helpful pointers:
In the recent Deep Residual Learning for Image Recognition paper it is written:

If one hypothesizes that multiple nonlinear layers can asymptotically
approximate complicated functions[...]
This hypothesis, however, is still an open question. See [28].


What to set in steps_per_epoch in Keras' fit_generator?

As mentioned in Keras' webpage about fit_generator():

steps_per_epoch: Integer. Total number of steps (batches of samples)
to yield from generator before declaring one epoch finished and
starting the next epoch. It should typically be equal to
ceil(num_samples / batch_size). Optional for Sequence: if unspecified,
will use the len(generator) as a number of steps.

You can set it equal to num_samples // batch_size, which is a typical choice.
However, steps_per_epoch give you the chance to "trick" the generator when updating the learning rate using ReduceLROnPlateau() callback, because this callback checks the drop of the loss once each epoch has finished. If the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than num_samples // batch_size without affecting the overall number of training epochs of your model.
Imagine this case as using mini-epochs within your normal epochs to change the learning rate because your loss has stagnated. I have found it very useful in my applications.


Observation window in Predictive modelling

It works fine if you make sure that window remains the same for both seen and unseen data.


Why is input preprocessing in VGG16 in Keras not 1/255.0

The pre-trained weights that are available on Keras are trained with the preprocessing steps defined in preprocess_input() function that is made available for each network architecture (VGG16, InceptionV3, etc).

For example

from keras.applications.vgg16 import preprocess_input

If you are using the weights that comes with keras for fine tuning, then you should use the corresponding preprocess_input() function for the network. It can be different from the original preprocessing steps mentioned in the paper.

from keras.applications.vgg16 import VGG16
model = VGG16(weights='imagenet', include_top=False)

If you want to stick to the original preprocessing steps, you can find pre-trained weights that is trained with the original preprocessing steps instead of using the weights that comes with Keras.

Maybe at the time of writing the blog post the weights were trained with different preprocessing steps.

Checkout this github issue to learn more.