Gibbs sampling is an example for the more general Markov chain Monte Carlo methods to sample from distribution in a high-dimensional space.

To explain this, I will first have to introduce the term state space. Recall that a Boltzmann machine is built out of binary units, i.e. every unit can be in one of two states - say 0 and 1. The overall state of the network is then specified by the state for every unit, i.e. the states of the network can be described as points in the space $\{0,1\}^N$, where N is the number of units in the network. This point is called the state space.

Now, on that state space, we can define a probability distribution. The details are not so important, but what you essentially do is that you define energy for every state and turn that into a probability distribution using a Boltzmann distribution. Thus there will be states that are likely and other states that are less likely.

A Gibbs sampler is now a procedure to produce a sample, i.e. a sequence $X_n$ of states such that, roughly speaking, the distribution of these states across the state space reflects the probability distribution. Thus you want most of the $X_n$ to be in regions of the state space with high probability (and low energy), and few of them to be in regions with low probability (and high energy).

To do this, a naive Gibbs sampling approach would proceed as follows. You start with some state $X_0$. To find the state $X_1$, you would pick some unit and calculate the conditional probability for that unit to be in state 1 ("on") conditional on the current value of all other units. Call this number p. You would then set the unit to 1 with probability p and pick the next unit to repeat this to get from $X_1$ to $X_2$ and so forth.

In the special case of a restricted Boltzmann machine, this can be greatly simplified. Instead of going through, say, first all hidden units and then all visible units and update them like this one by one, you can, in fact, update all hidden units in one step and all visible units in one step, because any two hidden units and any two visible units are independent. Thus, for a full cycle through all units of the state space, you would:

calculate the probability for all hidden units to be 1, given the value of the visible units
set the hidden units to 1 with this probability
calculate the probability for the visible units to be 1, again conditional on the value of the hidden units, and
set the visible units to 1 with this probability

This constitutes one full Gibbs sampling step and is your step 1 + the first part of 3 (the second part is then needed for the further calculation and not part of the sampling). The reason why we do this in the CD algorithm is that we actually want to approximate an expectation value and use a sampler for this.

This is a complex topic and hard to summarize in a few sentences. If you want to learn more about the mathematics behind this (Markov chains) and on the application to RBMs (contrastive divergence and persistent contrastive divergence), you might find this and this document helpful - these are some notes that I put together while learning about this.


Statistical inference on a very small datasets

There are two separate issues:

Sampling - Picking the optional ingredient level for next experiment to run. Given you have only have 4 explanatory variables, just plot them. Either all pairwise or a couple of 3d charts. With the outcome variable on the y or z axis. You'll then see the trend in the data. You can decide to get more data for interpolation (between the data points you already have) or extrapolation (data outside of the current range). There are frameworks, such as Bayesian Optimization, but that is too much work given the small dimensionality.
Inference - Predicting performance for new data. Given the data you have seen thus far (sample data), estimate parameters. In your example that would the estimating the contribution of each of the 4 ingredients, either individually or interaction. Those parameters could be scalar coefficients or distributions.


What do this Classification evaluation results mean to you? Do they are suspicious or not?

The statistics here are obviously very good, in fact too good for any practical data set. Your model is almost perfect... Unfortunately, it's practically useless and I'll explain.

In machine learning, if you see something like this you know you are in trouble. That can happen if there are problems with your data workflow. For example, you might have removed all outliers that you shouldn't, or you actually used a subset of your training data for the test set.

It's fine if you're just toying SVM, but you'll never encounter something like this in real life.


How can I know the name of the features selected by a Deep Belief Network?

In general you are extracting/creating 10 features that scan theoretically recreate the input i.e. the 41 features. The features on their own may not necessarily make a lot of sense and (depending on the dataset) may not be easily interpretable. One could draw a comparison to Principal Component Analysis and the result components as features.

One benefit of using Deep Belief Nets to pre-train your model and encode features is that the data must not be labelled. This leads to the final point, which is that your 10 features must not necessarily have names. You can just call them e.g. feature1, feature2, ..., feature10.

If you are having problems with the actual code and the way to push the 10 features further into the classifier, I would suggest you provide the code that you have so far and add more detail regarding your exact problem.


Neural networks are very good function approximators. Hence, they can approximate a wide range of nonlinear functions. Remember that linear functions are easier to represent than nonlinear functions. Hence, the neural network will clearly be able to approximate a linear function. This can be easiest seen if we only use linear activation functions. But we can also use a nonlinear ReLU and still be able to arbitrarily approximate the function on a compact set.

The question that we should have is rather: Is it overkill to approximate a linear function with a neural network when a linear regression would do the job? The answer should be clear that you should rather use a linear regression instead of a neural network.

The given example just wants to demonstrate that even without knowing the relationship between our predictors and criterion (this is sometimes called domain knowledge) the neural network will still be able to approximate the function without the need for additional domain knowledge.


Check this paper:

https://arxiv.org/pdf/1411.4280.pdf

In section 3.2 they discuss in detail why standard dropout fails in convolutional layers and the idea of spatial dropout. Fully connected networks learn via connections between single neurons while convolutions learn via features. In a sense, dropping out entire features instead of single neurons can be thought as the convolutional version of regular dropout.


How to get mean test scores from GridSearchCV with multiple scorers - scikit-learn

For multi-metric evaluation, the scores for all the scorers are available in the cv_results_ dict at the keys ending with that scorer's name ('_scorer_name').
so use

  grid.cv_results_['mean_test_(scorer_name)']

Ex: grid.cv_results_['mean_test_r2']


Why can it be that my neural network is predicting the contrary?

After being all day trying to figure this out I had to post this question to find clarity.
The problem is here

dw = 1./m * dz.dot(activations[d].T)

it should be

dw = 1./m * dz.dot(activations[d-1].T)


What is the upscaling factor in super resolution with deep learning?

The upscaling factor simply denotes how much the original will be upscaled. In the case of SRGAN, which uses 4x upscaling, this simply means that the output image will be four times as large as the input image. An input image of size 720 by 1080 pixels will be of size 2880 by 4320 pixels after being upscaled using SRGAN.


How to feed data to multi-output Keras model from a single TFRecords file

After playing around with tf.data.map operations I found the answer was easier than expected, I simply had to preprocess the data and put all the labels for each output of the model as a different key of a dictionary.

First I create a dataset from the tfrecords file

dataset = tf.data.TFRecordDataset(tfrecords_file)

Next, I parse data from the file

feature = {'image/encoded': tf.io.FixedLenFeature((), tf.string),
           'image/shape': tf.io.FixedLenFeature((3), tf.int64),
           'age': tf.io.FixedLenFeature((), tf.int64),
           'gender': tf.io.FixedLenFeature((), tf.int64),
           'ethnicity': tf.io.FixedLenFeature((), tf.int64),
 }

return tf_util.parse_pb_message(protobuff_message, feature)

dataset = dataset.map(parser).map(process_example)

At this point, we have a standard dataset we can operate with doing batching, shuffling, augmentation or whatever we wan. Finally, before feeding the data into the model, we have to transform it to fit the requirements of the model. The code below shows an example of both input and label preprocessing. Previoulsy, I concatenated all the labels, now I create a dictionary witht the names of the outputs in the model as keys.

def preprocess_input_fn():
    def _preprocess_input(image,image_shape, age, gender, ethnicity):
        image = self.preprocess_image(image)
        labels = self.preprocess_labels(age, gender, ethnicity)
        return image, labels

    return _preprocess_input

def preprocess_image(image):
    image = tf.cast(image)
    image = tf.image.resize(image)
    image = (image / 127.5) - 1.0
    return image

def preprocess_labels(age,gender,ethnicity):
    gender = tf.one_hot(gender, 2)
    ethnicity = tf.one_hot(ethnicity, self.ethnic_groups)
    age = tf.one_hot(age, self.age_groups)
    return {'Gender': gender, 'Ethnicity': ethnicity, 'Age': age}

In my model, Gender, Ethnicity and Age are the names of the last layers of the model, so my model is defined as having three outputs:

model = Model(inputs=inputs,
              outputs=[gender, ethnic_group, age_group])

Now I can use a dataset to fit the model by applying the preprocessing function first:

data = dataset.map(preprocess_input_fn())

model.fit(data, epochs=...)


Is there a definitive and more conclusive way of interpreting the R^2 score from a linear regression model in terms of prediction accuracy?

Quoting Dr. Bruce Ratner:

"R-sq is a first-blush indicator of a good model. R-sq is often misused as the measure to assess which model produces better predictions...The root mean squared error (RMSE) is the measure for determining the better model. The smaller the RMSE value, the better the model is (the predictions are more precise)."

I sourced this from Dr. Ratner's LinkedIn, so I can't link to it. His website is: dmstat1.com

Be careful with R-sq. The more variables you use the higher your R-sq value will be no matter how good or bad your model is. If you are using more than a few predictor variables you should also look at Adjusted-R-sq.

If you would rather get a measure of accuracy you should look at RMSE. If you are comparing two models also look at Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC).

To answer your question, if the R-sq is 0.7, it is incorrect to say that 70% of the data has been predicted accurately by the model. R-sq is a measure of how much variance is accounted for within the predictor variables you are using. R-sq of 0.7 means that your predictor variables can explain 70% of the variance in the response variable.

For example pretend that the price of a cup of coffee is influenced ONLY by the price of water and the price of coffee beans. If I only know the price of water and I predict the price of coffee using only the price of water my R-sq is 0.5, because I know only 1/2 of the things that influence the price of a cup of coffee.

In reality, it is virtually impossible to know ALL predictors that influence an outcome (R-sq = 1) because we either don't know what variables influence an outcome, or the variables that influence an outcome aren't contained in the dataset.


Here's some ways what I would try:

Shift humidity for n days, which makes Pearson correlation coefficient minimize.
Extract frequencies using Fourier Transform etc.
Do power spectral density estimation using Welch's method etc.


Suppose you are investigating if heart rate can predict if a person smokes. You measure bpm for 30x1m consecutive times, and ask if the person smokes in order to build your training model data set.

What would contribute to a better predictor? 30 observations from a person who smokes, or 1 observation from 30 people who smoke? Given that one person's heart rate won't change much over 30m, it seems clear that you'd rather have 1 sample from 30 people than 30 samples from one person. 30 samples from one person are not worth as much as 1 sample from 30 people.

Because the samples from one person are not independent.

I think if you put non-independent samples into a neural net then it won't affect the predictive power too much as long as the non-independence is similar across all your training data. In one extreme, if all smokers and non-smokers have the same heart rate over the 30m period, then all you've done is repeated your input data precisely 30 times and nothing will be changed (except it will take 30x as long to run...).

However, if smokers' heart rates are constant, and non-smokers' vary, then you add 30 measurements of each smoker's heart rate to your model, and a bunch of random measurements correlating those rates with non-smoking. Your NN is very likely to predict anyone with one of those smokers' heart rates is a smoker. This is clearly wrong - those 30 measurements from each smoker are only worth one measurement, and putting them in a NN will train the network wrongly.


What are the disadvantages of Azure's ML vs a pure code approach (R/SKlearn)

I learned the Azure ML platform after scikit-learn (Python) and caret (R).

Advantages:

Low or no coding involved. It is more like Orange in that you set up the ML exercise as a large process flow. It is easier for non-technical people to get started with.
Easy to convert to production-ready. You click a button and it deploys a web-service that you could send to a programmer to use. In Python, you would need to add another library and in R it is even dicier.

Disadvantages:

Fewer algorithms (e.g. no XGBoost) and other transformations (e.g. NLP) built-in. You can of course reference Python and R to do this, but that is much tougher than using the GUI. If you can figure out how to call Python/R from within Azure ML, you likely can do it on your own.
Price.


Adam uses mini batches to optimize. During optimization, you may need go down hill, the cost function, so quickly using a high learning rate. When you reach to points which are near to relatively optimal point you have to reduce the learning rate in order not miss the optimal point. In other words you have to decay learning rate to have more accurate steps by reducing the learning rate. Mini-batch optimizers have multiple steps during one epoch, which all of them may not be true but because they try to minimize the cost for each batch of input data, they finally reach to the relative optimal points.

For each epoch, TensorFlow uses same learning rate and after finishing epoch, the next epoch will be started using the current learning rate divided by the decay parameter. It should not be negative because you are using gradient descent which implies moving toward low-level places.

Recently I was looking the code of optimizers in Keras and I found that as the following code:

if self.initial_decay > 0:
       lr *= (1. / (1. + self.decay * K.cast(self.iterations,
                                             K.dtype(self.decay))))

learning changes each iteration and not each epoch.


On interpreting the statistical significance of R squared

The p-value is the strength of evidence against the null hypotheses. In this case the null is that the coefficient is equal to zero. So your p-value says that this is very weak evidence against the null so you model is likely to be describing the underlying system of the data.

R-squared describes the percent of variation that is explained by the model. Your value is very low; 14.5%. Of all the "activity" in the data your model is only explaining 14.5% of it.

So you have a situation were model is most likely explaining variation in data but not explaining very much of it.  I would suggest altering the model and refitting.


Classification when variables are in ranges


Syntax error but nothing appears to be wrong?

In the second line of the for loop

create_grid.fit(X_train.fillna(X_train.mean(), y_train.fillna(y_train.mean()))

You need an extra closing parenthesis like so

create_grid.fit(X_train.fillna(X_train.mean(), y_train.fillna(y_train.mean())))


Approximate Text compression by training model?

Your problem can be solved with a hash map. If the dataset is very large, you can use an approximate hashmap. Examples of approximate hashmaps are Bloom Filters and Cuckoo Filters.


Works find for me. Maybe you have some error in the definition of the train/test set?

library(ISLR)
library(glmnet)
# Load data
iris = iris[iris$Species=="Setosa" | iris$Species=="versicolor",]
iris$Species = as.factor(iris$Species)
# Make a test/train split
set.seed(123)
smp_size <- floor(0.75 * nrow(iris))
train_ind <- sample(seq_len(nrow(iris)), size = smp_size)
train <- iris[train_ind, ]
test <- iris[-train_ind, ]

m1 = glm(Species ~ ., data = train, family="binomial")
p = predict(m1,newdata=test, type="response")
table(test$Species, p>0.5)


Which data set to use to find correlation between Predictor and response variables? Test data set? Training data set? or the entire data set?

This depends on your research question. Do you want to make predictions? - Then you need to split your data set in training and testsamples.

However, if you are more interested in answering the following question: What impact does X1, X2, X3 and X4 have on Y, respectively? Then you are interested in estimation. For this, you do not need to split your sample, but have to test your data set for underlying model assumptions (e.g. heteroscedasticity, autocorrellation of the residuals etc.) to get unbiased/accurate estimations.

For OLS (linear regression) these model assumptions follow the Gauss-Markov-Theorem.

Most statistical tests for model assumptions are already implemented in R:

Test for autocorrelation - Breusch-Godfrey - bgtest
Test for heteroscedasticity - White's Test - het.test
Test for normality - Jarque Bera Test - jarque.bera.test

However, even the tests make some assumptions about your data - but these tests are the most common ones.

UPDATE - Example for clarification

According to your comment. Imagine this real-life problem. You want to build a model that can predict the chance of cancer (y), using variables like age, blood pressure, weight, cholesterol value etc.. (X1, X2, X3, X4)

To assess the nature of the variables and their relationships you do descriptive analysis (Mean, Variance etc.) + Correlation analysis.

Let's have a look at the following data. (Yellow records would be your test samples - the data you DON'T have in real life, as those patients haven't seen the doctor yet).

As you can see the mean and variance can differ already a lot depending on the fact if you include test samples or not in your calculations.

Now we are coming to the correlation part - the relationship between the independent variables or also called predictors

I highlighted the relationship between weight and cholesterol. Without the test sample, it seems like weight and cholesterol are uncorrelated or have only a slightly positive relationship. If we add the test data it turns out that the correlation turns negative.

Question: If the correlation between your variables would have an impact on your choice of variable selection for your model, would it make sense to include the test-sample in your correlation analysis? Especially, knowing that this data is not available in real life yet.

Remember Model Estimation: if you build a multivariate regression model mean, variance and covariance are used to find the best parameters to estimate your dependent variable (Cancer). So a model that was already trained with all available data is likely to make better predictions as it has already seen the data it should predict.

Summary

Whenever you plan to make predictions with a model you should work as closely to real-life assumptions. So you split your data in train - test data. You ònly use the training data to perform all your tests and checks and you will not touch the test set before making a prediction.


Age range prediction according to browsing history

This problem is quite broad and I would like to add some of my suggestions. Refer to good source of Age wise browsing data, age-wise social behaviors and their preferences publicly available. Here are some datasource that you may refer and you will have to do extensive research on lot of factors to predict age based on browsing data.

Datasource 1: https://www.google.com/publicdata/directory

Datasource 2: https://www.ciser.cornell.edu/ASPs/datasource.asp

Refer to this paper : https://link.springer.com/chapter/10.1007/3-540-44934-5_3

I would definitely say that this problem doesn't seem to have any perfect solution, only thing we may do is to try our level best by various sources and implementations. Learning algorithms on public datasets on behavior, survey,  social and mental age specific activities etc can help as well. I have also attached a link to a paper on similar problem. I hope this helps to some extent.


Restricting a weight vector (optimization parameter) to be in a certain domain using python ML library linear regression model

There are many, I'd recommend you the following: cvxpy (in turn interfaces some other solver like scs or cvxopt), cvxopt, casadi, even tensorflow (some optimizes have options for bounds).


How to get different length output using keras?

Its the number of features that has to remain consistent not the number of timestamps.
Outputs will be batches of one row predictions, so in your case its 1500 and than 1000. Model does not care, features should remain same. And beware of dataleakage with time series.


It can be explained with a simple example:

EXAMPLE REVIEW: The dashboard finishing is not good, indicators, headlights,
parking lamps and ac vents are having impressive finishing
The overall car is good.

The statement "overall car is good" indicates that there is a mix of positive and negative opinion about the car. In the aspect of the dashboard, the opinion is negative. However, the review apprising the aspects "indicators, headlights,
parking lamps and ac vents".
It might be clear by this example that the aspects are the properties or factors of the target product or event.


Understanding how many biases are there in a neural network?

The sources are both correct, they implement bias in different ways, and are counting slightly different things:

Your first source implements a separate bias vector in each output layer in addition to the weights, and is referring to the dimension of the bias vector when it is counting biases.
Your second source implements bias as both a separate fixed value $1.0$ in each input layer, and a larger weights matrix with an extra column containing the actual learned bias value. It is referring to the extra added value in each layer when counting the "biases" - more accurately it is counting the added bias "signals" and not the learned bias values, because the learned bias values are implemented inside the weights matrix when using this approach.

In both cases, the number of learned values added due to bias are the same, and are the same as this:

  the answer of the first should be 5 and for the second 4 + 3. Each neuron except for in the input-layer has a bias

In the second case the equivalent values appear in weight matrices as extra columns. It's really just an implementation difference. By adding a fixed bias signal to inputs, it simplifies the learning update for each layer to occur to only one matrix, as opposed to a matrix plus a separate bias vector (requiring different update rules for each). This is at the expense of needing to manipulate the inputs each time, and there is no great difference between the approaches in terms of efficiency. Some libraries take one approach, some take the other.


Multi-task learning, finding a loss function that "ignores" certain samples

I seem to have a workable solution using this minor tweak to square loss:

$L=(y-\hat{y})^2 \cdot \left(\frac{y}{y + \epsilon}\right)$

Where $y$ is the label and $\hat{y}$ is the prediction, and $\epsilon=1e-6$ is to avoid a divide by zero error (it's ~ the smallest number within 32 bit precision)

In tensorflow:

epsilon = 1e-6
loss = tf.reduce_sum(
          tf.mul(tf.square(prediction - label),
                 tf.div(label, label + epsilon))
)

The second term is 0 when the label is 0 (e.g. loss is 0 and no gradient is added for that sample), and it'll round to 1, or near enough, when the label is non zero. As long as the range of labels does not cross 0 this should work, and that's a simple matter of scaling the labels if it's not the case by default.


CSR scipy matrix does not update after updating its values

Sparse matrix is a matrix that has most of its elements as zero. Matrix on the other side of it, is Dense. csr_matrix creates an all zero element matrix and if you are updating the elements as

m[i,j] = 0.

Well, you're not really doing anything as elements are already zero. For example

import numpy as np
from scipy.sparse import csr_matrix
mat = csr_matrix((2,2), dtype = np.int16).toarray()
print mat,"\n"
for i in range(np.shape(mat)[0]):
    for j in range(np.shape(mat)[1]):
        mat[i,j] = i+j
print mat

gives

[[0 0]
 [0 0]]

[[0 1]
 [1 2]]

which is true, the first matrix,an all zero matrix created and the second, is the updated one.

Edit:

After substituting when $i==j$, the resultant matrix still has four elements but this time with $(0,0)$ and $(1,1)$ indices being $0$. So the result is

(0, 0)    0
(0, 1)    1
(1, 0)    1
(1, 1)    0
(4,)

But explaining your expected result

(0, 1)    1
(1, 0)    1
(2,)

The result has basically two elements with $(0,1)$ and $(1,0)$ indices being $1$ and $1$ and shape is just $(2,)$ when flattened which does not make sense as matrix is still of size $(4,)$ . So python output is basically giving all the values in the matrix with index. If I am rightly understanding what you are assuming, since the matrix is sparse, you are expecting it to give only the values where the elements are non-zero, well this does not do that.

Hope it helps.


Keras Neural Network training is stuck (gets stuck around epoch 6)

I had a similar problem. I guess the reason is incompatibility of Keras with your current anaconda. Just update it.


If you have enough data, use cross validation.

If you don't have a lot of data, use cross validation on a similar dataset, and transfer the window size (the UCR archive has a bunch of similar dataset)

Don't forget, that the best warping  window size depends on the amount of training data. As you get more data, you can have a smaller warping window, see fig 6 of http://www.cs.ucr.edu/~eamonn/DTW_myths.pdf

eamonn


How Do I Classify Dirty/Dusty Camera Images?

What kind of distortion does dirt/dust produce? If it is somewhat regular, for example more or less regularly sized speckles (greyish circles) it might be worthwhile to try to classify the logarithm of the real part of the fourier transform of the picture, instead of the original picture.

Also, knowing the physical properties of the distortion might guide you towards devising simple heuristics (for example based on some frequencies in the fuorier spectrum) that are as good as a ml-based approach.


I'm just guessing but I can think of two options:

Train the model from the original data, including features which represent the current trend so that the model can integrate it. It makes the job of the model more complex so it might not work very well.
Since you can untrend the data I assume that you can calculate the trend function. So you could train an independent model for this trend function and apply it on the predictions obtained with the untrended model. In this case performance depend how accurately the trend is predicted.


You do not want to remove all correlated variables. It is only when the correlation is so strong that they do not convey extra information. This is both a function of the strength of correlation, how much data you have and whether any small difference between correlated variables tell you something about the outcome, after all.

The first two you can tell before you do any model, the final one not. So, it may be very reasonable to remove variables based on the combination of the first two considerations (i.e. even if the extra variables may in principle contain some useful information, you would not be able to tell given the strength of correlation and how much data you have) before you do any modelling/feature engineering. The final point can really only be assessed after doing some modelling.


I'm not sure what you already know, so I'll answer how I'd do it. If you do not understand things, or already know some things I say, please leave a comment.

I would tackle it as a multi-class problem if your output is categorical, hence indeed only $10\%$, $25\%$ and so on. If you want to treat it as a regression problem, I'd say you need continuous deltas of popularity, so you need all kinds of values as the delta, and not only a fixed number of them. The training data should dictate how you approach the problem. Note that you can always reduce the regression to a multi-class problem (which asks for an instance which of the deltas it should assign) by bucketing to intervals.

For argument's sake let's assume you have multi-class data. One document has one of the classes, i.e. $10\%$. You then go ahead and encode these classes into a binary vector, which probably even is a part of the library you're using.

You'd then go ahead and split your data with the hold-out method to have some vlalidation examples set aside to see how your model actually works. You then go ahead and train some model, say a random forest. With the validation data you calculate some measure of error and see if it is enough. If not you go ahead and choose other models to train, other features to add and so on.

I'd suggest to try use another approach than bag-of-words such as word2vec or GloVe. You can also add the sentiment classifiers values as feature columns. You could then compare the different features and see how it performs.

Let me know if this helps you or if I misunderstood your question.


When to use Dense, Conv1/2D, Dropout, Flatten, and all the other layers?

What are Dense layers and when are they useful?

Dense layers are used when association can exist among any feature to any other feature in data point. Since between two layers of size $n_{1}$ and $n_{2}$, there can $n_{1} * n_{2}$ connections and these are referred to as Dense.

Where does Conv layers come in and when are they useful?

Coming to the conv layers, these are important when nearby associations among the features matter, example object detection. Neighborhoods matter to classify or detect. It is very less likely that the pixels at the opposite corners(very far away) are somehow helpful in these use cases. Filters does this job of getting associations among neighborhoods. This answer is great at understanding difference between 1D and 2D convolutions. I really dont want to repeat it.

Dropout and Flatten

Dropout is a way of cutting too much association among features by dropping the weights (edges) at a probability. The original paper from Hinton et.al is a quick and great read to grasp it. Reducing associations can be applied among any layers which stops weight
updation for the edge. The another key difference here is it has no weights associated with it. It is just there dropping things.

Flatten layers are used when you got a multidimensional output and you want to make it linear to pass it onto a Dense layer. If you are familiar with numpy, it is equivalent to numpy.ravel. An output from flatten layers is passed to an MLP for classification or regression task you want to achieve. No weighting are associated with these too. It is just flattening the hell out.

To improve the model

Try more layers, change activation functions, try batching. If there is any chance of improving features and creating any higher order ones, do it. Too many variables to speculate.

This question is real broad, but hope I covered it well.


Does the count of items in a transaction matter to apriori?

No it is not important and highly recommended to remove the duplicate items and sort the items in lexicographical order in each transaction. This is to improve the performance.

In association rule mining, an item is frequent iff it is repeated in multiple transactions not in a single transaction. This is why you don't need to have duplicate items in a each transaction.


Genetic algorithms(GAs): to be considered only as optimization algorithms? Are GAs used in machine learning any way?

Yes, as you found they (evolutionary algorithms such as genetic algorithms) are using for optimization tasks. One reason that they are not using mostly in machine learning could be their performance to converge to the optimum point. Also, implementation of the GAs for some domains could be problematic and it could not be generalized like gradient descent, as it's involved in at least 5 phases (mutation, crossover,  ...).


How to interpret skimage orientation to straighten images?

Based on the doc you provide, orientation is in radians, ranging from -pi/2 to pi/2 counter-clockwise:

orientation : float. Angle between the 0th axis (rows) and the major
axis of the ellipse that has the same second moments as the region,
ranging from -pi/2 to pi/2 counter-clockwise.

Moreover, as it is said in the regionpropos doc, since 0.16.0, they use "rc" corrdinates everywhere. They say for the majority of computation you could find the old results with a simple transformation, but for others, it is more complicated, such as the orientation:

For example, the new orientation is π/2 plus the old orientation.

Which means that the right formula to get the angle you want is this one:
angle_in_degrees = orientation * (180/np.pi) + 90

And the orientation refers to this angle on the image:

Now: If you want your major axis and the 0th axis align, then rotate your image by -angle_in_degrees:
from skimage.transform import rotate
rotate(image, -angle_in_degrees, resize=True)


Deciding what type of model to use for predicting the bottom decile of student grades

My first attempt was a neural network with binary classification,
  forcing the model to predict that the student is either in the bottom
  2%ile, or not. Having read a few posts on FHarrell.com, I was thinking
  that a logistic regression that outputs the probability of being in
  the bottom percentiles is better?

I would recommend you start with logistic regression as a baseline. Then when you have your baseline I would compare the LR-model with a tree-based ensemble since they tend to work really well on tabular data. I would recommend random forest, it is easy to work with and is not that prone to overfitting. Then if you are feeling fancy you could try gradient boosting with XGBoost or LGBM. They tend to give marginally better results, but takes some more effort to work with.

  Or alternatively, should I just do a non-binary linear regression to
  directly predict the students %ile? The reason I thought that this
  wouldn't be as useful for my purposes is because I don't care about
  making predictions between any of the other percentiles. For example,
  I don't care about distinguishing between an A and a B student, I am
  just looking to identify F students. Therefore I would be losing
  accuracy in the bottom 2%ile and gaining it in the other irrelevant
  %iles?

I agree.

  Finally, I think that the neural network makes sense given that there
  is interaction between the variables.

Neural networks are generally not that good on tabular data. Unbalanced, which your have with a ration of 1:50, is especially hard for NNs. Then they tend to classify everything as the majority class.


High level understanding of residual blocks

Suppose we want to fit a function $f(x)$.  We can either try to learn a neural network model $F(\cdot)$ so that $F(x) \approx f(x)$.  Or, in the residual network approach, we try to learn a neural network model $R(\cdot)$ so that $x+R(x) \approx f(x)$.

Why is the latter easier to learn?  There's no fundamental reason why it should necessarily be so, in general.  It depends on the specific data set.  However, one possible intuition is that we might expect that in some settings, $f(\cdot)$ might be approximately linear: i.e., as a first-order approximation, $f(x) \approx x$ might be a reasonable first-order approximation.  Then we want to model the error term: i.e., suppose $f(x)=x+r(x)$; then we want to model $r(x)$.    In some settings, the error term $r(x)$ might be simpler to model or smaller than $f(x)$.  In that case, a residual network architecture might work better.

If you like, you can think of this as being akin to a Taylor-series approximation.  The Taylor series for $f(x)$ is

$$f(x) = c_0 + c_1 x + c_2 x^2 + \cdots$$

Suppose $x$ is small (i.e., $|x| \ll 1$).  Then $f(x) = c_0$ is a zero-th order approximation; $f(x) = c_0 + c_1 x$ is a first-order approximation; and so on.  At each step, we expect that the residual/error term is probably smaller than the approximation.  You can think of a residual network as learning a Taylor series for $f(x)$, in the special case where $c_0 = 0$ and $c_1 = 1$.

There are other more sophisticated explanations / intuitions, but hopefully this gives one possible perspective.


What is the reason behind taking log transformation of few continuous variables?

This is done when the variables span several orders of magnitude. Income is a typical example: its distribution is "power law", meaning that the vast majority of incomes are small and very few are big.

This type of "fat tailed" distribution is studied in logarithmic scale because of the mathematical properties of the logarithm:

$$log(x^n)= n log(x)$$

which implies

$$log(10^4) = 4 * log(10)$$

and

$$log(10^3) = 3 * log(10)$$

which transforms a huge difference $$ 10^4 - 10^3 $$ in a smaller one $$ 4 - 3 $$
Making the values comparable.


I'm no Excel expert, as I generally use Python or R instead, but this might get you started until an Excel expert comes along.  In the meantime, it would help if you clarified your question.  And you should be aware that search will only find you the index of the first match, not all matches in the string.  If you only need the first hit, you can use

=MID(A1,SEARCH("popul",A1,1),IFERROR(FIND(" ",A1,SEARCH("popul",A1,1)),LEN(A1)+1)-SEARCH("popul",A1,1))

although I cannot claim this is the best way to do this.  You really didn't specify where you want the results to appear, how they should look, or if you only have one cell you need to search in.  It would also help to know the version of Excel you have.  I'll also present a crude way to return all the hits in the string:

Cell A1 contains the string, B1 has no formula, and if you run out of "n/a"s you can extend columns B, C, and D by filling down.  The formulas are as follows:

B3 and below use

=IF(C2+1<LEN($A$1),C2+1,"n/a")

C2 and below use

=IFERROR(FIND(" ",$A$1,SEARCH("popul",$A$1,B2)),LEN($A$1)+1)

D2 and below use

=IFERROR(MID($A$1,SEARCH("popul",$A$1,B2),C2-SEARCH("popul",$A$1,B2)),"")

As you can see, there's little to no error checking except to deal with the match at the end of the string.  In the end though, if you're going to use Excel for this you should probably create a user defined function or utilize VBA instead of in-cell formulas.


Classification followed by regression to handle response variable that is usually zero

This sounds entirely reasonable, and the usual name for this structure I have heard for this is just "pipeline" which also applies to other system-feeds-next-system structures - it might also be "machine learning pipeline" or "data processing pipeline".

There are ways to assess performance of a ML pipeline:

You can of course compare the final accuracy or loss value, with the simpler model. Has turning the model into a more complex multi-stage one actually improved things? Sadly nothing is guaranteed, although I would be hopeful in your case initially - in part because you could apply adjustments available to classifier models used to deal with class imbalance issues.
You can decide which part of the pipeline will gain you the most benefit by switching between pipeline-so-far input to each unit and perfect input from the training data. Then you can see how much incremental difference is possible by perfecting that unit in the pipeline.

In your case you have a two stage pipeline, so you can check whether it is worth focusing more effort on the classifier or regression parts by comparing the incremental improvements between:

The unadjusted output of the whole pipeline run end-to-end.
The output of the regression (or zero) assuming that the classifier was perfect.
A perfect score.

Whichever of the two differences gives you the largest difference (2) - (1), or (3) - (2) points at work being most rewarded for working on the classifier or regression stage respectively.

You can see a worked example of this per-stage analysis in Advice for Applying Machine Learning (slides 21, 22), amongst other places.


Am I using optim.SGD incorrectly in pytorch?

LT;DR: Executing SGD (Stochastic Gradient Descent) is almost like executing GD (Gradient Descent) but computationally cheaper, and probably the only way to execute GD on big datasets.  By evaluating at each sample - each game - you are losing the advantages of SGD and one could say you're "rolling your own" version of a computationally cheap GD.

Gradient Descent Summary

After you evaluate the error/loss function of you model, with whatever criterion you may use, you will then attempt to take the gradient of your loss function at the point where your network is.  To simply things let's assume that you are using cross-entropy as the criterion of the loss (which makes some sense if you are evaluating an entire matrix of probability results).  Let's say that the loss function is, where $H$ is cross-entropy, $f$ the NN evaluation, $X$ input data and $Y$ the labels:

$$
E = H(f(X), y)
$$

This is what nn.CrossEntropyLoss does (or any other criterion).
We want to find the minima of $E$, for that we can use $\nabla E$ but we cannot get the general formula for $\nabla E$ because we do not have the actual formula for $f$.  Yet, we can find the (vector) value of $\nabla E$ at the location we are in (where location is the value of every parameter/weight in our NN).  We can find $\nabla E$ for one specific configuration of $W$ which will be the matrix of all weights in the network.

$$
\nabla E_{W} = \frac{2}{N} X^T(XW - y)
$$

This is oversimplified - to actually find the matrix $W$ we need to perform backpropagation.  In pytorch we get that by doing a huge chain rule with E.backward().  i.e. this is what backward() does, it gets the matrix $W$.

Now we have a handful of very expensive matrix multiplications to get from $W$ to $\nabla E_W$.  Possibly prohibitively expensive if the dataset is big ($X$ is the full dataset).  Enters SGD.

Stochastic Gradient Descent

The only difference in SGD from GD is that SGD will not use the entire $X$ in the calculation above.  Instead SGD will select just a handful of samples (rows) from $X$ and use that as an estimation of $\nabla E_W$.  Often it is said that SGD takes only a single sample from $X$ but implementations vary (e.g. ASGD).

Finally step() will apply $\nabla E_W$ times the learning rate to all weights/parameters in the network.

Your Approach

Since you are going sample by sample and performing backward() and step() on each sample you are not really doing SGD (despite calling that function).  In your case the SGD optimizer has only a single sample to select from every time, therefore you are uniformly trying all samples in your dataset (as opposite to Stochastically).  (That uniformity will reduce the variance of your model, which may be dangerous in other ways, although not very relevant here)

Therefore yes: To use SGD you do need to feed batches of input, and it will "average" (sum) the  gradients across the sampled rows of $X$.  It will average/sum/accumulate (sum all components) the gradients thanks to the fact that some samples may result in opposite gradient values in some directions; and hopefully the taken samples will agree on the main direction of the gradient and increase the magnitude in that direction.

P.S. The fact that pytorch accumulates the gradients from each selected sample is also one of the reasons we need to call zero_grad() in the training loop.  You want to accumulate gradients within an SGD step but not accumulate across steps (SO question on exactly that).

Code

To actually generate batches you can use torch.randperm() I personally like this answer on SO about randperm but I'll add a summary here:

Having the matrix $X$ above as X (in code), one can create a batch of 256 samples with:

batch_size = 128
n_samples = X.size()[0]
permutation = torch.randperm(n_samples)

for i in range(0, n_samples, batch_size):

    indices = permutation[i:i+batch_size]
    batch_x, batch_y = X[indices], Y[indices]

    # nn(X), criterion(), backward(), step(), zero_grads(), ...

Which you need to perform per epoch (refer to linked answer for that and more - do not want to copy that code in full since I do not deserve the merit).

The tricky bit may be the fact that you need to generate the batch from both $X$ and $Y$, and keep the order between them.  Using indices generated with randperm allows for that.


Neural Network for Multiple Float Output

Just use an output layer with 12 neurons instead of 1. Qualitatively there is no difference. For regression the output activations should be linear, and you have a few choices for cost: RMSE, MAE, or Huber Loss to name a few.


How to find optimized x values (input features) after training in deep learning?

If I understand your question correctly, you want to find the best features to optimize your model's performance?

I think what you're looking for is feature selection, which is an entire subject within machine learning/statistics. There is a whole tag on here about feature selection, you'd be better off asking your question with that. I would poke around a little online to learn more about feature selection, and if after that you still want help here you should provide more information about the problem you're trying to solve, the data you're working with, and more details about your model.


Improving the pix2pix Architecture for Sketch to Image Translation on a Dataset of Sketches of People to Photos of People

I managed to obtain better results with less artifacts by implementing the code of the paper "High-Quality Facial Photo-Sketch Synthesis Using Multi-Adversarial Networks".
I found this code to be quite nice working for my purposes.
If you need, I can also provide a Colab notebook with my ported code.


Not sure what you are exactly looking for. There are many ways to do this. One simple way is like this,

word_list = ['natural language processing', 'sequential labeling', 'programmable', 'spell checking', 'techniques', 'forensics', 'important issue', 'categorial grammar', 'girls', 'applications']

Since you said, a list of key words,

stopwords = ['programmable','techniques', 'important issue', 'girls', 'applications']

resultwords  = [word for word in word_list if word.lower() not in stopwords]
result = ' '.join(resultwords)

print (result)

This will yield,

>> natural language processing sequential labeling spell checking forensics categorial grammar

Otherwise, if you just have a string of text, use split() method to put each word into a list splitting by space.

querywords = word_list.split()

resultwords  = [word for word in querywords  if word.lower() not in stopwords]
result = ' '.join(resultwords)

You can always warp them into a function and automate this. As I said there are many ways to do it, without knowing what have you tried, this is a simple way to do without using nltk library.


What are some nice algorithms/techniques for optimizing and predicting Click Through Rates (CTR)?

Logistic regression is very common and efficient.

This will provide you with a probability of click for a given impression. You will likely need additional information for optimization, like keywords or A/B testing.


If you want to tackle the problem from another perspective, with an end to end learning, such that you don't specify ahead of time this large pipeline you've mentioned earlier, all you care about is the mapping between sentences and their corresponding SQL queries.

Tutorials:

How to talk to your database

Papers:

Seq2SQL: Generating Structured Queries from
Natural Language using Reinforcement Learning  salesforce
Neural Enquirer: Learning to Query Tables in Natural Language

Dataset:

A large annotated semantic parsing corpus for developing natural language interfaces.

Github code:

seq2sql
SQLNet

Also, there are commercial solutions like nlsql


How are the gradients of a Neural Network calculated just by matrix multiplication?

Like the derivative of y = h_relu * w2 being h_relu, which doesn't make much sense to me

These are partial derivative and depends on w.r.t variable. In the end, we have to complete the chain of partial derivative from output to the respective weights.

Your Network is -
$$x \xrightarrow[\text{}]{\text{}} w_1 \xrightarrow[\text{}]{\text{  h  }} ReLU \xrightarrow[\text{}]{\text{  h_relu  }} w_2 \xrightarrow[\text{}]{\text{ }} y\_pred$$
By definition ReLU is - $f(x)=\max(0, x).$
Derivative of ReLU - $\frac{d}{dx}\text{ReLU}(x)$
=$1 \forall x > 0$
=$0 \forall x < 0$

In backpropagation -
$\frac{\partial y\_pred}{\partial w_2} $ = $h\_relu$  ;since, y_pred = $h\_relu*w_2$
$\frac{\partial w_2}{\partial h} $ = $1$  ;derivative of ReLU=1,ignoring negative part for simplicity
$\frac{\partial h}{\partial w_1} $ = $x$ ;since, h = $x*w_1$
$\frac{\partial y\_pred}{\partial h\_relu} $ = $w_2$ ;since, y_pred = $h\_relu*w_2$

So, Loss gradient at different weights will be -
 $L$ at the $y\_pred$
 $L*h\_relu$ at $w_2$
 $L*w_2$ at $h\_relu$
 $L*h\_relu*1*x$ at $w_1$


For this problem, I could provide you with 2 approaches, 1. Naive approach 2. Actually used for object detection.

Talking about the Naive approach, in this approach we can build an end to end pipeline for digits extraction using the following pipeline:

Take images and clean image (maybe take threshold or tune brightness and contrast to remove noise and artefacts)
Take the cleaned image and find the contours of the text inside the image over the text and then finally detect the numbers.

This is certainly one solution but has a lot of complications involved alongwith.
To solve this problem, here is other approach that is used in actual OCR. For this, we follow this pipeline:

Take a cleaned image
From left to right (In the order of actual human reading pattern) run a sliding window and after Non-max suppression, find the text.
For each text found using the sliding window, run the prediction and store the prediction in the required format of your choice.

Although this too contains a number of complications and problems related to a number of other problems, this is still the best method and can be tuned as per the requirement to extract the exact required text.


How to access substrings in pandas column and store it into new columns?

This is perhaps more suited for StackOverflow. I would also use a better/more descriptive title for the question itself; that way others that are facing a similar problem are able to find it.

The reason you are seeing that error is because of the nan values, which are of type float. So while most of the rows in df['location'] contain strings, every row instance of an nan in the column is a float, and str.index() is not available for floats.

Your check of if df.location[i] == np.nan: is pointless, because np.nan == np.nan is always False due to the very definition of nan. Refer to this question on the topic. Because your check fails, the loop enters the else block and encounters a float object attempting to invoke a string method.

In my opinion you are using a very complicated approach to get what you want.

Replace your code with this. It should give you what you are looking for. Any nan values encountered will be handled by python.

df['location']=df['location'].str.replace(" ","").str.strip('(').str.strip(')')
df['latitude']=df['location'].str.split(',').str[0]
df['longitude']=df['location'].str.split(',').str[1]

I tested this using the following code segment:

df=pd.DataFrame()

df['location']=(
"(37.785719256680785, -122.40852313194863)",
"(37.78733980600732, -122.41063199757738)",
"(37.7946573324287, -122.42232562979227)",
"(37.79595867909168, -122.41557405519474)",
"(37.78315261897309, -122.40950883997789)",
np.nan,
"(37.78615261897309, -122.405550883997789)")

df['location']=df['location'].str.replace(" ", "").str.strip('(').str.strip(')')
df['latitude']=df['location'].str.split(',').str[0]
df['longitude']=df['location'].str.split(',').str[1]

print(df[['latitude','longitude']])

This produces the output:

             latitude             longitude
0  37.785719256680785   -122.40852313194863
1   37.78733980600732   -122.41063199757738
2    37.7946573324287   -122.42232562979227
3   37.79595867909168   -122.41557405519474
4   37.78315261897309   -122.40950883997789
5                 NaN                   NaN
6   37.78615261897309  -122.405550883997789


How do I approach learning Data Science/ML the 'rightest' way?

Welcome to the community!

I normally don't answer questions like this, they're pretty broad and there are many others just like it. It seems like you're not actually asking how to get started, you already seem set on reading some books (which is great!), so let me answer that question, rather than a very broad 'how to get started'.

First of all, I'd like to challenge your assumption about learning from the bottom up, and offer an alternative opinion (yes, this part is just an opinion). I'd instead encourage you to get your hands on the high level tools and start using them right away. If everything we ever did required us to forego abstractions and learn everything from the beginning, we'd still be reinventing the wheel over and over. Start experimenting with the existing tools and supplement that with your reading. You'll develop an excellent understanding this way.

So to answer your question... I'd recommend the second book you quoted: Hands-On ML with Scikit Learn and Tensorflow. This book will teach you good habits right from the beginning, give you a good understanding, and get you using those tools right away.

Best of luck.


Explaining XGBoost functioning to non-technical people

The xgboost docs come with a nice introduction and examples. If it is really only about a few slides, this may be a good start: https://xgboost.readthedocs.io/en/latest/tutorials/model.html


Need help in deriving Policy Evaluation (Prediction)

Going from step (4.3) to (4.4) is turning the expectation (dependent on following policy $\pi$) into a more concrete calculation. To do this, you must resolve your random variables ($R_{t+1}$ and $S_{t+1}$) into the specific values from the finite sets of $\mathcal{R}$ and $\mathcal{S}^+$ - where individual set members are noted $r$ and $s$.

Recall that $\pi(a|s)$ is the probability of taking action $a$ given state $s$, so by taking a sum of any function depending on $(s,a)$ e.g. $\sum_{a} \pi(a|s)F(s,a)$ you will get the expected value of $F(s,a)$ starting from state $s$ and following policy $\pi$.

So you could write (4.3b) as:

$$v(s) = \sum_{a} \pi(a|s)\mathbb{E}_{\pi}[R_{t+1}+\gamma v_{\pi}(S_{t+1})| S_{t}=s, A_t=a]\qquad\qquad(4.3b)\\$$

Similarly, now you have written the expectation when given $s$ and $a$ (so you have all the possible cases of $a$ available to work with), you can use the transition and reward probabilities to fully resolve the expectation and substitute the random variables ($R_{t+1}$ and $S_{t+1}$) for their MDP model distributions described in $p(s',r|s,a)$, leading to equation (4.4). E.g.

$$v(s) = \sum_{a} \pi(a|s)\sum_{s',r} p(s',r|s,a)\mathbb{E}_{\pi}[R_{t+1} + \gamma v_{\pi}(S_{t+1})| S_{t+1}=s', R_{t+1}=r]\qquad\qquad(4.3c)\\$$

$$\qquad = \sum_{a} \pi(a|s)\sum_{s',r} p(s',r|s,a)(r + \gamma v_{\pi}(s'))\qquad\qquad(4.4)\\$$

Your last equation is a slightly different formulation of the same result, where:

$R_s^a$ is the expected reward when taking action $a$ in state $s$. It is equal to $\sum_{s',r} p(s',r|s,a)r$. It is also equal to $\mathbb{E}[R_{t+1}|S_t=s, A_t=a]$, by definition and independently of $\pi$.
$P_{ss'}^a$ is the state transition probability - the probability of ending up in state $s'$ when taking action $a$ in state $s$. It is equal to $\sum_r p(s',r|s,a)$ given $s'$

Technically using $R_s^a$ loses some information about the dynamics of the underlying MDP. Although not anything important to reinforcement learning, which deals with maximising expected reward, so in some ways it is more convenient to already characterise the MDP with expected rewards.


Python library for segmented regression (a.k.a. piecewise regression)

numpy.piecewise can do this.

  piecewise(x, condlist, funclist, *args, **kw)

  Evaluate a piecewise-defined function.

  Given a set of conditions and corresponding functions, evaluate each
      function on the input data wherever its condition is true.

An example is given on SO here. For completeness, here is an example:

from scipy import optimize
import matplotlib.pyplot as plt
import numpy as np

x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ,11, 12, 13, 14, 15], dtype=float)
y = np.array([5, 7, 9, 11, 13, 15, 28.92, 42.81, 56.7, 70.59, 84.47, 98.36, 112.25, 126.14, 140.03])

def piecewise_linear(x, x0, y0, k1, k2):
    return np.piecewise(x, [x < x0, x >= x0], [lambda x:k1*x + y0-k1*x0, lambda x:k2*x + y0-k2*x0])

p , e = optimize.curve_fit(piecewise_linear, x, y)
xd = np.linspace(0, 15, 100)
plt.plot(x, y, "o")
plt.plot(xd, piecewise_linear(xd, *p))


Keras: apply multiple filters to each feature map in CNN

You may try Keras DepthwiseConv2D layer

Depthwise Separable convolutions consist of performing just the first step in a depthwise spatial convolution (which acts on each input channel separately). The depth_multiplier argument controls how many output channels are generated per input channel in the depthwise step.

It will convolute each Channel separately. As shown in this depiction.

$\hspace{6cm}$Image credit - Blog by Chi-Feng Wang
With depth_multiplier argument, you can add more Filetrs i.e. more copy of the "triplet" shown in the depiction.

depth_multiplier:
The number of depthwise convolution output channels for each input channel. The total number of depthwise convolution output channels will be equal to filters_in * depth_multiplier.


These numbers refer to minibatches, not individual samples. The data is not fed to the model one by one, but in small groups called "minibatches" or simply "batches". The size of the minibatch (the number of elements to be included in each minibatch) can be specified as a parameter to the fit method. As you did not provide any value, it took the default value of 32. This is specified in the documentation:

batch_size: Integer or None. Number of samples per gradient update. If unspecified, batch_size will default to 32. Do not specify the batch_size if your data is in the form of datasets, generators, or keras.utils.Sequence instances (since they generate batches).

Your data size divided by the batch size gives us the number of batches, which is the number you see: ceil(55000 / 32) = 1719. As they are not divisible, the last batch would have a few elements less; specifically, the last batch has 24 elements (= 55000 - 32 * 1718) instead of 32.


Which data sets would help to predict (exponential) market trends?

From a quick look, my guess would be that maybe they mine data from some predefined target websites, typically social media: twitter, facebook, etc. I could imagine that they capture trends in the sense of "what people are talking about" on social media platforms, but maybe they have other sources and methods.
There are Natural Language Processing methods which can be used: the basis would be some form of topic modelling, then there could be some more specific applications such as event detection. We can't know how much manual post-processing is involved though, there could be a manual filtering to remove irrelevant stuff.


Outlier detection for unbalanced classes

you need to distinguish between these cases:

Data Imbalance
Data Imbalance + Very few number of samples (minority class)
Severe Data Imbalance + Very few number of samples (minority class)

20:60   vs.  10:20    vs. 100:1000  vs. 10:100

and these cases:

similarities between different classes.
wide variations within the same class.

You need to understand to which of these cases your problem belong.

if you have very severe data imbalance + very few number of samples + wide variation within the majority class and similarities between different classes. regular oversampling or down sampling techniques will not help you as well as most of the synthetic oversampling techniques designed specifically to deal with the data imbalance but the assumption is to have enough number of samples.

Try to focus more on ensemble techniques that designed mainly to deal with data imbalance.
SMOTE-Boost
RUSBoost
SMOTEBagging
IIIVote
EasyEnsemble


Making fake result in data mining using weka j48 algorithm

I don't understand the purpose when the quality of the tree is not important but you should be able to make a tree to have 100% accuracy on training data set easily. Just avoid any pruning and let the tree grow as maximum as possible. As far as I remember Weka set a pruning policy for J48 by default, you should disable that. Also check to see that the split nodes with minimum number of instances possible.

Now that I an considering better, there are cases when full accuracy is impossible. Think for example a simple toy data set with 10 instances. Suppose that all instances are the same for input variables, but for target variable 5 are positive and 5 are negative. Either way you take a decision for prediction, the accuracy on training data would be 0.5. In this case you simply do not have enough data to discriminate on.


Merge two datasets based on date ranges. R


An RNN might work for this :

Lines have a "sequence of points"
Network should "learn" that points tend to shift towards left (as Y increases) and should be able to apply similar shift for prediction of X for a given Y

An example : Prediction of points for a Sin Wave


Difference between train, test split before preprocessing and after preprocessing

You should absolutely adopt the first scenario. That's because the transformers that you use have some parameters (e.g. mean and standard deviation in case of standard scalar) and this parameters are learned from data like the parameters of your machine learning model. As you know, you should not use the validation and test data for learning the model parameters and for the same reason, you should not use them for learning the transformer parameters. As a result, you should just use the training samples for fitting your transformer parameters if you want to try a practical machine learning scenario.


Mathematics: Writing down a three-class classifier confusion matrix

No, you did it incorrectly, you need to switch rows with columns (transpose). Next time, please check if a similar question has not been asked before. For the explanation, see https://stats.stackexchange.com/questions/318780/how-to-understand-confusion-matrix-for-3x3 .


As far as I know, currently there are not that many projects and products that allow you to perform serious machine learning (ML) work from within Excel.

However, the situation seems to be changing rapidly due to active Microsoft's efforts in popularizing its ML cloud platform Azure ML (along with ML Studio). The recent acquisition of R-focused company Revolution Analytics by Microsoft (which appears to me as more of acqui-hiring to a large extent) is an example of the company's aggressive data science market strategy.

In regard to ML toolkits for Excel, as a confirmation that we should expect most Excel-enabled ML projects and products to be Azure ML-focused, consider the following two projects (the latter is an open source):

Excel DataScope (Microsoft Research): https://www.microsoft.com/en-us/research/video/excel-datascope-overview/
Azure ML Excel Add-In (seems to be Microsoft sponsored): https://azuremlexcel.codeplex.com


Advise resources on un-supervised learning

I would recommend going through Towardsdatascience and Medium articles. They can explain the methods in a very simple and light way by summarizing the theory and also inlcuding the practical side of things.
for example: https://towardsdatascience.com/introduction-to-unsupervised-learning-8f1b189e9050
If you want to dive more into the maths the famous ESLII (Elements of Statiscal Learning) https://web.stanford.edu/~hastie/Papers/ESLII.pdf and Pattern Recognition  of Bishop (http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf) are the way to go. These two are the "bible" of Machine Learning


What would be a good loss function to penalize big differences and reward small ones, but not in a linear way?

Square loss (MSE or SSE) does this. Let $y_i$ be an actual value and $\hat{y}_i$ be its estimated value (prediction).

$$SSE = \sum (y_i -\hat{y}_i)^2$$

$$MSE=\dfrac{SSE}{n}$$

Except for numerical issues of doing math on a computer, these are optimized at the same parameter values of your neural network.

The squaring is critical. If a prediction is off by 1 unit, it incurs one unit of loss. If the prediction is off by 2, instead of incurring 2 units of loss, there are 4 units of loss—much worse than being off by 1. If the prediction is off by 3, wow—9 units of loss!

(If you look at statistics literature or some Cross Validated posts, you may see $n-p$ in the denominator of MSE, where $p$ is the number of parameters in the regression equation. This does not change the optimal value but does have some advantages in linear regression, chiefly that it is an unbiased estimate of the error variance under common assumptions for linear regression that you are unlikely to make in a neural network problem.)


Assuming that the "best" model is the model that makes the most accurate out-of-training sample predictions, there are two types of measures that you can use. R squared will only increase as you add more variables hence it is not a suitable measure.

There are two common approaches:

(1) Indirectly estimate test error by making an adjustment to the training error (e.g. Adjusted Rsquared / BIC/ AIC). In your case, the simple linear regression performed slightly better in terms of Adj Rsquared.

(2) Directly estimate the test error using either a validation set or cross-validation.


What is Word2Vec?

Word2Vec is an implementation of word embedding techniques.

Word embedding tries to represent relationships that may exist between the individual words (those contained in your processing texts) by giving them each a vector with same predefined dimension. In this vector space words that share common contexts may be located closer. How to assign the words into the vector space? This work has often been done through neural network training.

What problems can it solve?

The main problem solved by Word2Vec (or word embedding) is that it creates a way to represent relationships between the words in your processing texts, as opposed to just treat the words as individual symbols. This makes any following data mining or machine learning more effective.

Besides Word2Vec, you can also use many other pre-trained embeddings like GloVe. If the dataset is large, it is better to train your own word embedding which will get a better performance than using pre-trained embeddings.


predict the probability of buying a product

Seems like Naive Bayes of e1071 package could help.

Given the model product_id ~ comp_id + area_id it will construct conditional probabilities that may reveal some pattern in your data. For specifics, however, some sample data is nescessary.


How to compare supervised learning algorithm and it's technique ensemble learning algorithm?

TL;DR

Since both SVM and Random Forest are supervised algorithms, you can compare the two like you would compare any other two supervised algorithms.

The fact that a Random Forest is an ensemble classifier doesn't really matter as long as you treat all trees in the forest as a single model.

Comparing two supervised algorithms

The simplest way to compare supervised algorithms is with a train/test split:

Split all your data into two sets, namely a training and a testing set (a common ratio is 0.8/0.2).
Train both models independently with the data from the training set.
Use your models to predict the data from the testing set.
Give a score to the predictions by comparing what the model predicted vs the true value from the testing set. If you have a classification problem, you could use the F1 score. If you have a regression problem, you could use the R-square score.
Pick the model with the best score.

Other ways of comparing two supervised algorithms

Instead of a train/test split, you could look into cross-validation.
Instead of a random train/test split, you could look into stratified or time splits.
Instead of comparing two different algorithms, you could compare the same algorithm against itself but with different hyper-parameters (i.e. Hyper-parameter optimization).
Instead of F1 score or R-squared you could use a metric that best fits your business case.


Why is orthogonal weights initialization so important for PPO?

See this paper's (Exact solutions to the nonlinear dynamics of learning in
deep linear neural networks) result:

Moreover, we introduce a mathematical condition for faithful backpropagation of error signals, namely dynamical isometry, and show, surprisingly that random scaled Gaussian
initializations cannot achieve this condition despite their norm-preserving nature, while greedy pre-training
and random orthogonal initialization can, thereby achieving depth independent learning times. Finally, we
show that the property of dynamical isometry survives to good approximation even in extremely deep nonlinear random orthogonal networks operating just beyond the edge of chaos.

I think this is an answer to your question.


LSTM Validation MSE always lower than Train MSE

One important thing to start with is to check that your targets are in [-1, 1] range because you have a 'tanh' as output function. You should also analyze the distribution of your targets (True class).

Then, one of the first steps that I would recommend is to try to overfit your model. Make a model 'complex enough' and overfit small proportion of your initial data set (20% maybe?).

If your model does not overfit, there are two possibilities:

Model architecture not suited for this task (maybe lstm) or model not complex enough
Your data is not adapted for prediction task.

If your model overfits (Good sign):

Try to increase the complexity of your model when increasing the size of the samples (training set). Do not use any dropout or regularization and increase the number of layers instead of the number of units/layer.


I would recommend to use adaptive threshold rather than global threshold. Adaptive threshold deals better with uneven illumination. I have modified your code accordingly. You may want to experiment further with the value of block size and C value.

I have also used RETR_CCOMP and only show contours which either have a parent (index 3 != -1) or don't have any children (index 2 == -1).

I don't guarantee that this will work for other images but it may be a good starting point for your further experimentation.

import cv2
import numpy as np
import imutils

image = cv2.imread('front2.png', cv2.IMREAD_COLOR)

# Convert to grayscale
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
h, w = gray.shape
s = int(w / 8)
gray = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY_INV, s, 7.0)

# Downsize image (by factor 4) to speed up morphological operations
gray = cv2.resize(gray, dsize=(0, 0), fx=0.25, fy=0.25)
cv2.imshow("Gray resized", gray)
cv2.imwrite('Gray_resized.png', gray)

# Morphological opening: Get rid of the stuff at the top of the ellipse
gray = cv2.morphologyEx(gray, cv2.MORPH_OPEN, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5)))
cv2.imshow("Gray removed noise", gray)
cv2.imwrite('Gray_removed_noise.png', gray)

# Resize image to original size
gray = cv2.resize(gray, dsize=(image.shape[1], image.shape[0]))

# Find contours
cnts, hier = cv2.findContours(gray, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_NONE)

# Draw found contours in input image
image = cv2.drawContours(image, cnts, -1, (0, 0, 255), 2)

for i, cont in enumerate(cnts):
    h = hier[0, i, :]
    print(h)
    if h[3] != -1:
        elps = cv2.fitEllipse(cnts[i])
    elif h[2] == -1:
        elps = cv2.fitEllipse(cnts[i])
    cv2.ellipse(image, elps, (0, 255, 0), 2)

# Downsize image
out_image = cv2.resize(image, dsize=(0, 0), fx=0.25, fy=0.25)
cv2.imshow("Output image", out_image)
cv2.imwrite('Output_image.png', out_image)
cv2.waitKey(0)
cv2.destroyAllWindows()


Right way to Fine Tune - Train a fully connected layer as a separate step

First of all, you can do this in automatic way by setting the layers to be trainable or not. By selecting this hyper parameter to be false you freez the layer. we know that the first few layers are features extraction and the assumption when you perform fine tuning that the original problem and your problem are correlated, thus the features should be also the same.
What is the right thing to do?
It is very difficult to find a direct and absolute answer, it depends mainly on the original problem and the new one. In fact, I prefer to fine tune the whole network using different learning rates.
You can use smaller learning rate for the convolution layers and larger rate with the fully connected layers.


Because cost_complexity_pruning_path refits the tree model on the data you provide before doing the pruning (source), you need to preprocess the data first.  So this should do it:
X_preproc = final_pipe[:-1].transform(X_train)
path = final_pipe.steps[-1][1].cost_complexity_pruning_path(X_preproc, y_train)


One mistake I see is the fact that you are using the learning rate twice, one when calculating the partial derivatives and once when updating b0 and b1. The learning rate should only be used when updating b0 and b1 and is not used in calculating the partial derivates to b0 and b1.


Multiclass Classification with large number of categories

You might have more luck with a Naive Bayes Classifier.  It can handle a large number of target classes, and is relatively fast to train, since you largely just calculate a bunch of univariate stats to plug into at prediction time.  It won't capture fancy interactions as much as a random forest though, so if you are concerned about "they only buy shoelaces if they bought shoes but NOT shoeshine" vs "they often buy shoelaces if they've bought shoes" then it may disappoint.  You may also want to incorporate a time component, but I'm not sure what you're doing.

https://en.wikipedia.org/wiki/Association_rule_learning may also be relevant.


Extract Pattern using Short Text Processing

Problem 1: "Tactic Titles" Are Really TEXT

i.e. they are not standard "tags". I say this because they seem so structured and tag-like. For instance in text analysis using NLP techniques you mostly deal with texts in which "Review Comments" and "Reviewing Comments" and "Comment Review" are supposed to be considered the same.

Actually the way you already tried is a valid way to do it.

Another idea could be using information theoretic approaches. It means using something like Tf-IDF approach but this time taking classes into account. Here you are going to look for words/n-grams which appear a lot in desired class (High) and not a lot in the other class. A simple formulation could be:

$$Score_{w_i} = \frac{N_{high}}{N_{low}+1}$$
where $N_{high}$ is the number of times the word $w_i$ appeared in class High and $N_{low}$ is the number of times the word $w_i$ appeared in class Low. The constant $1$ is just a smoother to avoid a zero denominator.

If you use it be careful about overfitting as you are learning features in a very implicit and hand-crafted way. To avoid overfitting you may prune your found words based on a min_count or max_count limitation.

Of course this is a simple score and you can improve it by modification (e.g. i did not consider the normalization which is better to be considered, etc.)

For prediction you can also simply use naive Bayes. Count words/phrases based on Bayesian approach (just count the frequency of words happening in each class) and normalize them to probabilities. Now you have most likely data given classes and the probability of classes. A new data comes and $P(C|D)=P(D|C)\times P(C)$ is calculated easily.

Problem 2: The "Tactic Titles" Are Just Fixed Tags (Maybe irrelevant to score)

Means that semantics are not correlated with classes i.e. two titles with similar terms or meanings might be in two different classes.

Then it's not a learning problem. Either you have the label of the new data in the dictionary so you just extract the class, or you don't have it so you don't know about it!

How would you want to know that "Partner Launch" is low priority but "Review Comments" is high if you have no meta-information form the organization behind it? How can you predict that based on this most probably "Write Comments" is High or Low? .If you argue that Machine Learning is supposed to learn that latent variable behind, then I would say yes! but please note that maybe that latent phenomenon is not correlated with Text. For instance if you get meta-data about which department is doing any of this tasks, you may end up with a more correlated/causal set of features. The long story short: Maybe the information is not in TEXT but some other aspects.


Bootstrap clarification for datasets with multiple columns

Here you have an example on how to carry out a bootstrap selection from any given set of data, based on (as you suggested) using loops and randomly selecting indexes to recreate a simulated new bag from the whole population of M&Ms. The criteria should be pretty the same for different kind of data, since what you actually do is simulate the selection of samples from a population.
It could be something like:
-bags you have from different factories:
bag_1 = ['blue']*15 + ['red']*15 + ['green']*10 + ['orange']10
bag_2 = ['blue']*14 + ['red']*16 + ['green']*9 + ['orange']*11
bag_3 = ['blue']*16 + ['red']*15 + ['green']*11 + ['orange']*8

-color distributions per bag:
import seaborn as sns
import pandas as pd

bags_df = pd.DataFrame({'bag': [1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3],
                     'color': ['blue','red', 'green', 'orange', 'blue','red',
                     'green', 'orange', 'blue','red', 'green', 'orange'],
                     'count': [15, 15, 10, 10, 14, 16, 9, 11, 16, 15, 11, 8]})

ax = sns.barplot(x="bag", y="count", hue="color", data=bags_df)

-now your population would be all the bags together inb a big bag:
super_bag = bag_1+bag_2+bag_3

-and here is where you do the actual bootstrap:
### GENERATE 10 BOOTSTRAPED BAGS
from numpy.random import seed, rand, randint, randn

# seed random number generator
seed(1)

bootstrap_bags_dict = {}
for i in range(10):
    bootstrap_bags_dict['sub_bag_'+str(i)] = []
    # bootstrap sample
    boots_indices = randint(0, 50, 25)
    for idx in boots_indices:
        bootstrap_bags_dict['sub_bag_'+str(i)].append(super_bag[idx])

and you get a dictionary with 10 bags like:

In case you want to see if there is any difference in, for instace, the red color distribution coming from factory 1 or 2, I would:

split the dataset by factory1 and factory2, get the red M&Ms count for each one
then shuffle all the M&Ms in a big bag as coming from the same factory; here, your initial hypothesis is that there is no difference in receiving red M&Ms from both factories
as explained here , you can compare by permutation and hypothesis testing if the difference between bootstrapped samples from your big bag (as considering both bags from the same factory) is statistically different compared to the difference between the 2 initial factories bags (it might sound hard to know, some code about it can be provided)


Decision Trees Nodes vs Leaves Definition

Leaf nodes are the nodes of the tree that have no additional nodes coming off them. They don't split the data any further; they simply give a classification for examples that end up in that node. In your example tree diagram, the nodes that say 'Large', 'Medium' or 'Small' are leaf nodes. The other nodes in the tree are interchangeably called split nodes, decision nodes or internal nodes.

In gradient boosting algorithms, a number of decision trees are grown. Each tree is grown until some stopping criteria is met. One kind of stopping criteria is the maximum number of leaves in the tree. At each stage of growing a decision tree, a leaf node is turned into a split node by creating a yes/no question (we call this a binary split), and two new leaf nodes are created which correspond to each side of the split. Once the total number of leaf nodes in the tree reaches the limiting value, the tree building algorithm stops and begins building the next tree.


I'm not entirely sure but it looks like sequence labeling might be what you need:

Sequences of varying length
Supervised: you would need to train a model with a sample of sequences annotated with a label at every step (not sure that this is your use case?)
Can handle any number of features

Conditional Random Fields is the state of the art method, there are many libraries
available.


How to make a numpy ndarray from several numpy arrays

If your purpose is to deal with some data analysis from your arrays and have a nice view of it, try this using pandas:

import pandas as pd

df = pd.DataFrame({'distA' : np.array([0.2, 0.3, 0.4]),
                   'distB' : np.array([0.3, 0.4, 0.5])})

df

In the case you want to make computation with vectors, matrixes, tensors, etc .. NumPy is probably a good option. I didn't understand exactly what you've asked, though. Can you explain me what you're asking?


how to export the data edited to a new excel file

You could simply use:

df2.to_excel("output.xlsx")

I suggest you take some time to look at the various functionalities pandas offers. Will be great help to you in future!


Math of Logistic regression cost function

The c (small one) term is bias or intercept added to the model. This is similar to intercept we add in case of linear regression. The library allows you to set bias term to zero too.
When set to multinomial model, the cost function will try to minimize cross-entropy loss. Hard for me to write down the equation here, but I always go back to this useful explanation of how cross entropy loss works and how it is minimized. In other words, a multinomial regression will work like a single layer neural network with logistic activation function.


Rather than discarding the dominant feature (which will discard information), try reducing the number of features randomly selected when making each partition. In scikit's syntax this is max_features (mtry in R's randomForest). By default this is set to compare the square root of all features (http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html), instead try much smaller (perhaps even 2 aka decision stump). You'll probably also need many more trees vs. higher settings. This will allow you to extract information from more subtle features without losing all that great information provided by the dominant feature.


what could this mean if your "elbow curve" looks like this?

First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:

Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).

Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).

The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).


Yes, this is correct, the Bayes Classifier is in fact theoretically optimal too. You can quite easily verify this, this is a link to the proof for the answer (https://cs.stackexchange.com/questions/72295/showing-that-bayes-classifier-is-optimal)


How much data do you need to build a classifier?

It's impossible to answer this question in general, because the answer strongly depends on the content of the data. More precisely it depends if the relations between the features and the target class in the training data are sufficient for the parameters of the model to be estimated as accurately as possible. In the most simple cases a few instances might be enough, for example a linear regression model with one single feature would need only two "perfect" instances. Practically the following factors have a strong impact on the number of instances needed:

the number of parameters to estimate and the complexity of the model: a more fine-grained model needs more detailed information, i.e. more instances
the number of classes, because there are more possible combinations for the model to learn and because it usually implies a higher number of parameters as well (more complex model)
the number of features, for the same reason
the amount of noise in the data, because finding the general patterns is more difficult if there are lots of inconsistencies/errors in the data, so statistically more instances are needed to distinguish the effect of chance from the real patterns.

So the only way to check how much data is needed for a particular task and dataset is to do an ablation study, i.e. a series of experiments in which a model is trained every time with a different number of instances. For example if one has 1000 instances, they could try to train a model with 100, 200, 300,...,1000 instances. Then by plotting the performance of every model one can observe whether the curve becomes stable at some point: if yes, this point shows how many instances are needed. If not (i.e. the performance keeps increasing significantly), then more than 1000 instances are needed.

I think the size may affect the accuracy/precision of a classifier, but I am not sure 100% of that.

Definitely.


Sentiment Analysis Using Neural Networks with Python

Here's how to create such a network. Convert your text into numbers using some tokenizer (Tensorflow's Keras has one). Break it off into equal blocks of some size x. Feed these blocks into an Embedding layer followed by LSTM or GRU (or their bi-directional variants) and finally have a three way classification (positive, neutral or negative) or for regression just a single sigmoid.
One trick though if you have class imbalance, train with full data in the beginning (some initial epochs). Then, for the further iterations, drop the neutral data, just train on positive and negative examples (since, you mostly have a lot of neutral data in these cases).


Calculate implicit rating from streaming behaviour for Recommendation Engine

Create a histogram of the watch_time_ms. If you are lucky - you may see a bi-modal distribution (i.e two peaks). The higher/lower peak could be interpreted as interested / uninterested behavior respectively. Then your threshold could lie somewhere in the valley between your two peaks in the histogram.
If your videos are variable length - you may also want to try normalizing the watch_video_ms by dividing with the total_video_length_ms before reviewing the histogram. You then end up with smooth values between 0 - 1 for each video.
If there is no obvious visible pattern in the histogram - then choose a percentile to divide between interested / uninterested views. For e.g. the top 25 percentile of views represent are the 1's whereas the remainder are 0's. Make sure you choose the percentile appropriately based on the problem domain / objective.


How to determine the best way to replace empty records in the dataset?

Replacing missing values with mean, median and mode is feasible when the number of missing data is not small enough. In your cases, only a handful of data are missing. We can use the rest of the data to come up with estimates better than mean, median and mode. Let's see the possible approaches:

Scenario 1: using the reading for breakfast and lunch for day 31, try to find the nearest neighbours in day in terms of breakfast and lunch values. You can use this nearest neighbour day's dinner value as an estimate of the 31st day's dinner.

Improvement to above approach:

1) Instead of just choosing the single nearest neighbour, you may also use a kNN model.

2) If you have the dates or can figure out the day of the week (like its Sunday, Mon, Tues etc) then instead choosing the nearest neighbour among all the days, you can choose only among the same day of the week. (Nonetheless, you can figure it out by simply subtract or adding 7 to the day you want to find the values of.) Say 31st was a  Sunday, then its likely that the routine would be similar to others Sundays on the data rather than a working day like Monday.

Scenario 2: You can use the values of the same days of the week to estimate the missing values. Simplest ones would be to take a median. A more powerful estimate would be by using a small neural network to estimate the next day's values given the values of the past few days (here, the number of days to consider as input would be a hyperparameter).


Feature Selection algorithm/library for CRF

Since sklearn-crfsuite provides a scikit-learn compatible estimator, any scikit-learn feature selection class that takes an estimator could work.


Go into the other link from home - to the cloudera manager.

From there, you'll see Hue can be restarted, but there is probably an alert that needs to be resolved in there first.

If I remember right there's some initial configuration that needs to be done on the quickstart VM that's spelled out as soon as you log into the manager application.


TensorFlow: how to print out all elements in a tensor

Basically what you want to do is showing the full numpy array in jupyter notebook. Use the following code:

import numpy
numpy.set_printoptions(threshold=numpy.nan)

Then run your code and see the entire activation map.


You were very close. Just switch the first two input arguments.

> aggregate(directLaborNov18['Actual_work'], directLaborNov18[c('Employee')],  sum)
   Employee Actual_work
1   Joe Smo          20
2 Sam Adams          15


When forecasting time series, how does one incorporate the test data back into the model after training?

I agree with what Emre has commented on your question.

If you have enough data, I would try cross-validating your model by training it on different time sections of your data. For example, train your model on Jan-March data and then make predictions on April's data. Then you could train the model on Feb-April data, and predict May's data. Essentially creating a few training and testing sets to see how well your model generalizes. Once you feel you are getting good results, you could train your final "production" model on all of your available data and then keep an eye on how it does when real data comes in and make adjustments on what you see.

I've used a similar approach in the past on accelerometer data to classify exercises from a specific workout, and it worked well. Maybe something similar will work for you.


Which are the latest Optimization techniques in artificial intelligence?

You are asking for suggestions of a technique which will give the best results in comparison to other methods, that's difficult to say a priori as it will depend on your particular features and dataset and will be guided by trial and error and comparison of different algorithms (this is also related to the No Free Lunch Theorem, see some discussion here). However, there are certainly many optimization techniques advances in Artificial Intelligence. In this answer I'll cover the ones I'm aware of.

Recently there have been several nice advances where new perspectives in optimization models have been developed classic problems Machine Learning. These works have been published by renowned Operations Research academic Dimitris Bertsimas. See for example this link. For example, I find the following publications from the list to be very interesting and promising (you can find free PDF versions at the last link):

Bertsimas, D., King, A., & Mazumder, R. (2016). Best subset selection via a modern optimization lens. The annals of statistics, 44(2), 813-852.

In this work, the authors present a Mixed Integer Optimization approach for solving the best subset selection problem (i.e. to choose a subset of features out of a larger set in linear regression given a collection of observations). Typically the Lasso technique is used, but in several computational experiments it was outperformed by this method. Moreover, this new theory develops criteria for proving that the adquired solution is indeed optimal (or near-optimal), which was not possible using heuristic methods.

Bertsimas, D., & Dunn, J. (2017). Optimal classification trees. Machine Learning, 106(7), 1039-1082.

In this case I cite the abstract:

  State-of-the-art decision tree methods apply heuristics recursively to create each
  split in isolation, which may not capture well the underlying characteristics of the dataset. The optimal decision tree problem attempts to resolve this by creating the entire decision tree at once to achieve global optimality. In the last 25 years, algorithmic advances in integer optimization coupled with hardware improvements have resulted in an astonishing 800 billion factor speedup in mixed-integer optimization (MIO). Motivated by this speedup, we present optimal classification trees, a novel formulation of the decision tree problem using modern MIO techniques that yields the optimal decision tree for axes-aligned splits. We also show the richness of this MIO formulation by adapting it to give optimal classification trees with hyperplanes
  that generates optimal decision trees with multivariate splits. (...) We comprehensively benchmark these methods on a sample of 53 datasets from the UCI machine learning repository. We establish that these MIO methods are practically solvable on real-world datasets with sizes in the 1000s, and give average absolute improvements in out-of-sample accuracy over CART of 1–2 and 3–5% for the univariate and multivariate cases, respectively. Furthermore, we identify that optimal classification trees are likely to outperform CART by 1.2–1.3% in situations where the CART accuracy is high and we have sufficient training data, while the multivariate version outperforms CART by 4–7% when the CART accuracy or dimension of the dataset is low.

In their computational experiments the model has even outperformed Random Forest and XGBoost, as can be seen on slide 29/41 of this presentation Bertsimas gave earlier this year. This is quite impressive, as these algorithms are among the most frequently used by winning solutions in Kaggle (see for example this link and this one).

Bertsimas, D., & King, A. (2017). Logistic regression: From art to science. Statistical Science, 32(3), 367-384.

This case is similar to the first one, but for logistic regression instead of decision trees. Again they achieved new algorithms for training a classic Machine Learning model, getting high quality solutions as well as near-optimality guarantees.

Besides this line of work, large scale convex optimization is another area where lots of attention have been drawn. Big data and machine learning/deep learning applications motivate the development of new algorithms and theory to achieve scalable and efficient solutions in this new contetxt. See for example Bottou, L., Curtis, F. E., & Nocedal, J. (2018). Optimization methods for large-scale machine learning. SIAM Review, 60(2), 223-311.

All the advances I mentioned are very recent, so don't expect to have access to available and scalable libraries to use them out of the box. Therefore, the usefulness of them for your particular project will depend on (1) the scope of your project (research or more short term objective?), (2) the size of your datasets, and (3) the available time you have for working on it and delivering a solution.


How to backup a Caffe (DL framework) project? The Caffe folder (downloaded from GitHub) can't be copied or compressed

Based on the comments, the problem is due to wrong file permissions for certain files. Assigning read permissions to the full directory should solve it:

chmod -R u+r <caffe directory>


Effect of embedding dimension in performance of embedding algorithms

You are probably just overfitting.

When you increase the number of parameters that are being trained in your model, the likelihood of them learning some aspect of the training data that doesn't generalize to unseen data increases. So there should be some sweet spot, where the model is able to express the underlying structure in the data sufficiently, without putting too much emphasis on those aspects that are specific to the training data.