I have played around with it and managed to get a working configuration.
Necessary changes:

axis=1 for BatchNormalization. Normally, this should be used when data_format="channels_first", but it worked for me despite TensorFlow backend
Standardize the input:

X_train = X_train * 2.0 - 1.0; X_test = X_test * 2.0 - 1.0
Avoid batchnorm after the last convolution in res_unit.

The working solution looks like this:

# build a model
def res_unit(x, prefix=''):
    x_shortcut = x

    x = Conv2D(16, (1, 1), padding='same', name='{}/{}'.format(prefix, 'conv1'))(x)
    x = BatchNormalization(name='{}/{}/bn'.format(prefix, 'conv1'), axis=1)(x)
    x = Activation(activation='relu', name='{}/{}/relu'.format(prefix, 'conv1'))(x)

    x = Conv2D(16, (3, 3), padding='same', name='{}/{}'.format(prefix, 'conv2'))(x)
    x = BatchNormalization(name='{}/{}/bn'.format(prefix, 'conv2'), axis=1)(x)
    x = Activation(activation='relu', name='{}/{}/relu'.format(prefix, 'conv2'))(x)

    x = Conv2D(32, (1, 1), padding='same', name='{}/{}'.format(prefix, 'conv3'))(x)
    x = Add(name='{}/add'.format(prefix))([x_shortcut, x])
    x = Activation('relu', name='{}/relu'.format(prefix))(x)

    return x

def main():
    (X_train, y_train), (X_test, y_test) = load_data()
    X_train = X_train.astype('float32') / 255
    X_test = X_test.astype('float32') / 255
    X_train = X_train * 2.0 - 1.0
    X_test = X_test * 2.0 - 1.0
    y_train = keras_utils.to_categorical(y_train)
    y_test = keras_utils.to_categorical(y_test)

    print(X_train.min())
    print(X_train.max())

    X_input = Input((X_train.shape[1:]))

    X = Conv2D(32, (3, 3), padding='same', name='conv0')(X_input)
    X = BatchNormalization(name='conv0/bn', axis=1)(X)
    X = Activation('relu', name='conv0/relu')(X)

    X = res_unit(X, prefix='block1')
    X = res_unit(X, prefix='block2')
    X = res_unit(X, prefix='block3')

    X = Flatten()(X)
    X = Dense(32)(X)
    X = Activation('relu')(X)
    X = Dense(10)(X)

    X = Activation('softmax', name='softmax')(X)

    model = Model(inputs=X_input, outputs=X)
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

I've got 59.07% validation accuracy with it. However, sometimes training does not converge, it feels like it is very sensitive to the random init.
Still don't know why making batchnorm work here is so hard...


Keras/Tensorflow: model.predict() returns a list. How do I match the output with my class names?

Model prediction output is a bunch of probabilities. In order to get category name you need use following snippet. It calculates the argmax of predicions and give it to CLASSES list:
print(CLASSES[np.argmax(predictions)])


Reinforcement learning: negative reward (punish) illegal actions?

I think that you should specify better what an illegal action is. Suppose to me in a highway with a self-driving car. I don't which are your legal limits values but suppose that they are between 20mph and 40mph. Of course, the car itself could drive at less than 20mph and let us suppose that its maximum possible velocity is 60mph. If your self-driving car is driving at less of 20mph or between 40mph and 60mph you should give a negative reward for each time-step it is outside legal limits. Instead, if your self-driving car is at more than 60mph you have a problem in your environment because 60mph is a physical limitation and should be handled by the environment. An easy solution would be to clip the action between 0mph and 60mph and give negative reward if it's in the range 0mph to 20mph or if it is in the range 40mph to 60mph.


A precision-recall curve (as well as a ROC curve) is made from the different thresholds which can be used to separate predictions between positive and negative instances. These thresholds naturally depend on the possible values predicted by the classifier.
For example, if the classifier predicts only 3 different values $a<b<c$, then the threshold $t$ can only be in these positions:

$t<a$
$a<t<b$
$b<t<c$
$c<t$

Each of these positions will correspond to one pair of values for precision and recall, so exactly one point in the curve. Therefore the shape of the curve (how smooth it is) depends on how many different values the classifier predicts.
Thus a stairs-shaped curve shows that the classifier predicts only a few different values. There can be various reasons for that, but since you mention ensemble learning I guess that the final predicted value is based on the proportion of individual learners predicting the positive class, right? If yes that would explain it: if there are $N$ individual learners then the final predicted value can only be $0,1/N,2/N,...,N$. This means a maximum of $N+1$ different values and therefore a maximum of $N+1$ points on the curve.


face landmark detection cnn loss not converging in tensorflow

Use a standard pretrained network like ResNet, Effnet or even VGG16 for feature extraction and put a regression layer in the end.
Now, talking about your network. FIrst of all, the image size is not square, make it! Don't use even valued kernels (3 and 5 are mostly used). Number of filters increase with depth, yours don't follow a pattern.
Lastly, 100 images are nothing for these behemoths, give it more data.
Better stick to a pretrained model.


By increasing number of features in convolutional layers, you are actually increasing the number of learnable parameters which were initialized to random values. So, randomness increases and that's why the cross entropy loss increases.Hope this is satisfiable justification to your question.

Ex, Suppose convolution layer with :

filter_size = (3, 3) and number_feature = 10 => number_parameters = 3*3*10 = 90
filter_size = (3, 3) and number_feature = 64 => number_parameters = 3*3*64 = 576


How can I preprocess text to feed into a SVM?

X needs to be the features for your Model and Y needs to be a target variable.
As you mentioned, you are using a IMDB dateset so, all the features which you want your model to use will be stored in X variable whereas the 'LABEL' columns will be stored in the Y variable.
Instead try this code:
X = vectorizer.fit_transform(df_train['text'])
Y = vectorizer.transform(df_train['label'])

classifier_linear = svm.SVC(kernel='linear')
classifier_linear.fit(X, Y)

Another thing you did wrong here was that,
you passed the target variable of you testing data.
You train your model from your training data and if you are confident enough on your model then you test it using the testing data.
Concepts which can help you: Cross-Validation, train_test_split.


How do i use the Gaussian function with a Naive Bayes Classifier?

You got the Bayes theorem wrong. There should be a plus in the denominator:

(child gauss x child prob)/(child gauss x child prob + adult gauss x adult prob)

(adult gauss x adult prob)/(child gauss x child prob + adult gauss x adult prob)


What are the most popular data science application use cases for consumer web companies

Satisfaction is a huge one that I run into a lot.  Huge referring to importance/difficulty/complexity.

The bottom line is that for very large services (search engines, facebook, linkedin, etc...) your users are simply a collection of log lines.  You have little ability to solicit feed back from them (not a hard and fast rule necessarily).  So you have to infer their positive or negative feedback most of the time.

This means finding ways, even outside of predictive modelling, to truly tell, from a collection of log lines, whether or not someone actually liked something they experienced.  This simple act is even more fundamental (in my biased opinion) than a/b testing since you're talking about metrics you will eventually track on a test scorecard.

Once you have a handle on good SAT metrics then you can start making predictive models and experimenting.  But even deciding what piece of log instrumentation can tell you about SAT is non-trivial (and often changes).


What is difference between detrend and normalization?

You detrend data in order to get rid of the linear trend in your data, which might cause spurious regression, misleading evidence that there is some relation between variables.

Normalization means adjusting values measured on different scales to a notionally common scale.
Example you give: Standard score $\frac{X - \mu}{\sigma}$ is just one possible method of normalization . It allows you to see where value lies in comparison to mean.

In my experience you would only detrend data in order to create time series model. On the other hand, normalization is frequently used to compare previously not comparable statistics or detect anomalies (in case of standard score). As a result, usage really depends on the use case and you rarely use both at the same time.

Wikipedia is good enough source to clarify this unambiguity.


A norm is a concept in linear algebra which assigns a size to a vector. Many different norms exist you can read up on their many uses here.

In this paper, they are training a recurrent neural network. These are used for time series data. The assumption is a hidden state should be similar for successive time inputs $t$ and $t-1$. Thus, the author proposes an additional constraint based on the relative size of the successive hidden state vectors as

$||h_{t}||_2 - ||h_{t-1}||_2$.

The more different the norm of these successive states, the greater the cost will be.


KNN is instance based so it will store all training instances in memory. Since you are using images this will add up quickly. KNN on untransformed images might not perform that well anyway, you could look into filter banks to transform your images to a bag-of-word-representation (which is smaller and more invariant).

However if it is accuracy you are aiming for I would recommend skipping all that (it is very 2012 anyway) in favor of using deep learning, fi: construct an auto-encoder and determine similarity on the encoded representation of an image (which could in turn be done using knn btw).


Increasing the max_depth and the n_estimators might help. Especially the max_depth (2 is way too low). Take a look at that.


How to build a database of image data for machine learning?

There are several approaches to this as you need both the input (images) and if your problem is a classification one, you need to reliably store the labels. You might also have some additional information about the images that could be useful for your problem:

you can store the images in such a way that all information is contained in the permanent store (for instance folder names with the labels that you want to learn and all the images of a given class within that folder). Keras has a method that allows you to create a dataset from a directory tf.keras.preprocessing.image_dataset_from_directory.

another way (which I prefer) is to store in a (SQL) database all of the metadata (label, image url in a table for instance). This is more flexible because you can easily change a label, add a new category without having to move images around. This also allows you to change the format and add additional data related to each image.


There is principal component analysis (PCA) in Spark's
Machine Learning Library (MLlib).


How to break down large SVM classification model?

I think that using SVMs for your model is the main problem.

SVMs (linear or otherwise) are originally intended for binary classification. There are various procedures for extending them to a multi-class problem. The most common methods involve transforming the problem into a set of binary classification problems. The most common strategy is using "One-Versus-All" classifiers (OVA classification), and choosing the class which classifies the test data with the greatest margin. Another strategy is to build a set of One-Versus-One classifiers, and to choose the class that is selected by the most classifiers. Both these strategies are computationally expensive both in training and in testing.
I know there are other, more complicated methods, but I don't think that any of them is intended for such a high number of classes.

The most trivial way would be to use a neural-network model for such a problem. If you want to use one of the more classic algorithms, perhaps using decision trees will work (or even k-nearest neighbors).


Graph isn't an attribute in TensorFlow? Very basic question

2 options.

reinstall and try again (dont ask)

option b) downgrade to versions 1.13.1 and lower (again dont ask)

ofcourse you can ask I am just lazy


What are the differences between biased and unbiased learners?

Biased in the context that you are speaking means, that your model overfits the training data and can not generalize well. It means your model performs very well on your training data, but can not do well on cross-validation and test data. It is customary to say that biased learners memorize the training data which is really true. Biased learners don't learn the data, they fit the data. For understanding the other usages of bias take a look at this question.

There is something that may be worth mentioning. You may have heard people saying that your model has a high-bias problem. It just means that your model can not learn the training data, whilst the biased learners overfits the training data, means fits the training data. The latter can not generalize well because it has fitted the training data, memorized it, the former can not generalize because it has not learnt even the training data so it has not learnt so much and can not generalize.


Fasttext error while loading wiki pre-trained data

You could use the load_model method instead of load_fasstext_format method. Also, you could FastText Library directly without really having to install gensim to do the same.


Does classification of a balanced data-set lead to any problem?

Actually, I guess it highly depends on the real data-set and its distribution. I guess the paper has referred to that is that on occasions that the distribution of each class varies, your model won't work well because of changing the distribution of each class. In cases like a disease prediction where the number of each class varies for different places, a model that is trained in the U.S won't work in African countries at all. The reason is that the distribution of classes has been changed. So in such cases that usually the negative and positive classes are not balanced in practice, balancing them will cause the problem of distribution changes. On these occasions, people usually use the real data-set which is not balanced and use F1 score for evaluation.


Machine Learning with intended missing values

The common case of missing values for which data is imputed or removed assumes that missing values appear randomly in the data, so the absence of value has no relevance to the task.
From your description, in your data the fact that a value is missing is significant by itself. So I'd say that yes, it makes sense in this case to represent this information as a categorical variable. Note that it can be represented as a special value for the score feature indeed, but it doesn't have to be the same variable.


How to convert Hindi/Telugu/Marathi text to vector for text classification problem?

Hindi text is in Unicode format and can be read as follows in Python:

separators = [u"।", u",", u"."]
text = open("indiantext.txt").read()
#This converts the encoded text to an internal unicode object, where
# all characters are properly recognized as an entity:
text = text.decode("utf-8")

#this breaks the text on the white spaces, yielding a list of words:
words = text.split()

counter = 1

output = ""
for word in words:
    #if the last char is a separator, and is joined to the word:
    if word[-1] in separators and len(word) > 1:
        #word up to the second to last char:
        output += word[:-1] + u"(%d) " % counter
        counter += 1
        #last char
        output += word[-1] +  u"(%d) " % counter
    else:
        output += word + u"(%d) " % counter
    counter += 1

print

It is not clear from the question as to what exactly you mean by pretrained model for converting into vector but assuming that you are asking about the language model for hindi, ULMFiT is something you must refer to


How to explain a relationship between Accuracy and F1 Score / F-Measure?

Saying that accuracy is measured to get how accurate the model performs, and F1 is how well the model performs

This doesn't mean anything, it's obviously too vague.
The first things to check in order to understand this relationship are the definitions of accuracy and F1-score.
Wikipedia has a good page which explains how different classification evaluation measures are related.
Observations on your results:

The accuracy and F1-score are almost identical everywhere. This suggests that your data is probably quite well balanced, i.e. the difference in the number of positive vs. negative instances isn't very big. Why? Because if the data was imbalanced then the model would over-predict the majority class, and this would cause the F1-score to be much lower than the accuracy: assuming the majority class is the negative class, the recall would be somewhat low but the accuracy could still be high because most instances (majority class) would be correctly predicted.
As a consequence, there's no insight to gain from analyzing the relationship between accuracy and F1-score since they're virtually identical. The small differences might be due to the geometrical mean between precision and recall. F1-score is more informative in case of imbalanced data, but this is not the case here.
The F1-score is calculated only on the training data. It would be more useful to calculate it on the validation data.
There's some serious overfitting happening especially with the high learning rates, but with the low learning rates the fact that difference between training and validation accuracy increases is also worrying. Maybe the model is too complex or there are not enough instances in the data. Ideally the two accuracy values should converge.


Is it better to augment both training and validation sets or just training set?

Is it better to augment data both training and validation sets or just
the training set in order to achieve the best accuracy possible on a
convolutional neural network?

Augmentation is always a good thing provided it represents the actual distribution of data. Augmenting your training set may give you enough samples to train high variance low bias models [as an increase in training set samples, which if are general enough, prevents overfitting]
Validation set augmentation gives you more samples to validate on so it's always good.
But there may be instances where the augmented data closely resembles the initial training set samples. This increases the number of certain samples data points compared to other samples. This may cause the model to overfit due to the biased nature of the dataset. So its always wise to make sure that the samples in your augmented set are not mere copies of your data. For eg: In images, instead of just rotating them, you can add in a random noise.


Fully endogenous models for predicting multivariate time series

I am still interested in alternatives that depart more fundamentally from VAR, but digging a little deeper, I found some interesting variations around VAR that I did not know about and could be very useful.

Of particular interest are regularized VAR models. Applications to macroeconomic forecasting are for example discussed here: https://arxiv.org/abs/1508.07497.

Even better, the paper comes with a fully funcitonal R-package: https://cran.r-project.org/web/packages/BigVAR/index.html, that is described in more details here: https://arxiv.org/abs/1702.07094.

Some of the regularized method in the package, like VARX-L, allows for exogenous variables. As I described in my answer, this is not what I am directly interested in. But VARX-L (and the BigVar package) only allow for exogenous variables and do not require the inclusion of such variables (the exogenous channel can be shut down with the model still being operational). Good to know I can start with a fully endogenous model and have to flexibility to add exogenous variables later if needed.


Advantages of CNN vs. LSTM for sequence data like text or log-files

The WaveNet paper is a good place to start for a discussion of Causal CNNs vs LSTMs for synthesis and classification.  In that paper they actually train the network to do both at the same time, for example.
For short tokenized sentences such as you describe, Transformers are probably the state-of-the-art though.  In fact, they have been used recently in place of CNNs even for image classification tasks.  So, you don't necessarily have to have an encoder portion at all.  And, I believe the general thought is that the Transformer is encapsulating the function of both RNNs and CNNs and is outperforming them both in benchmarked tasks.
So, any of these networks could probably do the job you describe, but I would tend to using a Transformer as my first go.  It is built for this kind of task and is quicker and easier to train than a RNN.  I do suspect though that you would get good results with a LSTM on such short sequences.  They can be difficult to train properly on very long data.


Do you have any real example of Data Science reports?

I think you can better go through various university thesis reports and data science related journal papers to get more details on "Including hypothesis, testing, reports, conclusions" of the above mentioned Data science related problems.

Check these links from Stanford university :

http://cs229.stanford.edu/projects2014.html

http://cs229.stanford.edu/projects2013.html

http://cs229.stanford.edu/projects2012.html


Re-training regression model on covid data

Your previous model won't be useful. In my opinion, you can not use it to make predictions in Covid Pandemic, since all bookings have had a huge decrease.
I would recommend build a new model with only data of the Covid Pandemic time. Then, you will have a more accurate model for these times. The only problem if you have less historical data, but for sure, is better have less data but useful, than huge dataset that is not representative of current times.
Be aware that you can probably add some useful features in this new model, for example, number of deaths for Covid per day of the country. For sure this feature will help you because if you have a huge number of deaths per day, probably you will have less booking, otherwise, if there are less deaths per day, there should be more bookings.


Reason for Training and test loss sudden increment after some epochs keras

This might be happening because of high learning rate.

The loss function is convex and the model needs to reach the minima. Learning rate defines the magnitude of step you take towards minima. If you take high learning rate, you might overshoot the minima, hence sudden increase in loss value.


Adding and multiplying higher values based on different columns of a dataframe

So, from what I understand from the problem, you want create a Score column, whereby, typically:
$Score = A \times B \times C \times D \times E$
If ID == VID and the value in QID for that entry is unique in the whole data frame, then $E = max(E)$.
For this, I would create additional columns which check for these conditions before making the score column. Therefore, I would recommend this:
import numpy as np
import pandas as pd

# unique QID
QID_counts = df.groupby("QID").size().reset_index()
QID_counts.columns = ["QID", "QID_count"]

df = pd.merge(left = df, right = QID_count, on = "QID")

# checking if IDs are equal to VID
df["ID_VID"] = df[df["ID"] == df["VID"]]
df["Unique_QID"] = df[df["QID_count"] == 1]

# checking if both conditions are met
df["Max E"] = df["ID_VID"] & df["Unique_QID"]
df["Max E"] = df["Max E"].astype(int)
# obtaining indices
max_E_idxs = df[df["Max E"] == 1].index

# updating E to suit conditions
df["Score_E"] = df["E"]
df.loc[max_E_idxs,"Score_E"] = np.max(df["E"])

# creating score
df["Score"] = df["A"] * df["B"] * df["C"] * df["D"] * df["Score_E"]
```


Why ML model produces different results despite random_state defined? And how to set global random seed for sklearn

Not every seed is the same.
Here is a definitive function that sets ALL of your seeds and you can expect complete reproducibility:

def seed_everything(seed=42):
    """"
    Seed everything.
    """
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True

You have to import torch, numpy etc.
UPDATE: How to set global randomseed for sklearn models:
Given that sklearn does not have its own global random seed but uses the numpy random seed we can set it globally with the above :
np.random.seed(seed)

Here is a little experiment for scipy library, analogous would be sklearn (generating random numbers-usually weights):
import numpy as np
from scipy.stats import norm
print('Without seed')
print(norm.rvs(100, size = 5))
print(norm.rvs(100, size = 5))

print('With the same seed')
np.random.seed(42)
print(norm.rvs(100, size = 5))
np.random.seed(42) # reset the random seed back to 42
print(norm.rvs(100, size = 5))

print('Without seed')
np.random.seed(None)
print(norm.rvs(100, size = 5))
print(norm.rvs(100, size = 5))

outputing and confirming

Without seed
[100.27042599 100.9258397  100.20903163  99.88255017  99.29165699]
[100.53127275 100.17750482  98.38604284 100.74109598 101.54287085]
With the same seed
**[101.36242188 101.13410818 102.36307449  99.74043318  98.83044407]**
**[101.36242188 101.13410818 102.36307449  99.74043318  98.83044407]**
Without seed
[101.2933838  100.52176902 101.38602156 100.72865231  99.02271004]
[100.19080241  99.11010957  99.51578106 101.56403284 100.37350788]


Which combination of 3 hyperparameters to combat overfitting of a convolutional neural network?

If you are asking specifically about overfitting, I would only keep dropout rate from your list of three. Other two can be any chosen from: number of filters in the convolutional layers, number of convolutions, number of dense layers (if any), number of neurons in dense layers.

Learning rate should be optimized but not for the purpose of combatting overfitting, at least the way I understand it. Using Keras, you can start with some learning rate - not necessarily very fine tuned, and slowly reduce it over time once the training reaches a plateau using Learning Rate Scheduler in Callbacks. Also, there are ways to find an optimal learning rate such as learning rate finder, so it would probably be a misuse of resources to optimize for learning rate using grid search (I never used them though!).

You can also use the Model Checkpoint present in the above Callbacks page to save the model as the validation loss improves only, and ignore the later epochs where overfitting becomes an issue. This way, you won't need to include number of epochs in your search.


Can we have a dataset with slight difference in target values for same value of feature variable?

I'm not sure what's the context of your question but there's no problem with the approach you've outlined. Different values of y for the same unique value of x (over different rows, such that for example you have: x = {1, 1}, y = {1, 2}) are a natural result of the noise usually assumed in the model you fit (e.g. $y = x + \epsilon$).
Hope this helps.


In my experience, when people claim to have an automated approach to feature engineering, they really mean "feature generation", and what they're actually talking about is that they've built a deep neural network of some sort. To be fair, in a limited sense, this could be a true claim. Properly trained deep neural networks can handle any number of pairwise correlations between individual features or groups of features. That said, without a great deal of up-front data pre-processing tools that know how to intelligently handle different types of input data (e.g., free text, images, etc.), none of this would be possible. Bottom line, it takes a great deal of manual effort to do something automatically.


Keras save model FailedPreconditionError

The error was caused because after training the model I opened another tensorflow session to save predictions to dataset and put model.save('model.h5') after. So moving this line up, right after the model finished training solved the problem.


Reasons and prevention of trivial (and less trivial) misclassification errors?

By looking at the ratings of the book, it looks like they are doing some sort of hierarchical clustering.
First of all it's a book, then a text book, then in medicine & health science.

This is a pretty difficult problem because:

You don't know how many and which topics you have, and the topics are constantly changing.
Since this is a clustering problem (or a semi-supervised problem in the better case), a closed feedback loop for these errors cannot be applied. Clustering is always more difficult than a supervised learning problem.

Apparently they're doing a pretty good job in general. This book in particular is a tough one...


Ordinal Integer variable vs Continuous Integer variable

There is a rule called there is no free launch. It means that there isn't a learning algorithm that solves all the problems. You as a machine learning practitioner should decide when and how to use which algorithm. Suppose that you want to recognize faces. This problem is a learning problem which if you increase the number of training data, you will get better results. In these cases neural nets and deep nets are highly recommended. In this case it is not logical to use non-linear SVM because it will be so costly and you may not even get good answers. the reason is that deep nets cares about local patterns but SVM considers all the input pattern simultaneously. Actually in your case, I guess your data is categorical. For categorical data, people often use decision trees.

To illustrate an example, once I decided to train a simple MLP to distinguish whether an input pattern is in correct position, to solve 8-queen problem. I solve the game using Genetic algorithm and made data for training the net. The data I brought to net was categorical in some extant. I used it and the net was so good for the trained data, but input features similar to training data which were a bit different had bad recall rate. I trained a decision tree, I get so much better result.

Which algorithm depends on your task and your input features.


What should be the value of non-rated field when finding cosine similarity

Implementation-side, there is a good reason to make 0 correspond to not rated.  Since most users haven't rated most books, 0 will be the most common value and the cosine similarity function can use sparse matrices internally to speed up the computation.  The sparse matrix shortcut is the main reason why people use cosine similarity in the first place.

On the other hand, it will throw off your recommendations if the algorithm interprets a 0 to be closer to a 1 than a 2. You do need to intentionally ignore those, but you can easily do so inside your cosine similarity implementation rather than filtering the data before applying it.


Keras neural network entirely different results on two different hardwares (one is AWS)

Consistency in models / predictions in Keras depend on the following:

Software Stack: The most uncommon problem, sometimes there are differences in different version of the numerical software which may change results.

Difference in Backends: Keras supports the TensorFlow backend, the Theano backend, and the CNTK backend. All of which will yield slightly different models. In the python console you can check the version with a message similar to:

Using TensorFlow backend.

You can switch backends using the following guide: Backend utilities | Keras

Setting seeds: In order to have consistency among models you need to set random seeds accordingly. In the case of Keras and Tensorflow backend, you need to set two random seeds:

In numpy

from numpy.random import seed

seed(123)

In tensorflow

from tensorflow import set_random_seed

set_random_seed(123)

There are other more intricate details for setting seeds in certain parts of the graph in Tensorflow explained better on this Stack Overflow answer and the set_random_seed() API documantation on Tensorflow's website.

Additional randomness added by GPUs: According to this MachineLearningMastery article there is even more randomness being added to a model when GPUs are used.

Intuitively in your problem, the source of randomness comes from not having set the appropriate seeds in numpy and Tensorflow, so start from there and start exploring the other options and documentations. Also remember that if you're using a different backend (Theano or CNTK), setting seeds will require a different function to be called.
An additional recommendation is to look for bugs in the script, version of downloaded binaries, warnings and errors during installation in case the result are too different. Something could be broken if one instance doesn't perform as expected and a fresh install might be necessary.


What's the proper way to do back propagation in Deep Fully Connected Neural Network for binary classification

The problem faced with back propagation with activation functions that has a max limit used in a Deep Neural Network (i.e. more then 2 layer network) is that the deeper the network is the quicker your back propagation will degrade.

Take for example your tanh activation function

If say you backpropagate from the output layer to the hidden layer and your tanh derivative is 0.25 then the next layer will only be limited by 0.25 from your tanh function your derived. Since you compute the next layer's derivative which in this case can be 0.5 of the total input and the previous layer's derivative totals to 0.25 then your derivative would be a total of 0.125. By the time the back propagation reaches the input layer, the weights would change by such a small fraction that it does not even matter.

This is why the ReLu function was such a big breakthrough as it does not have a max limit. There should be methods to counter this effect but ReLu is way more effective.


This has already been answered in stackoverflow. There is no need to save the queries you used to populate the database, you only need to dump the contents of the database to a file:

db1/neo4j-shell -path db1/data/graph.db/ -c dump > export_data.cypher

In order to load the database dump into another database, you just supply it to neo4j shell through the standard intput:

db2/bin/neo4j-shell -path db2/data/graph.db/ < export_data.cypher


Highly Imbalanced dataset fro classes more than 200

The situation you describe is a few-shot learning problem: you have a lot of classes and only a few examples for some of them.

Similarity metric learning with siamese neural networks is well suited for this task. The idea is to learn a general similarity metric between examples, then classify new examples as belonging to the class of the "closest" sample from the training set. It seems a bit complex but it's probably the best way to learn "across" classes.

The link I've shared above is applied to an NLP problem so should be able to reuse it fairly easily.


In supervised learning, why is it bad to have correlated features?

Correlated features in general don't improve models (although it depends on the specifics of the problem like the number of variables and the degree of correlation), but they affect specific models in different ways and to varying extents:

For linear models (e.g., linear regression or logistic regression), multicolinearity can yield solutions that are wildly varying and possibly numerically unstable.
Random forests can be good at detecting interactions between different features, but highly correlated features can mask these interactions.

More generally, this can be viewed as a special case of Occam's razor. A simpler model is preferable, and, in some sense, a model with fewer features is simpler. The concept of minimum description length makes this more precise.


Predict quantitative response variable from quantitative and qualitative variables

You should use one of the regression methods for predicting a quantitative variable. For the qualitative features, you can binarize them to pass to the learning algorithm. If you use Python, scikit-learn provide classes for doing that in the preprocessing module.


How to measure the contribution of each input channel to a network's accuracy?

Neural networks are black boxes, and figuring out its internal workings can be really hard. When you decide to use ML for your model, you are sacrificing accuracy for interpretability.
However, that does not mean it is totally impossible to interpret networks.  The most tedious way to determine feature importance would be to remove features and see how it impacts the model. If you're interested in research into the subject, I've encountered some research papers to use decision trees to understand networks.
https://openaccess.thecvf.com/content_CVPR_2019/papers/Zhang_Interpreting_CNNs_via_Decision_Trees_CVPR_2019_paper.pdf
https://arxiv.org/pdf/1802.02195.pdf


edX offers some spark courses. Try the following:

https://www.edx.org/course/introduction-apache-spark-uc-berkeleyx-cs105x?gclid=CjwKEAjwm8-6BRDgnb-Dk96UmRASJADbMycYmMKC3xWs_3jGV9pUd0k3gxIbQRHE2H4_hlXeNMw1BhoCqHjw_wcB

I believe this is through a partnership with Databricks


In my experience, it is very difficult to come up with good working custom objective functions for xgboost.

Custom objectives need to be continuous and need to have a convex gradient and non zero hessian, which is often not the case for custom loss functions.

One simpler method you could use is to define a custom validation metric based on your range +/-10, that can be used in conjunction with early stopping to optimize your hyperparameters of the xgboost model. So the model will not directly optimize for this, but will select hyperparamters that will minimize your custom loss. However, I would suggest to stick with the RMSE objective for this problem.


Machine Learning to Get X, Y Coordinates From Picture of Plot

Learning means you have examples of a complex behavior and can learn the dependency of the behavior on some parameters implicitly.

In your picture, one can see the dependency rather explicitly and the dependency is rather simple: A big black disk for every sticky-note.

Solution:

Take picture data (red, green, blue and each 0..255) and for each pixel calculate total squared difference to a black disk of exactly the same diameter as the black dots in the picture (sum differences for each color). This means you compute the difference pixelwise and then square it pixelwise and then add up over the area of the disk.
Find local minima of these differences below a threshold.
The positions of these local minima are the desired positions.

Unfortunately I'm not good enough in R but I believe it can be done in a few lines if loops exist in R.


What machine learning algorithm should I use for specific user configuration?

Unsupervised Approach

Unsupervised learning can be a good starting point. You can do a clustering (k-means/hierarchical) of existing user-base to find patterns in it (example: cluster 1 contains employees with Windows laptop with 4 GB RAM and photoshop pre-installed , cluster 2 is Mac with 16 GB memory). Then for a new employee, predict the nearest matching cluster and use the pre-dominant setup of that cluster.

The problem is there may be lot of lot of variation in the same cluster, especially on softwares installed.

Supervised Approach

Other alternative is to convert it to a supervised learning problem with hand-rolled categories as outcome (example: 'laptops with lot of RAM and graphics software', 'laptops for development') and then predict that category. You will need upfront work and domain knowledge to create the categories. But it will reduce the predicted variable space by a large margin.

But How to Create Categories?

Maths can help us a little bit here. Let us assume that you have an N dimensional one-hot encoded vector representing the hardware and software setup. Let us make the assumption that there are a handful of latent K categories of the setup encoded by a large dimensional vector of size N. To go about finding those, you can take a low dimensional projection of the encoding vector. This is probably a mathematical way of finding the categories instead of what you'd do manually on the output of clustering/unsupervised approach.

Radical Alternative: Recommendations
If you have enough data, you can pose it as 'recommending right setup to a new employee' problem. Take any standard movie recommendation kind of setting with users and movie ratings. Replace users -> employees and movies with a unique hardware + software setup. Use standard recommendation algorithm applied to binary data.

All these approaches are actually variant of low dimensional projection of the encoding vector + similarity calculation with existing users.

Hope this helps.


Is feature scaling necessary in reinforcement learning for the agent to learn successfully?

I assume that when you say

  values that are originally small get changed the same ratio

you are standardizing your features so that every dimension is on the same scale from [-1,1]

This is common practice as it helps many models converge faster.  It would not prevent your models from generalizing the data, so here are some other possible causes for your problem:

The ML algorithm you are using is not appropriate for your task
The data is either insufficient in quantity or quality (e.g. noisy data)
The preprocessing stage is lacking some steps such as dimensionality reduction


Using LSTM for multi label classification

Since the review of an author is bound to change in terms of the number of words being used in the review, I would suggest using a Keras Sequential() model to build an LSTM encoder for the review itself. The final hidden layer of the Review LSTM encoder can then be fed into another LSTM encoder with 3 words (phone, country and day). Think of the last LSTM encoder as a sequential 3 worded message. The final layer of this LSTM can then be joined with a softmax layer to predict the author.

The reason I suggested two different LSTM has something to do with word embeddings. Word embeddings are basically features that get modified in multiple hidden layers in an LSTM. Having two different word embeddings means that the same word in the Review Embedding may mean different in the other LSTM embedding, ie have different embedding vectors after training. It ensures that the features are independent. You can argue that you might need 4 different LSTM and Embedding Layers because of 4 different features. That is definitely a viable option but it might need a lot more time to train. Having more LSTM will add hyperparameters significantly and training time will be really high.

That being said you may not need an LSTM to solve this problem. You can easily convert the phone, country and day values into integers. There are 7 days in a week so the value of day can only be one from 0 to 6. That reduces the dimension of features significantly, reducing training time. You can probably try a TF-IDF for the review feature or a naive bag of word model and use a Random Forest or SVM Classifier for the prediction of Author. It would be interesting to perform multiple experiments with and without LSTM and observe results with accuracy and performance.


Evaluating Model Accuracy on a testing data set for a DecisionTreeReegressor Model

Accuracy in ML vocabulary is used mostly for Classification problem i.e. Count of correct prediction out of total.
In a common speaking language, it will mean the predictive correctness of the model esp. on test data.

  My understanding is that it's same as Score which can be calculated simply asregressor.score(X_test, Y_test)

I am assuming that you are using SciKit-Learn,
score method for DecisionTreeRegressor will return R-square coefficient.Offical Link

  score(self, X, y[, sample_weight]) Return the coefficient of determination R^2 of the prediction.

What should you do -
You should calculate two metrics - R-square and MAE/MSE.
Reason being - for an end-user/business person, MAE would be useful e.g. saying that model's prediction will be ~250$ away from the correct value on an average.

Challenge with MAE/MSE is that it doesn't say if it is good model unless you have an idea of the underlying data. e.g. Creating two models on pricing data of 2 different city - Boston/Tokyo and the MSE is 1000$/$1500. You can't conclude that the former is a better model from this data.
R-square helps here.
Adjusted R-square (Another regression metrics) - If your feature set is fixed, then you need not check this metrics. It was devised to fix an issue with R-square when the feature set is different for different models.

 Snippet to get RMSE, R-square, Adjusted R-square

#https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics

def reg_metrics(y_test, y_pred, X_train):
    from sklearn.metrics import mean_squared_error, r2_score

    rmse = np.sqrt(mean_squared_error(y_test,y_pred))
    r2 = r2_score(y_test,y_pred)

    # Scikit-learn doesn't have adjusted r-square, hence custom code
    n = y_pred.shape[0]
    k = X_train.shape[1]
    adj_r_sq = 1 - (1 - r2)*(n-1)/(n-1-k)

    print(rmse, r2, adj_r_sq)

Links to study -
Statistics by Jim
Wikipedia


Curve Fitting By Predict polynomial degree with ANNs

Interesting idea :)

The things I would consider:

Problem definition

Should it be classification or regression? If you define it as classification, you should select the number of classes, hence the maximum and minimum polynomial degree (e.g. 1,2,3,4,5,6,7,8,9 and 10 could be polynomial degrees which you recognize).

If you select regression, you don't need to define the maximum number of classes, but the output layers could be trickier to define.

Dataset

How do you define your samples? I guess that it's easy to generate samples (select some polynomial with degree $n$, sample points from it and then all those points define one sample for class (or regression value) n).

It would be interesting to create the image out of the sample (plot of points) and then maybe use a convolutional network to learn the classification (or regression).

Also, consider how dataset big should be. It's hard to say, you will have to test it and see how big it should be.

Losses

For classification probably cross entropy and for regression probably mean squared error.

Let me know the results.


What is the Time Complexity of Linear Regression?

It highly depends on the "solver" you are using.

Calling $n$ the number of observations and $p$ the number of weights, the overall complexity should be $n^2p+p^3$.

Indeed, when performing a linear regression you are doing matrices multiplication whose complexity is $n^2p$ (when evaluating $X'X$) and inverting the resulting matrix. It is now a square matrix with $p$ rows, the complexity for matrix inversion usually is $p^3$ (though it can be lowered).

Hence a theoretical complexity : $n^2p+p^3$.

Side notes

However, numerical simulations (using python's scikit library) seem to have a time complexity close to $n^{0.72} p^{1.3}$

This may be due to the fact that no implementation actually perform the full inversion (instead, the system can be solved using gradient descents) or due to the fact that there are other ways to calibrate the weights of a linear regression.

Source

An article from my blog : computational complexity of machine learning algorithms


How to integrate nvidia gpu with jupyter notebook for scikit-learn

There is no way to use GPU with scikit-learn as it does not officially supports GPU, as mentioned in its FAQ.


the task seems very hard to understand what you want to achieve as you say that there several equally good representation for each data key... In other words, you say following?

kd1 = l1 + l2 + l3 + l4 + l5 + l6
kd1 = l1 + l7 + l8 + l9   l5 + l10
kd1 = l11 + l12 + l13 + l14 + l5 + l10
kd1 = ...


Training data for multi-category classification algorithm

No, it is perfectly possible to train on multiple categories. What you need, though, is an exhaustive list of these categories (in supervised learning, that is).

Suppose you are trying to associate sentences with topics, and you have a list of possible topics topics = ['sports', 'soccer', 'politics'].
It sounds like your data look something like this:

sentence                       | topics
-------------------------------|----------------------------------
'Barack Obama loves soccer'    | ['politics', 'sports', 'soccer']
'The parliament is important'  | ['politics']
'Soccer is fun'                | ['sports', 'soccer']

Then you need to one-hot encode the topics:

X = [['Barack Obama loves soccer'], ['The parliament is important'], ['Soccer is fun']]

Y = [[1, 1, 1], [1, 0, 0], [0, 1, 1]]

And then you train a neural network to predict not one but three (= number of topics) values.


Ad click prediction: what are the negative examples?

You're overthinking it.  You might not need a threshold.  Start with the simplest approach you possibly can: If you showed the ad to a visitor, then that's a negative example.  Each time you show an ad to a visitor, you end up with an instance, whether positive or negative.  If you've showed the ad three times to the same visitor, you end up with three examples.  (Maybe all negative; or maybe one is positive and two are negative; but that's fine.)

Bonus tip: Do research methods for handling class imbalance.


Help Using Backprop Formulas from Deep Learning Book

My mistake was silly. I did not notice in the book that the weight matrices are numbered as the layer they are going into..so my $w^2$ above should be $w^3$ and then the result from the book is $\frac{\partial }{\partial w^3_{11}}=a^2_1(a^3_1-y)\sigma'(z^3_1)$ exactly as calculated by just doing the derivatives.


Find threshold in rate to determine reason for lost customer

You could use a two-sample $t$-test to determine whether software usage is significantly lower for lost users.

Plotting 'software usage rate'/'software usage rate relative to country average' against 'percentage of lost users' might give you an idea about a suitable threshold.


Pandas get_dummies() rows dropping after joining back with X

Can you post your code when you join the datasets? It's possible you're joining in a different way than you had intended. See the following helpful tutorial for figuring out exactly how you want to join datasets: https://pandas.pydata.org/docs/user_guide/merging.html
Additionally - I would check that the X and dummy dataframes have the same indexes. It's possible that in your join these forgotten entries simply don't share the same index in both dataframes.


Gathering the number of Google results from a large amount of searches.

With any search engine you will be limited by number of requests and any way of outcoming those limits will be a gray zone of violation of end user agreement (and, eventually, you will get banned for some time, of course). You should be looking into Search APIs of known search engines, for example, Bing gives you 5000 searches per month for free which - for a proof of concept research - might be enough. Also, 5k/month will give you some 20-30k until summer, so your data set will become bigger while you will be polishing your idea.

Also, Google's free tier search is limited to 100 requests per day. Which gives you completely legal 3k per month as well. Combined (given, you treat Google and Bing results as equal) you get 8k per month.


Need some tips regarding starting out with the field's specific programming languages, with a heavy focus on data visualization

R is a more compact, target oriented, package. Good if you want to focus on very specific tasks (generally scientific). Python, on the other hand, is a general purpose language.

That being said, and obviously this is a matter of opinion, if you are an experienced developer go for Python. You'll have far more choices in libraries and a far bigger potential to build big software.

Some examples of 2D scientific plotting libraries:

Matplotlib
Bokeh (targeted D3.js)
Chaco
ggplot
Seaborn
pyQtGraph (some significant 3D features)

Some examples of 3D scientific plotting libraries

Vispy
Mayavi
VTK
Glumpy

Some examples of libraries typically used in Data Science in Python:

Pandas
Numpy
Scipy
Scikit-learn
Scikit-image

Also check the list for other relevant Scikit packages.

As for starting software I would advise you to use any of the already prepared Python distributions that already come with a bunch of scientific libraries inside as well as software such as IDEs. Some examples are:

WinPython
Python XY
Anaconda
Canopy

Personally I'm a user of WinPython due to being portable (former user of Python XY, both are great). In any case these distributions will greatly simplify the task of having your scientific Python environment (so to speak) prepared. You just need to code. One IDE known to be specially good for scientists is Spyder. Yet these ones also will work:

PyDev
PyCharm
WingIDE
Komodo
Python Tools for Visual Studio

As for data visualization tips you'll see that the most common functions in the libraries mentioned above are also the most widely used. For instance a library like Pandas let's you call plots directly from the object so there is already an intuitive approach to data visualization. A library like scikit-learn (check the site) already shows examples followed by data visualization of the results. I wouldn't be too concerned about this point. You'll learn just by roaming a bit on the libraries documentation (example).


Time series data and ML - separating training/test data

When you are working with time-series data, the most recent data captures the most relevant information possible, so it is more prudent to include them in training data. So a more prudent decision would be to opt for Roll-Forward Partitioning.
Roll-Forward Partitioning: We start with a short training period and we gradually increase it, at each iteration of training, we train it on the current training period and make it forecast the next interval of data. It will require more training time, but it mimics what we would do during deployment where we would want to keep training our model at regular intervals to keep it up to date.
You can find more about it here, here and here.


What are useful evaluation metrics used in machine learning

For model evaluation there are different metrics based on your model:

Confusion matrix

Classification accuracy:
(TP + TN) / (TP + TN + FP + FN)
Error rate:
(FP + FN) / (TP + TN + FP + FN)
Paired criteria

Precision: (or Positive predictive value)
proportion of predicted positives which are actual positive
TP / (TP + FP)
Recall: proportion of actual positives which are predicted positive
TP / (TP + FN)

Sensitivity: proportion of actual positives which are predicted positive
TP / (TP + FN)
Specificity: proportion of actual negative which are predicted negative
TN / (TN + FP)
True positive rate: proportion of actual positives which are predicted
positive
TP / (TP + FN)
True negative rate: proportion of actual negative which are predicted
negative
TN / (TN + FP)
Positive likelihood: likelihood that a predicted positive is an actual positive
sensitivity / (1 - specificity)
Negative likelihood: likelihood that a predicted negative is an actual
negative
(1 - sensitivity) / specificity
Combined criteria

BCR: Balanced Classification Rate
½ (TP / (TP + FN) + TN / (TN + FP))
BER: Balanced Error Rate, or HTER
Half Total Error Rate: 1 - BCR
F-measure harmonic mean between precision and recall
2 (precision . recall) / (precision + recall)
Fβ-measure weighted harmonic mean between precision and recall
(1+β)2 TP / ((1+β)2 TP + β2 FN + FP)

  The harmonic mean between specificity
  and sensitivity is also often used and
  sometimes referred to as F-measure.

Youden's index: arithmetic mean

between sensitivity and specificity
sensitivity - (1 - specificity)
Matthews correlation: correlation between the actual and predicted
(TP . TN – FP . FN) / ((TP+FP) (TP+FN) (TP + FP) (TN+FN)) ^ (1/2)
comprised between -1 and 1 Discriminant power normalized likelihood index
sqrt(3) / π .
(log (sensitivity / (1 – specificity)) +
log (specificity / (1 - sensitivity)))
<1 = poor, >3 = good, fair otherwise

You can find much more here. Also there are some explanations here and you can find useful code snippet from here which are implemented.


Parameterization regression of rotation angle

The second way, predicting $x=cos(\alpha)$ and $y=sin(\alpha)$ is totally okay.

Yes, the norm of the predicted $(x, y)$ vector is not guaranteed to be near $1$. But it is not likely to blow up, especially if you use sigmoid activation functions (which are bounded by they nature) and/or regularize your model well. Why should your model predict a large value, if all the training samples were in $[-1, 1]$?

Another side is vector $(x,y)$ too close to $(0,0)$. This may sometimes happen, and could indeed result in predicting wrong angles. But it may be seen as a benefit of your model - you can consider norm of $(x,y)$ as a measure of confidence of your model. Indeed, a norm close to 0 means that your model is not sure where the right direction is.

Here is a small example in Python which shows that it is better to predict sin and cos, that to predict the angle directly:

# predicting the angle (in radians)
import numpy as np
from sklearn.neural_network import MLPRegressor
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import r2_score
# generate toy data
np.random.seed(1)
X = np.random.normal(size=(100, 2))
y = np.arctan2(np.dot(X, [1,2]), np.dot(X, [3,0.4]))
# simple prediction
model = MLPRegressor(random_state=42, activation='tanh', max_iter=10000)
y_simple_pred = cross_val_predict(model, X, y)
# transformed prediction
joint = cross_val_predict(model, X, np.column_stack([np.sin(y), np.cos(y)]))
y_trig_pred = np.arctan2(joint[:,0], joint[:,1])
# compare
def align(y_true, y_pred):
    """ Add or remove 2*pi to predicted angle to minimize difference from GT"""
    y_pred = y_pred.copy()
    y_pred[y_true-y_pred >  np.pi] += np.pi*2
    y_pred[y_true-y_pred < -np.pi] -= np.pi*2
    return y_pred
print(r2_score(y, align(y, y_simple_pred))) # R^2 about 0.57
print(r2_score(y, align(y, y_trig_pred)))   # R^2 about 0.99

You can go on and plot the predictions, to see that predictions of the sine-cosine model are nearly correct, although may need some further calibration:

import matplotlib.pyplot as plt
plt.figure(figsize=(12, 3))
plt.subplot(1,4,1)
plt.scatter(X[:,0], X[:,1], c=y)
plt.title('Data (y=color)'); plt.xlabel('x1'); plt.ylabel('x2')
plt.subplot(1,4,2)
plt.scatter(y_simple_pred, y)
plt.title('Direct model'); plt.xlabel('prediction'); plt.ylabel('actual')
plt.subplot(1,4,3)
plt.scatter(y_trig_pred, y)
plt.title('Sine-cosine model'); plt.xlabel('prediction'); plt.ylabel('actual')
plt.subplot(1,4,4)
plt.scatter(joint[:,0], joint[:,1], s=5)
plt.title('Predicted sin and cos'); plt.xlabel('cos'); plt.ylabel('sin')
plt.tight_layout();

Update. A navigation engineer noticed that such a model would be most accurate when the angle is close to $\frac{\pi N}{2}$. Indeed, near 0° and 180° the angle $\alpha$ is almost linear in $\cos(\alpha)$, and near 90° and 270° it is almost linear in $\sin(\alpha)$. Thus, it could be beneficial to add two more outputs, like $z=\sin(\alpha+\frac{\pi}{4})$ and $w=\cos(\alpha+\frac{\pi}{4})$, to make model almost-linear near 45° and 135° respectively. In this case, however, restoring the original angle is not so obvious.

The best solution may be to extract coordinates $(x,y)$ from both representations (in the second one, we need to rotate $(z,w)$ to get $(x,y)$), average them, and only then calculate arctan2.


Understanding time series anomaly detection using Autoencoder

The length of TIME_STEPS in one sequence is more of a Hyperparameter. That you should try to optimize.

Does this method split the data or is he just creating a 3D variable in the correct
format for the Convolutional Autoencoder?

It simply create dataset for a 1-dimsnional convolutional network.Something like this,
$\hspace{3cm}$
Each row is an instance. Convolution will happen across the row.

I have a dataset where there are stored the measurements of 30 devices

You may simply assume each data point as a Time-step assuming it was collected simultaneously.
But your data has an additional dimension i.e. Devices.

So you should either build one model for each device or take the average of each device and consider it as the single time-step.
Or you can apply a 1-D CNN with a 2-D kernel in your first layer, assuming each device as a channel. Check an example here.
In case, Keras doesn't allow a 2-D kernel, then use a 2D-CNN with kernel size "30xM". With that, the convolution will happen in only one direction.
 Beware that if you use the last 2 options, then your pre-processing function will have to change i.e each time-step will have "30x1" as a dimension.


Synthetic Gradients - what's the practical benefit?

But, hey! These predicted "synthetic gradients" will be very small (negligible) anyway - after all, that's why we start a new backprop session B. Weren't they too small, we would just parse all 6 timesteps in a single long bkprop "session A".

-That's not necessarily correct. We typically truncate and start a new backprop because of hardware constraints, such as memory or computational speed. Vanishing-gradient can be improved by other means, such as Gradient Normalization - scaling-up the gradient vector if it gets too small beyond certain layers, or scaling-down if it's about to explode. Or even using Batch Normalization

It's important to understand how to update any DNI module. To clear things up, consider an example of network with several layers and 3 DNI modules:

 input
   |
   V
Layer_0 & DNI_0
Layer_1
Layer_2
Layer_3 & DNI_3
Layer_4
Layer_5 & DNI_5
Layer_6
Layer_7
   |
   V
output

  The DNI_0 is always trained with a synthetic gradient arriving from DNI_3 (flowing through Layer_2, and Layer_1 of course), siting several layers further.

  Likewise, the DNI_3 is always trained with a synthetic gradient arriving from DNI_5

  DNI_0 or DNI_3 will never get to see the true gradient, because true grad is only delivered to DNI_5, and not earlier.

  For anyone still struggling to understand them, read this awesome blogpost, part 3

Earlier layers will have to be content with synthetic gradients, because they or their DNI will never witness the "true gradient".

Regarding training in parallel with minibatches instead of Parallelizing via Synthetic Grads:

Longer sequences are more precise than minibatches, however minibatches add a regulization effect. But, given some technique to prevent gradient from exploding or vanishing, training longer sequences can provide a lot better insight into context of the problem. That's because network infers output after considering a longer sequence of input, so the outcome is more rational.

For the comparison of benefits granted by SG refer to the diagrams page 6 of the Paper, mainly being able to solve longer sequences, which I feel is most beneficial (we can already parallelize via Minibatches anyway, and thus SG shouldn't speed up the process when performed on the same machine - even if we indeed only propagate up to the next DNI).

However, the more DNI modules we have, the noisier the signal should
 be. So it might be worth-while to train the layers and DNI all by the legacy backprop, and only after some epochs have elapsed we start using the DNI-bootstrapping discussed above.

That way, the earliest DNI will acquire at least some sense of what to expect at the start of the training.  That's because the following DNIs are themselves unsure of what the true gradient actually looks like, when the training begins, so initially, they will be advising "garbage" gradient to anyone sitting earlier than them.

Don't forget that authors also experimented with predicting the actual  inputs for every layer.

If your layers have expensive backprop (perhaps you have Batch-Normalization or some fancy activation functions), correcting with DNI might be a lot cheaper, once it's sufficiently well-trained.  Just remember that DNI is not free - it requires matrix multiplication and probably won't provide much speed-up on a simple dense layer.

Minibatches gives us speedup (via parallelization) and also give us the  regularization.
Synthetic Gradients allow us to infer better by working with longer sequences and (potentially) less-expensive gradient. All together this is a very powerful system.


How to apply multiple clustering algorithms to same dataset and make comparison?

Have a look at the scikit learn API, they have a section dedicated to this topic.
If you want to go further, have a look at chapter 3 "Unsupervised Learning and Preprocessing" of the book Introduction to Machine Learning with Python: A Guide for Data Scientists, there is a section on clustering methods (they also mention the evaluation of clustering methods which is very insightful)


To understand the intuition behind Adagrad, let's have a look at the graphs below, representing the evolution of the loss function when model's weights are updated according to different values of the learning rate in a 1-D search space:

Graph 1 (left - Too Fast!): when updating the weights of the model, we can see that it does not converge to the global minimum. It is due to the fact that the learning rate is too high and it keeps bouncing at the bottom.

Graph 2 (center - Too Slow!): in this case, the weights' update takes place very slowly, because the learning rate is too small, thus it takes too long to converge (or even never) to the global minimum.

Graph 3 (right - Spot On!): in this case, the learning rate is adapted depending on the value of the gradient, i.e. the higher the gradient, the lower the learning rate, or, the lower the gradient, the higher the learning rate. This makes it possible to increase the chances of convergence without having the issues commented before.

Adagrad is one of several adaptive gradient descent algorithms like ADAM, ADADELTA, etc. You can check more info here.

NOTE: the pictures above have been taken from here.


How to segment old digitized newspapers into articles

As far as I know topic segmentation is not a particularly easy task with clean data, so it's likely to be challenging with noisy old French.

It's not exactly the same problem so I'm not sure if this is useful but you might want to look into using stylistic features in order to help the model detect the changes between articles. There has been a fair amount of work on the task of style change detection as part of the PAN series (the task has run for 3 years, results and papers are available from the previous years).

Hope this helps.


But what about new entities (movie or production company name) that
  trained system hasn't seen how can we tag them. Re-training the model
  every time with new released movies won't be feasible.

A NER model should not have to be retrained to tag a new text it has not seen before. If trained successfully it will use information it learned from the labeled data and be able to apply it on new data. For your use case this could be information such as:

Capitalization - common for movie titles and production company names
Words used - frequently used words for either movie titles or company names
Where in text - movie titles perhaps appear early in texts

spaCy is a good library to get started with NER. Here is an example on how to train one using your own data.


Industrial application(s) of LDA (latent Dirichlet allocation)?

I can't answer for the systems in already existing companies, but I can definitely share an application of LDA in NLP.

Latent Dirichlet Allocation is a popular technique use for topic modelling in Natural Language Processing. The idea is to have corpus of natural langue text with lots of documents and the goal is to get the distribution of the words appearing in the corpus each (Distribution) being termed as a topic.

For visual analysis you can view this note book here. It makes use of the NLP library built for LDA, called pyLDAvis.

So lets say you have a corpus, and you want to build a tagger which labels the document(input text) into one of the N Classes, but you don't have labelled data. How do you start? One way is to use LDA to perform Topic Modelling, get the topics, then name the topics to N Classes. Label the data using LDA model, Use the loosely labeled data to build a classifier.


Sci-kit learn function to select threshold for higher recall than precision

As far as a fuction in scikit to implement a certain threshold for a higher recall, I don't think there is one.

But, depending on what model you're using, you can vary the threshold for probability outputs of the model to obtain a higher Recall.

For example, say you are using a Random Forest or Logistic Regression classifier model, then you can use the predict_proba function on your data set; this will return probabilities for each sample belonging to a particular class. If you then create a new array from the output of predict_proba using your threshold, then the model will have a Recall with your desired accuracy when it classifies new unseen data.

e.g pseudo code for binary classification

clf = sklearn.ensemble.RandomForestClassifier()
model = fit(X,y) # fit model to training datset
probs = model.predict_proba(X_new) # prediction on a new dataset X_new

threshold = 0.7 # threshold we set where the probability prediction must be above this to be classified as a '1'
classes = probs[:,1] # say it is the class in the second column you care about predictint
classes[classes>=threshold] = 1
classes[classes<threshold] = 0


How to filter rows in Python pandas dataframe with duplicate values in the columns to be filtere

I think you have defined midnight as today's 00:00. Therefore, the rows before today are out of your range.

I think this may work for this cases:

# Convert string to datetime format
df['Date Time'] = pd.to_datetime(df['Date Time'])

selected_rows = df[((df['Date Time'].dt.hour * 60 + df['Date Time'].dt.minute) >= 9 * 60) &
                   ((df['Date Time'].dt.hour * 60 + df['Date Time'].dt.minute) <= 17 * 60)]

The filter rules use the time only and ignores the date.


How to interpret the original time series approximated using principal component analysis?

Usually, the norm to asking any question on Stack Overflow or any other sister websites is that one is supposed to ask one question only unless they are very similar which doesn't seem to be in this case.
To answer your questions, $X^TX$ is called Sample Covariance (or Correlation) Matrix where $X$ is the data matrix of dimensionality $(m,d)$. So, the resultant matrix has a dimensionality of $(d,d)$ where $d$ is the dimensionality of the feature space.
And as you said, this matrix is made to go an eigendecomposition to get $w\cap w^{-1}$ where $\cap$ is a diagonal matrix of eigenvalues, arranged in decreasing order and $w$ is the normalized eigenvectors stacked according to the corresponding eigenvalues. The reason one might want to choose some $k$ number of dimensions is to reduce dimensionality.
Reduced dimensionality provides multiple benefits - reducing space complexity, faster computations etc. The problem arises when you specifically talk about time series. PCA, ICA does not take into account temporal dependence which might cause data to have suboptimal forecast. There are different ways to tackle this issue, one might want to use Forecastable Component Analysis, autoencoders etc to ensure they are reaping the benefits of not only gaining the benefits that PCA provides but also ensuring that the problems with PCA's are avoided.
To answer the second question, I'm NOT sure of the reason but one may want to drop the first principal because it is in the direction of maximum variance, i.e. it varies most in this direction.


Unsupervised clustering without of Data which is supposed to be on a linear function

I would rather look at mixture models. Or, if there is additional noise besides the lines, use computer vision algorithms, for example, some variation of the Hough transform.


To calculate my confusion matrix with recall and precision, my test set need to be equal(balanced)?

I don't think there is any reason to modify the matrix so keep it as it is. Even if you scale it what purpose does it serve? At the end of the day your model does not change even if you modify your confusion matrix.

In my opinion you can use other metrics e.g. f1-score (or f beta score), AUC score, etc to judge your model. Confusion matrix only provides visualization where your model "confused" and I would say it is less useful for binary classification (as you only have False positive or False negative). Metrics above serve as better judge for evaluating your model.

This is a related question which you can probably check.


How to build in symmetry of inputs into a Deep Neural Network?

This might not be what you're looking for, but you can use data augmentation to do this: from each sample, you can generate multiple samples by cloning it, generating a random permutation and permuting the input and the expected output (for an autoencoder, this should achieve the same goal). This doesn't guarantee complete permutation invariance but if you do it enough times for each sample it should get you pretty close.


Best approach for a simple self driving car

~ If we consider the problem as an image classification task,

Basically, we can use a CNN trained on images belonging to 7 classes ( which represent the steering levels ). We give an image to the CNN and it outputs a steering level. This is a classification problem which will take each frame ( image ) of the video and output the steering level. So as the car steers the probability for the centre steering level will increase assuming that the road is straight after taking the turn.

See multiclass image classification.

~ else, if we train an agent for the problem ( reinforcement learning ),

Reinforcement learning requires an agent ( in our case, the car ) which produces some action ( steering levels ) in an environment ( street, road etc. ) and gets an reward accordingly. The rewards need to be maximized.

We will have a CNN which encodes an image to a vector representation.

  So, consider two cases. We have an image which needs to the car to steer hard left. In the first case, the car turns soft left and loses its path. Here, the reward for this action becomes -100. In the other case, the car turns hard left, stays on the road and hence gets a reward of 100.

Basically, the agent gets a positive reward if it stays on the road doing some action ( steering level ) else it gets a negative reward showing that this action should not be made on such an image ( road ).

See here for learning RL.

Tip:

You can try combining both the above methods. Generate class probabilities ( steering levels ) from CNN. If the car does not steer well, increase the loss by a certain factor.

  Try thinking from a human perspective. Imagine driving a car and the cautions you take to avoid going out the road.


Machine Learning - Precision and Recall - differences in interpretation and preferring one over other

To make sure everything is clear let me quickly summarize what we are talking about. precision and recall are evaluation measures for binary classification, in which every instance has a ground truth class (also called gold standard class, I'll call it 'gold') and a predicted class, both being either positive or negative (note that it's important to clearly define which one is the positive one). Therefore there are four possibilities for every instance:

gold positive and predicted positive -> TP
gold positive and predicted negative -> FN (also called type II errors)
gold negative and predicted positive -> FP (also called type I errors)
gold negative and predicted negative -> TN

$$Precision=\frac{TP}{TP+FP}\ \ \ Recall=\frac{TP}{TP+FN}$$
In case it helps, I think a figure such as the one on the Wikipedia Precision and Recall page summarizes these concepts quite well.
About your questions:

if avoiding false-positives matter the most to me, i should be measuring precision; And if avoiding false-negatives matters the most to me, i should be measuring recall. Is my understanding correct?

Correct.

Suppose, I am predicting if a patient should be given a vaccine, that when given to healthy person is catastrophic and hence should only be given to an affected person; and I can't afford giving vaccine to healthy people. assuming positive stands for should-give-vaccine and negative is should-not-give-vaccine, should I be measuring Precision? or Recall of my classifier?

Here one wants to avoid giving the vaccine to somebody who doesn't need it, i.e. we need to avoid predicting a positive for a gold negative instance. Since we want to avoid FP errors at all cost, we must have a very high precision -> precision should be used.

Suppose, I am predicting if an email is spam(+ve) or non-spam(-ve). and I can't afford a spam email being classified as non-spam, meaning can't afford false-negatives, should I be measuring Precision? or Recall of my classifier?

We want to avoid false negative ->  recall should be used.
Note: the choice of the positive class is important, here spam = positive. This is the standard way, but sometimes people confuse "positive" with a positive outcome, i.e. mentally associate positive with non-spam.

What does it mean to have high precision(> 0.95) and low recall(< 0.05)? And what does it mean to have low precision(> 0.95) and high recall(< 0.05)?

Let's say you're a classifier in charge of labeling a set of pictures based on whether they contain a dog (positive) or not (negative). You see that some pictures clearly contain a dog so you label them as positive, and some clearly don't so you label them as negative. Now let's assume that for a large majority of pictures you are not sure: maybe the picture is too dark, blurry, there's an animal but it is masked by another object, etc. For these uncertain cases you have two possible strategies:

Label them as negative, in other words favor precision. Best case scenario, most of them turn out to be negative so you will get both high precision and high recall. But if most of these uncertain cases turn out to be actually positive, then you have a lot of FN errors: your recall will be very low, but your precision will still be very high since you are sure that all/most of the ones you labeled as positive are actually positive.
Label them as positive, in other words favor recall. Now in the best case scenario most of them turn out to be positive, so high precision and high recall. But if most of the uncertain cases turn out to be actually negative, then you have a lot of FP errors: your precision will be very low, but your recall will still be very high since you're sure that all/most the true positive are labeled as positive.

Side note: it's not really relevant to your question but the example of spam is not very realistic for a case where high recall is important. Typically high recall is important in tasks where the goal is to find all the potential positive cases: for instance a police investigation to find everybody susceptible of being at a certain place at a certain time. Here FP errors don't matter since detectives are going to check afterwards but FN errors could cause missing a potential suspect.


CycleGAN: Generator losses don't decrease, discriminators get perfect

This usually happens when your discriminator starts to just understand that the difference between your generated(fake) and real samples only differ by some n/255 factor. This is easy to learn, and if your generator cannot properly generate "good" samples then within a few 10-100 epochs your Discriminator will learn this highly trivial transformation.
Now when we reach a point where the generator cannot converge from pure random output it will get stuck generating the same random output with a constant high loss. Since I don't know the exact architectures of your network there are some solutions out of the box that has helped me.
Try adding some gaussian noise at the very beginning of your Discriminator network, this will make sure your Discriminator will not learn trivial transformations right away. I cannot comment on this channel yet, so if you can post your network architecture then I could possibly give you a better answer.


It seems you are passing the same parameters to both Python classes: ARIMAX and SARIMAX... and this should produce the same answer, I agree.

The implementations are however separate, it seems - ARIMA inherits from ARMA models and SARIMAX inherits from MLEModel. Have a look at the source code for ARIMA (search fro "class ARIMA") and SARIMAX (search for "class SARIMAX").

There are probably some different ordering of the operations, or different limits on the tolerances for MLE.


Can I apply feature selection before splitting by requiring selection occurs > 90% of time

As explained in the post you link, it depends how you select the features: if it doesn't involve the target variable, then it's probably fine. I'll assume the most common case, that is that the selection relies on the target variable.
There are two parts in this problem:

About the ensemble method for feature selection: let's say a particular feature is selected 90% of the time. This means it's not selected 10% of the time, so if you were using a single training set it would have a 10% chance not to have this feature selected. So in theory your model is likely to be better than a "regular model", because it includes a few features which wouldn't have been selected in the "regular model". Note however that this method could also have negative side effects, because the features are not selected as a whole but individually (i.e. it might not be the optimal subset of features).
About the risk of data leakage: by definition, applying this method on the whole data means using information from the test set, so a potential bias in the evaluation. It's true that the ensemble method decreases the risk of selecting a feature by chance, but for every feature there's a 10% chance that it wouldn't have been selected in the "regular model". Since this selection is based partly on the test set, you can't be certain that the evaluation is reliable.

Assuming your goal is to use cross-validation and that the feature selection process is computationally expensive, I can think of two ways to do this properly:

Select a random N% of the data (where N is the size of the training set, e.g. 90% if using 10 fold CV), do the feature selection on this data, then use this predefined set of features every time, independently from the training set. The idea here is that the selected set of features cannot take advantage of a selection on the whole data, so it's not particularly  optimized for the test set or any CV split. That should be enough to obtain a fair evaluation, although technically there is still data leakage in this way.
Do an additional split in your data: with the training set you apply exactly the process that you described for feature selection, then run CV on it. After this, apply the final model to the test set. For example take 80% of the data for the training set, do feature selection and CV based only on this training set, and after that use the 20% left of unseen instances as test set. There's no data leakage in this way, but the final evaluation comes from a single test set, not from CV (here the CV stage can be used to study performance variation or tune hyper-parameters).


Difference between PCA and regularisation

Lasso does feature selection in the way that a penalty is added to the OLS loss function (see figure below). So you can say that features with low "impact" will be "shrunken" by the penalty term (you "regulate" the features). Because of the L1 penalty, the $\beta_i$ can become zero (which is not the case with Ridge, L2). In the Lasso case you would "eliminate" a feature when it is "shrunken" to zero, and you could call this feature selection. Lasso can be used in "high dimensions", i.e. when you have many features ("columns") but not so many observations ("rows").

Principle components work in quite a different way. The first principle component is a normalised linear combination [of the original features] which has the largest variance. So you kind of "transform" the original features to a principle component (which is a "new feature" derived from the original ones), where you try to capture as much variance as possible in one principle component.
Principle components are uncorrelated (orthogonal). This can be very helpful when you do linear regression, in which (high) correlation between features can be a real problem. I see PCA as a tool for dimensionality reduction (not so much feature selection), since you can express many features in a (smaller) number of principle components.
So maybe a little too brief summary:

Lasso: "shrink" the estimated coefficients for features which are not too useful (but leaves the features as they are)
PCA: "combine" several features into one or more orthogonal "new" feature(s) (principle components) and use them in some type of model

For more details, refer to "Introduction to Statistical Learning" (available for free online). Ch. 6.2.2 covers the Lasso, Ch. 10.2.1 covers PCA.


Lime Explainer: ValueError: training data did not have the following fields

It looks like a dataframe/nparray mismatch.  Probably your model was trained on a dataframe; doing the training instead on an nparray may fix the problem.

I've seen such errors in working with xgboost; see e.g.
https://github.com/dmlc/xgboost/issues/2334 and
https://stackoverflow.com/a/52578211/10495893


Algorithm for finding best juice combinations

As described, you have no data describing individual people (such as age, sex, shoe size), but are searching for an  optimum value of the mix for the whole population. So what you want is a mix with the maximum expected rating, if you chose a random person to rate it from the population. In principle, this expected rating is a function taking two parameters e.g. $f(n_{apple}, n_{orange})$ - the amount of the third juice type is a not a free choice, so you only have two dimensions.

You can break down your problem into two distinct parts:

Taking samples from your population in order to find approximation to the function $f(n_{apple}, n_{orange})$
Using the approximation as it evolves to guide the search for an optimum value.

For a simple approach, you could ignore the second bullet point and just randomly sample different mixes throughout the event. Then train a regression ML on the ratings (any algorithm would do, although you'll probably want something nonlinear, otherwise you'll just predict one of the pure juices as favourite) - finally graph its predictions and find the maximum rating at the end. This would probably be fine when pitched as a fun experiment.

However, there is a more sophisticated approach that is well-studied and used to make decisions when you want to optimise an expected value of an action whilst exploring options - it is usually called multi-armed bandit. In your case, you would need variants of it that consider an "arm space" or parametric choice, as opposed to a finite number of choices that represent selecting between actions. This is important to you, since splitting your mix parameters up into e.g. in 5% steps, will give you too many options to explore given the number of samples you need to make. Instead, you will need to make an assumption that the expected rating function is relatively smooth - the expected rating for 35% Apple, 10% Orange, 55% Grape is correlated with the rating for  37% Apple, 9% Orange, 54% Grape . . . that seems at least reasonable to me, but you should make clear in any write-up that this is an assumption and/or find something published that supports it. If you make this assumption, you can then use a function approximator such as a neural network, a program like xgboost or maybe some Guassian kernels to predict expected rating from mix percentages.

In brief for a multi-armed bandit problem, you will use data collected as your experiment progresses to estimate the expected value for each choice, and on each step will make a new choice of mix. The choice itself will be guided by your current best approximation. However, you don't always sample the current top-rated value, you need to explore other mixes in order to refine your estimated function. You have choices here too - you could use $\epsilon$-greedy where e.g. 10% of the time you choose completely randomly to get other sample points. However, you might need something more sophisticated that explores more to start with and still converges quickly, such as Gibbs sampling.

One thing you don't say is at what level you are pitching this experiment. Studying the multi-armed bandit problem by yourself referring to blogs, tutorials and papers could be a bit too much work if this is for school science fair. If this all seems a bit too vague and a lot of work to study, then you can probably stick with a simple regression model from the data of a random experiment.

I suggest whichever approach you take, that you run some simulations of input data and see whether your approach works. Obviously there is a lot of guess work here. But the principle is:

Create a "true" model function - e.g. pick an imaginary favourite mix and make it score higher. Make it a simple and probably quite subtle function - e.g. score 5 for best result, and take away euclidean distance in "juice space" times a small factor (maybe 1.5) from it.
Create a noisy sampler that imitates someone in your experiment giving a rating to a specific mix. Ensure that the mean value from this matches the "true" function.
Try out your sampling and learning strategies, see how well they find the favourite mix.

I highly recommend this kind of dry run before putting your system to real use, otherwise you will have no confidence that your ML/approximator is working.

One more piece of advice about your estimator: You are expecting a large amount of variance in your data, and will not have a lot of samples. So to avoid over-fitting you will want to have a relatively simple ML model. For a neural network for example, you will probably want only one hidden layer with very few neurons in it (e.g. 4 or 5 might be enough). Finding a model sophisticated enough to predict a curve, but simple enough that it doesn't overfit given very noisy target outputs might take a few tries - this is the main reason why I suggest performing trial runs with simulated data.


Assuming the feature selection method is always the same, on an external data (or final training set), you would simply apply the exact same method. The actual set of selected features does not matter.
If there was any difference in the selection method, for instance if you are selecting different number of features, you would do like with any other hyper-parameter: select the best model according to the nested CV process, then apply the same hyper-parameters (including for instance number of features) when training the final model.


How to get common values between two multi sets?

Since you have multiple repetitions in A and B, and these two vectors are of different lengths (5913 and 3733), it is expected that vector C and D are of different size. However, C and D might contain the same unique elements.

Let's take the example proposed by Biswajit Banerjee:

A = c(1,1,2,2,3,3,4,4,5,5,6,6,9,11,15,17,19,19)
B = c(1,2,2,2,3,3,3,4,5,6,7,8,8,8)

C= A[(A %in% B)]
D = B[(B %in% A)]

all(unique(C) == unique(D))
TRUE

You could verify that your vectors C and D of length 2007 and 126 contains the same unique values. Does it solve your problem ?


Is this approach for Active Learning correct?

Your flow is correct. Model is retrained on new labeled data. Otherwise, the next candidates for labeling will be selected from the same region as previous candidates on which the model is least certain. By training on new labeled data, model will move on to new regions about which is least certain. Here is the diagram for pool-based method from page 5 of this survey on active learning:


Whether you need two different data sets depends on the arguments needed in function you are using. Some model fitting functions need a formula and a data argument. Some functions need a xand a y argument. Sometimes you have both options.

For example a linear model (lm) takes a formula and a data argument. Since you can only enter 1 data frame in the data argument, your dependent and independent variables have to be in one and the same dataframe. This is the same as in your example with the gbm function.

However for example a random forest (randomForestpackage) takes the arguments x and y. Therefore, you can have two dataframes (one with y data, one with x data).

Conclusion: it always depends on the function and package that you are using and the only way to find out is to read in the documentation which of those two ways you should be using.


There is a Python library called dateparser that will accept a wide variety of formats, including relative dates like "next Tuesday", and return exact datetime representation.