How to scale an array of signed integers to range from 0 to 1?

This is called unity-based normalization. If you have a vector $X$, you can obtain a normalized version of it, say $Z$, by doing:

$$Z = \frac{X - \min(X)}{\max(X) - \min(X)}$$


These techniques are not mutually exclusive; combining dropout with weight decay has become pretty standard for deep learning.

However, where weight decay applies a linear penalty, dropout can cause the penalty to grow exponentially.  This property of dropout can lead to hypothetical failures as proposed and proven in section 4.2 of this paper.

In general, research has consistently shown the benefits of dropout (with and without weight decay) for training deep networks.  A practical scenario in which weight decay is exclusively preferred over dropout would be quite the anomaly.


Different Test Set and Training Set Distribution

Great question, this is what is known in Machine Learning paradigm as either
  "Covariate Shift", or "Model Drift" or "Nonstationarity" and so on.

One of the critical assumption one would make to build a machine learning model for future prediction is that unseen data (test) comes from the same distribution as training data! However, in reality this rather simple assumption breaks easily and upcoming data (its distribution) changes over time for many reasons. For those who may not be familiar with this very important problem, I encourage looking here or post!

To me, your question falls into the same category. Although I do not have the perfect solution (an implementation to offer), but I think you may look:

This blog post gives you a simple way to handle the subsampling of training data with code provided in Python!
Check this research paper. They propose to solve the problem by reweighting the training data so that the distribution of training is closer to the distribution of test using Kullback-Leibler Importance Estimation Procedure base on "Kullback-Leibler divergence" theorem. I do not know if they provide an implementation or it can be implemented easily, but I think it might worth digging as it sounds a professional way to deal the distribution mismatch.

QUICK update (a good solution): I found a Python implementation of KLIEP algorithm of that research paper (last point) to find those weights. It rather seems easy to use! Basically it resamples the training by putting weights (via the KLIEP algorithm) so that the assumption of having a similar distribution of train and test holds true as much as possible.


Yes, it is very common and sometimes necessary to use the target variable for stratified sampling.

Consider the case of fraud detection, given a bunch of features about a person (e.g. income, gender, position etc) we want to predict the likelihood of that person has committed the crime (a boolean value indicating whether the person is a suspect). This dataset is likely to be very asymmetric with very few positive examples.

Now if we want to use k-fold cross-validation, we must stratify the samples using the target variable. If we don't, we might end up with a fold without any positive example at all and no metrics can be calculated from that fold.


What is the the cost of combining categorical variables?

It depends on whether you retain the original columns or not.  You are not providing any additional information to the NN either way, so it's just a matter of how many features to compute in each batch.  That said, for the same reason, you might as well just use a value (e.g. -1 or 0) for the missing cities.  Depending on your implementation language, this is probably quicker easier to implement in terms of data engineering, and will amount to the same thing in terms of results.  Obviously the NN will need numerical values, so effectively you could just label encode the city feature and assign missing values to one of the labels.

There are some very minor considerations, such as if you combine state and city, you will end up with a larger label set for the single column, which, if you normalise to an appropriate range (e.g. 0 to 1), which NN's often benefit from and which can help to avoid exploding gradients, you will have a finer graduation than a column each for city and state.  Depending on the size and complexity of your data, this could have an impact on the NN's performance (prediction-wise) but it is likely to be a very minor effect compared to all of the other decisions you'll need to make in the process.


Understanding these probabilities, and reading the pdf and cdf

The payoff aren't +30 and -10. They're +20 and -10. Successful projects return \$30M after investing \$10M.


How to asses employees performance using their data?

Assuming that the values denoting positive and negative behaviors are both positive, you can set score as -

score = -0.7*negative_behavior_value + 0.3*positive_behaviour_value


Which algorithm to use for efficient resource assignment?

For the most part, this probably doesn't need a ML solution. Instead it'll probably be based on assigning rules based on business logic. For example, If the Customer is from a certain country, assign +5 points. Then the tickets are assigned out based on that.

This way, it captures a lot of information and goals you're wanting to achieve as business.

If you wanted to go down the ML route, my hunch is a model such as RandomForest or a tree based solution would be helpful as they're relatively explainable. At least as a starting point.

Uber Engineering team talk about how they prioritize their tickets here, which is an interesting read.


What is the purpose of the discriminator in an adversarial autoencoder?

The purpose of having a prior distribution $p(z)$ in any generative adversarial network is to be able to smoothly match a latent code $z$ in a known distribution to an input $x$ in the domain and vice versa. The encoder of a simple autoencoder, without any additional measures other than the in typical pipeline
$$x \rightarrow E \rightarrow z \rightarrow D \rightarrow x'$$
would only require $x$ to approach $x' = D(E(x))$, and for that purpose the decoder may simply learn to reconstruct $x$ regardless of the distribution obtained from $E$. This means that $p(z)$ can be very irregular, making generation of new samples less feasible. Even with slight changes to the bottleneck vector, we cannot be sure that the encoder would ever be able to produce that code with any $x$.

In an adversarial autoencoder (AAE) however, the encoder's job is two-fold: it encodes inputs in $p(x)$ to the respective code in $q(z)$ so that:

it minimizes the reconstruction cost $f(x, D(E(x)))$ (where $f$ is a distance metric between samples, such as the mean squared error);
while it learns to adapt $q(z)$ to the prior distribution $p(x)$.

The latter task is effectively enforced because the discriminator receives:

positive feedback from codes in $p(z)$;
and negative feedback from codes in $q(z)$.

Even if the discriminator might not know anything about any of the two distributions at the beginning, it is only a matter of making enough iterations before it does. The ideal encoder will manage to trick the discriminator to the point of having approximately 50% accuracy in the discrimination process.

Also note that $p(x)$ may not be just a Gaussian or uniform distribution (as in, some sort of noise).
Quoting from Goodfellow's Deep Learning book (chapter 20):

  When developing generative models, we often wish to extend neural networks to implement stochastic transformations of $x$. One straightforward way to do this is to augment the neural network with extra inputs $z$ that are sampled from some simple probability distribution, such as a uniform or Gaussian distribution. The neural network can then continue to perform deterministic computation internally, but the function $f(x, z)$ will appear stochastic to an observer who does not have access to $z$.

Although denoising autoencoders rely on this aspect to learn a model that ignores noise from a sample, the same paper on AAEs (section 2.3) shows that combining noise with a one-hot encoded vector of classes can be used to incorporate label information about the sample. This information is only provided to the discriminator, but it still influences how the encoder produces $q(z)$.


Am I doing a log transformation of data correctly?

Yes, log transform seems a good solution for better interpretation. Overlap between these two datasets is really small, so, only by looking at the plot, you can say with high certainty, that they are significantly different from each-other.


Binary classification: best ways to pre-procees the data

A better clarity about the kind of problem that you are trying to solve will be really useful.

I any case the first thing I will suggest, is please go through your features and try to understand the relationship with the response. All tree based methods are highly prone to overfitting and hence they don't generalise well.
If you come to the conclusion that some variables show a linear relationship and some show a non-linear then you might be better off using Linear methods such as LogisticRegression or LinearDiscriminantClassifier(if your features show an indication that they are sampled from a normal distribution). Obviously you can use PCA or regularization methods if you are certain that some features are highly correlated and offer no prediction value.
Secondly, if you have done all the above then you are probably done with the feature selection step. The next approach is feature creation. See if certain variables are non-linear in nature. Then try adding higher degree polynomial terms. Try diagnosing interaction effects between features, try to include features for those interactions. If there are any features you think are redundant to the problem. Think again. See if you can extract any information out of them.

Kindly tag the link of the problem here.


What does the phrase 'underlying mapping' mean?

Functions map domains to ranges.  Neural networks learn such functions, so you can think of a neural network as a mapping of input spaces to output spaces.  Deep neural networks are stacked with many layers of course, and each of those can be viewed as sub-functions of the network with their own underlying mappings.  For example, each layer in a convolution network consists of some convolution layers + some other helper layers such as normalization and pooling.
The paper is about the discovery that residual connections in 'very' deep networks help them converge to an accuracy at least as good or better than a shallower version of the same network.
Basically, it was found that deep convolutional networks actually underperformed compared to shallower versions, but that was not due to overfitting.  Intuitively, this is confusing because you would expect to be able to construct a deeper network from a shallower version and have it at least perform as well.  How?  You take a network of say 5 convolution layers, and add 5 new ones.  If the 5 new layers learned an underlying mapping that was the Identity function (i.e. input to output in a one-to-one manner), then the networks should perform similarly.  This is not what happens in reality and the deeper network has a difficult time learning Identity mappings.
If we assume an underlying mapping function of a sublayer (one layer of convolutions), $H(x)$, that can learn a complex function, we find that it has difficulty learning mappings that help the network using SGD.  Their insight was to reformulate the underlying mapping as $H(x) = F(x) - x$. That is, some mapping that is a new function with the original input subtracted.  This naturally leads to 'wiring' the network with a residual and the underlying mapping becomes: $F(x) = H(x) + x$.  The Identity mapping is now trivial and the $H(x)$ mapping can learn to add new complexity.


Let's say you have a dataframe with 10,000 rows, and  you have only 1,000 unique ones.

You can do:

df_unique = df.drop_duplicates()
df_sample = df.sample(n)

df_final = pd.concat([df_unique, df_sample], axis=0)

In the above code, n is the amount of sample you want.
In this way you can assure that every unique row is in your dataset and you have more samples on it.


How synthetic sampling of data helps classifier to predict better result?

I think that in order to understand how the SVM handles the new synthetic
data, you should look at the loss function SVM uses, i.e. hinge loss and the behavior on an imbalanced dataset. Intuitively this function will try to fit the hyperplane that best separates the data. For example imagine you have a dataset that is not linearly separable and you have to classes A (with a 1000 samples) and B (with 50 samples). So the SVM will "move" the hyperplane to classify right all samples of A even if this means failing to classify correctly some samples of class B.Then when you create new samples of the classes B the classifier will try to find a balance between the error of the both classes. A more detailed explanation can be found in the 6.3 section fo this article

In general all classifiers tend to classify correctly the most dominating class because of the loss function they use. One way of dealing with it is generating synthetic data, other way is using a cost sensitive classifier, this is the cost of misclassification of the minority class is higher than the majority class. More details can be found in the following links:

https://svds.com/learning-imbalanced-classes/

http://machinelearningmastery.com/tactics-to-combat-imbalanced-classes-in-your-machine-learning-dataset/


How to generate and visualize results of all possible parameter values in Python?

It's probably fastest to do this using numpy, in which you can define the possible ranges of your values using numpy.arange as follows:
import numpy as np

demand = np.arange(100, 201, 1) # integers from 100-200 inclusive
ticket_qty = np.arange(0, 201, 1) # integers from 0-200 inclusive

If you then want to calculate the possibilities using the different values you can use the .outer method of the different universal numpy functions (i.e. numpy.subtract to subtract values) to get easily get the results without looping over the arrays.


Weighted loss functions vs weighted sampling?

First off, I would point out that both concepts can very well coexist. Let's take the following example:

Image classification including 2 classes and samples covering 2 domains. The classes are imbalanced and one domain is "harder" than the other. You could use weighted sampling to sample more examples from the "harder" domain while simultaneously use a weighted loss to counter the class imbalance.

Now let's look at some pros and cons:

Weighted sampling

Weighing each sample implies that you have knowledge of all samples and score them all. From a practical perspective, this is not always possible or feasible. Indeed, in cases where training samples are streamed, you don't control which samples come your way and weighted sampling is impossible. Or in cases where you continually update the weights of each samples.

On the other hand, it is much more practical to update the weights of your samples than to modify your loss throughout training.

Weighted loss

On the plus side, a weighted loss isn't dependent on how the instances are sampled, which can be more practical. On the down side, if you sample your data in such a way that most instances have low weights, your model will not update rapidly.


Is it necessary to tune the step size, when using Adam?

the lr (learning rate) parameter is the MOST influencial parameter for Adam (or any other optimization algorithm). I It does not mean step size... as that is not controlled by Adam but by the neural network and how you feed batches into it.

So yes, lr is very, very important, and it is the learning rate, not the step size.

UPDATE

Actually as mentioned by OP, the learning rate and step size are the same thing (my bad, I confused it with the batch size).

Generally, my recommendation is to try with learning rates of 0.0001, 0.001, 0.01 and 0.1.

The rule of thumb is that a smaller learning rate provides a much stable learning curve... but a much slower one while a larger learning rate will converge quicker... but it will become unstable. So you will have to do a bit of exploration. I normally use 0.001 and 0.0001 but that depends a lot on the problem you are dealing with.

Also, as compared with SGD, Adam includes momentum, essentially taking into account past behaviour (the further away such behaviour is in the past, the least relevant it is) that is controlled by the beta params, but I do not recommend to modify them, as normally it is not required. So, although Adam is a more stable learning method (and normally a quicker one) than SGD, the learning rate it is still the most fundamental parameter to play with.


Are GPTs close to real intelligence or just another Data In -Data Out -Data Permutation and Combinations?

To answer that we Need to have a formal Definition of real intelligence. You got multiple definitions, but which one satisfies you?
Lets say you Claim real intelligence is Awareness. Than if you ask a machine can you prove to me that you are Aware you wont be satisfied. But can you prove you are Aware.
I mean at the end of the day it does not even matter. What matter is we are solving some specialised takes with GPTs a lot better than with other language models. Thats it. GTP(3) has a lot of flaws, just Google it.


Neural Network for Multiple Output Regression

What you are describing is a normal multidimensional linear regression. This type of problem is normally addressed with a feed-forward network, either MLP or any other architecture that suits the nature of the problem.

Any neural network framework is able to do something like that.

The key to do that is to remember that the last layer should have linear activations (i.e. no activation at all).

As per your requirements, the shape of the input layer would be a vector (34,) and the output (8,).

Update: the usual loss function used for regression problems is mean squared error (MSE). Here's an example of multidimensional regression using Keras; the network is not an MLP but it should be Ok to illustrate the idea.


Python or R for implementing machine learning algorithms for fraud detection

I would say that it is your call and purely depends on your comfort with (or desire to learn) the language. Both languages have extensive ecosystems of packages/libraries, including some, which could be used for fraud detection. I would consider anomaly detection as the main theme for the topic. Therefore, the following resources illustrate the variety of approaches, methods and tools for the task in each ecosystem.

Python Ecosystem

scikit-learn library: for example, see this page;
LSAnomaly, a Python module, improving OneClassSVM (a drop-in replacement): see this page;
Skyline: an open source example of implementation, see its GitHub repo;
A relevant discussion on StackOverflow;
pyculiarity, a Python port of Twitter's AnomalyDetection R Package (as mentioned in 2nd bullet of R Ecosystem below "Twitter's Anomaly Detection package").

R Ecosystem

CRAN Task Views, in particular, Time Series, Multivariate and Machine Learning;
Twitter's Anomaly Detection package;
Early Warning Signals (EWS) Toolbox, which includes earlywarnings package;
h2o machine learning platform (interfaces with R) uses deep learning for anomaly detection.

Additional General Information

A set of slides, mentioning a variety of methods for anomaly detection (AD);
A relevant discussion on Cross Validated;
ELKI Data Mining Framework, implementing a large number of AD (and other) algorithms.


Yes embedding works on unseen images. That is the point of the approach - instead of learning to classify each face in the training set directly, the model learns a vector that is intended to be:

far apart when the face itself is different
similar when the face is the same, but other items in the image - e.g. background, hair style etc - are different

In order to use embeddings to recognise someone, you need one or more sample images of that person, so that you can generate a target embedding. Then you can predict any new image that generates a new embedding close enough to the target embedding(s) to be an image of the same person.

This can still suffer usual problems of overfitting to training set. So it is important to understand what training data was used for the model. Embeddings generated from images that are very different from the training data will not be as reliable as embeddings of images that are similar to the training data.


How can one use a validation set to reduce overfitting Naive Bayes?

It is not much about the algorithm you use. It is about the fact that you learned so much details from your training set so just tune the parameter inside a loop in which training and validation errors are calculated each time. In NB case, you do not have many parameters in that sense. Probably the features can be inspected instead of parameter. Then see the famous point at which the validation error is the minimum.

Do not use fixed splits. For each calculation in loop do $n$ times splitting + evaluation and take the mean and std of errors. Gives you better impression about the stability of the results and effectiveness of your algorithm.


Remove Local Outliers from Dataframe using pandas

You could use this approach here:

#------------------------------------------------------------------------------
# accept a dataframe, remove outliers, return cleaned data in a new dataframe
# see http://www.itl.nist.gov/div898/handbook/prc/section1/prc16.htm
#------------------------------------------------------------------------------
def remove_outlier(df_in, col_name):
    q1 = df_in[col_name].quantile(0.25)
    q3 = df_in[col_name].quantile(0.75)
    iqr = q3-q1 #Interquartile range
    fence_low  = q1-1.5*iqr
    fence_high = q3+1.5*iqr
    df_out = df_in.loc[(df_in[col_name] > fence_low) & (df_in[col_name] < fence_high)]
    return df_out


Are neural networks able to deal with non-normalised inputs?

Normalization helps to eliminate scale factors that might exist between variables in your data. Take, for example, the classic problem of predicting home prices. If you represent the square footage of your home in square millimeters, a large change in this value will have a relatively small effect on home price, implying a small gradient on this variable. If you represent that value in square kilometers, a small numerical change will have a large impact on price, implying a large gradient. Normalization isn't necessarily required, but can help to balance the problem by making all variables have "equal weight" in your model. If you were to include both the square millimeter and square kilometer variables in your training data, the neural network would likely spend a lot of effort optimizing on the square kilometer variable, since it is numerically more important. You can still do training with un-normalized data, but it will likely take longer, and possibly have worse output if your important variables are numerically smallest.


It looks like you should redefine your task as a regression problem instead of a classification problem, because your target variable is numerical.

The performance will be much better and it will avoid the need for a questionable pseudo-regression-like measure. Mean absolute error and mean squared error are standard regression evaluation measures.


Zero-inflated independent feature in tree-based models

Basically, for categorical random variable, the decision tree will try all potential splits and select the one that minimizes the mse. For continuous random variable, the decision tree tries to find the split which minimizes the mse. You may see that categorizing the data makes fewer potential choices for the split, and, therefore, is not optimal in accuracy. On the other hand, fewer splits to consider during the training decrease running time.
My answer is as follows.
If the train time permits, I would do option 1. This way, you don't lose any data. Maybe exact value of the feature is a key. It may the point especially if your response is unbalanced. Say, this feature may be a good predictor of your response if positive. For example, radiation level is zero for most of people, small positive values are OK, but large positive values are the key. However, we don't know how large values we should look for.
Option 2 speeds up training time which may be of benefit.
Option 3 creates unnecessary redundancy.


Subdivide a numerical vector with a normal distribution

You seem to be describing the method cut in pandas (documentation)

This method does exactly what you want, if you want to separate a dataframe into n equal-sized bins or manually specify the ranges.

Example:

df = pd.DataFrame(np.array([10, 50, 99, 140, 250, 300, 450, 499]), columns=['price'])

bins = pd.IntervalIndex.from_tuples([(0, 100), (100, 300), (300, 500)])
df['price'] = pd.cut(df['price'], bins)

Output:

You can simply specify the number of bins you want:

df = pd.DataFrame(np.array([10, 50, 99, 140, 250, 300, 450, 499]), columns=['price'])

df['price'] = pd.cut(df['price'], bins=3)

Output:

I hope this helps improve your code.


How to approach semi-supervised binary classification problem with few labels only from one class?

I see two approaches:

either you do all your process using only the positive class, so based on one class classification approaches such as isolation forest, one class SVM, recontruction error of autoencoder (trained only by your positive class), and so on... All those classifiers learn from one class.

or you can do it in 2 steps. First try unsupervised approaches (such as clustering) to extract negative (and other positive) class samples. Then use a binary classifier on positive vs negative class. In this approach, you want to be careful with the unsupervised split that give you the input dataset of your second step. You want to avoid mislabeling.


Training dataset decreasing in quality (Google data science blog)

Here are some things to consider. You may already be doing some of them, but it's hard to tell.

As new data arrive, add them to your database. Randomly sample, with replacement, your database. All other things being equal, take a 2/3 sample and reserve 1/3 for testing. Record the training error and the prediction error.
At intervals, take past training data sets and measure prediction error on a sample of new data only.
Prepare time series of the training error, contemporary prediction error and post-training prediction error.
Look for trends and temporal autocorrelation.
Consider possible seasonality in demand for the offered product.
Consider market changes (entry of new competitor, substitution goods, price points)

This is just the view from 30,000 feet. It's hard to say more without knowing more about your model.

Good luck.


In principle no you do not need to check for stationarity nor correct for it when you are using an LSTM.

The thing about stationarity is that it makes prediction tasks much more efficient, and stable. Think about stationarity in terms of a target. When what you are trying to predict is not stationary, it is like trying to shoot a moving target. It isn't impossible to do, but there is a better way to do things.

If you are trying to predict a stationary variable, then what you are trying to do is hit a non-moving, for lack of a better word, stationary target. It is staying in the same spot relative to you.

Correcting for non-stationarities in an ARIMA model is necessary because it is one of the fundamental assumptions that you make about that particular model. LSTMs do not make that assumption. So you don't have to do it with an LSTM. Seems like the answer to your question is the same as what the people you asked originally. Case closed, right?

Well... you remember shooting at moving targets is much more difficult than shooting at a stationary target right? You may want to force stationarity on a problem that you are going to run against an LSTM, simply because you may be able to eek out a little bit more performance by making it easier for the neural network to learn. So although you don't need to do it, it may still be a good idea and give you a boost in performance.


The code you provided only performs the forward propagation of a neural network without using gradient descent to backpropagate the loss. This means that your network parameters do not change, i.e. your model is not learning anything. The model will always output the same value for an input in your training data and the loss will always stays the same.


According to the official documentation, if you had previous versions of opencv-python and opencv-contrib-python installed, you should consider removing them first:

 pip uninstall opencv-python
 opencv-contrib-python

Also, can you try installing them as sudo?

sudo -H pip install opencv-python


It is not a good idea to keep the statefulness like your second figure, because this would prevent the parallelization of the computations.

A stateful RNN (LSTM, GRU, etc) saves the last hidden state and uses it as the initial state for the next batch. If you do it like in your first figure, all sentences in a batch can be computed in parallel. However, for the second figure, you need to compute the last hidden state in sequence 1 to use it for sequence 2, meaning that you need to compute the LSTM for sequence 1 only, then for sequence 2, then sequence 3, and so on, effectively making your batch size 1 and wasting GPU memory.


How is bias added in a convolutional layer

There are two ways bias is usually added to a convolutional layer:

Tied bias: where you share one bias per kernel
Untied bias: where you use use one bias per kernel and output

Check this blog post comparing the two.

Other than that, bias isn't really necessary in CNNs. There are many popular implementations which ignore it altogether (e.g. ResNet-152).


How to figure out if the problem is time series Forecasting or not?

1 - Is the data enough to perform a time-series analysis with credible results for the next quarter?

This dataset is about 700 lines per Region, correct? Assuming you have 5 regions that is 3500 samples. Well this is surely not a large dataset but you can try to perform forecasting models on it as it is not small either. If the model is relatively simple your model may be able to run it. In DS the only way to know what will work is by trying or by having someone that already tried that saying it work, or it don't.

2 - How should I tackle the task to forecast Region Wise Performance as well

Region Wise, you have even less data, but still you can perform it in the same way, and probably you can get this as a bonus from the general model for all regions.

Your only challenge is to encode the Region into a way that can be used for ARIMA, NNs or other regressive models.

Maybe data from the Region is more relevant to a shallow model than the region itself (such as number of employers, population size, level of technological advance ...)


Questions about a multivariate timeseries forecasting model - keras

From Keras docs:
https://keras.io/api/layers/core_layers/dense/

Input shape
N-D tensor with shape: (batch_size, ..., input_dim). The most common
situation would be a 2D input with shape (batch_size, input_dim).
Output shape
N-D tensor with shape: (batch_size, ..., units). For instance, for a
2D input with shape (batch_size, input_dim), the output would have
shape (batch_size, units).
If the input to the layer has a rank greater than 2, then Dense
computes the dot product between the inputs and the kernel along the
last axis of the inputs and axis 1 of the kernel (using tf.tensordot).
For example, if input has dimensions (batch_size, d0, d1), then we
create a kernel with shape (d1, units), and the kernel operates along
axis 2 of the input, on every sub-tensor of shape (1, 1, d1) (there
are batch_size * d0 such sub-tensors). The output in this case will
have shape (batch_size, d0, units).

The input to dense_1_1 is (None, 7, 420). The Dense layer will be applied to every [:,i,:] for in 1..7 slice of the input producing an output of size (None, 7, 60)
If you're expecting an output with shape (None, 60) you can reshape the input to (None, 420*7) but that would not make sense in this case since you would want to apply the Dense layer to each time slice of your input.


Keras Applications - using images larger than the default size

Have a look at where the reshaping happens. Just before that, you can insert a global average pooling layer in. This way you can handle any size.

However, I recommend cropping and scamming. Create multiple crops if necessary and average the results. That is likely still faster than using a bigger image.

How to use Xception

#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""See https://martin-thoma.com/image-classification/ for details."""
from __future__ import print_function

import numpy as np
import json
import os
import time

from keras import backend as K
from keras.preprocessing import image
from keras.applications.xception import Xception
from keras.utils.data_utils import get_file

CLASS_INDEX = None
CLASS_INDEX_PATH = ('https://s3.amazonaws.com/deep-learning-models/'
                    'image-models/imagenet_class_index.json')

def preprocess_input(x, dim_ordering='default'):
    """
    Standard preprocessing of image data.

    1. Make sure the order of the channels is correct (RGB, BGR, depending on
       the backend)
    2. Mean subtraction by channel.

    Parameters
    ----------
    x : numpy array
        The image
    dim_ordering : string, optional (default: 'default')
        Either 'th' for Theano or 'tf' for Tensorflow

    Returns
    -------
    numpy array
        The preprocessed image
    """
    if dim_ordering == 'default':
        dim_ordering = K.image_dim_ordering()
    assert dim_ordering in {'tf', 'th'}

    if dim_ordering == 'th':
        x[:, 0, :, :] -= 103.939
        x[:, 1, :, :] -= 116.779
        x[:, 2, :, :] -= 123.68
        # 'RGB'->'BGR'
        x = x[:, ::-1, :, :]
    else:
        x[:, :, :, 0] -= 103.939
        x[:, :, :, 1] -= 116.779
        x[:, :, :, 2] -= 123.68
        # 'RGB'->'BGR'
        x = x[:, :, :, ::-1]
    return x

def decode_predictions(preds, top=5):
    """
    Decode the predictionso of the ImageNet trained network.

    Parameters
    ----------
    preds : numpy array
    top : int
        How many predictions to return

    Returns
    -------
    list of tuples
        e.g. (u'n02206856', u'bee', 0.71072823) for the WordNet identifier,
        the class name and the probability.
    """
    global CLASS_INDEX
    if len(preds.shape) != 2 or preds.shape[1] != 1000:
        raise ValueError('`decode_predictions` expects '
                         'a batch of predictions '
                         '(i.e. a 2D array of shape (samples, 1000)). '
                         'Found array with shape: ' + str(preds.shape))
    if CLASS_INDEX is None:
        fpath = get_file('imagenet_class_index.json',
                         CLASS_INDEX_PATH,
                         cache_subdir='models')
        CLASS_INDEX = json.load(open(fpath))
    results = []
    for pred in preds:
        top_indices = pred.argsort()[-top:][::-1]
        result = [tuple(CLASS_INDEX[str(i)]) + (pred[i],) for i in top_indices]
        results.append(result)
    return results

def is_valid_file(parser, arg):
    """
    Check if arg is a valid file that already exists on the file system.

    Parameters
    ----------
    parser : argparse object
    arg : str

    Returns
    -------
    arg
    """
    arg = os.path.abspath(arg)
    if not os.path.exists(arg):
        parser.error("The file %s does not exist!" % arg)
    else:
        return arg

def get_parser():
    """Get parser object."""
    from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
    parser = ArgumentParser(description=__doc__,
                            formatter_class=ArgumentDefaultsHelpFormatter)
    parser.add_argument("-f", "--file",
                        dest="filename",
                        type=lambda x: is_valid_file(parser, x),
                        help="Classify image",
                        metavar="IMAGE",
                        required=True)
    return parser

if __name__ == "__main__":
    args = get_parser().parse_args()

    # Load model
    model = Xception(include_top=True, weights='imagenet')

    img_path = args.filename
    img = image.load_img(img_path, target_size=(224, 224))
    x = image.img_to_array(img)
    x = np.expand_dims(x, axis=0)
    x = preprocess_input(x)
    print('Input image shape:', x.shape)
    t0 = time.time()
    preds = model.predict(x)
    t1 = time.time()
    print("Prediction time: {:0.3f}s".format(t1 - t0))
    for wordnet_id, class_name, prob in decode_predictions(preds)[0]:
        print("{wid}\t{prob:>6}%\t{name}".format(wid=wordnet_id,
                                                 name=class_name,
                                                 prob="%0.2f" % (prob * 100)))

Why it works with any size

Look at the model.summary() of Xception, especially the output shape. Notice the global average pooling layer? Before that, the shape is determined by the input. Meaning until that point, it can be anything.

  Global pooling is another type of transition layer. It applies pooling over the complete feature map size to shrink the input to a constant 1 × 1 feature map and hence allows one network to have different input sizes.

Source: Me: Analysis and Optimization of Convolutional Neural Network Architectures


Negative Latent Factors in Factorized Machines

Found after some digging many concepts related to Non-Negative Matrix Factorization, which if properly setup constrain a FM algorith to come up with non negative factors (and therefore predictions.). Many useful material here:

https://en.wikipedia.org/wiki/Non-negative_matrix_factorization
https://towardsdatascience.com/parts-based-learning-by-non-negative-matrix-factorisation-1279ffd29618


I found the dataset. Its called the "Visual Domain Decathlon"

Heres a link: https://www.robots.ox.ac.uk/~vgg/decathlon/


As is common in R the answer depends and you should always double check to make sure including the target (or other) doesn’t affect the answer. But generally it won’t. Superfluous features are ignored as long as newdata includes all expected predictors present during training.


Changing the default scale of a plot to a "custom scale"

You are almost there when you tried to set the xticks values.

Try

ticks = [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100]
plt.xticks(ticks, ticks)

The two parameters are the tick locations and the tick labels and they could be different, if you wanted.

Here's a full sample code

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import random

point_min = 0 # low value for random data
point_max = 101 # high value for random data
point_count = 100 # number of data points

# generate a DataFrame
data3 = pd.DataFrame({'A':random.sample(range(point_min, point_max), point_count)
                    , 'B':random.sample(range(point_min, point_max), point_count)}
                    )

tick_freq = 10 # define how far apart you want your ticks

data3.plot(x='A', y='B', kind='scatter'
         , c=np.repeat(np.atleast_2d([.5, .6, .7]), data3.shape[0], axis=0) # marker color
         , figsize=(15, 5)) # wide plot to show all tick labels
plt.xticks(ticks, ticks)
plot;

The resulting graph looks as below


What is Deep learning approach to count the number of Diamonds in an image?

I say that go with Object Detection. Because you already have annoted images. And segmentation applied to locate objects and boundaries (lines, curves, etc.) in images. . In Recent days EfficientDet Works very well in object detection. And for implementation look here. Best of luck.


What are the exact differences between Deep Learning, Deep Neural Networks, Artificial Neural Networks and further terms?

I will try to explain it in the simplest way I can-

Deep Learning - Deep learning is a machine learning technique that teaches computers to do what comes naturally to humans: learn by example. It is a subclass of Machine Learning where (in layman terms) we use only Neural Networks. https://www.mathworks.com/discovery/deep-learning.html

Deep Neural Network (DNN) - This is just a term for a neural network with many layers and many nodes in layers, which is not usually found in a shallow network. There is no clear boundary between a shallow and deep neural network. I suggest you watch Andrew Ng's course on deep learning to get a better understanding.

Artificial Neural Network (ANN) - This is just another term for a neural network, all neural networks are ANN's.

Feedforward Neural Network - It is a type of neural network where there is no feedback connections. In technical terms, the information flows only in one direction (input to output) in the forward propagation stage.
https://towardsdatascience.com/deep-learning-feedforward-neural-network-26a6705dbdc7

  Shouldn't there also be for instance Deep Recurrent Neural Networks?

There is no need for Deep Recurrent Neural Networks, it would simply be Recurrent Neural Networks.

  Is it correct that Deep Neural Networks must feedforward Neural Networks?

No, RNN is a type of Deep Neural Network which is not a feedforward NN.

So, to answer your questions,

  Does the term Deep Neural Networks ONLY belong to feedforward Neural Networks?

No, the term Deep Neural Networks belong to all Neural Networks with multiple layers. (Architecture does not matter).

  Is the term Deep Neural Networks a collective term for ALL Artificial Neural Networks with multiple layers, or just for all feedforward Neural Networks with multiple layers?

Yes, it is a collective term for ALL Artificial Neural Networks with multiple layers.

  Would it be more accurate to use the collective term "Deep Learning Architectures" with strictly separated subclasses as Wikipedia suggests?

Deep learning architectures just refer to the different neural network architectures used for different tasks - CNN for image processing, RNN for sequence (text, audio etc) processing etc. So, when you are doing a particular task, you can use the associated architecture name. For general purpose, you can just use the term DNN.

Hope this clears your confusion.


Forecasting non-negative sparse time-series data

I have two ideas here, maybe they will be helpful.

Idea 1: Model time between events

You might think of your data as being generated by two processes: the first is a distribution over time intervals, and the second is a distribution over purchase amounts. So to model your data you could create one distribution (gaussian?) over the nonzero values in your dataset, and another over the lengths of sequences of zeros (poisson?).

Idea 2: Model customer inventory

Even though the sales events in your dataset are sparse, you could spend a little time to come up with a model of why the customer is making purchases when they do. In one possible model, the customer has an inventory that shrinks over time, and they make purchases when their inventory crosses some minimum threshold. You could use your sales data to fit the slope (for linear shrinkage) or rate (for exponential shrinkage) as well as the threshold.

This could get arbitrarily complex, since the customer under this model might have different thresholds or shrink rates at different times ... but for starters it could be a useful approach to get a sense of things.


How does "linear algebraic" weight training function work?

Your goal is to find a $w$ such that

$$Xw \approx y$$

and way to model this problem is to minimize the objective function:

$$\min_w\|Xw-y\|^2.$$

Differentiating with respect to $w$ and equate it to zero gives us

$$2X^T(Xw-y)=0$$

$$X^TXw-X^Ty=0$$

$$X^TXw=X^Ty$$

if $X^TX$ is invertible, then we have

$$w=(X^TX)^{-1}(X^Ty)$$

Remark:

We tend to avoid computing inverse and prefer gradient-based method. Complexity of the normal equation method is cubic.


Comparing multi-class results with binary classification results

(As everything in life) It depends.

Often, a single person (a sample) can be within more than a single class. You do have a set of 3 binary classifiers that are known, these are understood as three models to which we can ask the questions:

classifier 1, do this person have MI?
classifier 2, do this person have CAD?
classifier 3, do this person have CHF?

What is not obvious from the above is that (ignoring any physical constraints - I'm no expect in heart diseases) a person may have MI, CAD and CHF at once.  Or just two of them at once, or only one, or none of them.

If you build a classifier that will say that a person has: MI OR CAD OR CHF OR nothing (N); then the binary classifiers cannot be compared against that new classifier because the three classifiers produce a different set of results from the multiclass classifier.  But not all is lost, let's work a couple of options that can be explored:

There are physical limitations that allow us to assume that the classes are disjoint

If one can argue that it is physically impossible for someone to have two of: MI, CAD or CHF at once then we can compare the classifiers.  Note that this is an assumption that needs a proof outside of the ML process: e.g. the definition of the diseases may exclude the possibility of having them at once.

In this case we need to deal with the case where multiple binary classifiers may output a positive class for the same sample.  For example if both the MI and CAD classifiers outputs the positive class for a sample then we need to select either MI or CAD as the true classification.  Most binary classifiers can output (or at least estimate) class probabilities, one can use these probabilities to decide for the winning classifier (the output with the higher probability).

(Note: If the classifiers are different model types from each other it is likely that they will require probability calibration but this is beyond this answer.)

With this process you then have a simple set of 4 classes (N, MI, CAD, CHF) for all samples in the dataset.  You can use this set to compare against the output of the multiclass classifier.

It is (physically) valid for a person/sample to be in multiple classes at once

In this case the multiclass classifier is not classifying four classes, instead it needs to classify eight classes (N, MI, CAD, CHF, MI+CAD, MI+CHF, CAD+CHF, MI+CAD+CHF).

Finding the samples for each of these classes is likely to be difficult (and it is likely you will not find any samples at all for some of them).  Moreover you will need to rebuild your model to include more classes.

On the other hand this is the perfect case for the use of a multiclass and multilabel classifier.  Instead of creating the eight classes (which will likely be unbalanced) several models can predict multiple classes at once.  Notably Decision Tress (and forests) and Neural Nets are good at multiclass with multilabel classification.

If you can go for the multilabel classification you will have matrices as output of your model, for example for some 6 samples you may get:

MI  CAD CHF
0   0   0
1   0   0
1   1   0
0   0   1
0   1   1
0   0   0

This can be directly compare against the output of all three binary classifiers on the same 6 samples.

If you go for the 8 classes (and I strongly advise against that - it is always a pain to deal with very unballanced classes) you can directly compare the output of the binary classifiers against the list of results.  (e.g. class MI+CAD means that both the 1st and 2nd binary classifiers did output a positive result for a sample).

P.S. Why one would prefer multilabel classifiers over building the classes by hand?  Multilabel classifiers allow one to predict multiple classes at once, which do work as a "new class", yet they perform this as part of their decision algorithm.  As a rule of thumb a multilabel classifier is less likely to impact the separability of other classes by adding this "new class" as a combination of existing classes.  A new class built by hand can affect the separability of the other classes if the algorithm (or hyperparameters) weight classes based on their (low) support (number of elements in class against total number of samples).


Sum encoding is similar to one-hot encoding but the difference is that in sum encoding we take one value as '-1' and it is not compared to other value. Whereas in one-hot encoding we create one column for each value to compare against all other values.
It is referred as sum encoding because the intercept represents the grand mean and the contrast estimate is the mean for level 1 minus the grand mean.
In one-hot encoding intercept represents the mean for the baseline condition and contrast estimate represents simple the difference between one particular condition and the baseline.


When is something a Deep Neural Network (DNN) and not NN?

You are right. Mainly any network with more than two layers between the input and output is considered a deep neural network. Libraries like tensorflow provide efficient architecture for deep learning applications such as image recognition, or language modelling using Convolutional neural networks and Recurrent neural networks. Another thing to keep in mind, is the depth of the network also has to do with the number of units being used in the layer. Mainly, as your non-linear hypotheses get complex you will need deep neural networks.


How do we define the noisy-MAX canonical aggregator for Bayesian networks?

As cited in the OpenMarkov tutorial that you linked, the noisyMAX is to extend noisyOR to non-binary variables. You can read more in section 4.1.5 here.


R Recommender System for very sparse matrices required

I would look into the Soft Imputation method that has an implementation in R. It uses iterative soft-thresholding to compute missing values. Calculations are done with a matrix class called "Incomplete" to deal large sparse matrices and allows for quick calculation of scaling/centering rows and columns.

I've had good success using this completing a 10,000 by 10,000 very sparse matrix so I'd imagine it should do fairly well with your dataset.


Celsius to Fahrenheit conversion simple Neural Network

Hard to say without seeing code, but I can imagine several reasons:

You are using an activation function. You want 'linear' or no activation
Learning rate is much too low
Wrong network (should be one dense layer with one weight)
Poor batch size

This works pretty instantly:

from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Nadam

model = Sequential([Dense(1, activation=None, input_shape=[1])])
model.compile(loss='mean_squared_error', optimizer=Nadam(lr=0.1), metrics=['mean_absolute_error'])

F = list(range(-200,200))
C = [(f - 32) * 5 / 9 for f in F]

model.fit(F, C, batch_size=16, epochs=50)

print(model.predict(np.array([300])))
print(model.layers[0].get_weights())

[[148.88893]]
[array([[0.5555556]], dtype=float32), array([-17.777737], dtype=float32)]

300 F -> 148.8889 C, good, and it has learned the conversion is C = 5/9 * F - 17.7778, or C = 5/9 * (F - 32)


What loss function avoids overconfidence?

You could add a penalty by adding a small "distance" to every loss calculation. They you would have a "floor" below which no comparison could be below. The tricky decision is how large to make the penalty. There is a large literature on how to choose the "right" penalty that keep your overconfidence low but maximizes the predictive ability.

If you have a copy of "The Elements of Statistical Learning", then look at page 61 and the 10 + pages following that. There are also other items in the index covering penalized methods. The Wikipedia article on the Lasso methods is also a possible starting point.

When I type this into an R console on a new Linux box I'm outfitting:

 ??penalized

I get:

Help pages:

brglm::profileObjectives        Objectives to be profiled
mda::gen.ridge      Penalized Regression
mgcv::bam       Generalized additive models for very large datasets
mgcv::bam.update        Update a strictly additive bam model for new data.
mgcv::gam       Generalized additive models with integrated smoothness estimation
mgcv::jagam     Just Another Gibbs Additive Modeller: JAGS support for mgcv.
mgcv::null.space.dimension      The basis of the space of un-penalized functions for a TPRS
mgcv::pcls      Penalized Constrained Least Squares Fitting
mgcv::smooth.construct.bs.smooth.spec       Penalized B-splines in GAMs
mgcv::smooth.construct.cr.smooth.spec       Penalized Cubic regression splines in GAMs
mgcv::smooth.construct.tp.smooth.spec       Penalized thin plate regression splines in GAMs
quantreg::rq.fit.lasso      Lasso Penalized Quantile Regression
quantreg::rq.fit.scad       SCADPenalized Quantile Regression
rms::pentrace       Trace AIC and BIC vs. Penalty
VGAM::sm.ps     Defining Penalized Spline Smooths in VGAM Formulas
VGAM::summarypvgam      Summarizing Penalized Vector Generalized Additive Model Fits

I would have gotten more if I were using my older machine ( a macpro that died recently) but I havent' installed an lasso-type packages yet. There are quite a few:

> rownames(available.packages()[  grep("lasso", available.packages()[,1]) , ])
 [1] "biglasso"      "cglasso"       "CVglasso"      "DIFlasso"      "dpglasso"      "elasso"
 [7] "extlasso"      "gglasso"       "glamlasso"     "glasso"        "glassoFast"    "GPCMlasso"
[13] "grplasso"      "hglasso"       "iilasso"       "ipflasso"      "lasso2"        "lassopv"
[19] "lassoscore"    "lassoshooting" "lmmlasso"      "nnlasso"       "palasso"       "ppmlasso"
[25] "prioritylasso" "PUlasso"       "sealasso"      "sglasso"       "Tlasso"


Handling Categorical Features on NGBoost

It does not support at the time (it will come just as xgboost did not have to have it)

Given thats its a boosting method in the first place one can ask whats the history of xbgoost and subsequent cat and lgboost. XGBoost implementation of gradientboosting did not handle categorical features because it did not have to, it was sufficient enough as it was. What made xgboost special was the use of Hessian information. When other implementations (e.g. gbm in sklearn in Python) used just gradients, XGBoost used Hessian information when boosting. Which in turn was super faster.

TL;DR handling categorical features is not just a matter of convinience, it is also important for speed. This feature will come subsequently (if we contribute!) for time its all about representing uncertainty.


How to do time series regression without scikit and numpy in Python?

In case you want to perform a simple time-series regression without using any packages such as Numpy etc, you need to write and solve the model yourself. You can either use gradient descent or least squares to solve the model.

Let's look at a least squares solution. A simple model (omitting sibscripts) would look like:

$$ y = \beta_0 + \beta_1 y_{t-1} + u,$$

where $y$ is explained by a lag $y_{t-1}$ and $u$ is the error term (you can also use more lags if you want). You can "combine" this into a matrix with first column equal to one (the intercept) and the second column containing the lagged $y_{t-1}$. In matrix notation, the model looks like:

$$ y = \beta X + u .$$

Now you can solve by:

$$ \hat{\beta}=(X'X)^{-1} X'y.$$

Now it should be relatively easy (but still some work) to solve the problem without using packages such as numpy.

As an alternative to matrix notation and gradient descent, you can also solve a linear regression by other means, e.g. as demonstrated in this post.


Finding similarity between two histogram plots

I'm curious what other people will say, but one option is to use KL-divergence.  If your two histograms have the same x-axis, you can divide every column by the total count to convert counts to proportions. Then you could treat the histograms as probability mass, and use the KL-divergence.  KL-divergence is really a measure of the distance between two probability distributions, but histograms are an approximation of a discrete distribution. You'd pick one of your distributions to be P and one to be Q, then calculate $-\sum_x P(X)\log{\frac{P(X)}{Q(X)}}$ where $x$ is each bin on the x-axis of the histogram.  Be aware that this distance metric is directional, meaning you'll get a different answer if you swap P and Q.  The reason is that the KL-divergence is a measure of information transfer (like, the information you gain by moving from Q to P), but I think that's irrelevant to your question.


I'm afraid you have to repeat the K-fold CV a few times (with different seeds each time) and average the results. I'd guess that the high variance comes from the small size of the dataset.


ensuring that dependent variable decreases monotonically with independent variable

There are multiple algorithms that support monotonicity, i.e., can learn a monotonic decision rule.

It is easy to enforce a monotonicity constraint when using linear regression and logistic regression; you simply enforce that the corresponding coefficient for that variable is non-negative during optimization.

xgboost supports monotonicity constraints.  However, beware that it is "best-effort" and does not guarantee that the model will be globally 100% monotonic for the entire space of data values.

Neural networks can support monotonicity constraints, by enforcing that the weights are non-negative.

I don't know how to enforce monotonicity with decision trees or random forest classifiers.  See, e.g., https://cs.stackexchange.com/q/69220/755 for a counterexample showing that even if your training set is monotonic, the resulting decision tree might learn a non-monotonic rule.

Some references to check out: https://stats.stackexchange.com/q/257049/2921, https://stats.stackexchange.com/q/342651/2921, https://stats.stackexchange.com/q/341422/2921.


Should the baseline comparison be done on the training or the test set results?

You definitely want the comparison to be based on the test set:

Evaluating on the training set doesn't make sense for all the usual reasons.
Especially in the case of different features, comparing the performance on the training set could be badly misleading: if one of the model overfits, its performance on the training set will appear better but its real performance (on the test set) is likely worse.

Note that it might make sense to study what happens on the training set (e.g. to measure overfitting), but that cannot be the real evaluation used for comparing the models.


BERT is trained on a combination of the losses for masked language modeling and next sentence prediction. For this, BERT receives as input the concatenation of the special token [CLS], the first sentence tokens, the special token [SEP], the second sentence tokens and a final [SEP].

  [CLS] | First sentence tokens | [SEP] | Second sentence tokens | [SEP]

Some of the tokens in the sentences are "masked out" (i.e. replaced with the special token [MASK]).

BERT generates as output a sequence of the same length as the input. The masked language loss ensures that the masked tokens are guessed correctly. The next sentence prediction loss takes the output at the first position (the one associated with the [CLS] input and uses it as input to a small classification model to predict if the second sentence was the one actually following the first one in the original text where they come from.

Your task is neither masked language modeling nor next sentence prediction, so you need to train in your own training data. Given that your task consists of classification, you should use BERT's first token output ([CLS] output) and train a classifier to tell if your first and second sentences are semantically equivalent or not. For this, you can either:

train the small classification model that takes as input BERT's first token output (reuse BERT-generated features).
train not only the small classification model, but also the whole BERT, but using a smaller learning rate for it (fine-tuning).

In order to decide what's best in your case, you can have a look at this article.

In order to actually implement it, you could use the popular transformers python package, which is already prepared for fine-tuning BERT on custom tasks (e.g. see this tutorial).


How to set up multi cluster spark without hadoop on Google Compute engine

As explained in the comments by @Emre, there is a python script on github which you can use to set up a cluster of whatever size you like: Spark Cluster on Google Compute Engine | Ido Green.  It allows you to specify the instance type to use for the workers (slaves) and separately what to use for the master.

Note that it uses gcutil, which is deprecated in favor of gcloud compute


It depends on a lot of factors:

Does the function you're learning change overtime ? If you try to detect dogs in pictures, and you already have a good accuracy, then relearning is not necessary. But if you try to detect topics in news streams, then you'll need to update your model regularly.
Do you have initially enough data ? If you receive a stream of training data, then you should find an algorithm that can be trained in an online manner.
If you are looking for constantly improving the accuracy, then the more data you feed the better results you'll get. Regularly retraining a new model with the old data + the new ones might be sufficient.


Linear Regression Loss function for Logistic regression

You have taken x,y from random space. So, it should not follow any rule/logic.
Get a real data e.g. Iris(just 2 class) or using sklearn make_classification module
 Figure out the approx value of theta for a good model
 Then loop before and after this value
 Have y_true in probability(not the Class)


Cross-fold validation done on whole dataset or training set?

Rather than a train-test split, I would recommend n-fold cross-validation or, better yet, leave-one-out cross-validation.

  My question, is how do I evaluate my model if it appears that based on the train/since it seems like my data is getting different results based on the splitting.

The high variance in your error estimates is a consequence of a very small dataset.  When sampling from such a small dataset, it's super easy to wind up with train and test splits that aren't representative.  Cross-validation will give you a more stable error estimate.

  What I've done is: run the entire scrip in a loop and iterate over 10 different splits and then plot the error bars on the ROC curve. Is this reasonable?

It's a very good idea to run your model evaluation routine several times and average the results.  It sounds to me like you're manually running a less-controlled version of cross-validation.

I'm not sure what you mean when you say "plot the error bars on the ROC curve".  Is area-under-ROC your accuracy metric?  Or are you computing the accuracy then somehow plotting that on the same axes as an ROC curve?

This wasn't part of your question, but it may be helpful nonetheless.  When you have a dataset with more features than examples, then you are incredibly likely to run into problems related to the curse of dimensionality.  In the 302-dimensional space where your features reside, your dataset of just 77 examples is incredibly sparse.

You may want to consider simpler models (logistic regression, shallow decision trees, etc) and employ some type of regularization to discourage your model from learning spurious relationships.  You could also try to use a dimensionality reduction technique to eliminate some of your 302 features.


How do I get Multiple CSV files (csv file names will be column names) from a folder to a pandas dataframe?

How do you get to the second picture from that code? I wil give you rough set of steps to follow along with the main function names. you can search through the pandas documentation for more details.

initialise some empty lists:  ticker_names=[] dataframes=[]
Loop over the files and read them using df = pd.read_csv(file_name)
Take just the Close column of each single ticker's dataframe: df = df["Close"]
Keep track of the tickers' names in a list: ticker_names.append(ticker)
Bring them all together into one dataframe: final_df = pd.concat(dataframes, left_index=True, right_index=True). We uses the indices because you have date-time index values, and we want to keep them all.
Then give each column the correct name: final_df.columns = ticker_names

You can play around with each of these steps, and I would suggest looking at the **pd.merge** documentation.


What types of matrix multiplication are used in Machine Learning? When are they used?

There are two distinct computations in neural networks, feed-forward and backpropagation. Their computations are similar in that they both use regular matrix multiplication, neither a Hadamard product nor a Kronecker product is necessary. However, some implementations can use the Hadamard product to optimize the implementation.

However, in a convolutional neural networks (CNN), the filters do use a variation of the Hadamard product.

Multiplication in Neural Networks

Let's look at a simple neural network with 3 input features $[x_1, x_2, x_3]$ and 2 possible output classes $[y_1, y_2]$.

Feedforward pass

In the feed-forward pass the input features will be multiplied by the weights at each layer to produce the outputs

$\begin{bmatrix}
x_1&x_2&x_3
\end{bmatrix} * \begin{bmatrix}
w_{1,1} & w_{1,2} & w_{1,3} & w_{1,4}\\
w_{2,1} & w_{2,2} & w_{2,3} & w_{2,4}\\
w_{3,1} & w_{3,2} & w_{3,3} & w_{3,4}
\end{bmatrix} = \begin{bmatrix}
h'_1 & h'_2 & h'_3 & h'_4
\end{bmatrix}$

At the hidden layer these will then go through the activation function, if we assume sigmoid then

$\begin{bmatrix}
h_1 & h_2 & h_3 & h_4
\end{bmatrix} = \frac{1}{1+e^{\begin{bmatrix}
-h'_1 & -h'_2 & -h'_3 & -h'_4
\end{bmatrix}}}$

Finally we go through the next set of weights to the output neurons

$\begin{bmatrix}
h_1 & h_2 & h_3 & h_4
\end{bmatrix} * \begin{bmatrix}
v_{1,1} & v_{1,2}\\
v_{2,1} & v_{2,2}\\
v_{3,1} & v_{3,2}\\
v_{4,1} & v_{4,2}
\end{bmatrix} = \begin{bmatrix}
y'_1 & y'_2
\end{bmatrix}$

$\begin{bmatrix}
y_1 & y_2
\end{bmatrix} = \frac{1}{1+e^{\begin{bmatrix}
-y'_1 & -y'_2
\end{bmatrix}}}$

Backpropagation pass

In the backpropagation we will update the weights through gradient descent. Usually derivations will ignore the need for the Hadamard product by just representing the derivatives with indexes, or implying them implicitly. However the Hadamard product can be used to be more explicit in the following places.

$v^{new} = v^{old} - \eta \frac{\partial C}{\partial v}$

$\frac{\partial C}{\partial v} = \frac{\partial C}{\partial \hat{y}} \circ \frac{\partial \hat{y}}{\partial v}$.

$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y$

$\frac{\partial \hat{y}}{\partial v} = \frac{1}{1+exp(v^Th + b)} \circ (1 - \frac{1}{1+exp(v^Th + b)})$.

Let's look at why we can define this last equation as Hadamard product. The $(v^Th + b)$ is computed as (I ignored the bias term)

$\begin{bmatrix}
v_{1,1} & v_{2,1} & v_{3,1} & v_{4,1}\\
v_{1,2} & v_{2,2} & v_{3,2} & v_{4,2}
\end{bmatrix} * \begin{bmatrix}
h_{1} \\
h_{2} \\
h_{3} \\
h_{4}
\end{bmatrix} = \begin{bmatrix}
\sum v_{i,1} * h_{i} \\
\sum v_{i,2} * h_{i}
\end{bmatrix}$

$\frac{\partial \hat{y}}{\partial v} = \frac{1}{1+exp(v^Th + b)} \circ (1 - \frac{1}{1+exp(v^Th + b)}) = \begin{bmatrix}
\frac{1}{1 + e^{\sum v_{i,1} * h_{i}}} \\
\frac{1}{1 + e^{\sum v_{i,2} * h_{i}}}
\end{bmatrix} \circ \begin{bmatrix}
1 - \frac{1}{1 + e^{\sum v_{i,1} * h_{i}}} \\
1 - \frac{1}{1 + e^{\sum v_{i,2} * h_{i}}}
\end{bmatrix} $.

$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y = \begin{bmatrix}
\hat{y}_1\\
\hat{y}_2
\end{bmatrix} - \begin{bmatrix}
y_1\\
y_2
\end{bmatrix} = \begin{bmatrix}
\hat{y}_1 - y_1\\
\hat{y}_2 - y_2
\end{bmatrix}$

$\frac{\partial C}{\partial v} = \frac{\partial C}{\partial \hat{y}} \circ \frac{\partial \hat{y}}{\partial v} = \begin{bmatrix}
\hat{y}_1 - y_1\\
\hat{y}_2 - y_2
\end{bmatrix} \circ \begin{bmatrix}
\frac{1}{1 + e^{\sum v_{i,1} * h_{i}}} \\
\frac{1}{1 + e^{\sum v_{i,2} * h_{i}}}
\end{bmatrix} \circ \begin{bmatrix}
1 - \frac{1}{1 + e^{\sum v_{i,1} * h_{i}}} \\
1 - \frac{1}{1 + e^{\sum v_{i,2} * h_{i}}}
\end{bmatrix}$

As you can see all the matrix multiplications in both these steps are simple matrix multiplication but the Hadamard product can simplify the representation if used.

Convolutional Neural Networks

CNNs add an additional filtering step before passing through the weights. It passes a filter through the matrices to get a value that represents a neighborhood around a value. The filter takes the Hammond product and then sums all the elements of the resulting matrix.

For example, if we have the matrix in green

with the convolution filter

Then the resulting operation is a element-wise multiplication and addition of the terms as shown below. This kernel (orange matrix) $g$ is shifted across the entire function (green matrix) $f$. Performing a Hammond product at each step, and then summing the elements.

Note: The Hammond product is often defined for matrix of exactly the same dimensions, this is relaxed in CNNs due to the filter moving through the image successively. It is also possible to perform the Hammond product on the edges of the image where different padding techniques can be used.


How to paste string and int from map to an array in hive?

Assuming your map is called "M" and you want your array field to be called "A"

SELECT
...
array(concat_ws(":","string1",M[1]),
      concat_ws(":","string2",M[2]),
      concat_ws(":","string3",M[3]) as A
....
FROM table;


Finding roots of a non linear expression when multiplied with a linear expression

I don't think this equation can be solved in general. Since it is an infinitely long polynomial, it cannot be evaluated. It can only be done if the constants G_t:t+n followed a very specific pattern, so that the series becomes equivalent to a known function that we can evaluate. For example:

If $G_{t:t+n}$ were equal to [ 1, 1/2!, 1/3!, 1/4!, ...] (where 4! is the factorial of 4), then the sum would be equal to $e^\lambda$
If $G_{t:t+n}$ were equal to [1, 0, -1/3!, 0, 1/5!, ...] then the sum would be equal to $\sin(\lambda)$

But in general, if G were any random set of constants, the function cannot be evaluated.


Why does PAC learning focus on learnability of the hypothesis class and not the target function?

Fair warning, this is just an intuition and I'm not really expert in this kind of question. Nice question anyway :)
Theoretical models of learning like PAC are meant to be used to prove learnability results. Therefore what matters is not only that the definition corresponds to the intuition of what "learning" means, but also that it's actually technically possible to prove anything with this definition. I suspect that this is why (or one of the reasons why) the PAC definition is constrained to the particular class of functions that the algorithm deals with, since it makes it possible to theoretically determine what is the true best function in this class. This is needed to prove (or disprove) that the algorithm can always return it.
I also suspect that in the universe of all possible functions, the optimal function could always be some kind of infinite non-generalizing function which assigns exactly the true $Y$ for any $X$ (at least to the extent possible, i.e. when $X\neq X' \Rightarrow Y\neq Y'$). In this perspective restricting the class of target functions forces the algorithm to generalize, since it must represent the data in the finite space of the parameters of the class of functions.
This could explain why the definition proposed by OP is not usable:
if I'm not mistaken, there's simply no way to prove any learnability result with it.


The problem with 1000 classes is the ILSVRC2012 challenge which is using a subset of the full imagenet with "only" 1000 classes. The advantage of that is that it is fixed while imagenet itself is always expanding and new classes and examples may be added.

Training on the full ImageNet set is a very long task and will require either a very long time or a long time and a lot of computing powers (talking tens of GPUs). Although it's not short, training ILSVRC2012 can be done in one day up to a few days depending on your hardware resources and complexity of the model being training.


One approach would be to use tf-idf score. The words which occur in most of the queries will be of little help in differentiating the good search queries from bad ones. But ones which occur very frequently (high tf or term-frequency) in only few queries (high idf or inverse document frequency) as likely to be more important in distinguishing the good queries from the bad ones.


Nothing better if you could get more data and make classes (at least close to) balanced!

Choice of algorithm (I believe you using only RFC) entirely depends upon problem statement and as we all know, there is no free lunch in statistics, so you might've to try other algorithms (or just create a pipeline trying few more) as well.

Try over/under sampling and penalize your model by applying some custom matrix for miss classification, if required. Another point to keep note of is performance metric (avoid Accuracy paradox). Apart from deeper dive with F1, Recall & Precision; also try to look into [Kappa]1 or [ROC curves]2.

Based on the limited information (better idea would be to add a graph showing class imbalance) you've provided, this is the best I recommend. Hope it helps!


Dense? or TimeDistributedDense? after LSTM layer in Keras

When using the TimeDistributed, you need to have a sequence through time so that you can apply the same layer (in this case, Dense), to each time slice. In your code, you have return_sequences=False which does not produce a sequence, so you need to apply Dense.
Assuming you meant to put return_sequences=True, then Dense and TimeDistributedDense do the same task in this case.


How to calculate probability of non independence using bayes theorem?

I think in practice you would make the assumption of independence in any case. Considering all the dependencies can get quite complicated very quickly and you'd need many observations. For example, in your example A=1, B=2 and C=2 was never observed,but you can't really tell, if that is because P(A=1 , B=2, C=2 | C1) is small and there is too little data, or if P(A=1 , B=2, C=2 | C1)=0. In many practical applications the approximation with independence introduces less error than not having enough observations for all possible events.

There is hybrid approaches, though, where you would model some dependence, e.g. tree-augmented naive bayes. These approaches usually perform better, but are computationally more complicated as well.


Do 3D bar charts have advantages over 2D bar charts?

A relevant source seems to be

  A Psychophysical analysis of chart readability, 2001

Sadly, I don't have access to it.

Anecdotes

If you have an area chart, giving it a 3D-effect and rotating slightly can help readability. But I don't remember a context where you could not have used a line chart in the first place.

Bad sources

PRO: Looks good (targetdashboard.com)
CON: Values are harder to read from the axes only - you need to add values to
the bars themselves (targetdashboard.com)


Confusion with Notation in the Book on Deep Learning by Ian Goodfellow et al

Let $\mathbf{y}  = \mathbf{W}^T \mathbf{x}$
Then, $\mathbf{y}^T =(\mathbf{W}^T \mathbf{x})^T =\mathbf{x}^{T}(W^T)^T = \mathbf{x}^{T}W $. Note that $\mathbf{W}$ does not have to be a square matrix.
Let $e^{(i)}_{j} = \delta_{i,j} $.
Then,  $y_{i} = \mathbf{y}^{T}e^{(i)} = (\mathbf{x}^T W) e^{(i)} = \mathbf{x}^{T}(We^{(i)}) = \mathbf{x}^{T}W_{:,i}$
and thus
$h_{i} = g(\mathbf{x}^T W_{:,i}+c_{i}) = g(y_{i}+c_{i})$
On the other hand,
$f(..)  = w^{T} \max\{\mathbf{0},W^{T}\mathbf{x}+\mathbf{c}\}+b = w^{T} \max\{\mathbf{0},\mathbf{y}+\mathbf{c}\}+\mathbf{b}$.
Does that answer your question ?


Ways of filtering erronous email addresses using NLP?

More important than the algorithm is having an existing corpus of labeled data.  Most ML algorithms need to train on a huge amount of text before they start producing useful results (for example a NLP algorithm was trained to generate fiction by reading the entire Harry Potter series).

Do you have a list of known good and bad e-mail addresses?

You can still try to group email addresses even if your data isn’t labeled, but it’s harder.


Should I use a decision tree or logistic regression for classification?

Long story short: do what @untitledprogrammer said, try both models and cross-validate to help pick one.

Both decision trees (depending on the implementation, e.g. C4.5) and logistic regression should be able to handle continuous and categorical data just fine. For logistic regression, you'll want to dummy code your categorical variables.

As @untitledprogrammer mentioned, it's difficult to know a priori which technique will be better based simply on the types of features you have, continuous or otherwise. It really depends on your specific problem and the data you have. (See No Free Lunch Theorem)

You'll want to keep in mind though that a logistic regression model is searching for a single linear decision boundary in your feature space, whereas a decision tree is essentially partitioning your feature space into half-spaces using axis-aligned linear decision boundaries. The net effect is that you have a non-linear decision boundary, possibly more than one.

This is nice when your data points aren't easily separated by a single hyperplane, but on the other hand, decisions trees are so flexible that they can be prone to overfitting. To combat this, you can try pruning. Logistic regression tends to be less susceptible (but not immune!) to overfitting.

Lastly, another thing to consider is that decision trees can automatically take into account interactions between variables, e.g. $xy$ if you have two independent features $x$ and $y$. With logistic regression, you'll have to manually add those interaction terms yourself.

So you have to ask yourself:

what kind of decision boundary makes more sense in your particular problem?
how do you want to balance bias and variance?
are there interactions between my features?

Of course, it's always a good idea to just try both models and do cross-validation. This will help you find out which one is more likely to have better generalization error.


How does the validation_split parameter of Keras' fit function work?

You actually would not want to resample your validation set after each epoch. If you did this your model would be trained on every single sample in your dataset and thus this will cause overfitting. You want to always split your data before the training process and then the algorithm should only be trained using the subset of the data for training.

The function as it is designed ensures that the data is separated in such a way that it always trains on the same portion of the data for each epoch. All shuffling is done within the training sample between epochs if that option is chosen.

However, for some datasets getting the last few instances is not useful, specifically if the dataset is regroup based on class. Then the distribution of your classes will be skewed. Thus you will need some kind of random way to extract a subset of the data to get balanced class distributions in the training and validation set. For this I always like to use the sklearn function as follows

from sklearn.model_selection import train_test_split

# Split the data
x_train, x_valid, y_train, y_valid = train_test_split(data, labels, test_size=0.33, shuffle= True)

It's a nice easy to use function that does what you want. The variables data and labels are standard numpy matrices with the first dimension being the instances.


How to code an SVM's equation including kernels?

The SVM problem is a quadratic programming problem. It depends on whether you are willing to call a quadratic programming packages to solve your problem, if you are willing to do so, you just have to reformat the question in the input format of a programming language. Here is a resource for how to call quadratic programing solver in Python and here is how to do it in matlab.

Note that

$$\sum_{i=1}^m \alpha_i-\frac12\sum_{i,j=1}^my^{(i)}y^{(j)}\alpha_i\alpha_jK( x_i, x_j)=e^T\alpha-\frac12\alpha^TA\alpha$$

where $A_{ij}=y^{(i)}y^{(j)}K(x_i,x_j)$ , $K$ is the kernel function, and $e$ is the all one vector.

Now, suppose you are not willing to call a quadratic programming solver. If you read on the document that you attached, you will reach a section called sequential minimal optimization. Here is the wikipedia page of the algorithm.

The idea of the algorithm is that we can't update a single $\alpha_i$ at a time due to the linearity constraint, hence we do the update in pairs of $\alpha_i$ and $\alpha_j$. You might want to refer to Platt's paper for details including heuristic on how to choose the pair for update.


Convnets : do we have separate activation maps for images in a batch

Yes, separate maps. The number of the image in the batch is another dimension.

Max pool is performed on a per-image basis, other layers operate on an entire batch - e.g. batch normalisation.


xgboost: give more importance to recent samples

Just add weights based on your time labels to your xgb.DMatrix.
The following example is written in R but the same principle applies to xgboost on Python or Julia.

data <- data.frame(feature = rep(5, 5),
                   year = seq(2011, 2015),
                   target = c(1, 0, 1, 0, 0))
weightsData <- 1 + (data$year - max(data$year)) * 5 * 0.01

#Now create the xgboost matrix with your data and weights
xgbMatrix <- xgb.DMatrix(as.matrix(data$feature),
                         label = data$target,
                         weight = weightsData)


Tactics to avoid feeling overwhelmed by machine learning

We have to climb up a steep learning curve when we learn about machine learning. Your question is quite general: One of the tactics I use when learning is divide and conquer. Get some coarse overview about the whole area, then pick some particular area and dig deeper only there.
Perhaps the question is too general, the best tactic may vary and depend on the area you address.
But I am not sure if learning the math is always helpful (although it may always be interesting for those who care).

The algorithms can often be applied in a black box approach, and it may be sometimes not necessary to understand an algorithm in math terms (white box), but sufficient to know it's function, strengths and weaknesses (black box).

You may be the first one that tests that algorithm for the domain, so pure experimentation is useful in the end.


Best ML model for predicting yearly data with many blocks?

Your data consists of time series. Recurrent neural networks, RNNs, are applied successfully to problems like this data. Its structure is appropriate for you. Especially, you can use LSTM which is very powerful for time series data. Also there are a lot of papers in arxiv about its capabilities, pros and cons.

I think, linear regression can fail for your task.

I suggest you to read this and this articles to understand how LSTM works and what you can do with it. Then you can decide is it appropriate for your problem or not.


Is it a red flag that increasing the number of parameters makes the model less able to overfit small amounts of data?

It is not necessarily a red flag. Of course, without seeing the code, it is impossible to say that for sure.

What is likely happening here is that adding parameters to your model, it makes it harder for it to converge to some minimum. More parameters roughly mean that your model is able to explain "more complex stuff". And since you have a small amount of data, the explanation should remain rather simple. Therefore, your model is trying to explain something simple in a complicated way, and it might not be easy to do so.

Also, are you using dropout or regularization? If yes, this might also be an issue as these are explicitly use to avoid overfitting.


Gradient descent formula implementation in python

Your goal if to compute the gradients for the whole theta vector of size p (number of variables). Your temp is a vector also of size $p$, which contains the values of gradients of the cost function relative to each of your theta values.

Therefore, you want to substract point-wise the two vectors (with learning rate $\alpha$) to make an update, so no reason to sum the vector.


spikes in the cost vs iteration graph using ReLU activation function

When I run my implementation of MNIST digit recognition I even get those spikes for a sigmoid transfer function. You have to question yourself: how bad are those spikes? I presume you use stochastic gradient descent right. For every iteration you consider a batch (a subspace of the samples) and train on that. Some of those batches contain very difficult pictures and your network will fail at them, resulting in a high cost. This in turn will lead to a temporary high adaptations of your weights and biases. You see that it drops again quite quickly because the other samples will again send it in the right direction.

I think you have more spikes because your ReLU is just better at generalization and thus after a while you will get more peaks (but lower) because the network gets better and better. And in the end the most important criteria is your accuracy and not the cost ;). This has less problems with the spikes because it takes into account all the samples instead of a batch.

Concluding: stochastic gradient descent will go to a low point, but because of it random walk it might bump into some peaks, but in the end it will find a nice low cost.

If you still have some questions, feel free to ask, I am still quite new here and still have to learn how to give good answers.


difference between empirical risk minimization and structural risk minimization?

SVMs were developed by Vapnik (1995,1998), based on the structural risk minimization principle (Vapnik, 1982) from statistical learning theory.

The complexity of the class of functions performing classification or regression and the algorithm’s generalizability are related. The Vapnik-Chervonenkis (VC) theory provides a general measure of complexity and proves bounds on errors as a function of complexity. Structural risk minimization is the minimization of these bounds, which depend on the empirical risk and the capacity of the function class

More Details with Math & Paper

More on SRM

In brief:

Empirical risk minimization (ERM) is a principle in statistical learning theory that defines a family of learning algorithms and is used to give theoretical bounds on their performance. The core idea is that we cannot know exactly how well an algorithm will work in practice (the true "risk") because we don't know the true distribution of data that the algorithm will work on, but we can instead measure its performance on a known set of training data (the "empirical" risk).

  Note the h here stands for the f — our classifier. This function is called Empirical Risk Minimization (ERM).

Structural risk minimization (SRM) is an inductive principle of use in machine learning. Commonly in machine learning, a generalized model must be selected from a finite data set, with the consequent problem of overfitting – the model becoming too strongly tailored to the particularities of the training set and generalizing poorly to new data. The SRM principle addresses this problem by balancing the model's complexity against its success at fitting the training data.

  which is called Structural Risk Minimization (SRM). J(f) is the complexity of the model, usually can be the bound of vector space. λ≥0 is the coefficient choosing the strength of the penalty term.

More Info at


Why is it hard to grant efficiency while using libraries?

I don't think that everyone reaches for C/C++ when performance is an issue.

The advantage to writing low-level code is using fewer CPU cycles, or sometimes, less memory. But I'd note that higher-level languages can call down to lower-level languages, and do, to get some of this value. Python and JVM languages can do this.

The data scientist using, for example, scikit-learn on her desktop is already calling heavily optimized native routines to do the number crunching. There is no point in writing new code for speed.

In the distributed "big data" context, you are more typically bottleneck on data movement: network transfer and I/O. Native code does not help. What helps is not writing the same code to run faster, but writing smarter code.

Higher-level languages are going to let you implement more sophisticated distributed algorithms in a given amount of developer time than C/C++. At scale, the smarter algorithm with better data movement will beat dumb native code.

It's also usually true that developer time, and bugs, cost loads more than new hardware. A year of a senior developer's time might be $200K fully loaded; over a year that also rents hundreds of servers worth of computation time. It may just not make sense in most cases to bother optimizing over throwing more hardware at it.

I don't understand the follow up about "grant" and "disable" and "assert"?


This question is slightly philosophical, but can be explained in this way - If you model is trained on real photographs it will likely not generalize well to things like drawings unless they are photorealistic or contain the features that the model is using to classify the images. You would likely have to include drawings in your training data to be able to do this effectively.

An interesting project leveraging Machine Learning that may have some relevance to what you are talking about is a recent Google project called "Quick Draw!" (link here) You should check it out. Essentially, it takes input of human drawings and outputs a classification label for what it thinks you drew.


Righteous data analysis approach for this industrial process monitoring

The problem you are looking at is a common researched problem in the field of reliability engineering. If I understand you example correctly your goal is to gain an understanding in the following:

What manufacturing conditions cause certain parts (in your situation hoses) to fail sooner than others?

For each variable you mention you will have to establish whether these are constant over the life time of a part or whether they change. Here I measure time as the time that elapses once the part is used by your client in production. In reliability engineering the variables that do not change over time e.g. human operator of the manufacturing process, production batch, etc. are called time independent covariates the ones that change over time, e.g. amount of pressure put on a hose in production (if you have a sensor that reads that kind of information at your client site), are called time dependent covariates.

You will have to transform your data such that it becomes censored survival data. This type of data means that every dependent variable is amount of time in operation from production. Since some of your hoses will likely never have a defect they will leave your observation period. Note that you do need to take these points into consideration too. When you combine all these data points you get something called right censored survival data.

With the right censored survival data and the covariates together you can start to build a accelerated failure time model. Note that it is recommended to first start with a Kaplan Meier estimate of your survival rates before diving into AFT, but I just want to give you some pointers.

A book that I can highly recommend on reliability engineering is:

Practical Reliability Engineering
The New Weibull Handbook

Both don't dive deep into AFT but they do explain the fundamentals of physical systems and how this links to statistics. From experience I know that not many people are working on this field and that you can have a lot of wrong pointers.

In terms of bringing a model like this in production will heavily depend on the quality of your data. Typically sensor data can be of poor quality, moreover, user input tends to be even of worse quality. An article that gives you an idea of what kind of organizational challenges you may face can be found here. Expect a project like this to take several months to years.


How would I implement a model in Keras to generate embeddings based on documents?

For starters, it depends on why you're trying to create Document Embeddings.
There's the TF-IDF algorithm; it's simple to grasp and implement, and it facilitates using cosine distance as a metric. To train in parallel, you could probably feed both documents to the model and have a synchronized counter doing the calculations such as Term Frequency and Inverse Document Frequency, and then in the end you can use those values to scale the elements of your TFIDF matrix.
Alternatively, you can use the Doc2Vec algorithm as explained here.
It's an extension of the Word2Vec model, but parallelizing the training for this one might be more complicated.
I'm sure there are more ways, but these are the two most popular that I know of.


Gradient descent with infinite gradient value

Is the method replacing infinitely large values to some large but
  finite values correct?

Yes. For example, the same problem happens for the logarithm in cross-entropy loss function, i.e. $p_i \text{log}(p'_i)$ when $p'_i \rightarrow 0$. This is avoided by replacing $\text{log}(x)$ with $\hat{\text{log}}(x) = \text{log}(x+\epsilon)$ for some small $\epsilon$.

Similarly, you are changing $f(x)$ in the denominator to $\hat{f}(x) = max(\epsilon, f(x))$.

However, I would suggest $\hat{f}(x) = f(x) + \epsilon$ instead of a cut-off threshold. This way, the difference in $f(x_1) < f(x_2) < \epsilon$ would not be ignored unlike the max cut-off.


K-Means is a clustering technique NOT classification. You don't have the ground truth here to compare with. Hence accuracy doesn't make any sense.

You can train the model and with the test data predict which cluster the test data belongs to.

Try to visualize it, it shall be helpful.


Scikit model is not able to predict sequence correctly

I don't use Python so I can't tell you exactly what is going on but I had a quick look at your data:

A few remarks:

it looks like the vast majority of the points are created artificially by interpolation. Why not, but that's unlikely to reflect the reality of the price changes: I would expect much more variation/noise in a real dataset about car prices.
there's no need to add so many points to the dataset anyway. With all this data there's not even a need to train a model, since virtually every possible instance is already in the data.
It seems to me that there's something weird with the prediction year: normally the higher the prediction year the more the price decreases right? here year 0 has no decrease at all, year 1 has the highest decrease,..., and year 4 has little to no decrease. That could confuse the model.
because it's mostly an artificial dataset the relation is very simple: a basic regression future_price = past_price * a + b would already give quite good results, and the relation can be learned perfectly when adding the prediction_year feature. At least MLP and random forest should give near perfect results.
From a quick look at the code, I suspect that the problem has to do with the scaling. I'm not sure what is supposed to happen there since I'm not familiar with these functions: it might be that the predicted values need to be "un-scaled" at the end maybe? Anyway I don't think scaling it's needed at all here.

For the record the graph was done with R like this:

library(ggplot2)
d<-read.table('interpolated_dataframe.csv',sep=',',header=TRUE)
ggplot(d,aes(past_price,future_price,colour=factor(prediction_year)))+geom_point()


Methods for Determining Possible Causation Between Two Time Series

I find two possible solutions: Granger Causality and Convergent Cross Mapping (CCM).

Granger Causality is based on the t-test of lagged value of first variable with the second variable. The limitation is that it could not eliminate the confounding effect.

The CCM first build the manifold for the first variable X. Then for each time point, weights of neighbors are estimated, and used to estimate Y. Then correlation between estimated Y and Y are tested.


How many columns do X_train and X_test have?

I'd imagine (though can't confirm, as I don't have access to your data) that they have fewer (or more) than 18 columns.

This is because your code

example_measures = np.array([[4,2,1,1,1,2,3,2,1],[4,2,1,2,2,2,3,2,1]])
example_measures = example_measures.reshape(1,-1)

produces an array of shape (1, 18).

EDIT: I've tried matching your dataset, and got the following:

import numpy as np
from sklearn import preprocessing, cross_validation, neighbors
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.datasets import load_breast_cancer

X= load_breast_cancer().data
y= load_breast_cancer().target

X_train, X_test, y_train, y_test = cross_validation.train_test_split(X,y,test_size=0.2)

clf = LinearRegression(normalize=True)
clf.fit(X_train, y_train)

accuracy= clf.score(X_test, y_test)
print(accuracy)

If I call X.shape, I get (569, 30). So, if you want to make your own array to pass to clf, it needs to have 30 columns (one for each feature).


Is your question how to vectorize the equations for updating?  You have a vector of i and j, and for each update you are summing over j.  Calculate the vectors of each of the components and then sum the vector over j.

Providing actual code might help your question.  As is, it seems vague.