Unfortunately, there isn't any.  However, the AbstractDifferenceRecommenderEvaluator can be extended for the purpose. The class would look something like this:

public abstract class AbstractKFoldRecommenderEvaluator extends AbstractDifferenceRecommenderEvaluator {

private final Random random;
public double noEstimateCounterAverage = 0.0;
public double totalEstimateCount = 0.0;
public double totalEstimateCountAverage = 0.0;

private static final Logger log = LoggerFactory
        .getLogger(AbstractKFoldRecommenderEvaluator.class);

public AbstractKFoldRecommenderEvaluator() {
    super();
    random = RandomUtils.getRandom();
}

public double getNoEstimateCounterAverage(){
    return noEstimateCounterAverage;
}

public double getTotalEstimateCount(){
    return totalEstimateCount;
}

public double getTotalEstimateCountAverage(){
    return totalEstimateCountAverage;
}

/**
 * We use the same evaluate function from the RecommenderEvaluator interface
 * the trainingPercentage is used as the number of folds, so it can have
 * values bigger than 0 to the number of folds.
 */
@Override
public double evaluate(RecommenderBuilder recommenderBuilder,
        DataModelBuilder dataModelBuilder, DataModel dataModel,
        double trainingPercentage, double evaluationPercentage)
        throws TasteException {
    Preconditions.checkNotNull(recommenderBuilder);
    Preconditions.checkNotNull(dataModel);
    Preconditions.checkArgument(trainingPercentage >= 0.0,
            "Invalid trainingPercentage: " + trainingPercentage);

    Preconditions.checkArgument(evaluationPercentage >= 0.0
            && evaluationPercentage <= 1.0,
            "Invalid evaluationPercentage: " + evaluationPercentage);

    log.info("Beginning evaluation using {} of {}", trainingPercentage,
            dataModel);

    int numUsers = dataModel.getNumUsers();

    // Get the number of folds
    int noFolds = (int) trainingPercentage;

    // Initialize buckets for the number of folds
    List<FastByIDMap<PreferenceArray>> folds = new ArrayList<FastByIDMap<PreferenceArray>>();
    for (int i = 0; i < noFolds; i++) {
        folds.add(new FastByIDMap<PreferenceArray>(
                1 + (int) (i / noFolds * numUsers)));
    }

    // Split the dataModel into K folds per user
    LongPrimitiveIterator it = dataModel.getUserIDs();
    while (it.hasNext()) {
        long userID = it.nextLong();
        if (random.nextDouble() < evaluationPercentage) {
            splitOneUsersPrefs2(noFolds, folds, userID, dataModel);
        }
    }

    double result = Double.NaN;
    List<Double> intermediateResults = new ArrayList<>();
    List<Integer> unableToRecoomend = new ArrayList<>();
    List<Integer> averageEstimateCounterIntermediate = new ArrayList<>();

    noEstimateCounterAverage = 0.0;
    totalEstimateCount = 0.0;
    totalEstimateCountAverage = 0.0;
    int totalEstimateCounter = 0;

    // Rotate the folds. Each time only one is used for testing and the rest
    // k-1 folds are used for training
    for (int k = 0; k < noFolds; k++) {
        FastByIDMap<PreferenceArray> trainingPrefs = new FastByIDMap<PreferenceArray>(
                1 + (int) (evaluationPercentage * numUsers));
        FastByIDMap<PreferenceArray> testPrefs = new FastByIDMap<PreferenceArray>(
                1 + (int) (evaluationPercentage * numUsers));

        for (int i = 0; i < folds.size(); i++) {

            // The testing fold
            testPrefs = folds.get(k);

            // Build the training set from the remaining folds
            if (i != k) {
                for (Map.Entry<Long, PreferenceArray> entry : folds.get(i)
                        .entrySet()) {
                    if (!trainingPrefs.containsKey(entry.getKey())) {
                        trainingPrefs.put(entry.getKey(), entry.getValue());
                    } else {
                        List<Preference> userPreferences = new ArrayList<Preference>();
                        PreferenceArray existingPrefs = trainingPrefs
                                .get(entry.getKey());
                        for (int j = 0; j < existingPrefs.length(); j++) {
                            userPreferences.add(existingPrefs.get(j));
                        }

                        PreferenceArray newPrefs = entry.getValue();
                        for (int j = 0; j < newPrefs.length(); j++) {
                            userPreferences.add(newPrefs.get(j));
                        }
                        trainingPrefs.remove(entry.getKey());
                        trainingPrefs.put(entry.getKey(),
                                new GenericUserPreferenceArray(
                                        userPreferences));

                    }
                }
            }
        }

        DataModel trainingModel = dataModelBuilder == null ? new GenericDataModel(
                trainingPrefs) : dataModelBuilder
                .buildDataModel(trainingPrefs);

        Recommender recommender = recommenderBuilder
                .buildRecommender(trainingModel);

        Double[] retVal = getEvaluation(testPrefs, recommender);
        double intermediate = retVal[0];
        int noEstimateCounter = ((Double)retVal[1]).intValue();
        totalEstimateCounter += ((Double)retVal[2]).intValue();
        averageEstimateCounterIntermediate.add(((Double)retVal[2]).intValue());

        log.info("Evaluation result from fold {} : {}", k, intermediate);
        log.info("Average Unable to recommend  for fold {} in: {} cases out of {}", k, noEstimateCounter, ((Double)retVal[2]).intValue());
        intermediateResults.add(intermediate);
        unableToRecoomend.add(noEstimateCounter);

    }

    double sum = 0;
    double noEstimateSum = 0;
    double totalEstimateSum = 0;
    // Sum the results in each fold
    for (int i = 0; i < intermediateResults.size(); i++) {
        if (!Double.isNaN(intermediateResults.get(i))) {
            sum += intermediateResults.get(i);
            noEstimateSum+=unableToRecoomend.get(i);
            totalEstimateSum+=averageEstimateCounterIntermediate.get(i);
        }
    }

    if (sum > 0) {
        // Get an average for the folds
        result = sum / intermediateResults.size();
    }

    double noEstimateCount = 0;
    if(noEstimateSum>0){
        noEstimateCount = noEstimateSum / unableToRecoomend.size();
    }

    double avgEstimateCount = 0;
    if(totalEstimateSum>0){
        avgEstimateCount = totalEstimateSum / averageEstimateCounterIntermediate.size();
    }

    log.info("Average Evaluation result: {} ", result);
    log.info("Average Unable to recommend in: {} cases out of avg. {} cases or total {} ", noEstimateCount, avgEstimateCount, totalEstimateCounter);

    noEstimateCounterAverage = noEstimateCount;
    totalEstimateCount = totalEstimateCounter;
    totalEstimateCountAverage = avgEstimateCount;
    return result;
}

/**
 * Split the preference values for one user into K folds, randomly
 * Generate random number until is not the same as the previously generated on
 * in order to make sure that at least two buckets are populated.
 *
 * @param k
 * @param folds
 * @param userID
 * @param dataModel
 * @throws TasteException
 */
private void splitOneUsersPrefs(int k,
        List<FastByIDMap<PreferenceArray>> folds, long userID,
        DataModel dataModel) throws TasteException {

    List<List<Preference>> oneUserPrefs = Lists
            .newArrayListWithCapacity(k + 1);
    for (int i = 0; i < k; i++) {
        oneUserPrefs.add(null);
    }

    PreferenceArray prefs = dataModel.getPreferencesFromUser(userID);

    int size = prefs.length();
    int previousBucket = -1;
    Double rand = -2.0;
    for (int i = 0; i < size; i++) {
        Preference newPref = new GenericPreference(userID,
                prefs.getItemID(i), prefs.getValue(i));
        do {
            rand = random.nextDouble() * k * 10;

            rand = (double) Math.floor(rand / 10);
            // System.out.println("inside Rand "+rand);

        } while (rand.intValue() == previousBucket);
        // System.out.println("outside rand "+rand);
        if (oneUserPrefs.get(rand.intValue()) == null) {
            oneUserPrefs.set(rand.intValue(), new ArrayList<Preference>());
        }
        oneUserPrefs.get(rand.intValue()).add(newPref);

        previousBucket = rand.intValue();

    }

    for (int i = 0; i < k; i++) {
        if (oneUserPrefs.get(i) != null) {
            folds.get(i).put(userID,
                    new GenericUserPreferenceArray(oneUserPrefs.get(i)));
        }
    }

}

/**
 * Split the preference values for one user into K folds, by shuffling.
 * First Shuffle the Preference array for the user. Then distribute the item-preference pairs
 * starting from the first buckets to the k-th bucket, and then start from the beggining.
 *
 * @param k
 * @param folds
 * @param userID
 * @param dataModel
 * @throws TasteException
 */
private void splitOneUsersPrefs2(int k, List<FastByIDMap<PreferenceArray>> folds, long userID, DataModel dataModel) throws TasteException {

    List<List<Preference>> oneUserPrefs = Lists.newArrayListWithCapacity(k + 1);
    for (int i = 0; i < k; i++) {
        oneUserPrefs.add(null);
    }

    PreferenceArray prefs = dataModel.getPreferencesFromUser(userID);
    int size = prefs.length();

    List<Preference> userPrefs = new ArrayList<>();
    Iterator<Preference> it = prefs.iterator();
    while (it.hasNext()) {
        userPrefs.add(it.next());
    }

    // Shuffle the items
    Collections.shuffle(userPrefs);

    int currentBucket = 0;
    for (int i = 0; i < size; i++) {
        if (currentBucket == k) {
            currentBucket = 0;
        }

        Preference newPref = new GenericPreference(userID, userPrefs.get(i).getItemID(), userPrefs.get(i).getValue());

        if (oneUserPrefs.get(currentBucket) == null) {
            oneUserPrefs.set(currentBucket, new ArrayList<Preference>());
        }
        oneUserPrefs.get(currentBucket).add(newPref);
        currentBucket++;
    }

    for (int i = 0; i < k; i++) {
        if (oneUserPrefs.get(i) != null) {
            folds.get(i).put(userID, new GenericUserPreferenceArray(oneUserPrefs.get(i)));
        }
    }

}

private Double[] getEvaluation(FastByIDMap<PreferenceArray> testPrefs, Recommender recommender) throws TasteException {
    reset();
    Collection<Callable<Void>> estimateCallables = Lists.newArrayList();
    AtomicInteger noEstimateCounter = new AtomicInteger();
    AtomicInteger totalEstimateCounter = new AtomicInteger();
    for (Map.Entry<Long, PreferenceArray> entry : testPrefs.entrySet()) {
        estimateCallables.add(new PreferenceEstimateCallable(recommender, entry.getKey(), entry.getValue(), noEstimateCounter, totalEstimateCounter));
    }
    log.info("Beginning evaluation of {} users", estimateCallables.size());
    RunningAverageAndStdDev timing = new FullRunningAverageAndStdDev();
    execute(estimateCallables, noEstimateCounter, timing);

    Double[] retVal = new Double[3];
    retVal[0] = computeFinalEvaluation();
    retVal[1] = (double) noEstimateCounter.get();
    retVal[2] = (double) totalEstimateCounter.get();
    //retVal.put(computeFinalEvaluation(), noEstimateCounter.get());
    //return computeFinalEvaluation();
    return retVal;
}}

The implementation can be something like this:

public class RMSRecommenderEvaluatorModified extends AbstractKFoldRecommenderEvaluator {

private RunningAverage average;

@Override
protected void reset() {
    average = new FullRunningAverage();
}

@Override
protected void processOneEstimate(float estimatedPreference, Preference realPref) {
    double diff = realPref.getValue() - estimatedPreference;
    average.addDatum(diff * diff);
}

@Override
protected double computeFinalEvaluation() {
    return Math.sqrt(average.getAverage());
}

@Override
public String toString() {
    return "RMSRecommenderEvaluator";
}}

Source


Doubt to use accuracy or macro f1 measure in an unbalanced classification task

using accuracy for unbalance data means that correct classification for the most populous class members is more important than others. If the importance of correct classifying for all data records is equal in your problem accuracy is one of the worst choices.

There are some other good choices beside F1-macro which can be more helpful. Some of these metrics are as follows:

"Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN",
"MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC",
"AUCI", "G", "DP", "DPI", "GI"

Disclaimer:

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)


How does the given data gets plotted on a graph

First let's understand the problem. This is a classification task. The last column represents your target value and rest of the columns are features for a given id. The target value represents which class a given row will belong to. In the plot it is represented by red (Class - 1) and green (Class - 0). We use rest of the column values to plot a given point.

Now we just simply plot one value vs another. The graph that you have shown is for only two variable but in reality we will be using all the variable and the separation will not be a line, but a hyperplane. Since it is difficult to visualize, we usually use only two dimensions for demonstration purposes.

As an example, here is a plot for Age vs Grade marking different classes with different colors:

I hope you get the basic idea. Also, I suggest you to look at tutorials for kNN classifier. It will give you more understanding of the cocept.


Does the choice of error function impact the model parametrs?

Yes, it will impact because when you change the loss function, the numerical value of the loss function will change. So, this will change gradient values of the parameters during the back propagation. Therefore, change in loss function will impact the parameters.


Loss function for multivariate regression where relationship between outputs matters

MSE uses the euclidean distance function. You could simply use the Pearson correlation coefficient as similarity measure, so define a new distance function that decreases when the coefficient increases ($exp(-\rho)$ for instance).

Another possibility is to use the Wasserstein distance (or Wasserstein metric, or earth mover's distance). See the definitions on Wikipedia, especially this one which is not too complicated.
This metric is designed to compute a distance between distributions (continuous) or histograms (binned / discrete). It does not relate to correlation, but rather to the shape of the vector when plotted as an histogram. This is not what you asked for but it might be what you're actually looking for.


YOLO algorithm - understanding training data

For each bounding box you need

p_c: any object / no object (background)
b_x, b_y, b_w, b_h: x, y, width and height of the bounding box
c_i: object i / no object i

For e.g. 2 bounding boxes and 3 classes (e.g. car, person, traffic light) your input vector would look as follows (the superscript in brackets denote the index of the bounding boxes)

\begin{bmatrix}
p_{c}^{(1)}\\
b_{x}^{(1)}\\
b_{y}^{(1)}\\
b_{w}^{(1)}\\
b_{h}^{(1)}\\
c_{1}^{(1)}\\
c_{2}^{(1)}\\
c_{3}^{(1)}\\
p_{c}^{(2)}\\
b_{x}^{(2)}\\
b_{y}^{(2)}\\
b_{w}^{(2)}\\
b_{h}^{(2)}\\
c_{1}^{(2)}\\
c_{2}^{(2)}\\
c_{3}^{(2)}\\
\end{bmatrix}

The whole image is fed into the model. That is essentially why YOLO is so fast. It looks at the whole image only once.
This is done by the CNN. Basically each portion of a convolution corresponds to a grid cell. For example the upper right cell in an image would correspond to the upper right part of the filters in each layer. This is visualized at the left of this image:

Yes they have to have the same size. That's what most CNNs expect. All images in the training set must have equal sizes and so do the images of the test set.
The image gets shrinked and deformed into a square of size 608x608


How is bayesian risk computed to prune decision trees?

I'm going to try to answer my question. To (4), I would say yes. For (5), I believe that for each node, there's a specific partition that falls into the node when following the branches down the decision tree. For instance, for Node 2, there are 3 instances (where a=1) that results in [Yes, No, No] for the partitioned output class at Node 2. For (1), I would calculate the risks over each of these relevant examples and sum them. Then at node 2: $\frac{2}{3}(1)+\frac{1}{3}(1)+\frac{1}{3}(1) = \frac{4}{3}$ is the bayes risk where I use 0-1 loss. Then I would use these estimated risks in the pruning algorithm. Can anyone corroborate?
(2) was answered in the comments above, and I'm assuming for (3) that cross-entropy can be used for multi-class loss.


In general, you want to show your model all types of "resistors" it would see in the real world.

Is it feasible for you to manually go through your dataset and label each of these images as resistor A or resistor B? Do you care if your model can make this distinction? If the answer to either of these is no, I would just leave them all as one label and train the model. It's possible you don't have enough training data for your model to learn that these are different types of resistors, but you can evaluate that after you train.


Lack of activation function in output layer at regression?

Activation "linear" is identical to "no activation function". The term "linear output layer" also means precisely "the last layer has no activation function". Whether you use one or the other term might be down to how your NN library implements it. You may also see it described either way around in documents, but it is exactly the same thing mathematically:

$$a^{out}_j = b^{out}_j + \sum_{i=1}^{N^{hidden}} W_{ij}a^{hidden}_i$$

Where $a$ values are activation, $b$ are biases, $W$ is weight matrix.

For a regression problem with a mean squared error objective, this is the most common approach.

There is nothing stopping you using other activation functions. They might help you if they match the target variable distribution. About the only rule is that your network output should be able to cover possible values of the target variable. So if the target variable is always between -1.0 and 1.0, with higher density around 0.0, perhaps tanh could also work for you.


How to drop row where all columns BUT the timestamp are zeros in python

You can use difference to not look at the Timestamp column. Then sum over the horizontal axis looking for non-zero sum rows:

df.loc[df[df.columns.difference(['Timestamp'])].sum(axis=1) != 0]


Differerent ways to detect the appropriate number of topics

There's no "most suitable" way, but there might be one which works better with your data. The only way to know that would be to try all of them.

In case choosing the number of topics is an issue, you might be interested in using the non-parametric extension of LDA for topic modeling, which doesn't require you to specify the number of topics: this is called Hierarchical Dirichlet Processes, see for instance this introduction.


Would averaging two vectors in word embeddings make sense?

Averaging embeddings vectors could make sense if your aim is to represent a sentence or document with a unique vector. For words out of vocabulary it make more sense to just use a random initialisation and allow training of the embedding parameters during the training of the model. In this way the model will learn the representation for the out-of-vocabulary words by itself.

Alternatively, you could use external resources like WordNet [1] to extract a set of synonyms and other words closely related to a specific term, and then leverage the vectors of those close words (averaging them might have sense but it's always a matter of testing and see what happens, as far as I know there are no grounded rules established yet).

[1] https://wordnet.princeton.edu


How does for loops and elif work with python pandas?

Instead of just setting values for specific rows (i.e. rows where fclass is equal to the feature), you are setting the whole column (df.weight) with the value. Given that the last feature of df.class is probably bus_stop, which has a weight of 150 based on your logic, the final output will have a weight column filled with only values of 150.
There are multiple ways of getting to the output you want: (1) you can use .apply with a function containing the logic you want, (2) use the .loc method to set values for just the rows you want (so just the rows where the fclass column has a specific value) in combination with a loop, or even better, (3) use numpy.where/numpy.select to set all values at once without looping over the different features.
# method 1
def weight_function(row):
    if row["fclass"] == "bus_stop":
        return 150
    elif row["fclass"] == "taxi":
        return 100
    else:
        return 50

df["weight"] = df.apply(weight_function, axis=1)

# method 2
for feature in df["fclass"].unique():
    if feature == "bus_stop":
        df.loc[df["fclass"] == feature, "weight"] = 150
    elif feature == "taxi":
        df.loc[df["fclass"] == feature, "weight"] = 100
    else:
        df.loc[df["fclass"] == feature, "weight"] = 50

# method 3
import numpy as np

conditions = [df["fclass"] == "bus_stop", df["fclass"] == "taxi", ~df["fclass"].isin(["bus_stop", "taxi"])]
choices = [150, 100, 50]
df["weight"] = np.select(conditions, choices)


What companies would be great for entry level data science/ machine learning programmers to help fight for a good cause?

If you want to join companies for a good cause then my opinion would be look for startups. Startups have broader definition of work and more freedom with data. You can search through their goals and find what you need.


How to make a neural network generalizes better?

First make sure your dataset is labelled properly in 5 distinct classes from 0 to 4 and no more.
Then add dropout layer in between each layer with prob of 0.1 and gradually increase till 0.5 until you find  a good val score.
In your optimizer add weight decay term with value around 0.1.
Find some appropriate regularizer for your data type online.


I found my own answer. I had defined

w = theano.shared(rng.randn(feats), name="w")

and that was wrong.
The correct definition is:

w = theano.shared(rng.randn(feats, num_classes), name="w")

since the weights link 'feats'-number of input neuron to 'num_classes'-number of output neurons.


Are there any good NLP APIs for comparing strings in terms of semantic similarity?

You can use the Universal Sentence Encoder from Google and calculate the similarity between texts using the cosine similarity or angular distance between their vector representations.


Dataset - Sample pdfs for text processing?

A GitHub repo with ~1,000 pdfs is here.

Another GitHub repo has a corpus of pdf examples, including edge-cases, is here.


sine, cosine transformed cyclical features - am I losing information?

Are 0 and 23 supposed to be the same?
If not then you should use $( \sin(2i\pi/24),\cos(2i\pi/24) )$ instead of $( \sin(2i\pi/23),\cos(2i\pi/23) )$ with $i \in \{0,...,23\}$


When $T$ gets larger, i.e. $$T \rightarrow \infty $$ the probability distribution resembles the uniform distribution, so $q_i = \frac{1}{n}$. You can find the proof here, as well as some other interesting properties.

Given the previously mentioned fact, I guess he is referring to smoothing  in the sense that the noise if left out and what remains are the important patterns of the data, that is the distribution is made less 'spiky' as @Emre mentioned.


Back-propagation and stochastic gradient descent

Stochastic Gradient Descent (SGD) is an optimization method. As the name suggests, it depends on the gradient of the optimization objective.

Let's say you want to train a neural network. Usually, the loss function $L$ is defined as a mean or a sum over some "error" $l_i$ for each individual data point like this

$$ L(\theta) = \frac{1}{N} \sum_{i=0}^N l_i(\theta)$$

where $N$ is the number of data points and $\theta$ the model parameters.

For SGD you would randomly sample $i$ at each time step $t$ and do

$$ \theta_{t+1} = \theta_{t} - \alpha \nabla_{\theta}l_i(\theta_t)$$

with some learning rate $\alpha$ and the gradient with respect to the model parameters $\nabla_{\theta}$.

Backpropagation is now used to compute the gradient $\nabla_{\theta}l_i(\theta_t)$. As $l_i$ depends on a neural network with parameters $\theta$, this is not necessarily straight-forward, but it can be done quite efficiently using the chain rule in a smart way. This involves recursively computing the gradient of parameters in some layer using the gradients from higher layers, i.e. the gradients are computed starting at the network output and moving backwards. Hence the name backpropagation.

The wikipedia article on backpropagation goes through the math in a detailed manner.


How does scaling affect Logistic Regression?

It affects anything optimized by a form of gradient descent, because it affects the relative scale of the dimensions of the input. If A is generally 1000x larger than B, then changing B's coefficient by some amount is in a sense a 1000x bigger move. In theory this won't matter but in practice it can cause the gradient descent to have trouble landing in the right place in B's dimension.

I think the more significant effect may be regularization. All terms are penalized equally in a simple formulation of regularization. Again if A is generally 1000x larger than B, its coefficient will tend be 1000x smaller (all else equal) than B's, so will be far less penalized, when there's no particular reason to treat them differently.

Finally it affects interpretation, if you want to view the coefficients' magnitudes as correlating with importance, for the same reason.


Sum up counts in a data.frame grouped by multiple variables

You can do this using the xtabs function! Here's how I did it using your example data:

# Create example data...
name <- c("Maria", "Thomas", "Maria", "Maria", "Thomas", "Maria")
sex <- c("f", "m", "m", "f", "m", "m")
count <- c(97, 12, 5, 97, 8, 4)
data <- data.frame("name"=name, "sex"=sex, "count"=count)

# Create table...
xtabs(formula=count~name + sex, data=data)

which gives the following output:

        sex
name       f   m
Maria    194   9
Thomas     0  20


Since this dataset is already organised in a table, you can leverage standard SQL functions to perform a large part of the cleanup.
A record seems to be composed of 4 fields, for example:

university name, city, state, country
stanford law school - stanford - ca - united states of america

You could follow these steps to get a cleaner representation of this dataset:

Starting with the highest level (country), find unique values use it to identify all similar sounding words by listing soundex matches with itself (build a join query of the table with itself).
Use these suggested similar matches to fix all mistakes by updating the names.
Continue in this manner till you've fixed all four fields.
Identify missing states by using the city name to query the correct state from the rest of the table; if state and country are missing for "leland stanford junior university", then use the city name "stanford" to join it with itself and get the state/country name from the other records in the table.
For the university name, identify all abbreviations using grep to search for words ending in a dot character. Replace them with full expansions.
Next, breakup the university name into individual words and dump these into a temporary table in a single column. De-duplicate the values in the column so it only contains unique values.
Run the same soundex matching join query as in step 1 to identify similar sounding names, append these suggested similar names in a second column of the temp table.
Manually do a quick sanity check of what you've obtained as suggestions and delete any invalid matches.
Write a procedure to replace the words in each university name with suggested replacements, and you will have a much cleaner dataset.


How to bootstrap the AUC on a data-set with 50,000 entries?

Yes, sample the same size of your data set (with replacement), then find your AUC, say 10,000 times.  You will then have 10,000 different AUCs which will give you some idea of how confident you are in the AUC result.

When you sample with replacement, you will only find around 63% of the entries in your new bootstrapped data set will be unique (so many of these will have 2+ identical entries in the bootstrapped data set.  Every time you make a new bootstrapped sample, the 63% that are unique will change, as will how many times a record is duplicated in the bootstrapped sample.  This helps extract some information on the natural variability of your data.

So to recap, if you have 50,000 records, which means 50,000 probabilities/values and 50,000 class labels:

1) samples 1:50,000 with replacement.  You should find about 31,500 unique records in this new sample.

2) calculate the AUC

3) repeat steps 1 and 2, 10,000 times(pending computational times, I would shoot for at least 1000), and save those 10,000 AUCS.

4)  Now you have 10,000 AUCs.  If you take the 5th percentile and 95th percentile, you have the bootstrapped 95% confidence interval.  The big benefit here is now you can take the mean of those 10,000 AUCs, and give the standard error $(sd/sqrt(n))$ to give some idea of the variability.

Rather than just doing one AUC calculation on your full data and saying the AUC is $.77$, you may end up finding your AUC is $.75 +/- .03$, which is much more reliable to make a claim on.  Now in hindsight you can see your single AUC measure was a little optimistically high.


Is there an upper bound for k in nearest neighbors-based methods?

The predicted value by the nearest neighbor algorithm will actually just take the average value of all points that are close to the new point.

It's up to you to find the number of neighbors that you want to consider. If you have 2000 data points, this are the 2000 'neighbors' of which you will select a number of k neighbors for taking the average and using that average as prediction.

So the maximum number of k is the number of data points in your training set. In your case this is 2000.


See this post on machinelearningmastery to use scipy to find critical t value. This code below seemed to work.

from scipy.stats import t

#savings calc
diff = actualKwh - total

#compute precision of results, starting with standard error
stdErr = totalStd / math.sqrt(len(total.index))

# define probability & degrees of freedom
p = 0.95
degf = len(data.index) - 1

#print("dfLen=", len(data.index))

#print("degf=", degf)

# retrieve t value
t = t.ppf(p, degf)
#print("t =", t)

absPrec = t * stdErr
#print("absPrec =", int(absPrec))

relPrec = (100 * (absPrec / diff)).round(decimals=1)
#print("relPrec =", relPrec)


Where can I download the toy benchmark dataset for RNNs?

In https://arxiv.org/abs/1504.00941 the authors describe the toy dataset:

  We generated a training set of 100,000 examples and a test set of 10,000 examples as we varied T.

They give you values, e.g. $T = 400$. The start of $\S~4.1$ said that values in each example are uniformly drawn from the unit interval. Publishing the generated values on an FTP server would be silly. I suppose they might have published code, or the name of a PRNG algorithm together with a seed value, but supporting such code is not what they were focused on. They did supply hyperparameters, so you have enough information to generate similar examples.

The MNIST pixels, billion word corpus, and TIMIT datasets each come from their own sources, but clearly none of those is a "toy" problem as you asked about.


Bayesian Neural net with non probibalistic Data?

It is possible to predict a single value from a Bayesian Neural Network. Given a set of input data, conduct the forward pass to generate the resulting probability distribution. Then convert that probability distribution to a single specific value in one of the following common ways:

Sample - Take a random sample. That random sample will automatically be weighted by the posterior probability distribution. This type of sampling is similar to Thompson sampling.
Use a measure of central tendency. Given that posterior probability distribution, calculate the most useful measure of central tendency (e.g., mean, median, or mode).


How to process natural language queries?

Natural language querying poses very many intricacies which can be very difficult to generalize.  From a high level, I would start with trying to think of things in terms of nouns and verbs.

So for the sentence: How many books were sold last month?

You would start by breaking the sentence down with a parser which will return a tree format similar to this:

You can see that there is a subject books, a compound verbal phrase indicating the past action of sell, and then a noun phrase where you have the time focus of a month.

We can further break down the subject for modifiers: "how many" for books, and "last" for month.

Once you have broken the sentence down you need to map those elements to sql language e.g.: how many => count, books => book, sold => sales, month => sales_date (interval), and so on.

Finally, once you have the elements of the language you just need to come up with a set of rules for how different entities interact with each other, which leaves you with:

Select count(*)
  from sales
  where
   item_type='book' and
   sales_date >= '5/1/2014' and
   sales_date <= '5/31/2014'

This is at a high level how I would begin, while almost every step I have mentioned is non-trivial and really the rabbit hole can be endless, this should give you many of the dots to connect.


Can someone please explain what this sample function is upto?

From the link you provided:

  Sample a sequence of characters according to a sequence of probability
  distributions output of the RNN

  Arguments:
      parameters -- python dictionary containing the parameters Waa, Wax, Wya, by, and b.
      char_to_ix -- python dictionary mapping each character to an index.

  Returns:
      indices -- a list of length n containing the indices of the sampled characters

You are returning the indices from a dictionary you gave as argument. Why should you use char_to_integer indices.


Why does my model sometimes not learn well from same data?

Shuffle = True should give better results particularly when you are running for more epochs. But I am not clear why there is so huge difference in the accuracy when shuffle changes . One thing you can try is , to increase the number of epochs and see if the accuracy improves. For shuffle = False

Shuffle set to false, allows you to use the previously trained data. Setting this to true means that you either want to retrain or set the epoch to some value greater than 10. To learn, but this increases the chances of memorization (over fitting)


Why maximize ELBO in the variational autoencoder?

For the actual loss function of a VAE, we use $−\mathcal{L}$, more or less. Of course, it's expensive to actually calculate the expectation, which is why we use a single  sample each time, yes?

Yes. It turns out a single sample MC estimate has fairly low variance in this case. However, the Importance Weighted Autoencoder shows that taking multiple samples can be useful.

  It's usually explained that we treat () as being $\mathcal{N}(0,1)$. But simply plugging in $\mathcal{N}(0,1)$ to the KL divergence in the loss (as we do) just ensures that $_(|)$ gets close to it. We know nothing about $()$, right (besides the fact that if all $_(|)$ are close, then so is $_()$?

I prefer to think of it this way. There are in fact two models: a stochastic encoder (inference model) $q_\phi(x,z)=q_\phi(z|x) q(x)$ and a probabilistic decoder (generative model) $p_\theta(x,z)=p_\theta(x|z) p(z)$. Our goal is to enforce that $ q_\phi(x,z) $ and $ p_\theta(x,z) $ are close (small KL divergence).

Note that $q(x)$ and $p(z)$ are the empirical distribution and Bayesian prior, and are fixed ahead of time (not learned). This is for the vanilla VAE. However, these are very different from the aggregate marginals:
$$
q_\phi(z) = \int q_\phi(z|x) q(x) \, dx
\;\;\;\;\&\;\;\;\;
p_\theta(x) = \int p_\theta(x|z) p(z) \, dz
$$
In other words, $p(z)$ is something we choose as part of the model. Our job is then to ensure that the inference marginal $q_\phi(z)$ is close to it. In other words, we know $p(z)$, but we learn $q_\phi(z)$ (also called the inferred prior or aggregate posterior), which we would like to match it.

  How should we think about that KL divergence in the second formula? It's usually pointed out that for fixed $_$, maximizing $\mathcal{L}$ is equivalent to minimizing the KL div. But $_$ is not fixed; it's being learned. We could also improve $\log _()$ (our ultimate goal) by making that divergence worse, couldn't we? What justifies optimizing $\mathcal{L}$ alone (other than tractability)?

It's important to note that while $p_\theta(x)$ is being learned (well, an approximation is being learned), there is actually an "correct" one that can be computed via Bayes Rule:
$$ p_\theta(z|x) = \frac{p_\theta(x|z) p(z) }{p_\theta(x)} $$
when $ p_\theta(x|z) $ is fixed.
In other words, we don't have to learn an inference model; there is an optimal one given by Bayes rule. In theory we can just learn the generative model and use this rule to create the encoder.
But this is of course intractable to compute.

  How should we think about that KL divergence in the second formula?

We are doing variational inference. We cannot compute the true posterior $p_\theta(z|x)$. So instead we compute an approximation $q_\phi(z|x)$. The second KL says the approximation should be good. Notice something important: when the variational posterior approximation is perfect, then the ELBO equals the marginal likelihood. In other words, when our $q_\phi$ is perfect, then we are indeed optimizing the true marginal likelihood, as you noted we want to do.

  We could also improve $\log _()$ (our ultimate goal) by making that divergence worse, couldn't we?

No, decreasing it should always make the marginal likelihood worse. I don't have a proof, but I conjecture it is so.

  What justifies optimizing $\mathcal{L}$ alone (other than tractability)?

Well, it is a lower bound, so maximizing it guarantees we are "pushing up" the true marginal likelihood. As noted above, when the variational approximation is good, we are guaranteed to be doing the right thing.
There is a lot of literature on variational inference theory; I suspect one can say more under more constrained conditions.

  Basically, I'm confused by the common explanation that what we really want is to minimize that second KL divergence, and also that the best way to do that is to maximize the ELBO.

I don't know if it is the best way. It is certainly an efficient one. But in some cases, it may be better to perform inference more directly, e.g. Hamiltonian Monte Carlo methods.


is it possible get a overfit underfit comparation between models, with this chart? (homework)

Your chart seems to show that light GBM models are very inconsistent in terms of F1 score. The other two types of model tend to have lower validation accuracy than training accuracy, suggesting overfitting is occurring to some extent (but this is ubiquitous in machine learning so it’s not a deal breaker by any means). The best median validation performance is by RandomForest, however some outliers underperformed the models using bagging. Possibly a good approach would be to have an ensemble of RandomForest models.


Implicit Training Models in Spark MLlib?

Implicit feedback simply means that given a ratings-dataset, the zeroes are treated as "nothing was rated" as opposed to "the user rated this as a zero". The algorithm they use is based on the one in this paper. The code is pretty difficult to read but available here.

That being said, if you do have binary product choice variables (i.e., user A chose product B, but not the actual ratings), the implicit training could also work as is, even if was not originally designed for it.


When using GridSearchCV with regression tree how to interpret mean_test_score?

By default, GridSearchCV uses the score method of its estimator; see the last paragraph of the scoring parameter on the docs:

If None, the estimator’s score method is used.

And DecisionTreeRegressor.score (indeed, all/most regressors) uses R^2.

In response to your edit: you can specify scoring='neg_mean_squared_error'.  But note too that there's a linear relationship between MSE and R^2, so optimizing either of these are equivalent.


How do I approach grouped anomaly detection?

It seems that your solution does not have to include an "anomaly detection" in context of data analysis, but can more easily be solved with an iteration through the vector assuming a chronological order of the data (you did not mention any other feature so the problem is 1D).

I would suggest the following approaches:

1) Iterate through the vector and if you find N > threshold non-2xx responses in a row, mark starting index of a "cluster".

    vector<int> status, indices;
    int threshold = 10, counter = 0, flag = 0;
    status = read_data("data.csv");
    for(int i = 0; i < status.size(); i++) {
            if (status[i] < 200 && status[i] > 299) {
                    if (++counter >= threshold && !flag) {
                            indices.push_back(i - counter);
                            flag = 1;
                    }
            } else {
                    counter = 0;
                    flag = 0;
            }
    }
    return indices;

2) Once you cross over some non-2xx example, initialize the weight to, let's say 0.0 (zero). For every new negative example, subtract, let's say 0.2 and for a positive example add 0.05 or something and if you end up bellow -2.0 or above 2.0, reset the procedure and in case that it's < -2.0, return the starting index. You can play around with given numbers.

3) Some more complex strategy?

But yes, your solution does not have to include a machinery, try to start from something more simple and it will lead you in the right direction.


The R code below should help you clean your data.
Please note that I made some assumptions -- see the comments.

library(dplyr)
df = read.delim(['Path to your data'], sep = ",")

# I am making the assumption that record can be uniquely identified by the gameid, drive and down variable

df_drive = df %>% group_by(GameID) %>% mutate(max_drive = max(Drive), min_drive = min(Drive)) %>% select(GameID,Drive, down, max_drive, min_drive)
df_down = df %>% group_by(GameID, Drive) %>% mutate(max_down = max(down), min_down = min(down)) %>% select(GameID, Drive,down, max_down, min_down)

# Max and Min List
min_max = inner_join(df_drive,df_down,by=c("GameID","Drive","down"))

# Determine first and last row for each GameID
keep = min_max %>% mutate(idx = ifelse(Drive==min_drive & down == min_down, 'first', ifelse(Drive==max_drive & down == max_down, 'last', 'other'))) %>% filter(idx %in% c('first','last'))

games = inner_join(df, keep, by=c("GameID","Drive","down"))
games %>% filter(idx == 'first') %>% mutate(field_goal = ifelse(PlayType=='Field Goal','yes','no')) %>% select(GameID,yrdln, field_goal)

# I am making the assumption that sp and down will come from the last record.
first_play = games %>% filter(idx == 'first') %>% select(GameID,yrdln) %>% rename(first_play_yrdln = yrdln)
last_play = games %>% filter(idx == 'last') %>% mutate(field_goal = ifelse(PlayType=='Field Goal','yes','no')) %>% select(GameID,Drive,yrdln, field_goal, sp)  %>% rename(last_play_yrdln = yrdln)

df_final = inner_join(first_play, last_play , by=c("GameID")) %>% select(GameID, Drive, first_play_yrdln, last_play_yrdln, field_goal, sp)
df_final

You should get:

      GameID Drive first_play_yrdln last_play_yrdln field_goal sp
1 2009091000     2               22              33        yes  1
2 2009091001     1                5              10         no  1


GE Predix's machine learning (anomaly detection) capabilities

(full disclosure, I do work for GE)

GE itself has been using their GE Predix Anomaly detection algorithms to establish a relationship between anomalies & machine failure for their factories, in an effort to establish a preventive maintenance framework. The algorithms have been heavily tested to catch any point anomalies from a machine data.  Go ahead and test it out for your datasets: The algorithms are available for registered developers (in catalog) to use in their applications.

Technical approaches (algorithms): Univariate/Multi-variate anomaly detection algorithms.  It detect Shift anomalies , Unexpected trends in physical system behaviors, Anomalous patterns & detect changes in relationship among different variables. It also detects data with very high dimensions (>100). Mann-Kendall, Mann-Whitney, Kernel density estimate etc are some of the key algorithms used.


pytorch __init__() got an unexpected keyword argument 'train'

The error implies that it couldn't recognize the train parameter, in the ImageFolder class.
ImageFolder() does not have a parameter train, remove that it will fix the error. Instead of...
train_set = torchvision.datasets.ImageFolder(root=root,  train=True,transform=transform)

...you should have...
train_set = torchvision.datasets.ImageFolder(root=root, transform=transform).

For more check the documentation.


Best way to deal with realistically imbalanced dataset for Regression problem

You said that "an object with label = 0 exhibits random looking motion...whereas those with label = 1 are highly ordered objects". If you can see that, why should your model not be able to? Just train it and see if it works.

However, your dataset does not seem to be very large. You should think about transfer learning from some pre-trained model if your images/objects are not too strange.

There is no a priori need to handle an imbalance of labels, just be sure to compare your loss to a proper baseline, e.g. the label mean.

You said you have additional features, append them after flattening the convolutional layers. If the additional features are also 2D, just add them as a channel to your Input.


How to most effectively utilize historical data to train churn model

That's correct. You will have the same user multiple times. This can introduce a small bias. I would make a few points why this is better than working with your first approach where you have less data:

First, if the same user shows up in multiple slices that means he doesn't churn or churns later than others. So you bias the model towards figuring out what high retention users look like. As you say:

  So our model could potentially learn that if a particular set of features occurs many times in our data set, then the user with these features is less likely to churn

and this is actually a good thing if you have a good mix of user retention periods.

Second, you can include the time $t$ as a feature (or their signup date etc.). This will teach your model that users that haven't churned in the past are less likely to churn in the future and reduce the importance the model puts on these other patterns you mention.

More generally, you should not only have a set of static features that don't change for the same user from $t$ to $t+1$. You should also have set of features that is dependent on $t$, e.g. "the number of logins in the last 7 days". That way the model will see differences between the same user at time $t$ and $t+1$.


Conditional entropy calculation in python, H(Y|X)

##Entropy
def entropy(Y):
    """
    Also known as Shanon Entropy
    Reference: https://en.wikipedia.org/wiki/Entropy_(information_theory)
    """
    unique, count = np.unique(Y, return_counts=True, axis=0)
    prob = count/len(Y)
    en = np.sum((-1)*prob*np.log2(prob))
    return en

#Joint Entropy
def jEntropy(Y,X):
    """
    H(Y;X)
    Reference: https://en.wikipedia.org/wiki/Joint_entropy
    """
    YX = np.c_[Y,X]
    return entropy(YX)

#Conditional Entropy
def cEntropy(Y, X):
    """
    conditional entropy = Joint Entropy - Entropy of X
    H(Y|X) = H(Y;X) - H(X)
    Reference: https://en.wikipedia.org/wiki/Conditional_entropy
    """
    return jEntropy(Y, X) - entropy(X)

#Information Gain
def gain(Y, X):
    """
    Information Gain, I(Y;X) = H(Y) - H(Y|X)
    Reference: https://en.wikipedia.org/wiki/Information_gain_in_decision_trees#Formal_definition
    """
    return entropy(Y) - cEntropy(Y,X)


It highly depends, If you want to make inferences on the total population you need to determine what margin of error and what confidence level you can allow for. These will then help you determine the needed sample size.

If we would want to estimate the population mean $\mu$. The maximum difference between the sample mean $\bar x$ and $\mu$ is given by:

$$ \epsilon = z_{\frac{\alpha}{2}}\frac{\sigma}{\sqrt{n}}  $$

Where $n$ is the sample size, $\sigma$ is the population standard deviation and $z_{\frac{\alpha}{2}}$ is commonly called the critical value which is derived from the standard normal distribution. These values can be found in tables or using r or python, e.g 1.96 corresponds to a 95% confidence level.

Knowing $\sigma$ (you usually don't) will allow you to determine the sample size needed to approximate $\mu$ within $\pm \epsilon $ with a confidence level of $1-\alpha$. You can try using $\sigma =  \frac{1}{2}$ which is usually enough. There exists methods for determining $\sigma$ as well. Anyhow, you may rearrange the above relation as follows:

$$ n = \bigg ( \frac{z_\frac{\alpha}{2}\sigma }{\epsilon}  \bigg )^2 $$

In fact, this question is quite hard to answer. There are many factors that determine the sample size needed for your study. You should definitely go through some material on basic statstics starting with
Awesome probability theory and go on studying inferential statistics.

It is worth taking some time to understand what the total population is in your case and with regard to the inferential question, does the sample include enough relevant observations? I do not feel that enough info is given in your question to provide a good answer.

You may look at similar surveys conducted and see what sample sizes they have used.

Another common approach, when you don't know much about your population is using Slovin's formula :

$$n = \frac{N}{(1+N \epsilon ^2)}$$

Where $N$ is the total population. As an example given a confidence level of 98% $\rightarrow \epsilon = 0.02$  and a total population of 600. we would get

$$n = \frac{600}{1+600\times0.02^2} = 484.$$

Note that Slovin's formula  is easily misused but can give you a good starting point.

I would highly recommend studying statistics if you are planning to continue working with DS.

Hope this provided some help at least.


How to get a rebalance strategy with a cost matrix?

Is not very common to find cost functions where there is a cost associated with a correct answer C(b,b) (in your example).

But supposing there is, I think the solution to the classification could be trivial: I could say "All my predictions are 'b'" and in that way, I could have -10 as a cost many times and thus giving me a negative cost (depending on balance, of course).

I did not know the technique for applying cost you mention (rebalance accordingly), but for me, it would be more natural if the objective function changes to have this into account.

The following article talks about a possibility for addressing this (Instead of rebalancing, we should measure the cost sensitive matrix). And with XGBoost!

As far as I know, the cost function of a XGBoost classification can be personalized.

Cost Sensitive Classification

XGBoost change loss function - A related question in DS


Would you recommend feature normalization when using boosting trees?

Boosting trees is about building multiple decision trees. Decision tree doesn't require feature normalization, that's because the model only needs the absolute values for branching.

Wikipedia for decision tree:

Requires little data preparation. Other techniques often require data normalization....

However, it's always a good idea to normalize your features because:

It's easier to visualize and interpret your model
It's easier to compare another model (e.g. SVM) with the same data set


How would you a apply a cnn to do age estimation on static images?

With a CNN, one of the typical (and simplest) approaches is to perform classification or regression: you train a network with a set of labeled images (supervised learning) with the aim of, after the training, the network will be able to assign the correct label (among the available labels in the training set) to a new image (never seen before).

Basically, you need:

Decide which are your labels: they can be ages from 0 to 100, so you will have 101 labels. You can decide that you have 10 labels, 0-10, 10-20, and so on... For regression you can use real numbers within a range...
Training labeled data: set o pairs $(img,l)$ where $img$ refers to images and $l$ to labels, i.e. for each image there exists only one label among those you have decided to use
Testing labeled data: the same as above, but this images won't be used to train the network. They will be used to test its performance
CNN: design a Convolutional Neural Network (or get one done) to perform classification or regression. There are lots of examples out there
Train the CNN on your training data set, get the appropriate performance metric
Test the CNN on your test set after training, with the same performance metric
Compare training results with test results to evaluate the bias, variance and overfitting issues of the network

EDIT (following  Mark.F comment):

The network will be slightly different if you try to perform classification or regression: for classification, the network can only assign one of the values (typically integers in this scenario) that are available in the training data. For regression, the network will assign a certain value within a range for each image (for example 0-100). The cost functions for both types or network are different and usually also the last layers. What you do need for both approaches is labeled data.

The CNN are used with images because they use 2D filtering over them to extract the most important features. Basically, the CNN can learn the most important "aspects" of the images that best help to perform the desired task


SARIMAX with seasonality greater than one year

From the documentation of statsmodels:

s is an integer giving the periodicity (number of periods in season), often it is 4 for quarterly data or 12 for monthly data. Default is no seasonal effect.

So it depends on how you have your data recorded, if each entry is a year, your seasonality would be 4, however if you have monthly data, then it would be 36.


Get insights from Random forest::Variable Importance analysis

There is a number of different ways to view variable importance for random forests.
A great explanation of them

Variable importance measures

The most basic way is to look at how many times the variable was included in the trees in the random forest. This has the problem that it does not include how much of an effect the variable has on each tree
An improvement on this is to calculate the decrease in impurity that the variable cause in a tree when adding that variable to a tree. This is the default measure for the importance() function in the RandomForest R library. This has the problem of being biased towards continuous variables and ones with high carnality.
Another approach is the permutation importance. How this works is that it shuffles the variables in one column of the data and computes the decrease in accuracy. in R this is importance(type = 1). This is quite a good measure however it needs the features to be normalized to get the best accuracy (Shown in this paper). Its a little slower but still quite fast.
The last approach is the drop column approach. This works by dropping a feature then calculating how much the accuracy decreases from training the random forest on all the features. This is the slowest but most accurate.

Interpreting the results

These methods will all give a list of the feature with a measure of how good it is. This will give you an idea of how the performance of the random forest will change because of that feature.

Ie you might find Age is the most important feature for prediction viewing time. You can then look at a graph of age vs view time to see how those two variable interact.

With a drop column approach there is the possibility of finding that removing a feature will increase accuracy by the decrease in accuracy being an negative value.

For applying to the domain of the problem that is up to you and the real world problem that you are trying to solve with this model.

Hope this helps :)


Does K-Means' objective function imply distance metric is Euclidean

You're right and you're wrong.

  The objective/loss function of K-Means algorithm is to minimize the sum of squared distances

Yes, absolutely.

  written in a math form, it looks like this:
  $$J(X,Z) = min\ \sum_{z\in Clusters}\sum_{x \in data}||x-z||^2$$

Err... sort of. This is definitely the most popular formulation of kmeans, but a more appropriate formulation would be:

$$J(X,Z) = min\ \sum_{z\in centroids}\sum_{x \in data} distance(x,z)$$

In your formulation, you're concretely defining distance as euclidean distance, i.e. the L2 norm. But you can swap out L2 for any distance kernel and apply kmeans.

The caveat here is that kmeans is a "hill climbing" algorithm, which means each iteration should always be at least as good as the previous iteration, and so it must be the case that this improvement will be true for both the E and M steps. For most common distance metrics (L1, L2, cosine, hamming...) this is the case and you're good to go, but there are infinite possible distance metrics and if we're going to be technical about it, the probability that a random distance metric will satisfy this criterion is almost surely 0.

So, to circle back to your question: does the objective function as you formulated it imply the distance metric is euclidean? Yes. But does kmeans only apply to euclidean space? No, absolutely not. Use whatever distance metric you want and throw the EM algorithm at it and bam: you've got yourself a non-euclidean kmeans.

Generally, when people say "kmeans", they're talking about euclidean kmeans. But Kmeans is super easy to modify with a different distance metric and I think only the most pedantic people would argue that you shouldn't call it "kmeans" after such a modification. Although it's generally always described with the objective you posted (which, yes, does imply euclidean distance), you can really drop in pretty much any useful distance metric, throw EM at it, and it'll work. Some people might call it "___ kmeans" depending on the distance metric, but really it's all kmeans.

I think part of the reason kmeans often isn't described this way isn't formulated like this is because its often compared with gaussian mixture models (GMM). With a euclidean norm, kmeans is equivalent to a GMM with diagonal covariance (and hard cluster assignment), and is consequently often described as an easy way to fit a GMM with spherical clusters. But this comparison fails if we use a different distance metric.

I suggest you check out the CrossValidated discussion on this topic. Full disclosure: the highest voted answers disagree with me, but as you probably guessed I think they're being fairly pedantic.


In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have to define two dimensionalities, $d_t$ for the token and $d_p$ for the position, with the total dimensionality $d = d_t + d_p$, so $d>d_t$ and $d>d_p$. We would be decreasing the total size we devote to tokens in favor of positional information.
However, adding them together is potentially a super case of the concatenation: imagine that there is an ideal split of $d$ into $d_t$ and $d_p$ in terms of minimizing the loss; then, the training could converge to position vectors that only take $d_t$ elements, making the rest zero, and the positions were learned and happened the same, taking the complementary $d_p$ elements and leaving the rest to zero.
Therefore, by adding them, we leave the optimization of the use of the $d$ dimensions to the optimization process, instead of assuming there is an optimal partition of the vector components and setting a new hyperparameter to tune. Also, the use of the vector space is not restricted by a hard split in the vector components, but takes the whole representation space.


How can learn machine learning with Statistics background?

I would recommend using Kaggle or a similar website. There are a lot of competition so you probably can find one that motivates you.

One point that is imho often overseen is their very large collection of non-competition datasets. I am rather confident that you will find one that meets a personal interest of yours. You can then start to explore it and discuss results with others.

I really think that machine learning like other programming skills is best learned on practical examples. You will fail, learn from your mistakes only to fall into the next trap and this is very important as these experiences will help you later on!


Choosing the first node in a decision tree, basic example

Some corrections in the above:

While calculating entropy of a split, you should take into account the weights -  fraction of instances in each branch:

$$
entropy(A) = 0 * (2/6) + 0.81 * (4/6) = 0.54\\
entropy(B) = 0 * (3/6) + 0.91 * (3/6) = 0.45\\
entropy(C) = 0 * (2/6) + 0.81 * (4/6) = 0.54
$$

While choosing an attribute to split, you should choose the one with lowest entropy after the split. Entropy is the uncertainty, so you want to minimise it. As an example, if the split was perfect (all 0's on one side, and all 1's on the other), then entropy will be 0.

After selection of one node, the tree is built out recursively, as you mention.


Gracefully removing observations with outliers in N fields

This will achieve what you want. You can remove outliers from any column you wish, just pass that column number as an argument in the function.

id <- c(12,23,34,45,56,67)
num <- c(5,9,2,99,3,4)
prac <- data.frame(id, num)

remove_outliers <- function(x, col) {

  #find position of 1st and 3rd quantile not including NA's
  qnt <- quantile(x[ ,col], probs=c(.25, .75), na.rm = TRUE)

  H <- 1.5 * IQR(x[ ,col])

  x[ ,col] <- ifelse(x[ ,col] < (qnt[1] - H) | x[ ,col] > (qnt[2] + H), NA, x[ ,col])

  #get rid of any NA's
  x <- x[!is.na(x[ ,col]), ]
  x <- assign("dataset", x, envir = .GlobalEnv)
  return(x)
}

remove_outliers(prac, 2)


When should you use learning rate scheduling over adaptive learning rate optimization algorithm?

I'm not sure about other fields but recently in the field of deep neural network training there is this arXiv submission, The Marginal Value of Adaptive Gradient Methods in Machine Learning.

  Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient descent (SGD). We construct an illustrative binary classification problem where the data is linearly separable, GD and SGD achieve zero test error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to half. We additionally study the empirical generalization capability of adaptive methods on several state-of-the-art deep learning models. We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.


ValueError: Expected 2D array, got 1D array instead:

It seems to be because the predict method on your Pipeline object requires the input to match the input of the first object in your pipeline, which is the CountVectorizer. It any case, it only requires an iterable object, which your 1d array indeed is.

The classifier you train without the pipeline ends up being a MultinomialNB object, whose predict method actually requires a 2d array of shape num_samples * num_features.

Maybe you need to pass your test data through each of the individual steps manually before feeding anything into the final classifier?


ROC curves/AUC values as a performance metric

The results you posted are correct, I did a quick check with library(pROC) and got the same thing.  The important feature for AUC/ROC is that the cutpoint of calling a sample "1" or "0" is not set at 0.5.  For each of your predictors, the range of the numbers do not matter at all.  What does matter is how often higher numbers are associated with true positive labels.  Example:

Method 1:[1, 3, 5, 6, 9, 14] (predictor)
Method 1:[0, 0, 1, 0 ,1 , 1] (true label)
AUC: 0.8889

Method 2:[[0.01, 0.02, 200 , 250, 300, 1000000] (predictor)
Method 2:[0, 0, 1, 0 ,1 , 1] (true label)
AUC: 0.8889

The AUC measure is just a measure of how often true "1" samples have a higher number than true "0" samples.  The predictor does not have to even be bounded by [0,1], it can be any range.

Your plot would be more informative if it was sorted.  Consider the below sorted plot (crude, sorry) for each of the three methods, with the "1" labels highlighted in blue.  badaI clearly has the closest grouping of "1"s towards the higher values.


Best Way to tackle to time series classification problem?

Since I don't see the data and I cannot fully understand the data format, I am just giving some general advice. But first, I should mention that it doesn't make sense to consider each output (each 20-element array) as timeseries sequence unless there is a priority among them (by priority, I mean if you know a particular lab value is obtained before others)
So, if you have 20-element array of binary numbers as output, you might find out there is no two ouput exactly the same or there might be only a few observations with exactly similar output (because you can have $2^{20}$ combinations!). So, let's say if you have N observations, and you may see N distinct output array as well. Therefore, you may first cluster the labels into k groups and assign new labels to each group (the label of group i-th is i). Then, find the closest distance between your query and each group, so you can then find the appropriate class. Then, you can predict the ouput in array format (e.g. let's say you realize your query belongs to a group of 20 members). So, you can say, the output is expected value of output arrays in element-wise manner. So, e.g. your first element of 20-element output array might be 0.9. Or, you can be more precise and after finding the appropriate group, define a weight based on the distance between your query and members and find the weighted average of output arrays of that group!


Is there a library that would perform segmented linear regression in python?

No, currently there isn't a package in Python that does segmented linear regression as thoroughly as those in R (e.g. R packages listed in this blog post). Alternatively, you can use a Bayesian Markov Chain Monte Carlo algorithm in Python to create your segmented model.

Segmented linear regression, as implemented by all the R packages in the above link, doesn't permit extra parameter constraints (i.e. priors), and because these packages take a frequentist approach, the resulting model doesn't give you probability distributions for the model parameters (i.e. breakpoints, slopes, etc). Defining a segmented model in statsmodels, which is frequentist, is even more restrictive because the model requires a fixed x-coordinate breakpoint.

You can design a segmented model in Python using the Bayesian Markov Chain Monte Carlo algorithm emcee. Jake Vanderplas wrote a useful blog post and paper for how to implement emcee with comparisons to PyMC and PyStan.

Example:

Segmented model with data:

Probability distributions of fit parameters:

Link to code for segmented model.
Link to (large) ipython notebook.


Error while plotting Logistic Regression Classification

The error occurs because X_train and Y_train are pandas dataframes and you are trying to directly access them as arrays. There are two solutions:

Convert X_train and Y_train to arrays before using them as:

  X_set, y_set = X_train.values, Y_train.values

Whenever you are accessing their values use iloc so everywhere replace X_set by X_set.iloc


Macro and micro average for imbalanced binary classes

does it make sense to use micro and macro precision in binary classification problems when classes are imbalanced?

In general micro- and macro-average performance are not relevant in binary classification, whether the classes are balanced or not. Their value can be especially misleading if there is a strong imbalance, because it takes into account both the minority class (harder for the classifier) and the majority class (easier):

By definition micro-average gives more weight to the majority class, so the micro-average performance can be high even if the classifier does a terrible job at distinguishing the two classes.
The macro-average is not biased towards any of the two classes, still it's uselessly complex, it makes it harder to understand what's going on than simple performance on the positive class, which is normally the minority one (because that's the challenging one).

Of course there can be cases where it makes sense not to follow this standard evaluation setting, it's always a matter of choosing the appropriate way to evaluate a particular task.
The below example illustrates why micro- and macro-average are confusing in a standard case of imbalance:
              true A   true B
predicted A     90        9
predicted B      0        1

For A: precision = 0.91, recall = 1, f1-score = 0.95
For B: precision = 1, recall = 0.1, f1-score = 0.18
micro-average: precision = 0.91, recall = 0.91, f1-score = 0.91
macro-average: precision = 0.95, recall = 0.55, f1-score = 0.70

Assuming we don't know anything else than the selected performance measure, this classifier:

performs almost perfectly according to the performance of the majority class A,
performs very well according to micro-average,
performs decently according to macro-average,
performs terribly according to the performance of the minority class B.

Looking at the confusion table, it's clear that the classifier doesn't do a good job at distinguishing the two classes. So the most "honest" performance measure is the last one, i.e. the non-averaged performance on the minority class.


Ram usage of model keep increasing when training to previously trained model

I found it and answering as it might help somebody.

I saw pattern in this problem. Every time I load the model from my saved ones and then start them training again it will start eating my ram. Slowly it will reach to limit of my laptop.
But same model if i train from scratch will not do same.
I was loading model using

tf.keras.models.load_model("models/min__1576172002.model")

So Instead of training it again. I Build a new one and assign its weights = trained models weight. This way this problem is solved.

        prev_modal = tf.keras.models.load_model("models/min__1576224504.h5")
        model = Sequential()
        model.add(Dense(6,activation='relu',input_shape=(4,)))
        model.add(Dropout(0.2))
        # model.add(Flatten())

        model.add(Dense(5,activation='relu'))
        model.add(Dropout(0.2))
        # model.add(Flatten())

        model.add(Dense(5,activation='relu'))
        model.add(Dropout(0.2))

        model.add(Dense(2,activation='softmax'))

        model.compile(loss="mse", optimizer=Adam(learning_rate=0.001), metrics=['accuracy'])

        model.set_weights(prev_modal.get_weights())


ValueError while using linear regression

It seems in your code you build a pandas dataframe but you do not use it. I recreated your code line by line and was unable to get the same error. Try the following

from sklearn.datasets import load_boston
from sklearn.linear_model import LinearRegression
boston = load_boston()
X = boston.data
Y = boston.target

lineReg = LinearRegression()
lineReg.fit(X, Y)
lineReg.score(X, Y)

This results in an error of $0.7406$. Of course, this result is kind of meaningless because you should split your data into a training and testing set in order to accurately test your results.

You should do the following

from sklearn.datasets import load_boston
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

boston = load_boston()
X = boston.data
Y = boston.target
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

lineReg = LinearRegression()
lineReg.fit(X_train, y_train)
lineReg.score(X_test, y_test )

This will always give a different score because the data is shuffled before being split into your two sets. You MUST separate your training data are the start of your algorithm and do not pollute your training set with the testing set otherwise you risk biasing your model! This is very important.


My CNN model Accuracy doesn't increase (high loss and low acc)

Here are some hacks which you can use to improve then model.

The dataset seems to be inadequate. Try image augmentation.

Image augmentation basically applies to different transformations to your images. Like the rotation, scale, color, whitening etc. are changed. It helps the model to generalise better on the image. See here.

Use softmax classification function.

Instead of sigmoid, try softmax activation function since you are working on a classification task. Sigmoid is mostly reserved for binary classification tasks.

model.add(Dense(1, activation='softmax'))

Add more convolution layers.

Consider adding more Conv2D layers to the model because the image size is quite large. More the number of layers, the better feature extraction will take place. Due to less layers, your model is not able to extract smaller features which may be required for proper classification.

Tips:

Try adam optimizer instead of rmsprop.
Restrict the kernel size to ( 3 , 3 ).
Use a smaller batch size. Your batch size is 20 for 142 images. That makes only ~7 batches. Lower it to a number like 6 or 10.
Use Dropout layers in between the Dense layers.
The smaller learning rate always helps like 0.001 or 0.0001.


How to create new feature based on clustering result

To do clustering you can use sklearn's KMeans Clustering function - sklearn.cluster.KMeans with n_clusters=3 and other parameters as default. This will give you 3 clusters. After you have trained your model you can use the .labels_ attribute of the trained model to classify every example. You can do this in the following way:

>>> from sklearn.cluster import KMeans
>>> import numpy as np
>>> X = np.array([[1, 2], [1, 4], [1, 0],
...               [10, 2], [10, 4], [10, 0]])
>>> kmeans = KMeans(n_clusters=2, random_state=0).fit(X)
>>> kmeans.labels_
array([1, 1, 1, 0, 0, 0], dtype=int32)

To create a new column based on category cluster you can simply add the kmeans.labels_ array as a column to your original dataframe:

>>> df['categories'] = kmeans.labels_


Which method to use to remove trend from time series?

Detrend does a least squares fit (linear or constant) and subtracts this from your data points. You can look this up in the docs.

Simply taking the difference between consecutive data points will in general lead to other results.

In general the regression based detrending seems to be more reasonable. You could also think about using random sample consensus (RANSAC) to be more robust to outliers.


You can use the plot3Drgl library.  E.g.

library("plot3Drgl")
x <- c(1,2,3,4,5)
y <- c(6,7,8,9,10)
z <- c(11,12,13,14,15)
colors <- c(rep("red",3), rep("blue",2))
plot3d(x=x,y=y,z=z,col=colors)

Or you can use plotly:

library("plotly")
x <- c(1,2,3,4,5)
y <- c(6,7,8,9,10)
z <- c(11,12,13,14,15)
colors <- c(rep("red",3), rep("blue",2))
df<-as.data.frame(cbind(x,y,z,colors))
colnames(df)<-c("eat", "pork", "chops", "colors")

p <- plot_ly(data=df, x=~eat, y=~pork, z=~chops, color =~colors, colors=c('#BF382A', '#0C4B8E')) %>% ## Like a pipe?
  add_markers() %>%
  layout(scene = list(xaxis = list(title = 'eat'),
                     yaxis = list(title = 'pork'),
                     zaxis = list(title = 'chops')))
htmlwidgets::saveWidget(as_widget(p), "index.html")

This saves the output as an html file.  In order to get this to work, you'll also need to install pandoc.  On CentOS/RedHat do yum install pandoc pandoc-citeproc.  On Mac OSX, using homebrew, do brew install pandoc.


Using Apache Spark to do ML. Keep getting serializing errors

Most often serialization error in (Py)Spark means that some part of your distributed code (e.g. functions passed to map) has dependencies on non-serializable data. Consider following example:

rdd = sc.parallelize(range(5))
rdd = rdd.map(lambda x: x + 1)
rdd.collect()

Here you have distributed collection and lambda function to send to all workers. Lambda is completely self-containing, so it's easy to copy its binary representation to other nodes without any worries.

Now let's make things a bit more interesting:

f = open("/etc/hosts")
rdd = sc.parallelize(range(100))
rdd = rdd.map(lambda x: f.read())
rdd.collect()
f.close()

Boom! Strange error in serialization module! What just happened is that we had attempted to pass f, which is a file object, to workers. Obviously, file object is a handle to local data and thus cannot be sent to other machines.

So what's happening in your specific code? Without actual data and knowing record format, I cannot debug it completely, but I guess that problem goes from this line:

def vectorizer(text, vocab=vocab_dict):

In Python, keyword arguments are initialized when function is called for the first time. When you call sc.parallelize(...).map(vectorizer) just after its definition, vocab_dict is available locally, but remote workers know absolutely nothing about it. Thus function is called with fewer parameters than it expects which results in __init__() takes exactly 3 arguments (2 given) error.

Also note, that you follow very bad pattern of  sc.parallelize(...)...collect() calls. First you spread your collection to entire cluster, do some computations, and then pull the result. But sending data back and forth is pretty pointless here. Instead, you can just do these computations locally, and run Spark's parallel processes only when you work with really big datasets (like you main amazon_dataset, I guess).


How to compute conditional probability in code?

For this specific paper, the conditional distribution is calculated as follows:

$ P(f_i = v |f_{1:i-1} ) = \frac{\exp(o_{i,v})}{\sum_{v=1}^V\exp(o_{i,v})} $

on page 1702


The very obvious tip I would say is that K-means is not the algorithm you use for clustering text data in general. The nature of text data is a bit more complicated than structured data on which K-means is one of basic but still working algorithms (of course it also depends on the way you model the text i.e. how you convert a text dataset to a of numbers). Let me propose two things with one hint:

Hint

Document Clustering is also referred to as Topic Modeling. So you really need to have a look at this as I assume you didn't yet (according to using k-means for this problem). Now we see two standard algorithms for topic modeling but I strongly recommend you to see other algorithms as well.

NMF

Non-negative Matrix Factorization is a well-known decomposition method for non-negative matrices like TF-IDF or other variants of Bag-of-Words. You can simply apply it to your problem using SKLearn.

LDA

or Latent Dirichlet Allocation (don't get confused by Linear Discriminant Analysis). This is a pretty standard algorithm for topic modeling and implementations can be found in SKLearn, Gensim, NLTK, Spacy and other NLP/ML libraries.

Hope it helped! Good Luck :)


I suggest you to use as much data as possible. If the images are coming from two different cameras, it could be an original way to fight overfitting. I would use both datasets, and operate a train-validation-test split that is transversal to both.

Additionally, since the dataset is small, I strongly enourage you to use lots and lots of data augmentation, which is a very powerful technique to artificially increase the size of your dataset and fight overfitting.


I believe this is the representation you're after, please excuse the rough sketch but I think it explains the structure appropriately.

Single input going to three hidden units, each with a bias and tansig activation.
The outputs of the hidden layer are summed (via linear activation) with a bias to produce the output.


What is the difference between LeakyReLU and PReLU?

Straight from wikipedia:

Leaky ReLUs allow a small, non-zero gradient when the unit is not active.
Parametric ReLUs take this idea further by making the coefficient of leakage into a parameter that is learned along with the other neural network parameters.


TL;DR

If you are training the document-embedding model, then split the data before you convert the text into embeddings.

If you are using a pre-trained document-embedding model, then it won't matter and it is pre-processing step that it doesn't matter when you execute it.

Pipeline when training your own document-embedding model

Split your text data into train/validate/test sets.
Use your train set to train the document-embedding model.
Use your trained document-embedding model to convert train and validation sets to train your other model (e.g. classification model).
Test your final model by using your trained document-embedding model to convert the test set and test the trained final (classification) model.


How can I download data from online table quickly?

There doesn't appear to be an easy way - there is no mention of an API to get the data programmatically.

You could try contacting them via Twitter or something. Perhaps they would be happy to give you a dump of the data.

Otherwise, you might have to go down the route of writing a scraper e.g. using Selenium and BeautifulSoup. That would allow you to:

execute the search
download the file
navigate to the next search results page
repeat from step 2. until finished

They might be fairly easy, but do respect the websites wishes - perhaps they have a file called robots.txt which will tell you if they mind you doing that or not. In any case, build some time.sleep()s into your code, as not to bombard their server!


How to learn a classifier from a dataset with high imbalance

A common strategy for dealing with imbalance is to penalize harder the missclassifications that select the class with higher frequency.

In a binary classification problem you could penalize by dividing 1/n where n is the number of examples of the opposite class.

See the following from Prof. Jordi Vitriá

This is the loss function for structured output SVM.

The problem you mention is common in object recognition and object classification in images where much more background images are used than images containing the object. A stronger case happens with exemplar SVM's where just a single image of the object is used.


I will try to refer to the points I have some opinion about:

"What if it's not possible to label non-anomalous data to train the model?"
In this case, you face unsupervised learning problem. there are plenty of reading material regarding this topic and plenty of approaches.
here is one for example:
https://towardsdatascience.com/unsupervised-learning-for-anomaly-detection-44c55a96b8c1

this article may be good as well:
https://www.researchgate.net/publication/224375498_Anomaly_Detection_by_Combining_Decision_Trees_and_Parametric_Densities

if you can distinguish anomalies by their distribution, it sounds that you do know the anomaly data, am I wrong?

the issue with defining anomalies lower than 5th quantile, that it means that 5 % of your data that examined by the leaf will be considered as anomaly, so you if run you algorithm frequently, there is a good chance you will see a lot of anomalies.
maybe try using confidence intervals.

From what I have read, decision trees are not the classic method for anomaly detection.
read about fraud detection, isolation trees, using SVM and there are more ways.


Learning rate of 0 still changes weights in Keras

If your learning rate is set lower, training will progress very slowly because you are making very tiny updates to the weights. However, if you set learning rate higher, it can cause undesirable divergent behavior in your loss function. So when you set learning rate lower you need to set higher number of epochs.
The reason for change when you set learning rate to 0 is beacuse of Batchnorm. If you have batchnorm in your model, remove it and try.
Look at these link, link


Based on the context provided this is what I assume you have done:

Cat boost: Predicted the probability on 1 using all your x variables and bucket the probabilities and calculate the percentage of 1s in each bucket.

Logistic Regression: You have used 1 of the x variables which was continuous and divided into the bands mentioned and calculated the incidence rate of 1 in each band.

If both the above statements are true then

Your catboost model most likely is either underfit or did not learn anything from the model.
This inference is because the logistic Regression variables indicates that at least 1 variable has some prediction power on y.

It would be better to provide some additional details regarding the type of data and classes to analyze this further.


Linear regression using math or machine learning? Why even use machine learning for this?

Linear Regression is Linear Regression regardless of how you calculate/estimate the parameters.
The question becomes significant in case of a large multi-variate dataset where it is not easy/fast/possible to compute the parameters using algebraic equations (aka fitting simple line). In such cases Machine Learning techniques such as Stochastic Gradient Descent (SGD) can be helpful.


Clustering Customers on transactional behavior

As you have a target (column 93: "is_subscriber"), why not opt for a regression or classification method? Clustering is more beneficial for situations where you want to extract "common features" out of your dataset but you have no information on that dataset or the dataset implication, i.e., target.

Moreover, if you want to predict future customers as "potential subscribers" or not, a regression will provide you with the prediction formula. Then you can apply that formula to future customers to predict their behavior. You can obtain such result with Logistic Regression or, if you prefer a classifier, a Random Forest.


The sampling frequency, or sample rate, is the number of equal-spaced samples per unit of time. For instance, if you have 96 equally spaced observation per day, then you sampling rate is 96/day, or 96/24/3600=0.0011 Hz. Hz, which means per second, is widely used for sample rate.

The frequency of your data is different. I you signal is periodic or approximately periodic (say, the similar energy consumption patterns repeat everyday), the the frequency is the number of occurrence of that repeating pattern per unit of time. More generally, if your data does not look like periodic, it would help to conduct Fourier Analysis to understand the frequency components in your data. The Fourier Analysis (Fourier Transform) is the operation to transform time-domain data into the frequency domain.


I think the best way to deal with this problem is to use blending (although it is not a bad idea to start with random forest just to get sense about the problem you have). If you are not familiar with the word blending you can check this page:
http://mlwave.com/kaggle-ensembling-guide/

But let me explain what is the idea:
I believe that there is a kind of relation between the features you have: you can group all the features related to Weather together, The day, time ..etc together. Each set of features can be used to train a random forest. The output of these random forests can be combined by either linear or non linear regression blending techniques.

If the database you have is very large it is better to read more about blending techniques and do some experiments before going to your problem to make yourself familiar with it


Is there a possibility that there is no relationship between some inputs and outputs?

Yes and no

The YES

Let's consider at two distributions $F_X(x)$ and $G_Y(y)$ whose joint distribution is simply the product of the the two distributions: $H_{X,Y}(x,y) = F_X(x)G_Y(y)$. This means that the distributions are 100% independent: not just linearly (no correlation), but that there is absolutely no dependence between the two distributions. (Copula, Sklar's theorem, etc.)

The NO

For the above scenario to happen, there has to be absolutely no dependence between the two distributions. At the population level, this is calculated by Hoeffding's test of independence.

$$\theta =\int \bigg{[}H_{X,Y}(x,y) - F_X(x)G_Y(y)\bigg{]}^2 dH_{X,Y}(x,y) = 0$$

For empirical data, this would be tested with the hypothesis test $H_0: \theta=0, H_a: \theta\ne 0$.

Remember how we never accept a null hypothesis of equality because we know that it not quite true?

The YES (again)

It can be the case that, for all practical purposes, the distributions are unrelated. Yes, maybe if a butterfly flaps its wings in Ecuador it will be more likely that there will be an earthquake in Madagascar...but probably not much of an effect. This corresponds to $\theta\ne0$ but $\theta\approx0$ (equal for all practical purposes).


Should I report loss on the last batch or the entire training dataset?

The losses should be calculated over the whole epoch (i.e. the whole dataset) instead of just the single batch. To implement this you could have a running count which adds up the losses of the individual batches and divides it by the number of batches after the epoch ends.


Please go through mentioned link to explore the Recommendation rule.

https://medium.com/recombee-blog/machine-learning-for-recommender-systems-part-1-algorithms-evaluation-and-cold-start-6f696683d0ed

https://www.analyticsvidhya.com/blog/2018/06/comprehensive-guide-recommendation-engine-python/

https://www.datacamp.com/community/tutorials/recommender-systems-python

https://www.kaggle.com/gspmoreira/recommender-systems-in-python-101


Combine multiple classifiers to build a multi-modal classifier

Basically, you can do one of two things:

Combine features from both classifiers. I.e., instead of SVM-text and SVM-image you may train single SVM that uses both - textual and visual features.
Use ensemble learning. If you already have probabilities from separate classifiers, you can simply use them as weights and compute weighted average. For more sophisticated cases there are Bayesian combiners (each classifier has its prior), boosting algorithms (e.g. see AdaBoost) and others.

Note, that ensembles where initially created for combining different learners, not different sets if features. In this later case ensembles have advantage mostly in cases when different kinds of features just can't be combined in a single vector efficiently. But in general, combing features is simpler and more straightforward.


Why running the same code on the same data gives a different result every time?

It's not quite enough to set only the numpy random seed, as you've seen - the Keras documentation also notes that it's necessary to set:

the python hash seed
the core python rng
the random seed of your backend to keras
and force your backend to use a single thread.

The interaction of all of these will generally result in different loss values due mainly to changes in random weight initialisation, which data ends up in your train/val/test splits, and the order that data is passed to your model for training.

As others have mentioned, a small amount of variance is to be expected and it's certainly not worth hamstringing your performance over (by limiting yourself to a single thread); setting the seeds and rng should be enough to satisfy people your results are reproducible.

If your results still have unacceptable variance after taking those steps then it might be an indication that your model is brittle and may not generalise well to new data so you'll want to address that.

Final thought - when you say the values are different each time you run the model cell, are you deleting/overwriting the model you already trained? If you are running the model cell on existing model/weight values then that's the same as training for more epochs and would usually have a large impact on your results.


Need a Work-around for OneHotEncoder Issue in SKLearn Preprocessing


How to split a CSV into a dataframe without newline

You actually have a matrix, all items are grouped by row.

A very small example:

csv:
a,b
c,d

Will become:

[['a','b'],['c','d']]

The formatting of the output is indeed misleading.


True positives and true negatives, F1 score: multi class classification

In a multiclass problem there is one score for each class, counting any other class as a negative.

For example for class 1:

TP instances are gold standard class 1 predicted as class 1
FN instances are gold standard class 1 predicted as class 2,3 or 4
FP instances are gold standard class 2,3 or 4 predicted as class 1
TN instances are gold standard class 2,3 or 4 predicted as class 2,3 or 4 (here errors don't matter as long as class 1 is not involved)

In other words, the problem is evaluated as if it was a binary classification problem for every class individually. Doing the same process for every class independently (since the status of an instance depends on the target class), one obtains a different F1-score for each class.

After that, one generally calculates either the macro F1-score or the micro F1-score (or both) in order to obtain an overall performance statistic.


Okay, I got it. If anyone interested, they use 5x5 filter but with padding 2 and striding 1 so that with bias it doesn't change the 2D dimension of the output when applied on the result of max-pooling. The info on padding isn't present on the original paper...


Population stability Index vs Population Accuracy Index

Population stability refers to whether the  the distribution
of explanatory variables is changing over time. When this distribution changes  there is more concern over whether the model is currently fit-for-purpose since the data used to develop the model differs from the data the model is being applied to.
The Prediction Accuracy Index (PAI) is defined as the average variance of the estimated mean response at review divided by the average variance of the estimated mean response at development. In PSI we don't take into account the predictive ability.
To summarise,the Population Stability Index (PSI), interprets whether the distribution of development data and the distribution of review data are similar.  The PAI checks the predictive ability of the model on current data is not significantly worse than the predictive ability at development.
To know more refer this paper


What issue is there, when training this network with gradient descent?

what issue arises, when trying to train this network with gradient descent?

The activation function is sign function or signum function (A little modified).
So, its Derivative will be 0 at all the points
 Hence, the Gradient descent won’t be able to make progress in updating the weights and backpropagation will fail.


Can I cluster an aggregated data-set (grouped by) and apply dimensionality reduction?

There are no clear restrictions on can you cluster the data or not based on the data itself (in most of the cases), but you might face restrictions based on computational speed. The question is what you want to achieve in the end, but you can for sure cluster the data or reduce dimensions. You can also take a subset of your data, cluster it and then train ML model, something like catboost or lightgbm to predict clusters, instead of doing clustering on whole dataset. In my experience it will be faster to use model for cluster predictions.
For dimension reduction, you can go for PCA or TruncatedSVD, here are some useful links:
https://scikit-learn.org/stable/modules/unsupervised_reduction.html
https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html
But remember, that this techniques will reduce the number of features, not the rows, so you can still face computational issues.
If you want a way to map your high-dimensional data (around ~50 dimensions or less usually), you can use t-SNE
https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html