Discrete Ordinal Classification with Probabilities

Probit and logit ordinal regression model the cumulative probabilities $P(y \leq i | x)$. So you should be able to get class probabilities by taking the difference $P(y=i |x) = P(y \leq i|x) - P(y\leq i-1 |x)$. Then it just depends on the implementation whether or not it makes the probabilities available to you.

In R, for example, you can get these probabilities from the polr package.

[Even though you lose some information when doing one-vs-rest classification, ordinal regression models make some proportionality assumptions that (if not appropriate for your data) may not necessarily give you better results.]


How do I interpret loss and accuracy per epoch while training a CNN?

"loss" refers to the loss value over the training data after each epoch. This is what the optimization process is trying to minimize with the training so, the lower, the better.
"accuracy" refers to the ratio between correct predictions and the total number of predictions in the training data. The higher, the better. This is normally inversely correlated with the loss, but not always.

The "validation" counterparts are the same concepts but computed over the validation data, which is not used for training and hence "unseen" to your model. If the training loss and accuracy are good but the validation counterparts are bad, it means that your model is "overfitting", as it can't generalize to unseen data.
Normally, these measures are plotted together into a training and validation loss plot and a training and validation accuracy plot, so that you can better evaluate their behavior over time. You can see some examples on how to interpret different trends here.
In your case, the training is not finished, so we can't be sure. From what we can see, the accuracy is less than 30%, so it is bad, but maybe more training leads to improvements.


How can I include prior knowledge about monotonic features into a neural network?

If you plug this kind of data into a standard network, e.g. an MLP, you will usually hope that the model actually extracts this information itself. You could introduce a dummy variable that encodes this information, but you run the risk that the model learns to just follow the dummy variable and doesn't learn its own powerful abstractions and feastures from the data.

[EDITED: ] An example could be to create a weight based variable that is normalised to other physical characteristics, such as DV = weight / (height + waist circumfrence). This should then, according to your assumptions, scale nicely with the output obesity.

Another way that people might include prior information into a model, is to use a Probabilistic modelling, which incorporates ideas from Bayesian statistics. You can do things such as define a prior distribution for your outputs, conditioned on your inputs - essentially allowing you to provide information to the model (such as weight being correlated with obesity, ceteris paribus) - this then nudges the model to go along these lines.

If you would like to get into it, there are already some great libraries to make it really easy:

Stan - which has interfaces to many languages: Python, R, State etc.
Edward - probabilistic programming with modern GPU accelration and deep learning integration (Tensorflow and Keras).

A Linear regression example
A Bayesian Neural Network examples

This seems to be a nice overview of some of the methods and tools, but I haven't read through it all.

Another way that I could suggest is to play with the architecture of your model and the idea of auxilliary models. Have a look at my recent answer on a question talking about the Inception model from Szegedy et al..

The idea is that you have branches coming off the model at train time only, which also make predictions and produce error to be backpropagated through the preceding weights.

You could make a side model that predicts the obesity, based on e.g. the input weights and perhaps some related features extracted from the first layer of your neural network. This would make the idea or importance of this relationship more prominent during training and the weights would subsequently be tuned to take that into account.

At test time, you simply ignore these auxilliary branches.


Okay, so without going in to details,

Your target values are nominal, discrete values, hence the classification model would solve your problem.

Under classification there are variety of methods available,for example, there is decision tree, Naive Bayes etc.. . you may try all of these and check which gives the better confusion metrics.

Multi-Class means, there are multiple classes , such as in you colors.

If I have books features as input and i want to predict its genre, and the available categories i can put this book in are- fictional, romance, historical. all these labels makes it multi class because multiple genre.A book can be of any of the genre.

Multi-Label means, an instance can belong to more than 1 class.

Now, if this one book is fictional romance , then this makes it multi label. Because out of 3 available categories, this one book belongs to 2(it can be more than 2 also).

Now, This was just a short answer to get you started,please read on these key words to further details.

  1.) 1 target class - color

This is target attribute, color

  2.) Multiple values for that target class - (red, yellow, blue, green, pink)

Multiple class/labels for target attribute, red, yellow, blue... are classes/labels. Since there are more than 2 classes its multi class.

If you input map to 2 or more colors than 1 then it makes it multi label.


If your goal is solely predicting torque for a given engine, you can ignore hysteresis. A regression model will implicitly model hysteresis in the learned parameters of the model.

If your goal is building a model hysteresis for a given motor, then you should systematically measure performance with different recent histories. Then apply a time series model to model the effect of recent history.


Classification: How to manage data sets where one data row depends on another data row

I assume that you are solving a supervised classification problem, that is, you train your model on a labeled sample. I can think of two approaches to this problem.

I. Classify tags, use neighbor tags for the features

For each tag you can calculate features like:

(x, y) distances to the closest image
(x, y) distances to the closest image above this tag
number of images near this tag
...

Such features could be fed into any classifier like SVM or decision tree.

II Classify pairs of tags

For each tag $i$, you can consider all other tags $j$ and predict probability that $i$ is a caption of image $j$. This prediction may be based on concatenation of:

features of $i$
features of $j$
joint features of $i$ and $j$, like vertical and horizontal distance from $i$ to $j$.

After these probabilities are predicted, you can classify $i$ as a caption indeed, if any of these probabilities exceeds some threshold.


ignoring that is not differentiable at 00, as I think it is done in practice

yes see  ReLUs are not differentiable at zero

  If a neuron outputs 0 for every sample of the training data, it is basically lost, correct? Its weights will never be adjusted again?

yes see What is the "dying ReLU" problem in neural networks?


How do we choose which feature(column) should we normalize or scale

When to scale or normalize a column?

When you are using an algorithm that assumes your features have a similar range, you should use feature scaling.

The main example is distance algorithms like Euclidean distance.
Thus, the algorithms that are distance-based algorithms like K-means, SVM, etc also require feature scaling.
Gradient Descent Based Algorithms also require your data to be scaled to safely converge. From the article:

The difference in ranges of features will cause different step sizes for each feature. To ensure that the gradient descent moves smoothly towards the minima and that the steps for gradient descent are updated at the same rate for all the features, we scale the data before feeding it to the model.

For more, check this.

How to choose which feature/column in a dataset for scaling and normalization?

If the ranges of your features differ much then you should use feature scaling. If the range does not vary a lot like one of them is between 0 and 2 and the other one is between -1 and 0.5 then you can leave them as it's. However, you should use feature scaling if the ranges are, for example, between -2 and 2 and between -100 and 100.
Check this explanation for a better and detailed explanation.

And how to decide on which normalization algorithm to choose for a column?

I assume here you are asking which feature scaling method to use. The answer to this question also may vary due to context, however, the general answer is:

Use Standardization when your data follows Gaussian distribution.
Use Normalization when your data does not follow Gaussian distribution.

However, some algorithms do not require feature scaling and they can handle it. For example, since Decision Tree builds the model by splitting the features, it is invariant to features' scale.


In principle this seems close to a NER task, you could try to annotate a sample and train a sequence labeling model on it. However this would require quite a lot of work to get it right: annotation, then probably a good bit of trial and error to tune the right combination of features.

In a case like this I would rather go for a few carefully chosen regular expressions, they are likely to perform at about the same level without requiring as much work.


How to get rid of the expectation in Monte Carlo Policy Gradient method?

But how he removed the expectation part when talking stochastic gradient decent ?

A result that shows what happens in expectation can be estimated by sampling it, and that is precisely what stochastic gradient descent (or ascent in this case) methods do - they operate on individual samples on the assumption that this will on average produce reasonable direction for optimising parameters.

So there is no need to "get rid of" the expectation. In fact the sequence of equations is working deliberately towards it, because in most situations we do not know the full characteristics of the MDP and cannot calculate $d(s)$ (and maybe not even $\mathcal{R}_{s,a}$) - the expectation is used to remove the need for knowing those terms.

Importantly, $d(s)$ is the probability density of the state = the likelihood on a random sample of all states visited under a certain policy of finding the agent/environment in that state. This is often hard to calculate directly, even if you know the MDP and the policy, and it is intractable if you do not have a model of the MDP. However, when you take many samples then just by the act of sampling and using the value of $s$ that is observed, then you will approximate the true distribution of states in the long run.


Difference between Information Gain and Mutual Information for feature selection

Information Gain (IG) is the measure of entropy gained due to operations performed on an assigned dataset or random variable. Now, entropy here is just the variation in our dataset; so lesser the variation, lesser the entropy and in turn greater correlation among our dataset.

As you've hinted on a classification task, Mutual Information (MI) is a one-sided metric, whereas information gain is a two-sided metric. Actually both measure how relevant a feature is to a specific target class, & I've often observed various sources using the term information gain interchangeably with mutual information, as they are symmetric.

Important to note is while mutual information measures only the positive features, information gain measures both, negative as well as positive features of our data. For mathematical completeness, information gain of X given Y is given by IG(X|Y) = H(x) - H(X|Y), and mutual information I(X;Y) is given by I(X;Y) = sum_x sum_y P(X,Y) log {P(X,Y)/P(X)P(Y)}.

Following links might help grasp better understanding of these terms:

Part-1 | Probability Distribution
Part-2 | Information Theory

Hope it helps!


0.1 accuracy on MNIST fashion dataset following official Tensorflow/Keras tutorial

You haven't normalized your image dataset such as setting the pixel values between 0-1 which could help classifier converge faster.

Please do it by doing the operation below.

train_images = train_images / 255.0

test_images = test_images / 255.0

It seems you are using 50% of data for training as well as testing.
try to use the data in the ratio of 7:3 for training and testing.
Increase the epochs hope it will work.


Multivariate - Time series data pattern changes

One starting approach is doing the cross-correlation as you've said. The correlation will fail to capture the time shifted behavior. But you should calculate it for every pair of Milk/Drug you have.

The ccf() function in R does it for you.

If you're not successful i suggest you to take a look at the multiple time-series plotted side by side (like here) and check if really there exists this time-shifted pattern in your customer behavior.


Approaches to choosing number of bins in histogram

Another method is Bayesian Blocks from Studies in Astronomical Time Series Analysis. VI. Bayesian Block Representations by Scargle et al.

Bayesian Blocks is a dynamic histogramming method which optimizes one
of several possible fitness functions to determine an optimal binning
for data, where the bins are not necessarily uniform width.

Bayesian Blocks for Histograms


Sentences language translation with neural network, with a simple layer structure (if possible sequential)

Machine translation using traditional neural architecture (seq2seq models) had various issues due to rare-words, low accuracy and slow translation [1]. Even after using various mechanisms like attention and residual connections the performance was only comparable (not better than) statistical phrase-based machine translations [1].

I can only think of this paper as a successful attempt to use LSTMs in the encoder, decoder setting (8 encoder and 8 decoder layers) to get comparable results (there might be other attempts too). AWD-LSTMs [2] perform remarkably better than other models.

In the machine translation task, the model should understand proper relation between translated words and the words being translated and their positioning. This can only be achieved by using knowledge representation (word embeddings/encoding) from both the languages.

That's why we need to use both encoder and decoder layers.

If you ask me, I would say the following code (taken from link) is the simplest model structure possible using a simple LSTM/seq2seq model.

from keras.models import Model
from keras.layers import Input, LSTM, Dense

# Define an input sequence and process it.
encoder_inputs = Input(shape=(None, num_encoder_tokens))
encoder = LSTM(latent_dim, return_state=True)
encoder_outputs, state_h, state_c = encoder(encoder_inputs)
# We discard `encoder_outputs` and only keep the states.
encoder_states = [state_h, state_c]

# Set up the decoder, using `encoder_states` as initial state.
decoder_inputs = Input(shape=(None, num_decoder_tokens))
# We set up our decoder to return full output sequences,
# and to return internal states as well. We don't use the
# return states in the training model, but we will use them in inference.
decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)
decoder_outputs, _, _ = decoder_lstm(decoder_inputs,
                                     initial_state=encoder_states)
decoder_dense = Dense(num_decoder_tokens, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)

# Define the model that will turn
# `encoder_input_data` & `decoder_input_data` into `decoder_target_data`
model = Model([encoder_inputs, decoder_inputs], decoder_outputs)

From your question, it seems you only want to define the model using only Keras's sequential layer. If that is the case then you must know that variable encoder_states plays an important role in the aforementioned model's definition.

LSTMs are sequential models which means it works on a single word at a time and calculates hidden state for the next word in one iteration. The process is followed for all words in the input sequence (source language). Then the final hidden state is used in the decoder layer to compute the context for output sequence (destination language). That's why there is initial_state=encoder_states in decoder LSTM layer definition. Without encoder_states decoder LSTM won't know the context and your model will only give jibberish output.

are there successful attempts of doing sentence language translation with classic Sequential layers?

Sadly the answer is No.

You can only try to understand how machine translation works and get comfortable with the complexity of the machine translation model definition. As this is the simplest model possible.

For more information, you can go through these papers. 1 2 3

I hope it helps.


You have the null equivalent hexadecimal '\x00' in between each character. To remove those hexadecimal characters, first join the list with a space and then split it at '\x00' and then join again. Print statement simply looks like follows.

>>> print(' '.join( ' '.join(params).split('\x00')))


Scala mllib 2.2.0 k means. Number of runs

That's correct. It used to control making multiple starts at once but for various reasons this didn't work well, and was deprecated / made a no-op. I removed it for Spark 3.0.


Account for unknown error in time series data

I would set up a sensor as idle, purposefully to determine the decay in the signal. Then try an ARIMA or similar model on the data collected to find out what an approximation of the decay function is.

Further to this you should determine if the model still fits if they decay is put into a "stop/start" mode.

Afterwards you can apply a corrective function to remove or take into account the decay on your other data.


Shall I use the Euclidean Distance or the Cosine Similarity to compute the semantic similarity of two words?

First of all, if GloVe gives you normalized unit vectors, then the two calculations are equivalent. In general, I would use the cosine similarity since it removes the effect of document length. For example, a postcard and a full-length book may be about the same topic, but will likely be quite far apart in pure "term frequency" space using the Euclidean distance. They will be right on top of each other in cosine similarity.


How to make multiple regression perform better for outliers? (without reducing effect of them)

When it comes to outliers, one possible way of reducing the effects of the same is by means of a weighting mechanism.

When a standard regression is run on an entire dataset, all the observations in the dataset are being weighted equally.

Therefore, you could choose to run a robust regression where you use a weighting system, e.g. Huber, Bisquare, to adjust the weights so as to place a smaller weight on outliers in your dataset.

One way of detecting outliers in the first instance can be through the use of Cook's Distance, whereby if the distance is greater than 1 (as a rule of thumb), then the observation could be deemed to be an outlier.

You might find the following references helpful:

ROBUST REGRESSION | R DATA ANALYSIS EXAMPLES
Robust Regression — Reduce Outlier Effects


How to choose between different types of feature scaling?

You've listed a few different error metrics, I would pick one metric that is best suited to your problem. Trying to maximize several metrics at once makes it difficult to tell if your model is getting better. In any case, if normalization leads to the best score - then that is your answer. Since all other variables were already in range [0,1], then that's probably what I would have started with just to keep it consistent.
I am curious how much worse the z-score standardization is performing compared to normalization. From what you've written about the data - I'd be surprised if standardization vs. normalization produced significantly different results when only 3 out of 45 features are impacted.
This blog explains the topic better than I can About Feature Scaling. This article has several examples of training a MLP with scaled and unscaled variables How to use Data Scaling Improve Deep Learning Model Stability and Performance.
I'd also consider some simpler algorithms. It might be good to also train a logistic regression or similar as a comparison, fewer parameters will be easier to configure and less prone to error. A neural net will give you a boost if you have a lot of training data or a very non-linear problem.


How to get the number of steps until a certain accuracy in keras?

When you train the model using the .fit() call, it actually returns an object called history. Train the model using history = model.fit(x_train, y_train...) instead and then once training is complete, you can access the history.history dictionary to see this kind of info.

Refer to https://machinelearningmastery.com/display-deep-learning-model-training-history-in-keras/ for more information


It's a multiplicative factor, so nonintegers is OK.

But a value <1 shouldn't be used: while theoretically setting it equal to 0.5 should be the same as setting it equal to 2, the details under the hood may break:

      ...
      mtryCur <- if (direction == "left") {
        max(1, ceiling(mtryCur / stepFactor))
      } else {
        min(ncol(x), floor(mtryCur * stepFactor))
      }
      ...

(https://rdrr.io/cran/randomForest/src/R/tuneRF.R)
when stepFactor < 1, the ceiling(mtryCur / stepFactor) may exceed ncol(x), and the ceiling is the wrong direction to round, etc.

See also https://stackoverflow.com/q/27211765/10495893


Is there any way to define custom entities in Spacy

For pretrained models, spaCy has a few in different languages. You can find them in their official documentation https://spacy.io/models

The available models are:

English
German
French
Spanish
Portuguese
Italian
Dutch
Greek
Multi-language

If you want support for extra labels in NER, you could train a model in your own dataset. Again, this is possible in spaCy and from their official documentation https://spacy.io/usage/training#ner, here is an example

LABEL = "ANIMAL"

TRAIN_DATA = [
    (
        "Horses are too tall and they pretend to care about your feelings",
        {"entities": [(0, 6, LABEL)]},
    ),
    ("Do they bite?", {"entities": []}),
    (
        "horses are too tall and they pretend to care about your feelings",
        {"entities": [(0, 6, LABEL)]},
    ),
    ("horses pretend to care about your feelings", {"entities": [(0, 6, LABEL)]}),
    (
        "they pretend to care about your feelings, those horses",
        {"entities": [(48, 54, LABEL)]},
    ),
    ("horses?", {"entities": [(0, 6, LABEL)]}),
]

nlp = spacy.blank("en")  # create blank Language class
ner = nlp.create_pipe("ner")
nlp.add_pipe(ner)

ner.add_label(LABEL)  # add new entity label to entity recognizer

optimizer = nlp.begin_training()

move_names = list(ner.move_names)
# get names of other pipes to disable them during training
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != "ner"]

with nlp.disable_pipes(*other_pipes):  # only train NER
    sizes = compounding(1.0, 4.0, 1.001)
    # batch up the examples using spaCy's minibatch
    for itn in range(n_iter):
        random.shuffle(TRAIN_DATA)
        batches = minibatch(TRAIN_DATA, size=sizes)
        losses = {}
        for batch in batches:
            texts, annotations = zip(*batch)
            nlp.update(texts, annotations, sgd=optimizer, drop=0.35, losses=losses)
        print("Losses", losses)

If you want to use an existing model and also add a new custom Label, you can read the linked article in their documentation where they describe the process in details. Actually, it is quite similar to the code above.


Why are sigmoid/tanh activation function still used for deep NN when we have ReLU?

In certain network structures having symmetric activation layers has advantages (certain autoencoders for example)
In certain scenarios having an activation with mean 0 is important (so tanh makes sense).
Sigmoid activation in the output layer is still important for classification

In 95% of the cases ReLU is much better though.


Why is my loss increasing in gradient descent?

Based on your plots, it doesn't seem to be a problem in your case (see my comment). The reason behind that spike when you increase the learning rate is very likely due to the following.

Gradient descent can be simplified using the image below.

Your goal is to reach the bottom of the bowl (the optimum) and you use your gradients to know in which direction to go (in this simplistic case, should you go left or right). The gradient tells you in which direction to go, and you can view your learning rate as the "speed" at which you move. If your learning rate is too small, it can slow down the training. If your learning rate is too high, you might go in the right direction, but go too far and end up in a higher position in the bowl than previously. That's called diverging.

Also, good to note that it could be completely normal that your loss doesn't always decrease. This is particularly true if you use mini-batch gradient descent. In that scenario, your gradient may not always be completely accurate, and you might simply make a step in the wrong direction every once in a while.

I hope this explanation helps!


What are the approaches to aggregate categorical variables?

You problem is essentially you have high cardinality in your features, right? This will be relative to your problem, but you can look for mean encodings. Essentially, you will replace categories by the mean on target variable, however, this is highly prone to overfitting and you should take care.

The following two videos will give an excellent explanation:

https://www.coursera.org/learn/competitive-data-science/lecture/b5Gxv/concept-of-mean-encoding
https://www.coursera.org/learn/competitive-data-science/lecture/LGYQ2/regularization

Another idea is to group similar sequences into one segment, for example, let's say we have:

A->B->C and B->C

If this makes sense in your application, this could be transformed into one variable only.


Training a LSTM on a time serie containing multiple inputs for each timestep

The answer to this question highly depends on what relationship between the variables you are interested in.

If you are interested in the relationship between time and observation-value, treating the entities as different batches could make sense, under the assumption that the role of individual entities doesn't really matter to you. In this case, you would, for example, add the mean of each entity (or the overall mean) to all entities with missing values to get a constant number of observations per entity. But you could also simply average all values in each timestamp and include other features as min & max. This would most probably deliver better results.
If you are interested in the relationship between entities and observation-value, this is a matter of missing data in time series. There are a lot of techniques that can help you with that from simply imputing the mean to more sophisticated methods like a Kalman filter. However, in the end, you will have to ask yourself why these observations are missing and choose the appropriate method. But since you are using time-dependent models in your experiment, I assume, this is not of interest to you.
If you are interested in the interrelationship of all three variables, you are dealing with panel data. In this case, I don't see a reasonable possibility to model this with an LSTM. Maybe another RNN-architecture could work, however, the only paper I found was Tensorial Recurrent Neural Networks for Longitudinal Data Analysis from Mingyuan et.al. But in the end, it would not matter, since an ARIMA-model also isn't appropriate for panel data. Usually, you use a Difference-In-Differences approach for that kind of data. In this case, I would suggest changing the dataset for your experiment.


Why is there no end user Application, yet?

Listing 2 examples:
IBM Watson Analytics

Amazon ML use case

Preparing the data for supervised learning is require skills. Not all data came labeled and in form to be used as in need to solving problems.

Also many more platforms/Api are in market now but for sure you can't solve a problem only with 1 algorithm, is needed much more ... Hope it help.


assigning weight for attributes in dataset

Only for certain models.

For example for mlp you have this possibility:

The docs show you the attributes in use.

  Attributes:
  ...

  coefs_ : list, length n_layers - 1
  The ith element in the list represents the weight matrix corresponding to > layer i.

  intercepts_ : list, length n_layers - 1
  The ith element in the list represents the bias vector corresponding to layer > i + 1.

Just build your classifier clf=MLPClassifier(solver="sgd") and set coefs_ and intercepts_ before calling clf.fit().


What's the best metric for evaluate an estimator for a multi class problem with class imbalance dataset?

One standard metric is the top-1 or top-5 test error rate. For instance, for top-5, your model predict 5 most likely labels, and if none of the 5 labels is the ground truth label, you mark this instance as an error. This is usually a standard metric when people working with the ImageNet data. See example here usage here. This metric does not explicitly count for that the classes are imbalanced.

Another powerful metric is the Mean Average Precision (mAP), where you calculate the Average Precisoin (AP) for each class and then average among all classes. In this metric the smaller classes get equal weights compared to larger classes. The idea comes from the information retrieval community.  This metric is also suitable for multi-label classification. See example usage here.


How to load a csv file into [Pandas] dataframe if computer runs out of RAM?

As noted many times by the writers of pandas, the ideal size of memory for analyzing with pandas is around 5-10 times the load of data you are giving.

That being said, if you can afford to load data is chunks for pre-processing and dump only the columns that are needed, I recommended pandas.read_table option to load data in pieces (check option chunks).

pyspark as you mentioned might be good to go with. But there is dask which is built around python stack of pandas and numpy for distributed work loads which has support for pandas.DataFrame and numpy.array . But I never followed anyone who was successful in using it in production (or atleast mentioned that they had used it). May be some of the people here can vouch for it. There is another library called sframe to load the data way out of memory as the author suggested in one of his presentations. It keeps on serializing data onto disk.

Hope I summarized it well.


What is the procedure to create a bag of visual words model with SIFT?

Question 1: how many groups should I create?
There is no rule of thumb but typically if you have n categories of data you may want to set k as 10*n. This has known to work well, however you can always change this number by looking at the accuracy on your validation data set.

Question 2: Creation of histograms
Let us say you extract m features from each image and your dictionary contains k words, for each of these m features you would find the closest match to the k words and assign this word to that particular feature. This way you will have m number of visual words taking one of the k values. Now you count the occurrence of these visual words and creating a normalized histogram out of it. So each image will have a normalized histogram of the k visual words.

Question 3: Classification
Now that we know how to extract a histogram for a given image, we would compute the histograms on the training data and given a test image we extract its histogram. Now, how we classify this is entirely up to you. You could just find the closest match in the histograms of training data using euclidian distance or do a weighted sum etc.

A simple way of obtaining probabilistic data is to train a neural network with softmax at the output layer and given a test image pass its histogram through the network and obtain the probabilities.


Tensorflow training with batch size of (1, None, features), but model expects extra dimension

I managed to solve my problem with a generator, which expands the dimensions for each single batch to return shape (1, None, 2).

class SingleBatchGenerator:
    def __init__(self, X):
        self.X = X

    def __call__(self):
        for i in range(len(self.X)):
            xi = np.expand_dims(self.X[i], axis=0)
            yield xi, xi

X = [np.ones((np.random.randint(1, 100), 2)) for _ in range(100)]
gen = SingleBatchGenerator(X)

ds = tf.data.Dataset.from_generator(
    generator = gen,
    output_types=(tf.float64, tf.float64),
    output_shapes=((1, None, 2), (1, None, 2)),
)

autoenc.fit(ds.repeat(), steps_per_epoch=len(X), epochs=500)


Are there established good algorithms for incremental feature learning for a neural network? Do any python ML libraries implement such algorithms?

The opposite of what you describe is used more often. You start with many features and then prune the neural network at the end.

I don't believe any python neural network package does what you want.

But it's very simple to implement.

For example, using sklearn, you can just pass warm_start=True, so that it doesn't reinitialize weights every time you call fit(), and then you can increase the number of nodes in your feature layer, or any layer you wish.

from sklearn import datasets
from sklearn.neural_network import MLPClassifier
import numpy as np

X, y = datasets.load_iris(True)
X = (X-np.mean(X, 0))/np.std(X, 0)  # normalize

nhidden = 10
m = MLPClassifier(nhidden, max_iter=1000, warm_start=True).fit(X[:, :1], y)

# every 1,000 iterations, throw in a new feature
for nfeatures in range(2, X.shape[1]+1):
    m.coefs_[0] = np.r_[m.coefs_[0], np.random.randn(1, nhidden)]
    m.fit(X[:, :nfeatures], y)

# in the end, we have a neural network trained with all features
print(m.score(X, y))


Implementation of kmeans clustering using R

Actually, this is the expected behavior for running a k-means algorithm on the iris dataset. The iris dataset contains only two distinct clusters. So if you randomly set up three centroids, the third centroid will either end up on the right or on the wrong cluster, causing the algorithm to split that cluster into two (see the left picture).

Your outcome with 37% accuracy will most probably look somewhat similar to the left picture where the left cluster has been split into two clusters. The 85% accuracy will most probably come from those times in which the right cluster has been split into two clusters (you cannot be sure though without printing the results).

For testing purposes, you can simply use set.seed(123) before running the algorithm, so that you always get the same results. However, this doesn't solve the problem that the iris dataset isn't ideal for clustering purposes. I would suggest you check out other datasets or you give the R-package cluster.datasets a try.


A new comer in this field - How to start?

I would suggest you taking a Machine Learning and Statistics (Coursera, Udacity etc.) course straight away, since you have a profound experience in programming, so mastering any library wouldn't be of a big issue. Understanding of theoretical approaches is way more important than a particular library.

Main reason for that is the fact, that tutors of courses usually provide you with data big enough to understand the purpose of each algorithm/approach in the session/assignment so you will not only become proficient in some ML approach, but also understand which type of data can be then handled by a particular algorithm.

Also, I suggest you select a particular field of data science / machine learning, since data science, machine learning, statistics and so on, are just powerful tools and their application in computer vision, statistical modelling, natural language processing, fraud detection and other fields share something in common, but focusing on one field will allow you to progress faster.


Let us break this down into a few steps for better understanding:

Your linear classifier let us represent it with an equation y :
y = m1x1 + m2x2 + m3x3 + ..... where y takes into account all the features you have provided, multiplies it with random weights m .
Now you need to find the ideal values of m in order for you to get maximum accuracy. You could ofcourse brute force your way but there are optimized ways to do this. This is where gradient descent comes into play.
Gradient descent optimizes the values of m by taking into account the final output and understanding which features are important and which are not.
The loss function determines what is to be compared. In your example you are talking about two classes, in this case, the loss function will be a simple cross entropy loss function, that tells you how far you are from the actual values.
The loss is the difference in the predicted value vs actual value.
The only way you can minimize this loss is by playing around with weights. In order to understand which weights to play around with, you use simple first order derivatives to associate impact of output with each feature multiple by its weight.
Gradient descent is then used to compute the next set of weights that helps you reduce the loss by looking at the direction of weights to be used.
When you run this thru multiple epochs and steps, it stabilizes this weight matrix thereby improving performance of your model.

Hope this clears your question, would be happy to elaborate in case you need it.


How to train a model on a data where there are multiple data inside a data point?

If your list are of fixed size you can separate them in different columns.e.g. load1,load2...
If thats not the case, you need to define some statistics for the cycles like average load, max_load, min_load..


Is Recursive Feature Elimination finding best features subset?

No, RFE cannot guarantee that it finds the feature subset with optimal score.
As with most greedy processes, the point of RFE is to reduce the computational cost (fitting a model for each of the $2^m$ feature subsets), at the cost of perhaps not finding the actual optimum (but hopefully "close enough").
See also https://stats.stackexchange.com/questions/232461/question-about-recursive-feature-elimination


Handling dimensions for RGB data with Keras CNN

Strangely, if change the label's fake data to size=(500, 10) then it "works". But obviously that makes no sense, since it would mean I have 10 labels for each sample.

It would most certainly not mean that; it simple means that, for the type of softmax classification you are attempting here with CategoricalCrossentropy loss, your labels should be one-hot encoded, and not single digits, as here (hence the error).
You should

Either convert your single-digit labels to one-hot encoded ones using to_categorical (docs)
Or change your loss to sparse categorical cross-entropy (docs)

Changes in your code for the first approach:
labels_ohe = tf.keras.utils.to_categorical(labels, num_classes=10)
# [...]
model.fit(data, labels_ohe)

Changes in your code for the second approach:
model.compile(optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),
              loss=keras.losses.SparseCategoricalCrossentropy())

The MNIST dataset that works which I refer to can also be found in the linked documentation.

The MNIST labels, like the CIFAR10 ones, are also single-difit integers and not one-hot encoded; your MNIST model from the example you refer to you works because of the loss="sparse_categorical_crossentropy" setting.


Yes, just rearrange the formula for finding the normalised value. Where $x_i$ is the original attribute, and $z_i$ is the normalised value:

$$
\begin{align}
  z_i &= \frac{x_i - min(x)}{max(x) - min(x)} \\
  x_i &= z_i(max(x) - min(x)) + min(x)
\end{align}
$$


Propensity model with Only Positive Data

The model built using only positive value will be biased towards positive value and will always predict for the customers who bought any items before. But customers who didn't buy anything,  the model won't be able to predict anything.


Using Generative Adversarial Networks for a generation of image layer

In terms of generating an image "layer", that is just the same as generating an output image that can be overlaid on the input using standard graphics software. If you want pixel-level accuracy in the output then the output will need to be the same size as the input, otherwise it could be smaller, provided it is the same aspect ratio, and in which case it would need to be scaled up in order to be used as an overlay. In any case, as the output of a GAN can be an image (and often is), then this part is easy.

The "G" in GAN stands for Generative. The purpose of a generative network is to create samples from a population, where there are typically many possibilities. Those samples can be conditioned on some additional data, and that additional data could be an image, although many examples will simpler conditioning, such as a category that the training output is representative of.

One possibility for using a GAN, is where your population contains a range of traits, and you can calculate vectors that control that trait. So you can take an input image, reconstruct it in the GAN then modify it by adding/subtracting the trait-related vector. An interesting example of this is Face Aging With Conditional Generative Adversarial Networks, and similar examples are around of adding/removing glasses etc. For this to work for you, you would literally need images that had your points-of interest in them and ones without them, and then you would be able to control addition/removal of points-of-interest. The network would not detect these points in the input, instead it would add them into the output. From reading your question this does not seem to be what you want.

A similar paper uses a GAN to remove rain from photos, based on training many images with and without rain then learning the "rain vector", encoding new images with rain in them into the GAN's internal representation and subtracting this "rain vector".

GANs conditioned on input images (as opposed to categories or internal embeddings) are also possible - this example of image completion might be closer to your goal. If your points of interest are variable with many options feasible, then it could work for you.

However, if your points of interest are supposed to always be the same pixels in each image, then your goal might be better defined by strict ground truth and become more like semantic segmentation, which can be attempted with variations on CNNs, such as described in this paper by Microsoft. These are much easier to set up and train than GANs, so if you can reasonably frame your problem as pixel classification from the original image, this is probably the way to go.


Need rules of thumb for out of core larger than ram dataset on a laptop

I am not an expert but this is how I see things: use spark. Pyspark seems to be the flavor to use since it's python and that's the language that data science has settled on.

Dataset sizes are expanding way faster than RAM is on laptops. What "they" don't tell you is that scikit learn, pandas, R, etc all require your dataset and all intermediate steps to fit inside RAM. If you have an 8gb dataset, and you pivot it, your intermediate step could easily be say 30gb. Even if it's temporary and your final cleaned dataset to be fed into. A neural network shrinks back down to say 6gb, too bad. All of your steps must fit inside RAM.

In the real world, and even the limitation that datasets be limited to RAM sizes is quite limiting. This is what I wished I had known earlier. It's like saying here is a grocery store which is all yours (scikit learn has tons of features ) but you can only leave the store with a single shopping bag full of items). I would much rather have a grocery store with the standard 100 items and let me have a buffet on just those 100.

Yes, there are some libraries here and there for scikit learn that read from files in chunks, and then you add the chunks together, etc. But you will quickly see that that is reinventing the wheel and distracts from doing data science. The fact remains that these frameworks were built for in RAM usage. It's not a bad thing at all per se, but quickly becomes limiting.

What you want is to write the code once and not have it change whether the dataset is 1gb or 100gb. This is what pyspark is for. You can just learn their API and pyspark will handle chunking and distributing the workload such that it fits in the available RAM that you have. Of course if you are RAm limited computations take longer but this is ok. At least you will get an answer.

The downside is that if you want the very much appreciated ability to not have to worry about "is this dataset too large for my ram" question then you need to stay in pyspark's API. If they don't have a feature that say scikit learn has, or keras has, then too bad. Write your own or work around it.

Thankfully the API looks very complete.

Where does it lack? Well for say neural networks you aren't going to have all kinds of special activation functions and customization that you can have with Keras.

But in my experience and it really is very little, more data and better data beats the specific algorithm much of the time. I only mean to say that I would rather be able to process tons of data and use a vanilla classifier than be limited to in RAM dataset sizes and while being able to optimize 1000  hyperparameters.

You also don't want to be switching between different aPIs all the time as a beginner.

So if there is any chance that your dataset will be larger than RAM (I think that likelihood is a near certainty in the years ahead) I would most definitely learn pyspark. Even before scikit learn.


What is the difference between Inception v2 and Inception v3?

In the paper Batch Normalization,Sergey et al,2015. proposed Inception-v1 architecture which is a variant of the GoogleNet in the paper Going deeper with convolutions, and in the meanwhile they introduced Batch Normalization to Inception(BN-Inception).

  The main difference to the network described in (Szegedy et al.,2014)
  is that the 5x5 convolutional layers are replaced by two consecutive
  layer of 3x3 convolutions with up to 128 filters.

And in the paper Rethinking the Inception Architecture for Computer Vision, the authors proposed Inception-v2 and Inception-v3.

In the Inception-v2, they introduced Factorization(factorize convolutions into smaller convolutions) and some minor change into Inception-v1.

  Note that we have factorized the traditional 7x7 convolution into
  three 3x3 convolutions

As for Inception-v3, it is a variant of Inception-v2 which adds BN-auxiliary.

  BN auxiliary refers to the version in which the fully connected layer of the auxiliary classifier is also-normalized, not just convolutions. We are refering to the model [Inception-v2 + BN auxiliary] as Inception-v3.


Meaning of axes in Linear Discrimination Analysis

You could be asking yourself the same questions using PCA! Usually these linear transformations (or even non-linear for that matter, PCA, LDA, ICA, UMAP etc.) are used to reduce the high dimensional features space to serve one or multiple purposes. For example in PCA one could use it to compress the high-dimensional feature space to lower (usually 2 or 3) by finding the directions of maximal variance, either for better separation or a simple compression or better explainability via projected visualization for human eyes. Sometimes they are served for better class separation like the case of LDA, in a supervised fashion (in contrast to PCA). In case you missed it check out Sebastian Raschka's post on the difference between LDA and PCA for dimensionality reduction (be careful with LDA assumptions).

In order to answer the question about the axes, we better touch briefly on what LDA is aiming to do, borrowing the explanation from this blog post:

  LDA aims to find a new feature space to project the data in order to
  maximize classes separability. The first step is to find a way to
  measure this capacity of separation of each new feature space
  candidate. In 1988, a statistician called Ronald Fisher proposed the
  following solution: Maximize the function that represents the
  difference between the means, normalized by a measure of the
  within-class variability. The Fisher’s propose is basically to
  maximize the distance between the mean of each class and minimize the
  spreading within the class itself. Thus, we come up with two measures:
  the within-class and the between-class.

[The Answer]: As in the case of PCA: the first axis (by convention PCA 1) represents the axis holding the maximum variance in the data, PCA 2 the second in the rank and so forth. The same analogy holds true to LDA. Here though, LDA 1 (your x-axis in the graph, again by convention and the visual judgement) accounts for the maximum variation among the classes, and LDA 2 the second and so forth (still confused, you might one to check this video). You can define a simple linear model to separate them, that is it, that perhaps you couldn't using the original features.

[Bonus]: Although I am finished with the answer, and the following it is not super relevant, but I would like to bring a super nice example what these techniques might be able to answer from data. In the case of, again PCA, Tableau team is showing such capabilities, see this video. In real world problems, not a simple IRIS, these will come handy, and surely these possible even outside Tableau, but requires more efforts.


Custom metrics for unbalanced classes problem in RandomForest or SVM

I think I understand what you are trying to do. First, let me attempt to obtain clarity on two concepts that may be getting conflated (if not for you, then for other users).

When a random forest is being fit, it typically uses entropy or gini impurity for categorical outcome variables, and mean squared error or mean absolute error for numerical outcome variables -- this is the model building step. However, when the fit model is being evaluated, the scoring method is used to understand the models performance -- this is the step where we learn how accurate the model might be.

I've seen that sklearn allows specifying a custom scoring method, but as far as I know, I don't believe you can specify your own criterion parameter in the RandomForestClassifier or RandomForestRegressor methods/APIs.


What is the concept of Normalized Mutual Information in the evaluation of Clustering?

This is a method for evaluation of two clusterings in the presence of class labels so it is not proper for real clustering problems in which class labels are not available.

Imagine you have class labels and you want to evaluate a clustering or (compare two clusterings). The most natural idea is to use Purity score. It simply checks labels with clusters and the best case is, of course, when each cluster contains only and only one class label. This score, however seemingly natural, has a drawback. If you consider each cluster having only one data point, then Purity is maximized! So there should be an
awareness about the number of clusters when calculating the purity score.

The next idea is calculating the Mutual Information. Mutual Information considers two splits: (1) split according to clusters and (2) split according to class labels. Then it tells you how these two splittings agree each other (how much information they share about each other or how can you know about one of them if you know the other one). Like purity, MI also gets bigger when the number of clusters is large.

Then comes NMI which is bias-corrected for the phenomenon explained above and also normalizes the score between $0$ and $1$ (MI does not have an upper bound).

NOTE: I think your question was answered in first line. If you want to evaluate clustering you are not looking for external measures where labels are needed. I just explained a bit for sake of completeness of answer.


First off, I'm not sure that you even need machine learning for this. You probably know what features you are looking for in every specific job (which grades are most crucial). Won't it be easier to define a simple threshold per job?

For example, for job X, you need to have a grade above 80 in course A, above 90 in course A and so on. This can give you the list of all suitable candidates per job (but it won't tell you which job is the best fit per student).

A different approach would be to define the most important courses for each job (preferably weighted, because some are more important than others), than for every student calculate his score for every potential job (weighted MSE or MAE). This will give you the best fit for every student. You will than need to use some sorting algorithm to place all the students in the best fitted jobs overall (I assume that not all students can get their best fit as jobs don't have infinite positions).

For example, for student A and job X, the the score will be:

You will probably still need to use some threshold per job, since having a job being to best fit for the student, doesn't necessary mean that he meets its criteria.


Q-learning when minimising a total cost instead of maximising a total reward

I have a decision problem where the results are measured as a cost that I want to minimise. It seems like a good fit to Q-learning, but I am not sure how to adjust it to deal with a cost instead of a reward.

The simplest way to do this, without changing anything else about your learning algorithm, is to notice that

$$Reward = -Cost$$

So literally just optimise against the expected return based on sums over your negative costs, using standard Q learning. Everything will work as normal. Your best agent may still end up with a negative expected return (and negative Q values), but maximising this should still restult in an optimal policy.

If you really must use minimising cost as your objective for some reason, then there are a few small changes you need to make for that to work with Q learning.

The definition of your Q function becomes expected discounted sum of future costs:

$$Q(s,a) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k C_{t+k+1} | S_t=s, A_t=a]$$

(This literally just replaces $R_t$ with $C_t$)

Then your best guess at the optimal policy is the one that minimises expected future costs:

$$\pi(s) = \text{argmin}_a Q(s,a)$$

And your Q-learning update rule is also based on assumed optimisation of minimising the next step

$$Q(s, a) \leftarrow Q(s, a) + \alpha(c + \gamma \text{min}_{a'}[Q(s',a')] - Q(s, a))$$

This doesn't match either of your suggestions. If I was to correct your code, then using rewards (with r = -c), it would look like this:

q_dict[state1][act1] += alpha * (r + max(q_dict[state2].values()) - q_dict[state1][act1])

where alpha is the learning rate, and I assume there is no discounting (so it must be an episodic problem, not continuous).

If you wanted to use cost c directly, and find policies that minimise total cost, then it looks like this:

q_dict[state1][act1] += alpha * (c + min(q_dict[state2].values()) - q_dict[state1][act1])

i.e. you substitute c for r and min for max.

You ideas about having different starting values might make some difference to convergence rates. However, this is not related to whether you use a cost or a reward.

I advise against using the cost directly like this. Although it is simple and will work, whenever you read any RL articles you will have to keep adjusting whether to switch max for min. The convention of maximising the sum (or average) reward is much more common in RL tutorials, and whilst you are learning it, you will save yourself just a little bit of effort to follow this convention.


Label Encoder encoding the same class as two integers


Most of the critical points in a neural network are not local minima, as it can be seen in this question. Although it is not impossible to fall into a local minimum, the probability of it happening is so low that in practice it does not happen, except from very special cases as a single-layer perceptron.

Being local minima so hard to find, this means that it is highly unlikely to come across the global minimum in your optimization method. All that we do in deep learning is decrease the loss function to find fairly good parameters, but finding local and global minima is extremely unlikely.


Before going into modelling, I guess you can do a bit more of exploratory analysis(month by month, year by year). If you find any trend or seasonality and so on.

Why did you go to RF directly without using basing Techniques like ARIMA, ARMA, Exponential Smoothening and AR so on.

Sometime RF might not give you as good results as Base models, I think you don't have trend, this is from your graph(but not certain). If you can try doing some research and see if there are some external factors which are effecting your demand. Why did that happen and what is the root case for it.

To your model to understand, it needs some feature which explains its spikes somehow, it can achieved by doing feature engg/ research


Low memory error while performing degree 2 polynomial regression on (3000*1835) sized array

You are allocating a float64 (2400, 1686366) array (very big!), no surprise you get a memory error.
You say that you have 1835 columns, but the error message says 1686366, so which one is the correct number of columns? If it's 1835 then there's an error before feeding your set to the model.

Also, are you sure you need a float64 data type? You can reduce the memory consumption by using float32 instead of float64 (unless you really need to work with that precision)


Is it possible using tensorflow to create a neural network that maps a certain input to a certain output?

From your description, it seems that you are facing a regression problem, because you want your output to be certain values. This is different from classification problems, that have as output the probability of the input belonging to certain class.

The key to using neural networks for regression is that the output layer should have no activation, that is, it should be a linear layer. As pointed out by @JanvanderVegt, a common loss function for regression problems is the mean square error (MSE) between the current output and the MFCC features you computed.

If you google "tensorflow regression example" you can find dozens of complete examples, like this or this.


Should weights on earlier layers change less than weights on later layers in a neural network

This is expected and well established - Vanished Gradient

https://en.wikipedia.org/wiki/Vanishing_gradient_problem
This has the effect of multiplying n of these small numbers to compute gradients of the "front" layers in an n-layer network, meaning that the gradient (error signal) decreases exponentially with n and the front layers train very slowly.

While I'm not an expert in neural network (experts, please add an answer), I'm sure professional implementation for neural network is tricky and highly optimized. If you simply implement a vanilla text-book-styled neural network, you won't be able to use it to train a large network. Keep it small and you'll be fine.


How do we determine what is learnt by a ML model?

Shap values is your friend.

model = RandomForestRegressor(max_depth=6, random_state=0, n_estimators=10)
model.fit(X_train, Y_train)
import shap
shap_values = shap.TreeExplainer(model).shap_values(X_train)
shap.summary_plot(shap_values, X_train, plot_type="bar")

There are several other techniques with their own drawbacks. Some are:

Permutation Importance
Partial Dependence plots
LIME
Global Surrogate Models
etc.. You can add to this list.


Replacing mean by median over batch-size to lessen the impact of outliers

One shortcoming is that the median is often more computational expensive to calculate than the mean. The median can be calculated with a variation of quickselect which is linear worst-case performance. Calculating the mean only requires the sum and count of the numbers.


Preprocessing of Sudoku Dataset from Kaggle

You need to convert this large list of strings to a numpy array containing lists of integers. Here is how you can get the data into the correct format for a neural network:

import pandas as pd
from numpy import array

def preprocess():

    data = pd.read_csv('sudoku.csv')

    x = data[data.columns[0]].values
    y = data[data.columns[1]].values

    x = x.reshape(-1, 1)
    y = y.reshape(-1, 1)

    x = array([list(map(int,set[0])) for set in x])
    y = array([list(map(int,set[0])) for set in y])

    return x, y

x, y = preprocess()

print('\nx: ', x[0])
print('\ny: ', y[0])

I haven't thought enough about the problem to advise on how best to go about solving it, but at least this should get you started on playing around with different methods. Be aware that it will take a minute or so to run, as the script is now converting a large amount of strings to integers.


Any usable libs to build and visualise SOM in python?

You could have a try on this package.
There is a working example on this page.
If what you are interested is the Manifold learning,
you could also apply many packages from sklearn.


What (probabilistic models) can only output decisions when they are certain?

You can get probabilities for all labels. You can set a threshold for confidence if proba is greater than some value classify otherwise remain silent.


How to classify a set of words into one of the given labels

There are probably many variants but here are two simple approches:

Using pretrained word embeddings, you can calculate the semantic similarity between two words. For example you could use cosine to measure the similarity between the vector of a target word (e.g. "calm") and every word in the set (e.g. "cloud"). Then the mean across words in the set gives how much the set is associated to the target, and you can pick the target which has the max similarity.
Using WordNet to directly obtain a semantic distance/similarity between words. The method is similar to the one above.

Note that there are many improvements that can be done to these basic ideas, for example you could use a predefined set of words related to "calm" instead of just the word "calm" (you can get the most similar words from WordNet for instance). There also many options for the aggregation across the set of words.


Is it recommended to compute the average of metrics for classification?

Yes, you can try k-fold validation, where indicidual models are generated from various subsets of the data and performance measured on a validation set.
You can the compute the average and variance of model accuracy, other performance metrics to understand how stable your model is.


How can I improve a machine learning model?

It is questionable whether doing the PCA and reducing the dimension to 2D was a good idea. You can try experimenting with the number of components in PCA or even trying some non-linear dimensionality reduction like LLE:

  Locally linear embedding (LLE) seeks a lower-dimensional projection of the data which preserves distances within local neighborhoods. It can be thought of as a series of local Principal Component Analyses which are globally compared to find the best non-linear embedding.


Local (contrast) normalization

Local Response Normalization (LRN) layer implements the lateral inhibition we were talking about in the previous section. This layer is useful when we are dealing with ReLU neurons. Why is that? Because ReLU neurons have unbounded activations and we need LRN to normalize that. We want to detect high frequency features with a large response. If we normalize around the local neighborhood of the excited neuron, it becomes even more sensitive as compared to its neighbor.

2 options in keras. First it is already implemented check here but thats "old" keras. New, Google implementation can be found here


Why do convolutional networks work so well for images?

The convolution neural networks take into consideration that an image already has a two-dimensional structure. This is a domain knowledge provided by humans and not something the neural network can pick up.

For example, consider if I would give you the raw pixels shuffled in a flattened array (or any other signal for that matter). Would you be able to reconstruct its 2D structure? Probably this would be almost impossible (unless someone builds an algorithm to solve jigsaw puzzles at the pixel level!)

Applying convolution with various strides we are looking for patterns in the patches of the image and not on the individual pixels. Using different strides we are explicitly expressing the amount of resolution that we are investigating the patterns of the image.

So given that these convolution filters are trainable we are able to have an architecture that works well with images. In fact, it would work well with any signal that would have a 2D structure.


How to train millions of doc2vec embeddings using GPU?

One option is to switch to a deep learning framework that supports distributed training.


What are some solutions for dealing with time series data that are recorded at uneven intervals?

So, the question essentially asks how to model sequential data with inconsistent time intervals.
I would say option 2 would be more logical in my mind.
I would propose to have the timestamp when the image was taken from the camera as an input feature for each example. Here, it would still be in a format which you could model this using an RNN/LSTM (any sequential model architecture).
By adding the timestamp feature to your input, you at least have given the model the time representation of the images. You could represent the timestamp fo the image as number of seconds (minutes/hours, whichever is more appropriate to the dataset you have) from the earliest image you have in the dataset.
Of course, it's best to normalise this so you don't end up with very large values, which could reduce the overall generalisation performance of the model.


How to (better) discretize continuous data in decision trees?

No, you probably don't want to try all possible cut points in a serious implementation.  That's how we describe it in simple introductions to ID3, because it's easier to understand, but it's typically not how it is actually implemented, because it is slow.  In particular, if there are $n$ data points, then you'll need to test $n-1$ candidate thresholds; using the naive algorithm to calculate the information gain of each of those candidate thresholds takes $O(n)$ time per candidate, for a total of $O(n^2)$ time.

In practice, there are optimizations that speed this up significantly:

Don't try all possible thresholds.  Instead, pick a random sample of 1000 candidate thresholds (chosen uniformly at random out of the set of $n-1$ candidate thresholds), calculate the information gain for each, and choose the best one.
Use dynamic programming to efficiently compute the information gain of all $n-1$ splits, in total of $O(n)$ time, by reusing computation.  The algorithm is pretty straightforward to derive.


Can a machine learning model be used as some kind of compression?

You're right I'm not sure if I followed exactly what you said, however for your overall question of can a Machine Learning model be used as a form of compression - yes!
A good example is an Autoencoder, which is a neural network that 'breaks down' the input into basic elements and then reconstructs the input using only the elements required.  This is formally know as Dimensionality Reduction.
Some of the practical applications are removing noise for example from images, which if you think about is removing data you don't need but preserving the overall look.  Much like traditional image compression, it won't look exactly the same.


Terminology in machine learning: exogenous features vs external features

Exogenous simply means a value that is determined outside the context of your model & is then imposed on your model. Endogenous means the model determines the value.
I don't know about "external" as this word seems to depend on context. But you would be right to say these variables are exogenous.


FREAK is only a feature descriptor, so in order to use it you must provide a set of keypoints for it to extract your features e.g., FAST, AGAST, SIFT’s keypoint detector, etc. The original paper uses AGAST.
You might need to process the keypoints since the region requirements are different (A keypoint for SURF might lay too close on the border to compute FREAK).
keypoints = SomeDetector(image)
freakExtractor = cv2.xfeatures2d.FREAK_create()
keypoints,descriptors= freakExtractor.compute(image,keypoints)

The FREAK: Fast Retina Keypoint paper says:

Rostenand and Drummond proposed in [A] the FAST criterion for corner detection, improved by Mair et al. in [B] with their AGAST detector. The latter is a fast algorithm to locate keypoints. The detector used in BRISK by Leutenegger et al. in [C] is a multi-scale AGAST. They search for maxima in scale-space using the FAST score as a measure of saliency. We use the same detector for our evaluation of FREAK.

[A] E. Rosten and T. Drummond. Machine learning for highspeed corner detection. Computer Vision–ECCV 2006, pages 430–443, 2006.
[B] E. Mair, G. Hager, D. Burschka, M. Suppa, and G. Hirzinger.
Adaptive and generic corner detection based on the accelerated segment test. Computer Vision–ECCV 2010, pages 183–196, 2010. 2
[C] S. Leutenegger, M. Chli, and R. Siegwart. Brisk: Binary
robust invariant scalable keypoints. 2011.

Sources: Here and This SO question


How to specify scale_pos_weight value at runtime in Hyperopt?

Use Sklearns class_weight.compute_class_weight, which will return an array of weights.
My problem here is binary hence i had two elements in the array. We need to increase the weightage of the minority class. The target variable 1 was my minority class, thus I set the maximum scale_pos_weight so to the value from the second element in the array.
 clas_wts_arr = class_weight.compute_class_weight('balanced',np.unique(y_trn),y_trn)
 self.scale_pos_wt = clas_wts_arr[1]

As part of the space dictionary passed to the objective fn defined within fmin we include scale_pos_weight like the following
'scale_pos_weight' :hp.choice('scale_pos_weight',scl_ps_wt)

if the fmin object has been defined as follows
            best = fmin(fn=objective,
                    space=space,
                    algo=tpe.suggest,
                    max_evals=self.NUM_EVALS,
                    trials=trials)

Best will have the index of the best value in the array . Using this index we get the actual value of scale_pos_weight . We use this value to set the scale_pos_weight in best .
 best['scale_pos_weight'] = scl_ps_wt[best['scale_pos_weight']]

Now, use can set best as parameter to you classifier.


Calculate the similarity between pairs of time series data

Euclidean distance metric is unsuitable for time series...In short, it is invariant to time shifts, ignoring the time dimension of the data. If two time series are highly correlated, but one is shifted by even one time step, Euclidean distance would erroneously measure them as further apart. You might use it to compare your time series if you are extracting their features. Don't need to go and use directly PCA. You can use simple descriptive stats such as mean, min, max, kurtosis, etc...Then you can compare it with Euclidean.
I advise you to give it a try with DTW. ;)
How does it work?
http://alexminnaar.com/2014/04/16/Time-Series-Classification-and-Clustering-with-Python.html   (you can check that blog post, it explains it nicely)
Good luck to you!


How to count the number of times a sequence in a panda (time) series is less than a specific duration

Here's my idea:

# dataframe example
b = [1,0,0,0,0,0,0-1,1,0,0,-1,0,0,0,0,0,0,1,0,0,-1]
minutes = ['2019-01-02 15:' + str(x) + ':00' for x in range(len(b))]
df = pd.DataFrame({'a': minutes, 'b': b})

# keep only rows which are equal to 1 or -1
df_tmp = df[df['b'] != 0]

# convert date column to datetime if needed
df_tmp['a'] = pd.to_datetime(df['a'])

# create a new column which is the time difference between each row
df_tmp['diff'] = df_tmp['a'].diff().astype('timedelta64[m]')

# keep only the rows equal to -1
df_tmp = df_tmp[df_tmp['b'] == -1]

# count the values which are smaller or equal to 5
(df_tmp['diff'] <= 5).sum()


How to trust the labels generated using ML models?

So when we generate labels via machine learning models like clustering above, is it a recommended approach? Only if you can really make highly distinct 2 clusters/groups. This will be highy unlikely, especially for complicated and high dimensional datasets. One of the reasons is that clustering algorithms are just weaker than the supervised algorithms. If you can find a good representation (have a look at representation learning from Bengio), i.e. highly discriminative embeddings, than it might work.

Is it used in industries/real time where people don't have ground truth and only rely on labels based on ML models? Its an option, one can definately try it, but not rely on it.

How can we trust these labels generated? As long as you can validate it with out of fold set with ground truths, or with humans looking at the clusters, there is no problem.

Are things like this being used in Industries and how do they tackle the trust issue? Its one of the possible solutions, personally I always try first transfer learning. Especially for problems like yours, chances are there is already some pretrained model. Only thing you need is some labeling tool, for 1000 samples (it takes a couple of hours to do it but its worth it). Have a look at this tool.


Explanation for Why Logistic Regression can be so Accurate in Sentiment Classification?

Some people have this preconceived notion that logistic regression shouldn't perform well because it is too simple. While that might be true for certain types of complex problems, the reality is that logistic regression performs relatively well in many different scenarios.
It's difficult to give a definitive answer as to why a model performs well without actually seeing the data and the model you've constructed.
That being said, it could be that the problem you're trying to solve is not very difficult so even a simple logistic regression model yields good results. Another thing to consider is that while 88% accuracy sounds good, it might actually only be OK for this type of problem (and dataset) - it would be informative to try other non-neural-network based classifiers and see how well they perform.


How to treat data transformation choices as hyperparemeters?

So, the question talks about how to treat transformation choices as hyper parameters.
How I would go about it is the following:
Use one baseline model architecture for the data and then repeat the following:

Instantiate the baseline model (effectively make sure all of the weights are initialised)
Create the transformed dataset
Train the model
Compute generalisation performance measures (AUC, precision, recall, whatever).

Then compare the generalisation performance across all of the data transforms to find the "best" transformation which improves a generalisation metric which is appropriate for your task.


The impact of using different scaling strategy with Clustering

Welcome to the community!
Some points which might help:

Clustering, as an unsupervised task, can not be evaluated and usually some external criteria are used to find the best clustering.
According to the point above, better to make those assumptions as direct as possible. Starting with EDA (inspecting histograms, plotting boxplots, etc.) gives you a better initial point for analysis. Trying to understand the underlying structure of data directly from decrease of WSCC seems pretty indirect to me thus more difficult. For example a simple EDA tells you if data needs scaling urgently or not. Specially iin your data, ranges are not "dramatically" different, but if you see histogram, you may see, for instance, a exponential feature which needs a log-transform instead of scaling.
The last but not the least, the larger the number of clusters get, the smaller the WCSS become. You see it in plots. So the question is if these three scaling show a significant difference in dropping point of WCSS, which they don't and boosts this idea that scaling is not the biggest help here. Otherwise your code is pretty right (and well-written ;) )

I hope it could help to some extent.


How is the equation for the relation between prediction error, bias, and variance defined?

If:
$$Err(x)=E[(Y-\hat{f}(x))^2]$$
Then, by adding and substracting $f(x)$,
$$Err(x)=E[(Y-f(x)+f(x)-\hat{f}(x))^2]
$$
$$= E[(Y-f(x))^2] + E[(\hat{f}(x)-f(x))^2] + 2E[(Y-f(x))(\hat{f}(x)-f(x))]$$
The first term is the irreducible error, by definition. The second term can be expanded like this:
$$E[(\hat{f}(x)-f(x))^2] = E[\hat{f}(x)^2]+E[f(x)^2] -2E[f(x)\hat{f}(x)] $$

$$=E[\hat{f}(x)^2]+f(x)^2-2f(x)E[\hat{f}(x)] $$

$$= E[\hat{f}(x)^2]-E[\hat{f}(x)]^2+E[\hat{f}(x)]^2+f(x)^2-2f(x)E[\hat{f}(x)] $$

$$= E\big[(\hat{f}(x)-E[\hat{f}(x)])^2\big] + (E[\hat{f}(x)]-f(x))^2 $$

$$= Bias^2+Variance$$

Then the only thing that is left is to prove that the third term is 0. This is seen using $E[Y] = f(x)$.

Edit

I am not that sure on how to prove
$$E[(Y-f(x))(\hat{f}(x)-f(x))] = 0$$
If we assume independence between $\epsilon = Y - f(x)$ and $\hat{f}(x)-f(x)$, then the proof is trivial, as we can split the expected value in two products, the first of them being $0$. However, I am not so sure about the fact that we can assume this independence.


If ReLU is so close to being linear, why does it perform much better than a linear function?

A neural network is a succession of layers (dense/linear, convolutional). After each layer, you need a non-linear activation function. Why do you need a non-linear activation function? Because composing two linear transformations is equivalent to having a single linear transformation:

$ (x \cdot W_1 + b_1) \cdot W_2 + b_2 = x (W_1 \cdot W_2) + (b_1 \cdot W_2 + b_2)$

Therefore, stacking linear transformations would be pointless, because the whole network would be equivalent to a linear transformation.

A ReLU may seem similar to a linear function, but its non-linear nature gives the network the ability to model non-linear functions.
No, there is nothing unique about $0$ being the point of change. Most neural network layers have a bias term that makes the point of change irrelevant, as during the training the bias will be adjusted to the proper value.
While in deep learning everything is data-dependent, the fact that the ReLU's changing point is irrelevant is independent from the data (see previous answer).

The advantage of ReLU is against other non-linearities, like sigmoid and tanh, which suffer the vanishing gradient problem. ReLU itself has its own problem, namely the "dying ReLU problem".


Recommendations to generate a training set of images

1) wouldn't recommend switching cameras, could consider some type of resolution mapping to help though.
2) Likely not, but resolution alterations likely would.
3) Deeplearning => require more data, use transfer learning so you can use a less though.
4) try to keep the data as similar to it would be in production and try to keep the images general to avoid bias. (example: image recognizer predicting any image w snow to be a wolf)


How to create a table to display relative frequencies of selected words (eg. with, can, will) from any text corpus in nltk package in python

Finding frequency of words for the Gutenberg Corpus. You need to just put Gutenberg text files as input.

cfd = nltk.ConditionalFreqDist(
    (fileids, word)
    for fileids in gutenberg.fileids()
    for word in gutenberg.words(fileids))
modals = ['with', 'can', 'will']
cfd.tabulate(fileids, samples=modals)

Output :
                        with  can will
        austen-emma.txt 1187  270  559
  austen-persuasion.txt  643  100  162
       austen-sense.txt  971  206  354
          bible-kjv.txt 5961  213 3807
        blake-poems.txt   59   20    3
     bryant-stories.txt  276   75  144
burgess-busterbrown.txt   59   23   19
      carroll-alice.txt  175   57   24
    chesterton-ball.txt  799  131  198
   chesterton-brown.txt  726  126  111
chesterton-thursday.txt  544  117  109
  edgeworth-parents.txt 1304  340  517
 melville-moby_dick.txt 1659  220  379
    milton-paradise.txt  876  107  161
 shakespeare-caesar.txt  131   16  129
 shakespeare-hamlet.txt  212   33  131
shakespeare-macbeth.txt  134   21   62
     whitman-leaves.txt 1111   88  261

Hope This answers your question :)


What does the "dual" parameter in sklearn.svm.LinearSVC and sklearn.svm.LinearSVR do?

ELI5-style, here we go.

Kidwarts is a world-famous magical kindergarten for talented 5-year old wizards. Most of the day the kids spend playing in a large hall, which is full of fluffy puppies, magical seesaws, jumping chairs, tiny flying carpets (remote-controlled using kid-approved magic-wands), and all the other cool toys one could even imagine. Each kid has their favourite spot in the hall, where they would typically spend most of the day, petting their favourite puppy or jumping on their favourite jumping chair.

One sunny Tuesday morning miss Fluffypuff (the kindergarten's supervisor) brought a huge magic mirror to the center of the playhall. The mirror has two faces. If one looks into the mirror from one side, they would see meadows full of beautiful rainbow ponies hopping around. From the other face the mirror shows heroic fables of brave young wizards riding dragons and fighting evil witches. Obviously, it would be best if the mirror could be oriented in the room in such a way, that the "pony" face would be turned towards where most of the girls play, while the "dragon" one could be observed mostly by the boys. Everyone knows that 5 year old girls love ponies while boys adore dragons!

It is not an easy task, however. Although many boys indeed mostly hang around the corner of the hall with the flying carpets while a lot of girls prefer the corner with puppies, a bunch of kids are spread all around and could become unhappy and start crying using magical tears if the mirror is positioned unfavourably to them. Miss Fluffypuff has been strugging to find the best spot for the mirror the whole day.

Here is how she did it. First she hovered the mirror right in the middle of the room, turned in a somewhat random direction. She then waited and observed who of the kids started crying because they did not like what the mirror showed them. Based on that information, she nudged the mirror slightly towards the most unhappy kids, so that they would get to see more of their preferred side (or less of the "bad" side). After the whole day of such nudging the mirror seems to have converged to a more or less optimal orientation, with just a few kids left unhappy. To them miss Fluffypuff gave pieces delicious magical candy, so they were still quite happy after all!

In the evening Miss Fluffypuff was excited to tell the mirror story to her friend, Mister Puppypaw. "Ah, I would have done things differently" said Mister Puppypaw, twirling his gorgeous mustache with his fingers.
My method would be the following: I would first pick two kids in the room, and ask them to decide on their favourite orientation for the mirror. Then I would see who of the other kids complains about the mirror, add them to the team and have the new team decide together again. If at this point there were still unhappy kids, I would continue adding them to the "council" and let them find a compromise which suited everyone. We would thus end up with the situation, where the configuration of the mirror is decided to satisfy precisely the group of kids who care about it, while those without opinion would not have to be bothered. This would be super fair, wouldn't it?

Miss Fluffypuff agreed, but noted that her method also seemed totally fair in the end. They kissed and lived happily ever after. The End.

In this story, Miss Fluffypuff was solving the primal problem. That is, she was searching for the orientation $\mathbf{w}$ of the mirror directly. Mister Puppypaw, instead, proceeded in a roundabout, dual manner. He assumed that we can always compute the orientation $\mathbf{w}$ by having each kid vote on their preference, and aggregating these votes:

$$\mathbf{w} = \sum_{i} \alpha_i \mathbf{x}_i, $$

where $\mathbf{x}_i$ is the favourite position of kid $i$ and $\alpha_i$ is the vote (which will be $0$ if the kid does not care, positive if the kid prefers one face of the mirror, and negative otherwise). Consequently, the problem of Mister Puppypaw is not in finding the best orientation $\mathbf{w}$ directly, but rather in figuring out the vector of "votes" $\mathbf{\alpha}$ (from which the orientation $\mathbf{w}$ can be derived in a straightforward manner). YOu can also say that his main problem would be to find a subset of kids whose vote would be nonzero - these would be the "support vectors".

When $\mathbf{w}$ is just a two-dimensional vector and the number of kids is large, as is in the original story, the method of miss Fluffypuff is probably the best. Keeping track of a two-dimensional orientation of the mirror is simpler than managing a list of hundred or so "votes" or figuring out a proper subset of "kids who care".

If there were only one or two kids, however, it would instead be faster to just ask them rather than play this "mirror-nudging" game. Similarly, if the mirror had a million different knobs which needed to be configured, it would be easier to let even a hundred kids simply vote on their preferences (assuming each kid has a clear idea what million-knob configuration fits him best), rather than carefully tune these knobs in tiny increments, checking who of the kids started crying after every change.

In the end, either of the two methods would produce the same result, though.


Consequences of using XGBoost regressor for small dataset(< 500 rows)

The one other problem you have because of this is you are stuck with just this solution and this may not be the best one. Yes, you can use regularization but still, what if some other better solution exists than this. Since, in boosted trees, you aim is to  reduce your errors, each iteration of boosting tries to overfit on the examples which the last iteration predicted wrong.
But there is a way to confirm if this is the best solution. Do cross-validation. Or try LightGBM and CatBoost also, since, they are also good libraries for the same purpose. And finally, you combine all three of them to come with an ensemble that will you the best of the three worlds.


Help me understand how word-as-vector representations are constructed

I am assuming that you mean a vector representation of words, not to be confused by the vector representation produced in a bag of words approach that represent a document in vector space. Word2vec is an approach in which you train a model to represent words as a function of the provided context.

The answers that follow are:

1) No: You'll need some representation of context in which the word is used. For example: Skip-grams.
2) No: That is a user defined parameter
3) < I do not understand this question >


Data scaling for large dynamic range in neural networks

If it is a classification problem,  then you will use sigmoid or softmax to make the output value in (0,1) and all the value must sum to 1 as per the rule of probability.


Train neural network for regression with negative samples

An option might be to tinker with the cost function: Usually one uses the difference between the target output ($out_{t}$) and the actual output ($out_{a}$) as a part as the cost function. Let's say

$$
J = \sum_{i=1}^N (out_{t,i} - out_{a,i})^2
$$

is the cost function, where N is the number of samples. Now we can adjust this to the cost function

$$
J = \sum_{i=1}^N type_i \cdot (out_{t,i} - out_{a,i})^2
$$

where $type_i$ is 1 for normal samples and -1 for "negative samples". Alternative one might also use

$$
J = \left( \sum_{i=1}^M (out_{t,i} - out_{a,i})^2 \right) + \left( \sum_{i=M+1}^N \frac1{(out_{t,i} - out_{a,i})^2} \right)
$$

as the cost function where the samples up to $M$ are the normal ones and from $M+1$ to $N$ are the negative samples.


Group similar words under one topic and assign them a title

If you have already selected the keywords you want grouped, why not write a function that finds all occurrences of words in the list and replaces it with your one, core word?

What you're describing sounds like a more advanced version of stemming.


Binary classification model for unbalanced data

Since you are doing binary classification, have you tried adjusting the classification threshold? Since your algorithm seems rather insensitive, I would try lowering it and check if there is an improvement.
You can always use Learning Curves, or a plot of one model parameter vs. Training and Validation error to determine whether your model is overfitting. It seems it is under fitting in your case, but that's just intuition.
Well, ultimately it depends on your dataset, and the different models you have tried. At this point, and without further testing, there can not be a definite answer.
Without claiming to be an expert on the topic, there are a number of different techniques you may follow (hint: first link on google), but in my opinion you should first make sure you choose your cost function carefully, so that it represents what you are actually looking for.
Not sure what you mean by pattern intuition, can you elaborate?

By the way, what were your results with the different algorithms you tried? Were they any different?


sklearn cross_validate without test/train split

Its the first case - there are 3 test_scores since the default setup is 3-fold cross validation.
Which means that the training set is split into 3 subsets and at each cross validation step the model is trained on 2 subsets and tested on the one subset left out.

Its all in the docs


hierarchical clustering doesn't work as expected

Likely your precomputed distance matrix is bad.

Judging from your plot, it's probably a half-filled similarity matrix instead. And it likely is not in the required form for linkage.

  The input y may be either a 1d condensed distance matrix or a 2d array of observation vectors.

Did you make your distance matrix "condensed"? Otherwise it will be treated as data matrix, not distance matrix, because python does not have type safety but uses duck typing - if it quacks like a data matrix, it is a data matrix. Then you end up computing Euclidean distances on half-filled rows of this matrix...


What is the name for this chart which splits a quantity by allocation to a class hierarchy?

This is called a Sankey-Diagram:
"Sankey diagrams are a specific type of flow diagram, in which the width of the arrows is shown proportionally to the flow quantity." (wiki

An easy package to make these graphs would be "d3Network" in R. But many other options are available.


Actual problems in Data Science/Machine Learning connected with music

You could begin your journey by picking up one of the topics present in the magenta project started by google. It's open-source so you could pick up some great ideas there.
Link to Magenta: https://magenta.tensorflow.org/
Another aspect you could go ahead with is trying to combine to genres of music and produce something new. There are also surveys in recommendation for music. Spend a day getting to know what's there and if you could take up a certain drawback from a paper and go ahead with it!
Finally, if nothing clicks, you could refer to chapter 8 from here: https://arxiv.org/abs/1709.01620
This paper is a survey on deep learning for music generation! The 8th chapter focuses on current problems and future prospects.


Macro F1 result higher than accuracy for imbalanced dataset

They calculated the "standard" F1 score as defined for binary classification tasks:

precision = 0.656
recall = 0.792
f1 = 2 * (precision * recall) / (precision + recall)
f1

gives

0.7176132596685083

Other versions of the F1 score are used for multiple classes as you can see here under "Extension to multi-class classification":

  The F-score is also used for evaluating classification problems with more than two classes (Multiclass classification). In this setup, the final score is obtained by micro-averaging (biased by class frequency) or macro-averaging (taking all classes as equally important). For macro-averaging, two different formulas have been used by applicants: the F-score of (arithmetic) class-wise precision and recall means or the arithmetic mean of class-wise F-scores, where the latter exhibits more desirable properties.

Alternatively, see here for the scikit learn implementation of the F1 score and its parameter description.


This will affect how the backpropagation for each of these outputs will cause the intermediate nodes within the network to be updated. If the output nodes' error were equal then the gradient descent process of the intermediate nodes will favor the resulting error of each of these outputs equally. By more heavily weighting an output node it means that the gradient descent will favor a set of parameters that perform better on that output node.


value saveAsTextFile is not a member of org.apache.spark.sql.DataFrame

This is not standard part of the API of DataFrames. You can either map it to a RDD, join the row entries to a string and save that or the more flexible way is to use the DataBricks spark-csv package that can be found here.

If it's just one column you can map it to a RDD and just call .saveAsTextFile(filename)


This raises many questions that you may or may not be able to answer, and this may not be helpful but hopefully it is.

Regarding the data, what are it's origins?  Is it all electronically collected or is it an amalgamation of, say delivery driver logs that could be easily falsified or recorded incorrectly? Asked another way, how do you know that the order, based on timestamps, is not correct?

On the other hand, if you know the relationship between the points on the route, why bother with the timestamp values?  From your comments, it seems as though the participant traveled in a straight line.  If that is the case, you know the order based on the coordinates. You could order them by distance from the mean of the coordinates.

Since there is some disagreement between the "known" route and the "recorded" route I think to resolve the data you need to understand what causes this disagreement?  Is the timestamp produced by one device or multiple?  If one device, maybe someone is not driving the advertised route.  If multiple devices, maybe one is set to the wrong timezone or just has the wrong time.

Again, not really an answer, but hopefully helpful.


when DataFrame has mutliIndex,how to visualize it?

So your question is how to plot the top 3 game genres for each game platform (console). You are definitely on the right track in using the groupby method.
It is a bit long winded, but this would definitely do the job:
c = df.groupby(["Platform", "Genre"]).size().reset_index()
c.columns = ["Platform", "Genre", "Frequency"]

for p in df["Platform"].unique():
    """
    This line will take a subset of the c dataframe such that it only contains
    entries for a particular platform.
    The .sort_values() method will then arrange the frequency in descending order.
    The .head(n) will then take the top n entries and therefore, the top three genres
    for a particular platform.
    """
    df_ = c[c["Platform"] == p].sort_values("Frequency")
    sns.barplot(x = "Platform", y = "Frequency", hue = "Genre", data = df_)
    plt.show()

Here, you can edit the code so that for each platform, each graph appears on a separate axis.


How to do this complicated data extrapolation, prediction modeling?

I am not sure I understood the problem,however if you are trying to predict sales amount my guess is ARIMA might not be the right choice as it will not consider external variables.My suggestion is to gather related features such as how did the other movie from same genre did at that time of the year in that region,weather,star presence,presence of other popular matches or tournament at that time and etc.Let me know your thoughts.


Algorithm for segmentation of sequence data

Please see my comment above and this is my answer according to what I understood from your question:

As you correctly stated you do not need Clustering but Segmentation. Indeed you are looking for Change Points in your time series. The answer really depends on the complexity of your data. If the data is as simple as above example you can use the difference of vectors which overshoots at changing points and set a threshold detecting those points like bellow:

As you see for instance a threshold of 20 (i.e. $dx<-20$ and $dx>20$) will detect the points. Of course for real data you need to investigate more to find the thresholds.

Pre-processing

Please note that there is a trade-off between accurate location of the change point and the accurate number of segments i.e. if you use the original data you'll find the exact change points but the whole method is to sensitive to noise but if you smooth your signals first you may not find the exact changes but the noise effect will be much less as shown in figures bellow:

Conclusion

My suggestion is to smooth your signals first and go for a simple clustering mthod (e.g. using GMMs) to find an accurate estimation of the number of segments in signals. Given this information you can start finding changing points constrained by the number of segments you found from previous part.

I hope it all helped :)

Good Luck!

UPDATE

Luckily your data is pretty straightforward and clean. I strongly recommend dimensionality reduction algorithms (e.g. simple PCA). I guess it reveals the internal structure of your clusters. Once you apply PCA to the data you can use k-means much much easier and more accurate.

A Serious(!) Solution

According to your data I see the generative distribution of different segments are different which is a great chance for you to segment your time series. See this (original, archive, other source) which is probably the best and most state-of-the-art solution to your problem. The main idea behind this paper is that if different segments of a time series are generated by different underlying distributions you can find those distributions, set tham as ground truth for your clustering approach and find clusters.

For example assume a long video in which the first 10 minutes somebody is biking, in the second 10 mins he is running and in the third he is sitting. you can cluster these three different segments (activities) using this approach.