I'd say the distinct handling of the ordered and unordered factor in decision trees is more convention and
implementation detail than a necessity.

But it is also an important optimization feature. See the documentation of the  rpart here

  We have said that for a categorical predictor with $m$ levels, all $2^{(m-1)}$   different possible splits  are  tested..

and

  Luckily,  for any ordered outcome there is a computational shortcut that allows the program to find the best split using only $m-1$ comparisons.

As you see, the ordered factor may be processed much effectively.

My advice therefore - as a part of the feature ingeneering decide whether to use a factor ordered or unordered:

Use ordered factor only if it is highly correlated with the output variable, otherwise fall back to an unordered factor

Bellow is a simple example, how can a scattered output variable with an ordered factor as a feature fool the decision treee to be very deep and ineffective.

> df
  X Y
1 1 0
2 2 1
3 3 0
4 4 1
> str(df)
'data.frame':   4 obs. of  2 variables:
 $ X: Ord.factor w/ 4 levels "1"<"2"<"3"<"4": 1 2 3 4
 $ Y: num  0 1 0 1

Notice that the output variable $Y$ is highly uncorrelated with the ordered factor.

> fit <- rpart(Y ~ X, method="class", data = df, control=rpart.control(minsplit = 1))
> print(fit)
n= 4

node), split, n, loss, yval, (yprob)
      * denotes terminal node

 1) root 4 2 0 (0.50000000 0.50000000)
   2) X=1 1 0 0 (1.00000000 0.00000000) *
   3) X=2,3,4 3 1 1 (0.33333333 0.66666667)
     6) X=3,4 2 1 0 (0.50000000 0.50000000)
      12) X=1,2,3 1 0 0 (1.00000000 0.00000000) *
      13) X=4 1 0 1 (0.00000000 1.00000000) *
     7) X=1,2 1 0 1 (0.00000000 1.00000000) *

Which leads to a deep (and unscalable) decision tree.

Making the factor unordered results in the optimal  decision tree.

> df
  X Y
1 1 0
2 2 1
3 3 0
4 4 1
> str(df)
'data.frame':   4 obs. of  2 variables:
 $ X: Factor w/ 4 levels "1","2","3","4": 1 2 3 4
 $ Y: num  0 1 0 1
>

> fit <- rpart(Y ~ X, method="class", data = df, control=rpart.control(minsplit = 1))
> print(fit)
n= 4

node), split, n, loss, yval, (yprob)
      * denotes terminal node

1) root 4 2 0 (0.50000000 0.50000000)
  2) X=1,3 2 0 0 (1.00000000 0.00000000) *
  3) X=2,4 2 0 1 (0.00000000 1.00000000) *


You can find some data and maybe combine from multiple sources, starting with:

https://www.kaggle.com/kimjihoo/coronavirusdataset (what I could come up that includes patient ages from the first Google results)

And for the global stats: https://github.com/datasets/covid-19/blob/master/data/time-series-19-covid-combined.csv or


Neural Net Accuracy: Test Set vs Real World Data

Many resources teach the process of splitting data into training, validation and test sets. This is what you want to do for "closed" datasets where it's not possible to get additional data.

This assumption of a closed dataset is often not true in the real world, where it may be feasible to collect more data. Statistically speaking, it is a lot more desirable to define a test set as a new data sample that was collected separately from your training data. This might be more representative of how the model will behave in production, but sometimes even this is not enough:

A few weeks back I built an image classifier for cars. I trained it using a mix of existing datasets and the results of a web scrape. Ultimately, it was deployed through an iOS app where it was supposed to do make predictions in real time. In this case, it was not enough to just create a test split or collect a new sample from the web. We needed to shoot our own images that were representative for the use case in order to make realistic assumptions about the app's performance.


Is it possible to decompose a scalar value to a inter-dependent vector neural network?

Yes, you can do that by interchanging the position of decoder and encoder in an autoencoder.

In an autoencoder, you give a long vector as input - the encoder reduces it to a short length vector (compressed) - the decoder now takes this compressed vector as input and upsamples it to the size of the original vector. The autoencoder is trained by taking the Mean Square Error (MSE) of the output of decoder with respect to the input vector. This enforces the compressed vector representation to contain the information of the input vector.

Now coming to your case. You simply need to pass the single scalar value to a decoder that upsamples, say your 3 layer fully connected neural networks. Let this output be denotes as "latent representation". Now pass this "latent representation" to the encoder which uses this "latent representation" to output just a single scalar value. Use MSE objective to enforce the the above single scalar output to match the input scalar value. Once the training is done of the above reverse autoencoder, the "latent representation" will give you the required vector containing the information about the scalar value you wished to represent as a vector.


How can I find correlation between features?

Correlation is a metric that can be used when two features have the same number of total observations. Given your scenario, each router can fail for different number of times. So, we can't estimate correlation between two routers since they will be having different number of observations.

One thing you can do would be to convert every router into a time series signal, using the provided duration and start time. You can they compute the correlation then. But, it might not be really informative of the actual situation, as the signal will be predominantly sparse


Combining time-series data from different devices

Assuming that duration is the same across devices, or that it is possible to resample the data such that it is (ex: median across 1 minute intervals).

You can present the numeric data (value) as a line plot, with each sensor values as its own series. You may want to (or not!) normalize the values within each series to make them more comparable.

For the state, map each distinct state to a numerical value. Ex: { "resting": 1, "running": 2, ... }. Then use a point plot with time as X axis, and this numeric state value as Y. Mark the y ticks with the state labels. Each sensor should have its own color.

Compare and contrast

When working with multiple related data series, often the the similarities and dissimilarities between dataseries can be just as informative (or more) than each series in itself.

If the data is supposed to represent the same thing. Then consider computing the median of the value across all sensors, and then for each sensor the difference from group-mean. Plot in a line plot like above.

If you expect agreement in "state", then you can compute the mode across the sensors. And then for each sensor compute when the mode differs. Plot this in a state plot similar to above, but now have "mode" as a series - and for each sensor series only plot values that differ from mode. This can also be represented time-independent as a confusion matrix, either overall or for each sensor.


The whole idea of a validation set is that the model does not know about this data, so you can get an unbiased estimation of the model's performance. Then based on this unbiased estimation you find the best parameters for the model. The problem is, that finding the hyperparameters is in itself a way of training your model. So with optimized hyperparameters the model starts to overfit on the validation set. That is why to check the real accuracy of the model you actually need a different part of data, that your model never saw. This is the testing part of data.

Therefore in your case, when you do not have any hyperparameters, you can just use the division on train and test.

If you want a higher accuracy of estimating your model's accuracy than it is better to use Cross Validation instead of just dividing on train and test. Neural networks usually do not do full cross validation because it means increasing the computation time several times (5 for 5-fold cross validation).

With hyperparameters the ideal way is to do a double cross validation. One for validation set and one for test set. This is too expensive computationally so it is only used on very simple models, like a ridge regression.

Also very few models, that I know of, do not have hyperparameters. And usually those that do not have hyperparameters perform poorly compared to the ones that have. Ridge regression is often better than linear regression. Neural networks with variable number of layers perform better than fully automated neural networks.


using sklearn class weight to increase number of positive guesses in extremely unbalanced data set?

You can try to compute class weights and assign these values to model via weight classes function. One more reminder about weights; probably major classes weight will be less than 1 so you need to round it to 1 otherwise model won't learn major class this time.


Can I run Orange widgets from normal Python scripts?

I don't think Orange was ever intended to be used that way, but if you can convert your data into Orange.data.Table, you might be able to instantiate the widgets you need with it.

See the if __name__ == '__main__' block at the bottom of most widgets for some example.


Relation between convolution in math and CNN

Using the notation from the wikipedia page, the convolution in a CNN is going to be the kernel $g$ of which we will learn some weights in order to extract the information we need and then perhaps apply an activation function.

Discrete convolutions

From the wikipedia page the convolution is described as

$(f * g)[n] = \sum_{m=-\inf}^{\inf} f[m]g[n-m]$

For example assuming $a$ is the function $f$ and $b$ is the convolution function $g$,

To solve this we can use the equation first we flip the function $b$ vertically, due to the $-m$ that appears in the equation. Then we will calculate the summation for each value of $n$. Whilst changing $n$, the original function does not move, however the convolution function is shifted accordingly. Starting at $n=0$,

$c[0] = \sum_m a[m]b[-m] = 0 * 0.25 + 0 * 0.5 + 1 * 1 + 0.5 * 0 + 1 * 0 + 1 * 0 = 1$

$c[1] = \sum_m a[m]b[-m] = 0 * 0.25 + 1 * 0.5 + 0.5 * 1 + 1 * 0 + 1 * 0 = 1$

$c[2] = \sum_m a[m]b[-m] = 1 * 0.25 + 0.5 * 0.5 + 1 * 1 + 1 * 0 + 1 * 0 = 1.5$

$c[3] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0.25 + 1 * 0.5 + 1 * 1 = 1.625$

$c[4] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0 + 1 * 0.25 + 1 * 0.5 + 0 * 1 = 0.75$

$c[5] = \sum_m a[m]b[-m] = 1 * 0 + 0.5 * 0 + 1 * 0 + 1 * 0.25 + 0 * 0.5 * 0 * 1 = 0.25$

As you can see that is exactly what we get on the plot $c[n]$. So we shifted around the function $b[n]$ over the function $a[n]$.

2D Discrete Convolution

For example, if we have the matrix in green

with the convolution filter

Then the resulting operation is a element-wise multiplication and addition of the terms as shown below. Very much like the wikipedia page shows, this kernel (orange matrix) $g$ is shifted across the entire function (green matrix) $f$.

taken from the link that @Hobbes reference. You will notice that there is no flip of the kernel $g$ like we did for the explicit computation of the convolution above. This is a matter of notation as @Media points out. This should be called cross-correlation. However, computationally this difference does not affect the performance of the algorithm because the kernel is being trained such that its weights are best suited for the operation, thus adding the flip operation would simply make the algorithm learn the weights in different cells of the kernel to accommodate the flip. So we can omit the flip.


You haven't specified what happens for values other than '기아' ('KIA') and '현대' ('HyunDai').
Try

SELECT
CASE
    WHEN Maker='기아' THEN 'KIA'
    WHEN Maker='현대' THEN 'HyunDai'
    ELSE Maker
END AS new_Maker

FROM table_name


Understanding Transfer Learning of Word Embeddings

So Named Entity Recognition is a mechanism where you ask your network to learn about how to detect entities given word vectors as the input.
The theoretical aspect of word embeddings is that based on your construction of sentences, the word embeddings for Orange and Apple are very similar i.e their cosine angle is very small.
In Named entity recognition you use these word embeddings and feed them into a network where the data you are training one has tags for each of the word embeddings i.e entities or normal words. So your network is actually understanding the relationship of the word embeddings and how to tag them. This makes it incredible for us to see that Apple gets detected even though its not in the training set, precisely where word embeddings help us quite well because the word embeddings are usually trained on a large corpus of data containing the words apple, orange and other tokens. This is where transfer learning helps because you are using the word embeddings trained in an unsupervised manner and then used to learn about entities.
Hope that helps. I can elaborate if required.


At this moment, the answer is no. Tensorflow uses CUDA which means only NVIDIA GPUs are supported.

For OpenCL support, you can track the progress here.

BTW, Intel/AMD CPUs are supported.


Fuzzy C-means is implemented in Python and you just need to google it e.g. this one,  however you can implement it yourself as well.

My answer will be more about your task. You have categorical data which means any data point in your problem is on the corner of a high-dimensional simplex. A simple example is 3 points on three vertices of a triangle. How would you cluster them and even more, how would you fuzzy cluster them? (because they are always ON EACH OTHER which makes fuzzy clustering a bit meaningless)

I suggest this:
Find a similarity score between your points. For instance, as a very initial and simple approach, capture the normalized number of attributes in which two points have the same value.

a = [1,0,0]
b = [1,0,1]
similarity = 2/3

You can think of more similarity measures (what about Chi-Squared test?). When you have these scores between sample pairs, you can construct the affinity matrix and apply spectral clustering on it.

But where is the "Fuzzy" term? (or soft-clustering characteristic)

Spectral clustering works as follows:

You get the affinity matrix
Capture the eigen vector corresponding to the second smallest eigen value
That eigen vector is a representation of your data. Cluster that instead of data itself! Here you can use soft-clustering methods for clustering eigenvector elements.
See these steps in Section.2 of this.


How do I store a question-and-answer combination for machine learning?

Typically, people do not use one hot encoding as it takes up a lot of space (vocabulary size explodes) and is difficult to learn as well. Instead, they project words into a high dimensional vector space where each dimension represents some sense or direction.

For example, if you consider the words red and green - they are opposites in the context of a traffic light but are similar in the sense that they are both names of colours. A vector representation of the word will incorporate such relations by having similar values for some dimensions and different values for the others.

For more details, you can read this paper.

As for your next question, standard practice is to limit the sentence length through truncation and pad smaller sentences with additional 0 vectors.


Andrew Ng's explanation actually covers the YOLOv2 which uses anchor boxes. YOLOv1, which is the paper you linked, does not use anchor boxes so its not exactly the same.

They key to understanding how the bounding boxes are formed is to first understand how the output is encoded. To which, I'll recommend this link: https://hackernoon.com/understanding-yolo-f5a74bbc7967

Briefly speaking, and I'll be using the example from the paper, for S=7, B=2 and C=20, our output is a 7x7x30 tensor that encodes where (bounding box coordinates) and what the objects (probability of class) are. To achieve this, we construct a fully-connected layer at the end of our CNN that will give us 7x7x30 (rather forcefully). Hence on our first forward pass, each cell will have 2 random bounding boxes. A loss is calculated. The weights of the CNN will then be adjusted according to reduce that loss (opitimisation). Then the following passes will produce bounding boxes closer to the ground truth.


From development environment to production

There isn't a well established way of estimating the number of data points that you'll need. It's much more an art than a science. As you gain more experience, you'll learn some common sense lessons (in hindsight) along the way. For instance, you should never have more parameters than data points; if you're building random forests, you should not have more trees than data points. If you're doing deep learning, you should never have more neurons than data points (these are extreme examples). As a rule of thumb, try to avoid having 10 times more features than you have data points. 300 data points is very small, and you so you should probably limit yourself to linear models.

As Hobbes mentioned, cross validation and/or holdout sets are the proper way to judge the readiness of your model irrespective of the datapoint-to-feature ratio. If the error on the training set is 10% better than the error on the test/validation set, you're probably overfitting and your model is not fit for production. The 10% threshold is very much a rule of thumb.

Also worth mentioning, once you feel comfortable that your model is fit for production, you should never deploy it and then just walk away. If your model has value, you should do ongoing post production monitoring. In the real world, your model's performance will inevitably degrade over time (regardless of how strong/accurate it was at the time of the model build) as the environment that your model was trained for evolves. This is certainly true for finance and insurance industries.

Production monitoring can also be used to pull the plug if your model performs far worse than you anticipated. If you have a very low opinion of the potential stability of your model, deploy it silently (don't allow the predictions to go to downstream systems) and measure how your model would have done had you hypothetically deployed it.

A last piece of advice: when you do predictions on new documents you'll nearly always come across words that you have not accounted for in your training set. You should find a way to build this awareness into your model. For example, you could choose to use only keywords as predictors or you could create indicator variables for new words. Either way, you should not expect the vocabulary size to remain the same.


why do we operate with graphical models in VAE, if there are no probabilites involved?

Not sure which type of code you were looking at but e.g. here they do sampling of random variables. The encoder deterministically maps the input X to mean and standard deviation vectors. Using these vectors you sample Z from factorizing Gaussians with these means and standard deviations. The samples of Z are the again mapped deterministically back to X by the decoder. The reparametrization trick is used to propagate the gradients through the network while only having to sample from standard Gaussians.


predict_proba returns different results on Python 2 & 3

Maybe...
Have a you seen if one of the predict_proba produces 2 columns?

I think Python 3 produces two columns and the one comparable is the second.


How to deal with disconnected components in isomap?

I believe here is a solution: https://ieeexplore.ieee.org/document/8282172.
Short summary: Build the network and get disconnected graphs. Calculate the center of each small graph and connect the centers with each other. Then scale the edges connecting different small graphs by some number greater than 1.


What kind of regression model should I do?

One thing you can do about your 'participation' variable is including the beginning and end of your data window. Let's say your data ranges from $start=$ 1/1/2016 to $end=$ 1/1/2017. Instead of only calculating the difference between the second and the first post, you'd calculate the difference between the first post and 1/1/2016 and then the difference between the second and the first post. (If some people join the platform after 1/1/2016 then you'd take the minimum of the join date and 1/1/2016). And you'd also calculate the difference between 1/1/2017 and last post.

So, if someone only had two posts $p_1$ and $p_2$, you would get the difference $(p_1-start, p_2-p_1,end-p_2)$. The differences $p_1-start$ and  $end-p_2$ would then be larger than the differences $p_1-start$ and  $end-p_n$ for some with $n>>2$ posts.


How to resample one dataset to conform to the distribution of another dataset?

One simple way is to transform your distribution linearly. That should work fine if the distribution of your data has changed approximately linearly.

  Question is how to change the distribution of DS_A to match
  distribution of DS_B, with respect to multiple features?

That being said you could transform your feature distribution DS_A by

Substructing mean(DS_A) and add the mean(DS_B)
Divide with the standard deviation of DS_A and multiply with the standard deviation of DS_B

Long story short:You change the mean and the standard Deviation of the DS_A distribution to match the DS_B.

Here is a code in python that apply this transformation to two gaussian distributions

import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt

m1, m2 = 0, 10
s1, s2 = 1, 3

x1 = np.random.normal(m1,s1, 1000)
x2 = np.random.normal(m2,s2, 1000)

sns.distplot(x1, hist_kws=dict(alpha=0.1), color='red', label='Distribution 1')
sns.distplot(x2, hist_kws=dict(alpha=0.1), color='green', label='Distribution 2')

estimated_x1_mean = np.mean(x1)
estimated_x1_sd   = np.std(x1)
estimated_x2_mean = np.mean(x2)
estimated_x2_sd   = np.std(x2)

x2_new = (x2 - estimated_x2_mean + estimated_x1_mean)  * estimated_x1_sd / estimated_x2_sd
sns.distplot(x2_new, color='blue', hist_kws=dict(alpha=0.1, edgecolor='black'), label='Distribution 2 after Transformation')
plt.legend()

And the result


Using Generative Adversarial Network to generate Single Image

A generative adversarial network is comprised of two parts which are necessary for the training, the generator and the discriminator. They are playing a game against each other, the discriminator tried to learn what is real and what is fake, and the generator tries to create fake instances which will fool the discriminator.

At the input of the discriminator you will use real images, for example the MNIST numbers. At the input of the generator you will put just random noise vectors. These vectors will then be shaped by the layers of the generator to create some image, hopefully resembling the MNIST dataset. If it fools the discriminator, it wins that round. Otherwise, it will tune its weights using backpropagation in order to produce better quality results in the next round.

Once training is complete, you will detach the discriminator and use only the generator. Then for every single random noise vector fed to the input of the generator you will receive an associated image.

Here is a GaN implementation. You will need to put a path "./gan/images/" i the directory where you are running this GaN.

import numpy as np
import time
from tensorflow.examples.tutorials.mnist import input_data

from keras.layers import Input
from keras.models import Model
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Dense, Activation, Flatten, Reshape
from keras.layers import Conv2D, Conv2DTranspose, UpSampling2D
from keras.layers import LeakyReLU, Dropout
from keras.layers import BatchNormalization
from keras.optimizers import Adam, RMSprop
from keras.datasets import mnist

import matplotlib.pyplot as plt

class GAN():
    def __init__(self):
        self.img_rows = 28
        self.img_cols = 28
        self.channels = 1
        self.img_shape = (self.img_rows, self.img_cols, self.channels)

        optimizer = Adam(0.0002, 0.5)

        # Build and compile the discriminator
        self.discriminator = self.build_discriminator()
        self.discriminator.compile(loss='binary_crossentropy',
            optimizer=optimizer,
            metrics=['accuracy'])

        # Build and compile the generator
        self.generator = self.build_generator()
        self.generator.compile(loss='binary_crossentropy', optimizer=optimizer)

        # The generator takes noise as input and generated imgs
        z = Input(shape=(28,28,))
        img = self.generator(z)

        # For the combined model we will only train the generator
        self.discriminator.trainable = False

        # The valid takes generated images as input and determines validity
        valid = self.discriminator(img)

        # The combined model  (stacked generator and discriminator) takes
        # noise as input => generates images => determines validity
        self.combined = Model(z, valid)
        self.combined.compile(loss='binary_crossentropy', optimizer=optimizer)

    def build_generator(self):

        noise_shape = (28,28,)

        model = Sequential()

        model.add(Reshape((28*28,), input_shape=noise_shape))
        model.add(Dense(256))
        model.add(LeakyReLU(alpha=0.2))
        model.add(BatchNormalization(momentum=0.8))
        model.add(Dense(512))
        model.add(LeakyReLU(alpha=0.2))
        model.add(BatchNormalization(momentum=0.8))
        model.add(Dense(1024))
        model.add(LeakyReLU(alpha=0.2))
        model.add(BatchNormalization(momentum=0.8))
        model.add(Dense(np.prod(self.img_shape), activation='tanh'))
        model.add(Reshape(self.img_shape))

        model.summary()

        noise = Input(shape=noise_shape)
        img = model(noise)

        return Model(noise, img)

    def build_discriminator(self):

        img_shape = (self.img_rows, self.img_cols, self.channels)

        model = Sequential()

        model.add(Conv2D(32, kernel_size=(3, 3),
                 input_shape=img_shape))
        model.add(MaxPooling2D(pool_size=(2, 2)))
        model.add(LeakyReLU(alpha=0.2))
        model.add(Dropout(0.25))

        model.add(Flatten())
        model.add(Dense(64))
        model.add(LeakyReLU(alpha=0.2))
        model.add(Dense(1, activation='sigmoid'))
        model.summary()

        img = Input(shape=img_shape)
        validity = model(img)

        return Model(img, validity)

    def train(self, epochs, batch_size=128, save_interval=50):

        # Load the dataset
        (X_train, _), (_, _) = mnist.load_data()

        # Rescale -1 to 1
        X_train = (X_train.astype(np.float32) - 127.5) / 127.5
        X_train = np.expand_dims(X_train, axis=3)

        half_batch = int(batch_size / 2)

        for epoch in range(epochs):

            # ---------------------
            #  Train Discriminator
            # ---------------------

            # Select a random half batch of images
            idx = np.random.randint(0, X_train.shape[0], half_batch)
            imgs = X_train[idx]

            noise = np.random.normal(0, 1, (half_batch,28,28,))

            # Generate a half batch of new images
            gen_imgs = self.generator.predict(noise)

            # Train the discriminator
            d_loss_real = self.discriminator.train_on_batch(imgs, np.ones((half_batch, 1)))
            d_loss_fake = self.discriminator.train_on_batch(gen_imgs, np.zeros((half_batch, 1)))
            d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)

            # ---------------------
            #  Train Generator
            # ---------------------

            noise = np.random.normal(0, 1, (batch_size,28,28,))

            # The generator wants the discriminator to label the generated samples
            # as valid (ones)
            valid_y = np.array([1] * batch_size)

            # Train the generator
            g_loss = self.combined.train_on_batch(noise, valid_y)

            # Plot the progress
            print ("%d [D loss: %f, acc.: %.2f%%] [G loss: %f]" % (epoch, d_loss[0], 100*d_loss[1], g_loss))

            # If at save interval => save generated image samples
            if epoch % save_interval == 0:
                self.save_imgs(epoch)

    def save_imgs(self, epoch):
        r, c = 5, 5
        noise = np.random.normal(0, 1, (r * c, 28,28,))
        gen_imgs = self.generator.predict(noise)

        # Rescale images 0 - 1
        gen_imgs = 0.5 * gen_imgs + 0.5

        fig, axs = plt.subplots(r, c)
        cnt = 0
        for i in range(r):
            for j in range(c):
                axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')
                axs[i,j].axis('off')
                cnt += 1
        fig.savefig("gan/images/mnist_%d.png" % epoch)
        plt.close()

if __name__ == '__main__':
    gan = GAN()
    gan.train(epochs=30000, batch_size=64, save_interval=200)


why does it not lead to the conclusion that the cost function is nearly flat in the dimensions created by the parameters associated with the earlier layers.

Your conclusion sounds very reasonable - but only in the neighborhood where we calculated the gradient.

E.g. (this is an unrealistic example, but bear with me) consider the case in which the cost function is $C(x,y)=100x^2+y^2$, while $x$ is a weight in a later layer and $y$ is a weight in an earlier layer.
I used https://academo.org/demos/contour-plot/ to draw a contour map of $C$, and then drew $4$ steps (arrows starting at the top left) that a gradient descent algorithm might take.
(For an explanation about contour lines and why they are perpendicular to the gradient, see videos 1 and 2 by the legendary 3Blue1Brown.)

The gradient descent algorithm only sees the slopes where gradients are calculated, so here it thinks that the cost function is quite flat in the dimension of $y$:

Imagine a scenario in which the arrows above are even more densely packed, i.e. the steps make an even smaller progress in the dimension of $y$. This would make the gradient descent algorithm really slow.

Here is another intuitive example for an unfortunate case in which the gradient tells us "the cost function is nearly flat in this dimension, so there is no need to move in that dimension".

$\frac{\partial C}{\partial y}$ is very small here, so a gradient descent algorithm would go right very slowly:

As for your other questions, I would try to clarify Nielsen's explanation.

Let's say we have a $[784,30,30,30,10]$ network for the MNIST problem, and it has a severe vanishing gradient problem.
Therefore, after we randomly initialize the weights in the first hidden layer (i.e. the first $30$ neurons layer, whose weight matrix's size is $30\times 784$), they are virtually stuck that way (as the gradient in the earliest layer is extremely small).

Now, consider an unfortunate case in which we randomly initialize the weights in the first hidden layer such that the weights of some pixels are so small, that these pixels' input values are virtually discarded.
To clarify, these input values are multiplied by very small weights, so their weighted value is always close to $0$, regardless of the input value. The weighted value of each input value is the only thing the network knows about the input, and so it is as if these input values were discarded.

In other words, some of the pixels in the input image are invisible to the network.
To demonstrate that, I painted these "invisible pixels" in red:

Let's see how our network sees some MNIST image:

Is it a $6$? Is it a $5$? Even a human like me finds it difficult to say for sure..
What about a picture of an $8$ or $9$? would we be able to identify the digit then?

Nielsen claims that it would be impossible to effectively train such network to identify digits, as it simply has no access to some crucial information - the invisible pixels.


Predict the date an item will be sold using machine learning

A machine learning problem can be separated into a few modular parts. Of course these are all massive in reach and possibility. However, every problem you encounter should be thought of in this way at first. You can then skip things where you feel necessary.

Data pre-processing and feature extraction
The model
Post-processing

Data pre-processing and feature extraction

This and feature extraction are the two most important parts of a machine learning technique. That's right, NOT THE MODEL. If you have good features then even a very simple model will get amazing results.

Data pre-processing goes from your raw data and remolds it to be better suited to machine learning algorithms. This means pulling out important statistics from your data or converting your data into other formats in order to it being more representative. For example if you are using a technique which is sensitive to range, then you should normalize all your features. If you are using text data you should build word vectors. There are countless ways pre-processing and feature extraction can be implemented.

Then, you want to use feature selection. From all the information you extracted from your data not all of it will be useful. There are machine learning algorithms such as: PCA, LDA, cross-correlation, etc. Which will select the features that are the most representative and ignore the rest.

In your case

First, let's consider the data pre-processing. You notice that type might not be an integer value. This may cause problems when using most machine learning algorithms. You will want to bin these different types and map them onto numbers.

Feature selection: besides using the techniques I outlined above, you should also notice that productID is a useless feature. It should for sure NOT be included in your model. It will just confuse the model and sway it.

As a general rule of thumb, the amount of data that is suggested to have for shallow machine learning models is $10 \times \#features$. So you are limited by the size of your dataset. Also, make sure the outputs are quite well distributed. If you have some skew in your dataset, like a lot of examples where the item was sold right away, then the model will learn this tendency. You do not want this.

The Model

Now that you have your feature-space, which might be entirely different from the original columns you posted, it is time to choose a model. You are trying to estimate the time of sale for an algorithm. Thus, this can be done in two different ways. Either as a classifier problem or as a regression problem.

The classifier problem would separate the different times of sales into distinct bins. For example

Class 1: [0 - 5] days
Class 2: [5 - 10] days
etc...

Of course the more classes you will choose to have then the harder it will be to train the model. That means the resolution of your results is limited by the amount of the data you have available to you.

The other option is to use a regression algorithm. This will learn the tendency of your curve in higher dimensional space and then estimates where along that line a new example would fall. Think of it in 1-dimension. I give you a bunch of heights $x$ and running speeds $y$. The model will learn a function $y(x)$. Then if I give you just a height, you will be able to estimate the running speed of the individual. You will be doing the same thing but with more variables.

There are really a ton of methods that can do this. You can look through some literature reviews on the subject to get a hold of all of them. But, I warn you there is A LOT. I usually start with kernel-Support Vector Regression (k-SVR).

To test your model. Separate your dataset into three parts (training, validating, testing) if you have sufficient data. Otherwise two parts (training, testing) is also fine. Only, train your model on the training set and then evaluate it using the example it has not seen yet which are reserved in the testing set.

Post-processing

This is the step where you can further model your output $y(x)$. In your case that might not be needed.


How to automatically find the sentiment?

Here is a tutorial, that should help you get started.


I think this will help.

My understanding is that one-to-many and many-to-many(like in 4th case of your pictorial) are in a way similar to autoregressive networks, where you utilize the prediction you've made and use it predict further ahead.


Struggling to integrate sklearn and pandas in simple Kaggle task

I have never used sklearn_pandas, but from reading their source code, it looks like this is a bug on their side. If you look for the function that is throwing the exception, you can notice that they are discarding the y argument (it does not even survive until the docstring), and the inner fit function expects one argument more, which is probably y:

def fit(self, X, y=None):
    '''
    Fit a transformation from the pipeline

    X       the data to fit
    '''
    for columns, transformer in self.features:
        if transformer is not None:
            transformer.fit(self._get_col_subset(X, columns))
    return self

I would recommend that you open an issue in their bug tracker.

UPDATE:

You can test this if you run your code from IPython. To summarize, if you use the %pdb on magic right before you run the problematic call, the exception is captured by the Python debugger, so you can play around a bit and see that calling the fit function with the label values y[0] does work  -- see the last line with the pdb> prompt. (The CSV files are downloaded from Kaggle, except for the largest one which is just a part of the real file).

In [1]: import pandas as pd

In [2]: from sklearn import neighbors

In [3]: from sklearn_pandas import DataFrameMapper, cross_val_score

In [4]: path_train ="train.csv"

In [5]: path_labels ="trainLabels.csv"

In [6]: path_test = "test.csv"

In [7]: train = pd.read_csv(path_train, header=None)

In [8]: labels = pd.read_csv(path_labels, header=None)

In [9]: test = pd.read_csv(path_test, header=None)

In [10]: mapper_train = DataFrameMapper([(list(train.columns),neighbors.KNeighborsClassifier(n_neighbors=3))])

In [13]: %pdb on

In [14]: mapper_train.fit_transform(train, labels)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-e3897d6db1b5> in <module>()
----> 1 mapper_train.fit_transform(train, labels)

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn/base.pyc in fit_transform(self, X, y, **fit_params)
    409         else:
    410             # fit method of arity 2 (supervised transformation)
--> 411             return self.fit(X, y, **fit_params).transform(X)
    412
    413

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn_pandas/__init__.pyc in fit(self, X, y)
    116         for columns, transformer in self.features:
    117             if transformer is not None:
--> 118                 transformer.fit(self._get_col_subset(X, columns))
    119         return self
    120

TypeError: fit() takes exactly 3 arguments (2 given)
> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sklearn_pandas/__init__.py(118)fit()
    117             if transformer is not None:
--> 118                 transformer.fit(self._get_col_subset(X, columns))
    119         return self

ipdb> l
    113
    114         X       the data to fit
    115         '''
    116         for columns, transformer in self.features:
    117             if transformer is not None:
--> 118                 transformer.fit(self._get_col_subset(X, columns))
    119         return self
    120
    121
    122     def transform(self, X):
    123         '''
ipdb> transformer.fit(self._get_col_subset(X, columns), y[0])
KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
           n_neighbors=3, p=2, weights='uniform')


I will take as reference fairseq's implementation of the Transformer model. With this assumption:

In the transformer, masks are used for two purposes:

Padding: in the multi-head attention, the padding tokens are explicitly ignored by masking them. This corresponds to parameter key_padding_mask.
Self-attention causality: in the multi-head attention blocks used in the decoder, this mask is used to force predictions to only attend to the tokens at previous positions, so that the model can be used autoregressively at inference time. This corresponds to parameter attn_mask.

The weight mask, which is the combination of the padding and causal masks, is used to know which positions to fill with $-\infty$ before computing the softmax, which will be zero after it.
You don't need to preserve any zeros in the output, as the attention blocks take care of that (see answer (1)). In the original Transformer article, the attention works without bias, but the bias does not change performance. Actually, in fairseq the bias are used by default.
Yes, padding_idx is certainly used to zero out padded tokens.


How to extract the positions of employee from raw text

Have a look at this to see how a Knowledge Base can be built. I would say the best way is to build your own Knowledge Base based on your corpus if you have enough data.

The idea (simplified) for starting is to have patterns such as "NAME is POSITION" by seeing some of data. Through this you find many names and positions. Then you extract new patterns from them and scan whole data again. Do it until no new pattern shows up. Then you end up with lots of is-a relations which is what you need.

For extracting positions you may need sentence segmentation. Have a look at slide 56 onward to get an overview.

PS: If you do not have enough data, you may crawl web and get texts and try to make your lexical databases. Target websites with more probability of having such a context.

Hope it helps. Good Luck!


Is it cheating to stratified sample the whole dataset based on a previous evaluation result?

By having seen the results on the held-out test set, learning something and applying that to the model, you have essentially added some bias to your model. So if we are being strict, yes... You cheated a little bit  ;)

Without having a solid (meaning unbiased) validation metric, you cannot objectively compare the two results. You used prior information to make an improved modelling decision, which is great :)   However, that information flowed directly from your test data, which introduced the bias.

It can be difficult when you have limited data, I know. You could maybe look into data augmentation to synthetically increase your dataset. In your case, adding random rotations and mirrored images could help a great deal!


This is a multi-label classification task. Several labels can be assigned to each text. You can do this task using LSTM and sigmoid functions. The sigmoid functions are in the output layers and number of them is equal to number of your tags.


What does the pair of vertical lines in empirical entropy formula mean?

$D$ is a set which is further divided into sets $D_{1}$ and $D_{2}$. In this context the vertical lines represent the cardinality of the set i.e. the number of elements in the set. Hence, $\alpha$ represents the ratio of number of samples of $D_{1}$ to $D$.


What is the input space of a neural network (or other supervised learning algorithms)?

I guess it depends on the algorithm, but linear models, as well as neural networks, treat all variables as continuous. The algorithm will not explode or anything if you supply 0.2 at prediction stage. However, your algorithm is trained on data. The algorithm can at best do what it has learnt from the training data. For this reason, do not expect anything meaningful when you feed an example with a value that has not been seen in the whole training set, or that it does not follow the training set distribution.


How do I handle with my Keras CNN overfitting

as your data is very less, you should go for transfer learning as @muneeb already suggested, because that will already come with most learned parameters and then you can train that model using your custom dataset.

you can try out pre-trained models from here

If you want to go for your existing custom configured model only, try adding another Dense layer before the output.

like a Dense(128, activation='relu').

Share your results here, so that we can explore more.


Machine Learning Best Practices for Big Dataset

I'll list some practices I've found useful, hope this helps:

Irrespective of whether the data is huge or not, cross validation is a must when building any model. If this takes more time than an end consumer is willing to wait, you may need to reset their expectations, or get faster hardware/software to build the model; but do not skip cross validation. Plotting learning curves and cross-validation are effective steps to help guide us so we recognize and correct mistakes earlier in the process. I've experienced instances when a simple train-test set does not reveal any problems until I run cross-fold validations and find a large variance in the performance of the algorithm on different folds.
Before sizing up a dataset, eliminate the records with missing values of key variables and outliers, columns of highly correlated variables, and near zero variance variables. This will give you a much better estimate of the real usable dataset. Sometimes you may end up with only a fraction of the available dataset that can actually be used to build a model.
When sizing up a dataset for building a model, it is easier to estimate the computing resources if you enumerate the dataset in rows and columns and memory size of the final numeric matrix. Since every machine learning algorithm is ultimately going to convert the dataset into a numeric matrix, enumerating the dataset size in terms of GBs/TBs of raw input data (which may be mostly strings/textual nominal variables/etc.) is often misleading and the dataset may appear to be more daunting and gigantic to work with than it is.
Once you know (or estimate) the final usable size of your dataset, check if you have a suitable machine to be able to load that into memory and train the model. If your dataset size is smaller than memory available/usable by the software, then you need not worry about the size any longer.
If the dataset size is larger than the memory available to train a model, then you could try these approaches (starting from the simplest ones first):

Use a machine with more memory: If you're using a cloud service provider then the simplest approach could be just to provision more memory and continue building the model as usual. For physical machines, try to procure additional RAM, its price continues to reduce and if your dataset is going to remain this big or grow bigger over time, then it is a good investment.
Add nodes to the cluster: For Hadoop and Spark based cluster computing deployments, training on a larger data-set is as easy as adding more machines to the cluster.
Quite often classification tasks require training on data with highly imbalanced classes, the ratio of positive to negative classes could sometimes be as large as 1:1000 or more. A straightforward method to improve accuracy in these cases is to either over-sample the minority class or under-sample the majority class, or do both together. If you have a large dataset, under-sampling the majority class is a very good option which will improve your algorithm's accuracy as well as reduce training time.
Build an ensemble: Split the dataset randomly and train several base learners on each part, then combine these to get the final prediction. This would most effectively make use of the large dataset and produce a more accurate model. But you need to spend more time to carefully build the ensemble and keep clear of the usual pitfalls of ensemble building.
If you're using an ensemble, train many single-thread models in parallel. Almost all ML software provide features to train multiple models on different cores or separate nodes altogether.
Evaluate multiple different algorithms on the time taken to train them for your specific dataset vs. their accuracy. While there is no universal answer, but I've found when using noisy data, SVMs take much longer time to train than carefully built ensemble of regularized regression models, but may be only slightly more accurate in performance; and a well built neural network may take a very long time to train as compared to a CART tree, but perform significantly more accurately that the tree.
To reduce time taken to build the model, try to automate as much of the process as you can. A few hours spent automating a complex error-prone manual task may save your team a hundred hours later in the project.
If available, use those algorithm implementations which use parallel processing, sparse matrices and cache aware computing, these reduce processing time significantly. For example, use xgboost instead of a single-core implementation of GBM.
If nothing else works, train the model on a smaller dataset; as Emre has suggested in his answer, use learning curves to fix the smallest sample size required for training the model, adding more training records than this size does not improve model accuracy noticeably. Here is a good article which explores this situation - largetrain.pdf


Ordinal features to decision tree in Python

Ordinal variables are treated exactly the same as numerical variables by decision trees.  (And so, you might as well encode them as consecutive integers.)

As for (unordered) categorical variables, LightGBM (and maybe H2O's GBM?) supports the optimal rpart-style splits [using the response-ordering trick when suitable, else trying all splits when not too expensive].  If you want a single decision tree, just set hyperparameters accordingly.

See also:
Why decision tree needs categorical variable to be encoded?
Ordinal Attributes in a Decision Tree


WEKA Multilayer Perceptron with large dataset

That is a lot of attributes.  Moreover, WEKA's default MLP size of the hidden layer is "a", where the following preset sizes are given:

a = (features + classes) / 2
i = features
o = classes
t = features + classes.

It is expected that the MLP will take longer.  Some things you can try:

use dimensionality reduction such as PCA to reduce the number of features
reduce the number of folds if you are using k-fold cross validation
reduce hidden layer size (which will probably negatively affect accuracy)
use another model such as a SVM, which would be a lot faster

I recommend going through this high level introduction to neural networks.  It is important to gain some intuition about the models you are using and the options you have available before tackling your problem.


What is the difference between Gradient Descent and Stochastic Gradient Descent?

For a quick simple explanation:
In both gradient descent (GD) and stochastic gradient descent (SGD), you update a set of parameters in an iterative manner to minimize an error function.
While in GD, you have to run through ALL the samples in your training set to do a single update for a parameter in a particular iteration, in SGD, on the other hand, you use ONLY ONE or SUBSET of training sample from your training set to do the update for a parameter in a particular iteration. If you use SUBSET, it is called Minibatch Stochastic gradient Descent.
Thus, if the number of training samples are large, in fact very large, then using gradient descent may take too long because in every iteration when you are updating the values of the parameters, you are running through the complete training set. On the other hand, using SGD will be faster because you use only one training sample and it starts improving itself right away from the first sample.
SGD often converges much faster compared to GD but the error function is not as well minimized as in the case of GD. Often in most cases, the close approximation that you get in SGD for the parameter values are enough because they reach the optimal values and keep oscillating there.
If you need an example of this with a practical case, check Andrew NG's notes here where he clearly shows you the steps involved in both the cases. cs229-notes
Source: Quora Thread


How to place XGBoost in a full stack for ML?

Yes, it is a full-strength Machine Learning paradigm.

XGBoost is basically Extreme Gradient Boosting.

It only takes in numeric matrix data. So, you might want to convert your data such that it is compatible with XGBoost.

The wide range of parameters of the xgboost paradigm is what makes it so diverse. Boosting can be done on trees and linear models, and then more parameters can be defined depending on the model you have selected.

So, yes it is a complete paradigm in itself. But, when you want more than the limitations of xgboost like linear and tree models, then you can use the concept of ensembling.

In the case of ensembles, the tools/libraries which can be used depends on the data scientist who is conducting the experiment. It can be Keras or Theano or TensorFlow, or anything which he/she is comfortable with. (opinion-based)


Why neural networks do not perform well on structured data?

... someone pointed out that neural networks do not work very well with the structured data (data in tabular format) as compared to the unstructured data (like representing each pixel in an image).

It's difficult to propose a universal analogy, but perhaps one moderately complex example which is easy to understand will suffice.

In the link you provided user JkBk mentions the "no free lunch theorem", let's add to that the "infinite monkey theorem". The fact is you can have one or more free lunches, just not as many as you like. Similarly you can not have infinite monkeys, but you can have a lot of them. Limits on each.

Take for example this data from the World Health Organization, let's only look at age versus height for boys aged 5 to 19. Let's look at a subset of the data in tabular form, I presume you understand the concept of "height-for-age".

Process this chart with your brain, what does it tell you. It tells you exactly everything averaged for a subset of the data. Can you draw any inferences from it, how about another several tables of data?

A neural network would have to process the data in all the tables and come up with a means to say "this age = this height" and "this height = this age" as an average.

Here is one of several tables:

Now let's look at the whole dataset in a chart:

See how at age 13 there is the start of a growth spurt, that continues almost to the age 15 when it starts to slow down. Between 15 and 17 it starts to level off, and from 17 to 19 there's almost no growth at all. From 5 to 11 a simple algorithm can easily solve the equation in either direction, beyond 11 the algorithm is not so simply but it still can be solved with a polynomial.

For a neural network once it gets on track and thinks it is coming up with a solution a monkey wrench gets thrown in to the works. Look at the "Alice and Bob example" (from 'no free lunch'), trying to solve the problem rationally requires knowledge and intelligence to avoid getting trapped.

Alternatively an Equation Solver can intelligently brute-force a solution, providing a relatively simple equation that can be used efficiently on large volumes of data.

Once you have a simple equation you could apply it to a huge dataset, like the Census, to check for errors such as age not matching height. A neural network would be better for discovering that age versus height was different than the average in a particular area and supply the reason (distance to inexpensive high quality food, versus earnings, and a sprinkle of education) whereas a simple polynomial wouldn't have the additional knowledge buried within it to make deductions.

A neural network doesn't comprehend (literally or figuratively) why a solution that was working later fails; it needs to continue, to learn to ignore it's success, to reach a working solution.

A non neural network solution doesn't care (literally or figuratively) whether it was on track, figured something out and learned to apply it, it only knows to minimize error and eliminate variables.

That doesn't mean that each couldn't switch roles and learn, or be programmed, to do the job of the other; but then the neural network would be pedantic about precision throughout (slow) and the solver would over analyze (calculate left or right with high precision) rather than simply weigh the outcomes with adequate precision for each step.


I understand CART as a Classification and Regression Tree, as just a Decision tree. Decision Trees are greedy and deterministic.

Random Forest is an ensemble of decision trees, so you will be using random trees ensembling to select features.

I am guessing, your question is not really descriptive about the situation, that since the decision tree is not pruned and uses all data it gives a good solution there is no need to take care of overfitting or pruning They normally achieve better in the train than Random Forest because they overfit easily.

Taking out features will just give the tree less information, so less power.

  Does it mean that CART needs a large number of features to produce a good model?

Decision trees do good with a small number of features, it is not related.

Just to check, are you evaluating in the test set right?


activation functions in multiple layers in CNNs

Not only for Convolutional Neural Networks (CNNs), also for DNNs (Deep Neural Networks) and RNNs (Recurrent Neural Networks), we use activation functions at every layer. Sigmoid (for binary classification), softmax (for multiclass classification) or some other types are usually used at the final output layer, each specific for the kind of labels that we have to compare with the predictions.

However, other neurons require activation functions as well, especially for nonlinearity purposes; most popular ones are ReLUs(Rectified Linear Units), Leaky ReLU, tanh,.. and so on. We nearly always use an activation function for every neuron in Deep Learning. For a detailed insight, have a look at:

https://www.youtube.com/watch?v=Xvg00QnyaIY

Also specificly for Convolutional Nets:

https://www.youtube.com/watch?v=jPOAS7uCODQ&list=PLkDaE6sCZn6Gl29AoE31iwdVwSG-KnDzF&index=7

Hope I could help, please do not hesitate to ask more. Good Luck!


Log analysis dataset with labeled cybersecurity issues

In reference with your little found data either augment it or apply cross validation on top of it.
else
Look for your expected data in https://datasetsearch.research.google.com/


How does the meta Random Forest Classifier determine the final classification?

Each data point $x$ that you are predicting necessarily only lands in one leaf node for each tree. So if I have N trees, and $K_A$ is the number of trees in which $x$ ends up in a leaf node which predicts A, then we can say $x$ is class $A$ with "probability" $K_A/N$, and similarly for other classes. Then you can just make an argmax prediction for which class $x$ belongs to based on the "probabilities" (I keep putting this in quotes because it isn't an actual probability, but it's sometimes ok to think of it this way.)


How to build a recurrent neural net in Keras where each input goes through a layer first?

What should the input look like?

You are right to think a 2D tensor, but usually we add one more dimension for the batch. You can indeed have a variable length number_of_x_inputs, but to train during batch processing all inputs in a single batch will need to have the same shape. (Setting batch size to 1 will get around this.) During inference, you can have whatever length you want. See below code sample.

How do I pass each input vector through the same transformation before feeding it to the RNN layer?

Use TimeDistributed. The example below passes all vectors $x_i$ through the same feed forward network (Dense(5, ...)), but you should be able to swap that out for whatever f you were thinking.

from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed

x_dimension = 16
num_classes = 2

model = Sequential()

model.add(TimeDistributed(Dense(5, activation='relu'),
          input_shape=(None, x_dimension)))
model.add(LSTM(32, return_sequences=True))
model.add(LSTM(8))
model.add(Dense(num_classes, activation='softmax'))

print(model.summary(90))

This prints the following model:

Layer (type)                            Output Shape                        Param #
==========================================================================================
time_distributed_1 (TimeDistributed)    (None, None, 5)                     85
__________________________________________________________________________________________
lstm_1 (LSTM)                           (None, None, 32)                    4864
__________________________________________________________________________________________
lstm_2 (LSTM)                           (None, 8)                           1312
__________________________________________________________________________________________
dense_2 (Dense)                         (None, 2)                           18
==========================================================================================
Total params: 6,279
Trainable params: 6,279
Non-trainable params: 0
__________________________________________________________________________________________


I gave some explanations here that may help.
The radius $r_{ij}$ is the radius of the neighborhood of the winner. Only those neurons that are closer than $r_{ij}$ to the winner are allowed to update.
The formulas that you show here say that this radius is not constant but decreases with each iteration. Initially you take a large radius, and then make it smaller and smaller with each iteration.
Since you probably initialize your weights randomly, you might want to change many of them in the beginning of training but you might want to disturb less and less weights as they become more trained. That's why you decrease the radius.
This formula is just a particular policy of change of the radius with each iteration. In this case it is exponential. You can choose another policy or keep the radius constant or decrease it slightly (not exponentially) which may be reasonable if you use some domain knowledge when initializing the weights. Also, you can keep the learning rate constant if you like. The radius of the neighborhood and the learning rate are hyper-parameters. It is up to you how to choose them.


Models systematically underestimate values on the test set, why?

Welcome to data science SO. It seems your model has a bias towards a lower estimate of the true values. This can happen for a variety of reasons. Say you are trying to fit a straight line on data that was generated by a second degree polynomial. The straight line fit will have a bias. Typically bias happens because the model you are trying to build is insufficient able to capture the dynamics of the data you are trying to fit it on. I understand this is very generic but without more info about what model you used or any plot of the test set it is hard to help you more.


Which ML method for multiclass (non-binary) text classification should I choose (from SparkML)?

If you want to categorise your text using machine learning techniques, you have to get fixed length features from text to train any ML model. You can do
That using bag of words, tf-idf, averaging word vectors.
If you are using any deep learning based models, you can use LSTM with word vectors or CNN’s.


The cross-entropy error function in neural networks

One way to interpret cross-entropy is to see it as a (minus) log-likelihood for the data $y_i'$, under a model $y_i$.

Namely, suppose that you have some fixed model (a.k.a. "hypothesis"), which predicts for $n$ classes $\{1,2,\dots, n\}$ their hypothetical occurrence probabilities $y_1, y_2,\dots, y_n$. Suppose that you now observe (in reality) $k_1$ instances of class $1$, $k_2$ instances of class $2$, $k_n$ instances of class $n$, etc. According to your model the likelihood of this happening is:
$$
P[data|model] := y_1^{k_1}y_2^{k_2}\dots y_n^{k_n}.
$$
Taking the logarithm and changing the sign:
$$
-\log P[data|model] = -k_1\log y_1 -k_2\log y_2 - \dots -k_n\log y_n = -\sum_i k_i \log y_i
$$
If you now divide the right-hand sum by the number of observations $N = k_1+k_2+\dots+k_n$, and denote the empirical probabilities as $y_i'=k_i/N$, you'll get the cross-entropy:
$$
-\frac{1}{N} \log P[data|model] = -\frac{1}{N}\sum_i k_i \log y_i = -\sum_i y_i'\log y_i =: H(y', y)
$$

Furthermore, the log-likelihood of a dataset given a model can be interpreted as a measure of "encoding length" - the number of bits you expect to spend to encode this information if your encoding scheme would be based on your hypothesis.

This follows from the observation that an independent event with probability $y_i$ requires at least $-\log_2 y_i$ bits to encode it (assuming efficient coding), and consequently the expression
$$-\sum_i y_i'\log_2 y_i,$$
is literally the expected length of the encoding, where the encoding lengths for the events are computed using the "hypothesized" distribution, while the expectation is taken over the actual one.

Finally, instead of saying "measure of expected encoding length" I really like to use the informal term "measure of surprise". If you need a lot of bits to encode an expected event from a distribution, the distribution is "really surprising" for you.

With those intuitions in mind, the answers to your questions can be seen as follows:

Question 1. Yes. It is a problem whenever the corresponding $y_i'$ is nonzero at the same time. It corresponds to the situation where your model believes that some class has zero probability of occurrence, and yet the class pops up in reality. As a result, the "surprise" of your model is infinitely great: your model did not account for that event and now needs infinitely many bits to encode it. That is why you get infinity as your cross-entropy.

To avoid this problem you need to make sure that your model does not make rash assumptions about something being impossible while it can happen. In reality, people tend to use sigmoid or "softmax" functions as their hypothesis models, which are conservative enough to leave at least some chance for every option.

If you use some other hypothesis model, it is up to you to regularize (aka "smooth") it so that it would not hypothesize zeros where it should not.
Question 2. In this formula, one usually assumes $y_i'$ to be either $0$ or $1$, while $y_i$ is the model's probability hypothesis for the corresponding input. If you look closely, you will see that it is simply a $-\log P[data|model]$ for binary data, an equivalent of the second equation in this answer.

Hence, strictly speaking, although it is still a log-likelihood, this is not syntactically equivalent to cross-entropy. What some people mean when referring to such an expression as cross-entropy is that it is, in fact, a sum over binary cross-entropies for individual points in the dataset:
$$
\sum_i H(y_i', y_i),
$$
where $y_i'$ and $y_i$ have to be interpreted as the corresponding binary distributions $(y_i', 1-y_i')$ and $(y_i, 1-y_i)$.


What exactly is a trained model for machine-learning?

A machine learning model is an algorithm which learns features from the given data to produce labels which may be continuous or categorical ( regression and classification respectively ). In other words, it tries to relate the given data with its labels, just as the human brain does.

Mathematical functions are used to map the features ( produced as a result of feature extraction ) to their labels.

  Models trained on a high amount of data can generalise themselves
  better. Generalisation is the ability of a model to give generalised
  predictions across varied or diverse data. It is not biased towards
  the data on which it was originally trained.

Production of models has these basic steps involved:

Collection of suitable data.
Preprocessing of data for training the model.
Training the model.
Evaluating the model.
Hosting the model for production.

Deploying a model:

This could be done in many ways. You can serve a model or run it on an IoT device. You basically want to freeze all the trainable parameters so that they are constants. The model's learning capabilities are removed so that it could only make predictions. There are some models which exhibit Online Learning.

Different frameworks like TensorFlow, Keras, PyTorch etc. have their own methods of saving models. Like in Keras, we can save a NN model as a hd5f serialized file. With TensorFlow Lite, we can run a model on an IoT device.

Firebase MLKit hosts the model in the cloud created with TensorFlow.

  The best deployment platform is determined by its usability,
  scalability and developer friendliness.


what is the difference between "fully developed decision trees" and "shallow decision trees"?

[Later edit - Rephrase everything]

Types of trees

A shallow tree is a small tree (most of the cases it has a small depth). A full grown tree is a big tree (most of the cases it has a large depth).

Suppose you have a training set of data which looks like a non-linear structure.

Bias variance decomposition as a way to see the learning error

Considering bias variance decomposition we know that the learning error has 3 components:

$$Err = \text{Bias}^2 + \text{Var} + \epsilon$$

Bias is the error produced by the fit model when it is not capable to represent the true function; it is in general associated with underfitting.

Var is the error produced by the fit model due to sampled data, it describes how unstable a model is if the training data changes; it is in general associated with overfitting.

$\epsilon$ is the irreducible error which envelops the true function; this can't be learned

Considering our shallow tree we can say that the model has low variance since changing the sample does not change too much the model. It needs too many changed data points to be considered unstable. At the same time we can say that has a high bias, since it really can't represent the sine function which is the true model. We can say also that it has a low complexity. It can be described by 3 constants and 3 regions.

Consequently, the full grown tree has low bias. It is very complex since it can be described only using many regions and many constants on those regions. This is why it has low bias. The complexity of the model impacts also variance which is high. In some regions at least, a single point sampled differently can change the shape of the fitted tree.

As a general rule of thumb when a model has low bias it has also high variance and when it has low variance it has high bias. This is not true always, but it happens very often. And intuitively is a correct idea. The reason is that when you are getting close to the points in the sample you learn the patterns but also learn the errors from sample, when you are far away from sample you are instead very stable since you do not incorporate errors from sample.

How can we build ensembles using those kind of trees?

Bagging

The statistical model behind bagging is based on bootstrapping, which is a statistical procedure to evaluate the error of a statistic. Assuming you have a sample and you want to evaluate the error of a statistical estimation, bootstrapping procedure allows you to approximate the distribution of the estimator. But trees are only a simple function of sample "split the space into regions and predict with the average, a statistic". Thus if one builds multiple trees from bootstrap samples and averages the trees can be considered i.i.d. and the same principle works to reduce variance. Because of that bagging allows one to reduce the variance without affecting too much the bias. Why it needs full depth trees? Well it is a simple reason, perhaps not so obvious at first sight. It need classifiers with high variance in order to reduce it. This procedure does not affect the bias. If the underlying models have low variance high bias, the bagging will reduce slightly the already small variance and produce nothing for bias.

Boosting

How does boosting? Many compare boosting with model averaging, but the comparison is flawed. The idea of boosting is that the ensemble is an iterative procedure. It is true that the final rule for classification looks like a weighted average of some weak models, but the point is that those models were built iteratively. Has nothing to do with bagging and how it works. Any $k$ tree is build using information learned from all previous $k-1$ trees. So we have initially a weak classifier which we fit to data, where all the points have the same importance. This importance can be changed by weights like in adaboost or by residuals like in gradient boosting, it really does not matter. The next weak classifier will not treat in the same way all the points, but those previously classified correctly has smaller importance than those classifier incorrectly. The consequence is that the model enriches it's complexity, it's ability to reproduce more complex surfaces. This is translated in the fact that it reduces the bias, since it can go closer to data. A similar intuition is behind: if the classifier has already a low bias, what will happen when I boost it? Probably a much unbearable overfit, that's all.

Which one is better?

There is no clear winner. It depends too much on data set and on other parameters. For example bagging can't hurt. It is possible to be useless ut usually does not hurt performance. Boosting can lead to overfit. That is because you can go eventually too close to data.

The is a lot of literature which says that when the irreducible error is high, the bagging is much better and boosting does not progress too much.

Can we decrease both variance and bias?

Pure bootstrap and bagging approaches serves a single purpose. Either reduce variance either reduce bias. However modern implementations changed various things in how those approaches works. Sampling can be used in boosting and it seems to work towards reducing also the variance. There are bagging procedures which takes some ideas prom boosting, for example iterative bagging (or adaptive bagging) published by Breiman. So, the answer is yes, is possible.

Can we use other learners other than trees?

Of course. Often you will see boosting to use this approach. I read some papers on bagging svm or another learners also. I tried myself to bag some svms but without much success. The trees are the preferred way, however, for a very simple reason, they are simple to build, simple to adapt and simple to control their effect. My personal opinion is that it is not everything said regarding ensembles of trees.

PS: a last note on the number of weak classifiers: this depends entirely on the complexity of the data set mixed with the complexity of the learner. There is no recipe. Often 20 of them are enough to get most of the information and additional ones are only for tiny tuning.


Why Does XGBoost Keep One Feature at High Importance?

According to the docs "gain" is the default, which is calculated by this formula, where I don't think intuition would help. General consensus is, that feature importance is a tricky concept, as long as your model performs fine, you shouldn't worry about it. Especially with the linked formula for gain you can see that the first split may be the most important by far, depending on gamma, so that every other feature importance is low by default.
If, on the other hand you chose "weight", that would be just be the number of times the feature is chosen to split. That may mean that one feature contains a lot of unique values and can be chosen a lot of times and still improve accuracy.
All of this depends heavily on the data and parameters you are using. There isn't any general answer to this pattern.


If you have 4,500 examples of each category, you’re doing better than random guessing. This sounds like a hard problem where the examples will have only subtle differences, so that’s an accomplishment. (I assume you’re doing some kind of out-of-sample testing.) Consider how you do in classifying perhaps 20 examples of each. I would be curious to hear how you do. This is quite a different problem if you get 38/40 than 20/40.

The way to apply a CNN, though, would be to convert to a spectrogram and then run the CNN on the spectrogram array. Converting to a literal picture is unnecessary and might harm performance. There are many ways to convert your signal to two dimensions of time-frequency space. You’ve probably tried a Fourier transform. Check out wavelets.

Since you have time series data, consider recurrent neural nets and long short-term memory. These can be combined with CNN. You will find examples of CNN/LSTM code on GitHub, I’d imagine.

You have a fairly small data set, however, for deep learning approaches. Consider simpler models like logistic regression. Your in-sample performance may suffer while improving out-of-sample performance.

Finally, consider a proper scoring rule like Brier score. Frank Harrell has written much about this on the statistics Stack, Cross Validated. Shamelessly, I will mention that you may be interested in a question of mine on CV where I somewhat challenge the idea of proper scoring rules when there is a hard decision to be made: https://stats.stackexchange.com/questions/464636/proper-scoring-rule-when-there-is-a-decision-to-make-e-g-spam-vs-ham-email. My example comes from NLP, but there could be a speech situation (say text dictation, where, at some point, you have to decide to print “dad” or “bad” or “bicycle”).


How to count occurrences of values within specific range by row

You can apply a function to each row of the DataFrame with apply method. In the applied function, you can first transform the row into a boolean array using between method or with standard relational operators, and then count the True values of the boolean array with sum method.

import pandas as pd

df = pd.DataFrame({
    'id0': [1.71, 1.72, 1.72, 1.23, 1.71],
    'id1': [6.99, 6.78, 6.01, 8.78, 6.43],
    'id2': [3.11, 3.11, 4.99, 0.11, 2.88]})

def count_values_in_range(series, range_min, range_max):

    # "between" returns a boolean Series equivalent to left <= series <= right.
    # NA values will be treated as False.
    return series.between(left=range_min, right=range_max).sum()

    # Alternative approach:
    # return ((range_min <= series) & (series <= range_max)).sum()

range_min, range_max = 1.72, 6.43

df["n_values_in_range"] = df.apply(
    func=lambda row: count_values_in_range(row, range_min, range_max), axis=1)

print(df)

Resulting DataFrame:

    id0   id1   id2  n_values_in_range
0  1.71  6.99  3.11                  1
1  1.72  6.78  3.11                  2
2  1.72  6.01  4.99                  3
3  1.23  8.78  0.11                  0
4  1.71  6.43  2.88                  2


Who is actually sharing physical RAM in a distributed sytem that has virtual shared memory? (Server and/or clients.)

Either the host server or the clients can create DSM, or both.  It's likely that a server hosting the applications would manage this.  The host server may create shared memory, or it may direct the clients to do this, and share the memory among themselves, preserving its own memory.

Distributed Shared Memory is an abstraction.  A "server" creates shared memory, and a "client" makes a copy of it in its own memory.  The O/S or middleware keeps them in sync. A "client" for purposes of working of host server, may be a "server" regarding creating shared memory.  This client/server terminology is used in both contexts. When you read about shared memory you will often see the terminology used this way - the computer initiating the creation of DSM (distributed shared memory) is the "server" for that purpose.


Keras + Tensorflow CNN with multiple image inputs

Yes it is wrong, each (68, 59, 59) input should go through one model not an array of them.

You can treat each of 68 images as a channel, for this, you need to squeeze your data axes from (-1, 68, 59, 59, 1) to (-1, 68, 59, 59) to have a 59x59 image with 68 channels corresponding to Input((68, 59, 59)), and set data_format='channels_first' in conv2D, to let the layer know that channels are in the first dimension (it expects them to be in the last dimension by default). This is similar to an RGB image that has 3 channels corresponding to Input((59, 59, 3)). The rest is the same.
If 68 images are consecutive frames from a movie, you can use conv3D to extract motion patterns across neighbor frames too; this is done by 3D kernels instead of 2D kernels. It requires  (-1, 68, 59, 59, 1) data shape corresponding to Input((68, 59, 59, 1)). Also, we should use the default data_format='channels_last' since now there is only one channel as the last dimension. Commonly, temporal axis is placed third, i.e. (-1, 59, 59, 68, 1), which can be accomplished by moving the axes.


Creating an easy but not trivial dataset

You might use a Markov chain for generating a simple enough dataset. You can estimate the probabilities of punctuation by scanning a large corpus of text (e.g. wikipedia), then you can use the model to generare fictious sentences with punctuation marks. You can set the order of the Markov chain so as to make the generated dataset increasingly closer to real-world text. The Markov chain should be defined on the set of words, however, so you have to carefully choose the vocabulary.


While Yolov3 is a great model it may underperform for certain datasets. A simple example would be this. Take a model and train it on cifar 10. The same model might significantly perform lower on cifar 100 or imagenet!
Things you could do to improve your model:

Introduce Batch or group normalization?
Study the data better, if possible see if the data is used elsewhere and how they approach it.
Epochs isn't the problem, so that should stay the same.

If nothing continues to work then you might have to opt for some other model. Try yolov5 or rcnn!


Tensorflow F-RCNN first stage input and output strides

Let me try to explain what exactly stride means generally, then you'll be able to address your specific problem.
Let's say we have an images of size 7x7. Let's take a kernel of size 3x3. When you slide the kernel over image with :
1. stride=1

2. stride=2

Essentially stride means how much gap you should leave between two kernel position while applying convolution operation.
Now this concept is generic and applies to any operation where we need to slide a kernel over an input.

For Faster-RCNN model, features and anchors are two different things, so probably they're unrelated. Confirm this with official paper though.
For stride size, there is no restriction, you can take it whatever number you want, just make sure it should be less than or equal to half of the size of image(you can think why logically).


Generating random numbers from best probability distributions?

Good question. Lots of options.

Most would recommend you use CDFs instead of histograms. So convert your observed distribution to empirical CDF (in R it’s ecdf() - I dint know python). Then, plot another line that is the theoretical CDF using your best-fit parameters over the span of a to b, where a to b gives you at least the same coverage as your original data set limits.

Alternatively you could go for a chi-square option that’ll give you observed vs predicted and can be plotted as histogram but I don’t find that better in most cases, though it might make quantitative comparison more accessible


Multi-dimensional regression with ensemble models?

I think there are some algorithms in scikit-learn that support this kind of multi-output with correlated inputs/outputs (e.g. not using one learner per output, this can be achieved using the MultiOutputRegressor meta-estimator): RandomForest, DecisionTree and LinearRegression.
See : https://stats.stackexchange.com/questions/153853/regression-with-scikit-learn-with-multiple-outputs-svr-or-gbm-possible

and

http://scikit-learn.org/stable/auto_examples/ensemble/plot_random_forest_regression_multioutput.html#sphx-glr-auto-examples-ensemble-plot-random-forest-regression-multioutput-py
for a comparison between the two approaches.


When exactly should I use weighted loss?

As with almost every problem in real life, regarding how to handle class imbalance there is no golden rule to follow for any case and systematically obtain good results, only some recipes that you can try.

When your class distribution is imbalanced (Train: class A:10%, B:90% and Test: class A:10%, B:90%)), one recipe that can improve your results is using reweighting to give more importance to the minority class. There are, however, other approaches, like oversampling the minority class, or subsampling the majority class. Another option would be to use a loss function that is tolerant to class imbalances.

Having a different distribution between the training and test sets (Train: class A:10%, B:90% and Test: class A:50%, B:50%) can also be handled with the approaches above. However, you should think about why you have such a difference: while the existence of class imbalance in the training and validation data can be caused by the distribution of the data in real life, the differences in class imbalance between training and test data is a consequence of the creation process of those datasets, either caused by a conscious decision or by chance. Here are two examples that could lead to this situation:

In this case, the 50%/50% ratio reflects the reality of the data that will be received in the production environment, so we decided to define a test set with that ratio and using the rest of the data for training. It happened that the rest of the data had this imbalance. In this case, you should mitigate the problem of the data imbalance in the training data, maybe with some techniques like reweighting or resampling.
In this case, the 90%/10% ratio reflects the reality of the data that will be received in the production environment. However, we created the test set and the data set separately, maybe with less quality control for the training data. This led the team in charge of the training data to make some decisions about the data collection and the team in charge of the test data to take different decisions. The differences in the class imbalance between training and test data and casual, not driven by any conscious decision. In this case, this is a problem in how you have created your datasets. What you have to do in those cases, is to re-split your data to have the same distribution in both sets, trying to have a test set that is as close as possible to the real data and, if possible, having the same distribution in the training and test sets.

In all these cases, nevertheless, we should take into account the function we use to measure how good our model is. If the real data has a strong class imbalance, we should not choose the accuracy but something like the AUC.


Does not using more filters in deeper CNN creates more images?

No, your understanding is not correct.
Each of the 64 filters of the second layer will be applied to each of the 32 channels from the output of the first layer, resulting in 64 channels in the output of the second layer.
When the input of a convolutional layer has multiple channels, the convolution filter itself has the same number of channels. In your example, if we are using $3\times3$ filters, each filter in the second layer will be a tensor of dimensions $3\times3\times32$. Therefore, the filter "covers" the full depth of the input. Then, you simply perform the element-wise multiplication of the filter with the overlapping region in the input and add all the resulting elements together. Applying just 1 filter, we obtain a result with 1 channel.
This way, the number of channels of the output of a convolutional layer is the same as the number of filters in the convolution.


How to evaluate the performance based on rate data

It depends on what you want to achieve, and how you'd define performance.

Suppose it is a running exercise program with 3 exercises in a week with increasing targets.

The first formula calculates the average exercise success, the second formula defines the weekly success, where the exercises are weighted with the goals.

There is no right or wrong, just interpretation of the numbers.

Just to confuse you a bit more: suppose:

Goal: 8, Achieved 1, Goal: 2, achieved 1
Goal: 2, Achieved 0, Goal: 8, achieved 3.

Case 1. beats Case  2. on both instances in formula 1, but loses in formula 2.


Getting mean and covariance matrix for multivariate normal from keras model

Given that the covariance matrix has to be positive definite, the cholesky decomposition is a good way to solve this problem.
So the output of the network will be the mean vector mu and the upper triangular part of the cholesky matrix (denoted T here). The diagonal of this matrix must be positive elements (the diagonal of the covariance matrix are standard deviations):
p = y_train.shape[1] # dimension of the covariance matrix
inputs = Input(shape=(6,))
layer1 = Dense(24, activation='relu')(inputs)
layer2 = Dense(12, activation='relu')(layer1)
mu = Dense(p, activation = "linear")(layer1)
T1 = Dense(p, activation="exponential")(layer1)# diagonal of T
T2 = Dense((p*(p-1)/2), activation="linear")(layer1)
outputs = Concatenate()([mu, T1, T2])

Now let's define the loss function. Firstly, let's define the function that will extract the outputs of the network:
def mu_sigma(output):
    mu = output[0][0:p]
    T1 = output[0][p:2*p]
    T2 = output[0][2*p:]
    ones = tf.ones((p,p), dtype=tf.float32)
    mask_a = tf.linalg.band_part(ones, 0, -1)
    mask_b = tf.linalg.band_part(ones, 0, 0)
    mask = tf.subtract(mask_a, mask_b)
    zero = tf.constant(0, dtype=tf.float32)
    non_zero = tf.not_equal(mask, zero)
    indices = tf.where(non_zero)
    T2 = tf.sparse.SparseTensor(indices,T2,dense_shape=tf.cast((p,p),
         dtype=tf.int64))
    T2 = tf.sparse.to_dense(T2)
    T1 = tf.linalg.diag(T1)
    sigma = T1 + T2
    return mu, sigma

Now for the loss function:
from tensorflow_probability import distributions as tfd
def gnll_loss(y, pred):
    mu, sigma = mu_sigma(pred)
    gm = tfd.MultivariateNormalTriL(loc=mu, scale_tril=sigma)
    log_likelihood = gm.log_prob(y)
    return - tf.math.reduce_sum(log_likelihood)


Same validation accuracy, different train accuracy for two neural networks models

Based on this, both models generalize equally well. However both have overfit, the second one more significantly. You simply want to avoid that. For example, I'd expect the best validation loss to be closer to training loss if you're using early stopping. You can turn up things like dropout.

If you do so, I think you'll find the augmented model ends up producing a better model (lower validation loss).


Logically speaking, I think "X is Green Circle" is a reasonable conclusion. I find the idea in the paper in your question is quite similar to this paper: KRNN: k Rare-class Nearest Neighbour Classification.

Intuitively, for example, if a new data point is close to one rare class' point and one common class' point, it is more likely to belong to the rare class.

There's no conflict here because a data point will be much more likely to be close to a common class' point. However, once it's already close to a rare class's data point, it's more likely to belong to this class.

That said, I didn't check your calculation, I just don't think the "X is Green Circle" conclusion is illogical for this algorithm.

[Update]

After reconsidering this problem, I think Weighted kNN want to put an emphasis on the rare class data point because that is the class of interest (e.g. Anomaly detection).

It's possible that Accuary is not the metrics here but a Weighted Accuracy metrics that penalize a misclassified rare class data point harder so that we can detect more rare class data point.


Do we need to tune same model differently for different window sizes in time series data classification?

In my opinion it should be tuned again.
Reasoning:
Using different window size is almost equivalent to using different features(as in a non time-series modeling). Tuning hyper-parameters is usually done after features were selected(I.E for a given set of features).


Predict the probability that a user is active on a website

I’ll throw out a couple of hints / clarifications.

Your ‘test’ set is of no use to you since you have the dependent variables missing, so just ignore it to begin with.
You will want to split you ‘training’ data into two sets to begin with, one for training a model and one for testing the model and computing the effectiveness of the model.
Hopefully you have lots of records in the training set because you have so many independent variables. Some models are very sensitive to low sample size and high dimensionality.
You will need to decide if the feature columns need to be encoded. You’ll have to pay particular attention to any categorical features because they need to be one hot encoded.
You will need to choose a model or ensemble of models. You may start by trying a binary classifier. You will need to creat one for each of your timeframes. Each model should take the ~890 features and produce one result with a 0 or 1.
You will need to figure out how to evaluate your models to understand how the choices you are making to tune each model are influencing the results.

This is not a comprehensive list, but hopefully it is helpful to get you started.


There will not be any pre-trained models to cluster these words.

In fact, in order to build your own clustering model you will need more metadata about each observation/word in your array.

At the moment any model would only be able to "see" the name of the package/software in your array. So the best you could hope for is a model that clusters these words based on their spellings.

Now let's pretend you find a brief description of each software, now you could do a bit more. With this longer text you could use supervised or unsupervised methods to cluster these softwares into groups (see topic models, k-means etc) based upon similar words in the descriptions.

Long story short, there is not a pre-built model to do this, and to build one yourself you're going to need more information about each observation.


How do linear learning systems classify datapoints that fall on the hyperplane

Linear, binary classifiers can choose either class (but consistently) when the datapoint which is to classify is on the hyperplane. It just depends on how you programmed it.

Also, it doesn't really matter. This is very unlikely to happen. In fact, if we had arbitrary precision computing and normal distributed features, there would be a probability of 0 (exactly, not rounded) that this would happen. We have IEEE 754 floats, so the probability is not 0, but still so small that there are much more important factors to worry about.


why the RFE to select features is changing when the os or computer is changed

Since you do not provide any information about the creation of X_train and y_train I cannot be sure that this is the issue, but I would guess that it has to do with some change in the order of the features. Three-based algorithms' results are affected when the order of the input features changes, so make sure that this order does not change every time you run your code (e.g. you can sort them alphabetically before RFE).

You might find this discussion interesting.


Suggestion on tag clustering visualization

Instead of clustering, association rules is likely to be much more appropriate on such data.

Nevertheless it's obviously possible to treat cooccurrences as a similarity measure and cluster this for example using a variant of hierarchical clustering (for similarities, not distances!) Or by transforming the similarities to distances. You can also try to treat these as affinities for tSNE or MDS, for example (again, your usual library will likely not just work, but you will need to write some code yourself), but it's probably much easier to simply visualize the cooccurence matrix as a "heat map", after arranging similar tags close to each other with hierarchical clustering.


How to give a 3D Tensor as input to LSTM

LSTM takes as input 3 dimension tensors (batch_size,time_step,input). So before adding a LSTM() layer you need to either use Flatten() or TimeDistributed(Flatten()) layer.


What is the typical accuracy of masked language models during BERT pretraining?

It is very hard to tell and researchers usually don't measure it because it's not a comparable number such as GLUE score. It depends on size of your vocabulary and also the masking strategy (for example newer BERT successors use span-masking). BERT is a subword language model (uses WordPiece tokenizer) so if you use large vocab then sentences are usually tokenized to "smaller" subword units that are easier to predict. I trained a BERT-like model for Czech language, which is morphologically more complex than English, and got top@1 test accuracy ~0.55, top@3 test accuracy ~0.68 after 1.5M steps with 30K vocab size.


Welcome to the site!

As you know KMeans is an Unsupervised learning and it helps you to find out if there are any patterns in the data. Yes, the procedure which you are following to find some commonality/patterns in the data. But this is not generally used for prediction. FYI you can use K-means for predicting too, recently I came across that but I don't know whether it would yield the desired results.

If the data is categorical you need to apply One-Hot Encoding, which converts the categorical data into numeric, you can go through the link for better understanding. If not so you cannot apply K-Means algorithm.

If you cannot convert the categorical data into numeric data then you can use this package ClustMixType this is in R and you can use KMODES in Python.

If you have your target variable ready, as you haven't mentioned the data target variable, if the target variable is Numeric you can use

Neural Network
Regression
SVM
Random Forest and many more

if the target variable is Binary(is he a criminal yes/no)

Neural Network
SVM
Logistic Regression
Random Forest
Naive Bayes Classifier
KNN and many more.

Please go through this Link, for better understanding on Mixed Data type Clustering

Let me know if you need any help.