Python: convert variables into correct format for DataFrame

First, you need to convert list of list into a list.
From link, you can convert a list of lists into a list by declaring the following function.
flatten = lambda t: [item for sublist in t for item in sublist]

Now all you need is to create dataframe using created lists.
data = {"Newspaper":flatten(newspaper), "Title": flatten(title), "PubDate": flatten(pubDate)}
pd.DataFrame.from_dict(data)


Idenitity between TD(0) algorithm and Policy Evaluation in Dynamic Programming when alpha is equal to 1

No - Dynamic programming estimates the value of the next state by first looking at all possible next states. Temporal difference 0 estimates the value of the next state by only looking at a single next state.


difference btw data distribution and frquency distribution

If you don't know the definition, how could you calculate anyway?

I looked at a few reliable definitions,

https://www.spss-tutorials.com/frequency-distribution-what-is-it/
https://www.statisticshowto.datasciencecentral.com/data-distribution/
http://makemeanalyst.com/observational-studies-and-experiments/population-distribution-sample-distribution-and-sampling-distribution/

The differences are subtle, and sometimes depend on who you ask.

What I conclude is that the frequency (or sample) distribution is statistics on an actual sample, counted per bin, maybe percentages added to the statistics.

The (population) data distribution is the distribution that you'd expect from the whole population.

For fair coin tosses, the data distribution would be 50/50, though a sample distribution of 10 could give 6/4.

My advice, either use the textbook definitions, or present the statistics that you see fit
for your analysis. Repeat the definition if necessary.

If you have a large enough random sample the frequency distribution becomes an estimate for the data distribution anyway (but sometimes you have to prove this to show your sample is random).

When you have $n$ features, you repeat this for all features.
E.g. when you have people's 'gender', 'married', 'smokes', 'employed', features, you have to repeat for all these features.


What is the difference between dcast and recast in R?

See ?reshape2::recast: The function

  conveniently wraps melting and (d)casting a data frame into a single
  step.

library(reshape2)
recast(df_sample, Countries+variable~Type, id.var=1:2)
#   Countries variable  a    b
# 1 Australia    X2014 30  -40
# 2 Australia    X2015 50  -10
# 3 Australia    X2016 10 -100
# 4 Australia    X2017  5  -10
# 5       USA    X2014 10  -20
# 6       USA    X2015 20  -40
# 7       USA    X2016 15  -10
# 8       USA    X2017  5   -5

So, it's just a shortcut for these two steps:

(tmp <- melt(df_sample, id.vars=1:2))
#    Countries Type variable value
# 1        USA    a    X2014    10
# 2        USA    b    X2014   -20
# 3  Australia    a    X2014    30
# 4  Australia    b    X2014   -40
# 5        USA    a    X2015    20
# ...
dcast(tmp, Countries+variable~Type)
#   Countries variable  a    b
# 1 Australia    X2014 30  -40
# 2 Australia    X2015 50  -10
# 3 Australia    X2016 10 -100
# 4 Australia    X2017  5  -10
# 5       USA    X2014 10  -20
# 6       USA    X2015 20  -40
# 7       USA    X2016 15  -10
# 8       USA    X2017  5   -5


Generate sentences from keywords by adding formal word

I am also fairly new to this, but am working on the same type of problem where I have to predict the sentence based on previous data. Like you want giving X value predict y value.
So, basically you need to create a dataset as per your requirement and once your dataset is ready you can train your model with encoder-decoder or LSTM layers.
Please find below links that may be helpful to you to fulfill your requirements. See as X and Y value input output prediction.

Gmail style smart compose
Neural Machine Translation with Attention
Sentence Prediction using LSTMs


NLP: Compare tags semantically with machine learning? (finding synonyms)

It's a complex problem, the standard method is to train a model which represents this semantic similarity function using a large text corpus (i.e. not only the tags themselves), by considering that the meaning of a word is provided by its context words.

A more direct approach is to use a pre-trained word embeddings vectors that can be compared directly to find their similarity.

A similar idea is to use WordNet, which is a database of words with their semantic similarity to each other.


An autoencoder is meant to do exactly what you are asking. It is a means to take an input feature vector with $m$ values, $X \in \mathbb{R}^m$ and compress it into a vector $z \in \mathbb{R}^n$ when $n < m$. To do this we will design a network that is compressed in the middle such that it looks this.

We train this network by comparing the output $X'$ to the input $X$. This will cause $X'$ to tend towards $X$, thus despite the feature compression in the network, the output will preserve sufficient information about the input such that the input $X$ can be recovered.

Once this network is trained, we can then truncate everything after the layer which outputs the vector $z$. Then you can use the feature vector $z$ as the input features to train a different neural network which you can use to classify your instances as normal or not. During this process you will NOT tune any of the weights of the autoencoder. It will only be used as a feed-forward network.

The autoencoder

We will train an autoencoder on the MNIST dataset.

First we will download all the data

from keras.datasets import mnist
import numpy as np

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.

print('Training data shape: ', x_train.shape)
print('Testing data shape : ', x_test.shape)

  Training data shape:  (60000, 28, 28) Testing data shape :  (10000,
  28, 28)

Let us see the distribution of our output classes for the MNIST data

import matplotlib.pyplot as plt
%matplotlib inline

training_counts = [None] * 10
testing_counts = [None] * 10
for i in range(10):
    training_counts[i] = len(y_train[y_train == i])/len(y_train)
    testing_counts[i] = len(y_test[y_test == i])/len(y_test)

# the histogram of the data
train_bar = plt.bar(np.arange(10)-0.2, training_counts, align='center', color = 'r', alpha=0.75, width = 0.41, label='Training')
test_bar = plt.bar(np.arange(10)+0.2, testing_counts, align='center', color = 'b', alpha=0.75, width = 0.41, label = 'Testing')

plt.xlabel('Labels')
plt.xticks((0,1,2,3,4,5,6,7,8,9))
plt.ylabel('Count (%)')
plt.title('Label distribution in the training and test set')
plt.legend(bbox_to_anchor=(1.05, 1), handles=[train_bar, test_bar], loc=2)
plt.grid(True)
plt.show()

Let us look at some examples of the MNIST dataset

import matplotlib.pyplot as plt
%matplotlib inline

# utility function for showing images
def show_imgs(x_test, decoded_imgs=None, n=10):
    plt.figure(figsize=(20, 4))
    for i in range(n):
        ax = plt.subplot(2, n, i+1)
        plt.imshow(x_test[i].reshape(28,28))
        plt.gray()
        ax.get_xaxis().set_visible(False)
        ax.get_yaxis().set_visible(False)

        if decoded_imgs is not None:
            ax = plt.subplot(2, n, i+ 1 +n)
            plt.imshow(decoded_imgs[i].reshape(28,28))
            plt.gray()
            ax.get_xaxis().set_visible(False)
            ax.get_yaxis().set_visible(False)
    plt.show()

show_imgs(x_train, x_test)
print('Training labels: ', y_train[0:10])
print('Testing labels : ', y_test[0:10])

  Training labels:  [5 0 4 1 9 2 1 3 1 4] Testing labels :  [7 2 1 0 4 1 4 9 5 9]

These are some imports we will use or not for making our model. Note: not all of these are needed but I'm too lazy to sift through and pick the useful ones.

from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K
from keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D, Flatten, Reshape
from keras.models import Model
from keras import backend as K

Let us build our model. Notice we have the encoder, this maps the input from the higher dimension to the constrained dimension in the middle of the network. It goes from a vector of dimension 784 at the input to a vector $z$ of dimension 128. Then we have a decoder that is a mirror of the encoder which will try to decompress the vector $z$.

input_img = Input(shape=(28, 28, 1))  # adapt this if using `channels_first` image data format

input_img = Input(shape=(28, 28, 1))  # adapt this if using channels_first image data format

x = Conv2D(16, (3, 3), activation='relu', padding='same')(input_img)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
encoded = MaxPooling2D((2, 2), padding='same')(x)

# at this point the representation is (4, 4, 8) i.e. 128-dimensional

x = Conv2D(8, (3, 3), activation='relu', padding='same')(encoded)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
x = Conv2D(16, (3, 3), activation='relu')(x)
x = UpSampling2D((2, 2))(x)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)

autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

You can see a description of the model using

autoencoder.summary()

Now let us train this model

from keras.callbacks import TensorBoard
epochs = 50
batch_size = 128

autoencoder.fit(x_train_reshaped, x_train_reshaped, epochs=epochs, batch_size=batch_size,
               shuffle=True, validation_data=(x_test_reshaped, x_test_reshaped), verbose=1,
               callbacks=[TensorBoard(log_dir='/tmp/autoencoder')])

This will take a while. I will get back to you when it is done training. However, once this is done we will take the autoencoder model and we will separate the encoder and decoder part. We will trash away the decoder and only use the encoder.

Then we will use that as a forefront to any model we want to use to classify these digits. We will pass every image through our encoder, to get this compressed information vector $z$ and we will use that as the input to our classification model.

The classifier

The classification model will then look something like

# The known number of output classes.
num_classes = 10

# Input image dimensions
img_rows, img_cols = 28, 28

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
x_test_reshaped = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

The classification model

model  ==  Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

We will train the classification model

# Save the model# Save t
model_json = model.to_json()
with open("weights/model.json", "w") as json_file:
    json_file.write(model_json)

# Save the weights using a checkpoint.
filepath="weights/weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]

epochs = 100
batch_size = 128
# Fit the model weights.
model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          callbacks=callbacks_list,
          validation_data=(x_test_reshaped, y_test_binary))

score = model.evaluate(x_test_reshaped, y_test_binary, verbose=0)
print('Model accuracy:')
print('Test loss:', score[0])
print('Test accuracy:', score[1])

print('Predict the classes: ')
prediction = model.predict_classes(x_test_reshaped[0:10])
show_imgs(x_test)
print('Predicted classes: ', prediction)


Answering your question: yes, depending on the hyperparameters you choose, you could overfit the considered normal data, if you fit your separating hyperplane between normal and novel points being too much "shaped" on your input data.
There are, for instance in case of one-class support vector machines, some important hyperparams like nu or gamma:

nu: with this one, you tell the oc-SVM the fraction of novel points (i.e. anomalies) you want to consider in your input data; this way, you don't overfit your model by not considering normal all the input data (it also depends on your use case, how sure you want to be about normality of the input data points...)
You can test it with the scikit-learn package this way:

with nu=0.01

VS with nu=0.1, where you tell the model to consider a higher fraction of points as being abnormal:

so, the lower nu value is, the more you are "overfitting" your novelties detector (which could be better or worse depending on how well you know your input data).

gamma: now, look at the effect of the value of gamma, which is crucial for overfitting your model:

with gamma=0.1 (and rbf kernel), you have the following decision surface:

VS with gamma = 10

where, with this last option, your are overfitting so much.


df.where(condition, replacement, inplace=True)

Condition is assumed to be boolean Series/Numpy array. Check out where documentation - here is an example.


Validity of cross-validation for model performance estimation

Cross-validation is used to estimate the performance of a certain type of model on a specific dataset.

  one model has to be chosen as the model which is actually used for prediction on real-world samples (e.g. in a product).

Selecting one of the models obtained during cross-validation is not appropriate, and proceeding in this way would indeed cause the problem that you mention. The correct methodology is to train the final model on the full training data after cross-validation (i.e. independently from the models trained during CV). This way the performance obtained through CV is representative of the expected performance of the final model.


Image classification: Strategies for minimal input count

After running a few tests and collecting some training data I got a rough impression of how much data is needed. The question is still not answered entirely but I figured these insights might help others:

So how many images did I need?

Architecture:
- Pre-trained Inception V3 model with Imagenet data
- Removed last layer

Data / Images per category:

101 images
258 images
45 images
1064 images
259 images
607 images
161 images

Complexity:
Mixed complex sceneries with simple images.

Result:
The quality depends a lot on how difficult the task is. In my scenario the images to be classified where rather similar to the ones in the training data. In these situations all categories with 160 images and above are recognised rather reliably.

I will do some more testing and update my answer.


Feature engineering for time series (audio signals)

As of 2019, the most common approach is to convert the raw audio waveform to a time-frequency representation ("spectogram"). The most commonly used spectrogram for general audio is log-scaled mel-spectrograms.
This allows to use a 2D CNN, very similar to what one would use for images.


Can 1D-CNN method apply to real-time time series classification?

The inputs of a CNN must have the same shape during prediction as when it was trained. So if you have a CNN trained on 50 time-steps windows, then you can make predictions on a stream of input by updating the data in this window continuously. Each new time step you push the most recent row onto the end, and drop the earliest row.
Of course it is possible to make a CNN model that take 1x19 as the input also. But such a model would not have any ability to detect patterns over time, which would probably limit its predictive power quite a lot.


Generally, there are three main solutions for imbalance data classification.

Over-sampling the thin class by generating artificial data or some other ways. This solution can be helpful in situations in which generating accurate artificial data is possible.
Under-sampling the populous class. This solution also can be suitable for situations in which the ratio of classes population is not very low (for example 30% to 70% rather than 1% to 99%)
Using a proper loss function which can handle imbalanced data. There are many metrics that can evaluate the performance of the classifiers at the presence of imbalanced data.

Disclaimer:
If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)


your data looks like a count process. The default ARIMA parameters assume a normal distributed continues error term. So the standard ARIMA models are using this assumption. As far as I know there are some ARIMA models with poisson distribution as an error term, but I guess you just should google for timeseries for count processes. Something like pyflux could work.


How to cluster multiple time-series from one data frame

For an applied solution to your problem, I highly recommend reading the following: TSclust: An R Package for Time Series Clustering which can be found here. It presents the TSclust package in R and provides code.

Note that time series data is special, and cannot be treated like other data. So typical clustering techniques are not appropriate.


How do I create a new column to an existing data frame which is the product of other columns?

Use the below code

df['Fruits_total'] = df[['Apples','Lemon','Banana']].sum(axis=1)

axis = 1, shows that you are summing up the rows. This is what you missed in your code.


How is this score function estimator derived?

This is simply a special case (where $p_\psi = N(0,1)$) of the general gradient estimator for Natural Evolution Strategies (proved in another reference, look it up):

Outline of derivation based on the general formula for the gradient estimator:
$$\nabla_\psi E_{\theta \sim p_\psi} \left[ F(\theta) \right] = E_{\theta \sim p_\psi} \left[ F(\theta) \nabla_\psi log({p_\psi}(\theta)) \right]$$
If
$$\epsilon \sim \mathbb{N}(0, 1) = \frac{1}{\sqrt{2 \pi}}e^{-\frac{\epsilon^2}{2}}$$
then
$$\psi = \theta + \sigma \epsilon \sim \mathbb{N}(\theta, \sigma) = \frac{1}{\sigma\sqrt{2 \pi}}e^{-\frac{(\psi-\theta)^2}{2\sigma^2}}$$
Thus: $\psi = \theta + \sigma \epsilon \sim \mathbb{N}(\theta, \sigma) \Longleftrightarrow \epsilon = \frac{\psi-\theta}{\sigma} \sim \mathbb{N}(0,1)$
So:
$$\begin{align}
\nabla_\theta E_{\psi \sim N(\theta,\sigma)} \left[ F(\theta + \sigma \epsilon) \right] &= E_{\psi \sim N(\theta,\sigma)} \left[ F(\theta + \sigma \epsilon) \nabla_\theta (-\frac{(\psi-\theta)^2}{2\sigma^2}) \right] \\
&= E_{\epsilon \sim N(0,1)} \left[ F(\theta + \sigma \epsilon) \nabla_\epsilon (-\frac{\epsilon^2}{2}) \frac{d(\frac{\psi-\theta}{\sigma})}{d\theta} \right] \\
&= \frac{1}{\sigma} E_{\epsilon \sim N(0,1)} \left[ F(\theta + \sigma \epsilon) \epsilon \right] \\
&= \nabla_\theta E_{\epsilon \sim N(0,1)} \left[ F(\theta + \sigma \epsilon) \right]
\end{align}$$
note: scalar variables were considered in above steps for simplicity, but easy to extend/derive for vector variables


What Kernel is suitable for the following data for SVM classification?

I think the Rbf (Radial basis function) kernel (Wikipedia for rbf) is the most suited for this problem, as its contour plots are circles.

For instance,  this blog solves very similar problems using the same kernel.


the idea is that when reporting nDCG@5 for the two results (associated to the same query) you have to use the same ideal discounted cumulative gain (in your example it would be the one from 1,1,1,1,1 which is the highest). Then the result 1,1,1,0,0
ends up penalized (nDCG@5 < 1) for the missing documents.


How to compute score and predict for outcome after N days

This could be seen as a "simple" binary classification problem. I mean the type of problem is "simple", the task itself certainly isn't... And I'm not even going to mention the serious ethical issues about its potential applications!
First, obviously you need to have an entry in your data for a patient's death. It's not totally clear to me if you have this information? It's important that whenever a patient has died this is reported in the data, otherwise you cannot distinguish the two classes.
So the design could be like this:

An instance represents a single patient history at time $t$, and it is labelled as either alive or dead at $t+N$ days.
This requires refactoring the data. Assuming data spans a period from 0 to $T$,  you can take multiple points in time $t$ with $t<T-N$ (for instance every month from 0 to $T-N$). Note that in theory I think that different times $t$ for the same patient can be used in the data, as long as all the instances consistently represent the same duration and their features and labels are calculated accordingly.
Designing the features is certainly the tricky part: of course the features must have values for all the instances, so you cannot rely on specific tests which were done only on some of the patients (well you can, but there is a bias for these features).

To be honest I doubt this part can be done reliably: either the features are made of standard homogeneous indicators, but then these indicators are probably poor predictors of death in general; or they contain specialized diagnosis tests
for some patients but then they are not homogeneous across patients, so the model is going to be biased and likely to overfit.

Ideally I would recommend splitting between training and test data before even preparing the data in this way, typically by picking a period of time for training data and another for test data.
Once the data is prepared, in theory any binary classification method can be applied. Of course a probabilistic classifier can be used to predict a probability, but this can be misleading so be very careful: the probability itself is a prediction, it cannot be interpreted as the true chances of the patient to die or not. For example Naive Bayes is known to empirically always give extreme probabilities, i.e. close to 0 or close to 1, and quite often it's completely wrong in its prediction. This means that in general the predicted probability is only a guess, it cannot be used to represent confidence.

[edit: example]
Let's say we have:

data for years 2000 to 2005
N=1, i.e. we look at whether a patient dies in the next year.
a single indicator, for instance say cholesterol level. Of course in reality you would have many other features.
for every time $t$ in the features we represent the "test value" for the past 2 years to the current year $t$. This means that we can iterate $t$ from 2002 (2000+2) to 2004 (2005-N)

Let's imagine the following data (to simplify I assume the time unit is year):
patientId birthYear year     indicator
1         1987      2000     26
1         1987      2001     34
1         1987      2002     18
1         1987      2003     43
1         1987      2004     31
1         1987      2005     36
2         1953      2000     47
2         1953      2001     67
2         1953      2002     56
2         1953      2003     69
2         1953      2004     -    DEATH
3         1969      2000     37
3         1969      2001     31
3         1969      2002     25
3         1969      2003     27
3         1969      2004     15
3         1969      2005     -    DEATH
4         1936      2000     41
4         1936      2001     39
4         1936      2002     43
4         1936      2003     43
4         1936      2004     40
4         1936      2005     38

That would be transformed into this:
patientId yearT age indicatorT-2 indicatorT-1 indicatorT-0   label
1         2002  15  26           34           18             0
1         2003  16  34           18           43             0
1         2004  17  18           43           31             0
2         2002  49  47           67           56             0
2         2003  50  67           56           69             1
3         2002  33  37           31           25             0
3         2003  34  31           25           27             0
3         2004  35  25           27           15             1
4         2002  66  41           39           43             0
4         2003  67  39           43           43             0
4         2004  68  43           43           40             0

Note that I wrote the first two columns only to show how the data is calculated, these two are not part of the features.


How to extract audio features for each video frame using pyAudioAnalysis

I don't know all your calculations, but looking at the doc in the code here, short_window and short_step should be in samples (probably as well as mid_window and mid_step).
However in your code:
#mid_feature_extraction(
    signal, sampling_rate,
    mid_window, mid_step,
    short_window, short_step
)
midFeat,shortFeat,midFeatLabels=MidTermFeatures.mid_feature_extraction(
    signals, SamplingRate,
    0.043*SamplingRate, 0.043*SamplingRate,
    0.00016*SamplingRate, 0.00016*SamplingRate
)

it seems that short_step=short_window=0.00016*16000=2.56 which is not in samples. So it will be cast to integer and will be equal to 2 instead of 2.56.
Hope it helps.


Feature selection where adding features are deteriorating model

I guess the measurements from accelerometer-gyroscope-magnetometer are noisy and redundant in some sense. This means you can find some sort of correlation among the values of the measurements, for instance a correlation between the values from the accelerometer and the gyroscope.

PCA captures the principal directions of variation on your data, removing the correlation between the measurements and also reducing the noise, therefore increasing the accuracy. From the graph it can be seen that the accuracy just diminishes a little when using all the features.

Other factor I will consider is the magnitude of the features, a feature with a very large magnitude affects the behavior of K-NN.


Fields list of YearPredictionMSD (identify year of music from Million Song Dataset)

You don't get a full data dictionary with this dataset. Instead the explanation (from your link) is:

  90 attributes, 12 = timbre average, 78 = timbre covariance
  The first value is the year (target), ranging from 1922 to 2011.
  Features extracted from the 'timbre' features from The Echo Nest API.
  We take the average and covariance over all 'segments', each segment
  being described by a 12-dimensional timbre vector.

This is basically features derived direct from the audio. This is typical for problems in signal analysis, and the UCI dataset has been cut down so that it is purely about matching audio summaries to year of production.

The data dictionary from the original MSD dataset shows much more metadata. There are 54 feature types, but actually thousands of features - each type can be repeated many times. The UCI subset is not only subset by row, but by columns too, and only uses some of the MSD data.


What are the possible approaches to fixing Overfitting on a CNN?

To deal with overfitting, you need to use regularization during the training:

Weight regularization - The first thing you have to do (practically always) is to use regularization on the weights of the model. L1 or L2 regularization update the general loss function by adding another term known as the regularization term. As a result thee values of weights decrease because it assumes that a neural network with smaller weights leads to simpler models. Therefore, it will also reduce overfitting.
If you are not sure what you need, just use L2.

Keras - Usage of regularizers
Dropout - Add dropout layers after dense layers (by the way, there are also advantages to using dropout after the convolution layers, it helps with occlusions). Just make sure not to use it at the final dense layer (the one with the same size as the number of classes).
Data Augmentation - The simplest way to reduce overfitting is to increase the size of the training data. Use data augmentation to potentially expend your training set to "infinity". Keras's data augmentation is really simple an easy to use:

Keras Image Preprocessing

If you implement these 3 steps, you will see drastic improvements (probably even just after the first one).

Further corrections and improvements (nothing to do with overfitting):

Your batch normalization layer should come after the non-linear activation, or more accurately, it needs to come before the next convolution layer.
Add an additional dense layer or 2 (only if the results are not good enough).


Keras Sequence Generator leads to huge memory usage when used with fit_generator

Q1: Yes, in theory memory should be released
Q2: This is not a question
Q3: Yes, fit_generator can work perfectly fine on a CPU.

Your data generator looks fine as far as I can see, based on my own experiences with implementing them, so I can only advise on how to proceed to find the root-cause for now as a potential remedy for the actual problem.

The first thing I would try is to set multi-processing to False and workers to 1. I have had some issues myself a while back when using the multi-processing functionality, and while I do not remember if it was memory related or race-condition related, I would start by checking if things work fine.
The second thing I would look at is alternative ways of loading the files and see if these things still lead to memory leaks.


Where to find list of Tensorflow pretrained models available in download.tensorflow.org/models

It seems the models hosted on this are not listed anywhere publically (or at least my research failed!). However, there are official and several nonofficial GitHub repositories with high-level TensorFlow model definitions and pretrained weights. For example:

Nonofficial: that includes NASNet, ResNeXt, ResNet, InceptionV4, InceptionResnetV2, Xception, DPN (need PyTorch).
Official: contains a wide range of official and research models such as resnet, wide-deep, inception, delf, and tcn.

Hope it helps.


NLP algorithms for categorizing a list of words with specific topics

Extending the basic idea stated in comments by @krayyem, you are looking for an Ontology and/or Taxonomy. The short story is to (1) build them based on your data or (2) use existing ones.

Building from your data

You need lots of Is-A pairs which indicate what kind of thing the concept is e.g. Maradona is a football-player and football is a sport. For this purpose, you extract info from your text based on some patterns and update info then update pattern based on info and the loop goes on till it does not change anymore. see this answer though.

Solution you are probably looking for

Use existing knowledge bases that some of which are in Python e.g. wordnet. You can find more in the links I provided for the answer I linked above.

How to proceed kind of idea

Find the dominant keyword in the found keywords. You may simply just count and go through the concept graph (your ontology, the knowledge base) and find the parent concept. If the graph is not accessible in this form, you may create it yourself in Python with existing APIs and/or graph libraries in Python like Networkx.

If more input is needed please comment here. Good Luck!


Why the accuracy of my bagging model heavily affected by random state?

Can someone explain why the huge gap

It simply means that there's a quite high variance depending which random set of instances is picked. How many times do you re-sample the instances in the bagging process? Probably increasing the number of runs will decrease the variance. As mentioned in a comment, the most common reason for variance in performance is a sample which is too small (and/or a number of features/classes which is too high). It's likely to cause your models to overfit.

and should I just use the accuracy with highest value in my research paper or takes the average with random seed=None?

Never ever use the highest performance across random runs, this is cherry-picking and it doesn't reflect the true performance. The possibility to select a random seed is for reproducibility purposes, and selecting the one which gives the best results is the opposite of the principle of reproducibility.
Since you're using bagging, you should decrease the variance (that's the whole point) by increasing the number of runs. If you can't do that for any reason, then don't use bagging: simply repeat the regular process splitting-training-evaluating $N$ times (with a fixed proportion training/testing data) or use cross-validation, and report the average performance (preferably report the variance as well, e.g. standard deviation).


What common/simple problem would work well as a web app?

Maybe you could try solving easy classification problems like with Iris Dataset or Titanic Dataset. You'll find many tutorials dealing with those subjects, and they are basic and famous exercices for someone starting in Data Science.


Word2Vec operates on words and you want to compare 'texts' (series of words of varied length). For that, doc2vec might more appropriate.
You have very short 'texts' (names of campaigns) so generating embedding from them (only from them) want give greate effects. You could start with some pretrained vectors but anyway probably you want achieve much more. But the questions is:
It's not sure from your examples and explanation what will be nature of texts you want to find the most similar campaign name with. On one side you are writing about finding the one most frequent with a given word - then you could just create statistics of words and campaigns. On the other side, in your example, you pass whole text - then above mentioned apply, finding most similar text using text vectors.


Proof of the connection between V and Q in Reinforcement Learning

It follows very simply from the definitions of V and Q.
These are the relevant definitions:
$$G_t \space \dot{=} \space \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$$
$$V^{\pi}(s) \space \dot{=} \space \mathbb{E}_{A \sim \pi}[G_t|S_t=s]$$
$$Q^{\pi}(s, a) \space \dot{=} \space \mathbb{E}_{A \sim \pi}[G_t|S_t=s, A_t=a]$$
You can ignore the definiton of return, $G_t$, I included it for completeness, and to simplify the other definitions.
The only difference between the definitions of V and Q is adding the selection of $a$ in the conditions. If you remove the action selection in the definition of Q, and condition it instead on following the policy, you are back to the definition of V.
You can also expand the expectation in the definition of V using the policy function to show the same thing:
$$V^{\pi}(s) = \mathbb{E}_{A \sim \pi}[G_t|S_t=s]$$
$$V^{\pi}(s) = \sum_{a}\pi(a|s) \mathbb{E}_{A \sim \pi}[G_t|S_t=s, A_t=a]$$
$$V^{\pi}(s) = \sum_{a}\pi(a|s) Q^{\pi}(s,a)$$
Convert this back to an expectation over the policy:
$$V^{\pi}(s) = \mathbb{E}_{A \sim \pi}[Q^{\pi}(S_t,A_t)|S_t=s]$$


This is not at all a typical clustering problem, so I doubt any of these algorithms will help. If you want to try clustering, you will need to do appropriate feature extraction. Don't expect things to work on the raw data. But I guess once you have good features, the problem will already be solved.

Instead of trying to frame this as a clustering problem, look at it either from a sequential pattern view point, or even better: look at the few questions on how to learn regexps from a set of strings.


Predictive model to maximize sum of dependent variable?

Interesting problem which potentially involves many aspects of ML, here are a few thoughts:

At first sight I thought that this looks more like an optimization problem, not a regular classification problem. In this case I would suggest to maybe try things like genetic learning, because it can find an optimal assignment for individual elements which maximizes a global cost or reward function.
However I suspect that the problem is not entirely well defined: "but some that we classified as junk sold for so much in the auction that it wasn't worth sending any to junk.": I'm clueless about the business model but wouldn't it damage the reputation of the company if they were trying to sell all the junk cars at an auction? If I'm right then it matters that only a small proportion of cars are sent to the auction, and then the problem might be related to resource allocation (which is a specific kind of optimization problem).
Another way to look at it is that the vast majority of the cars are junk. Thus it might be relevant to see the problem as anomaly detection, in the sense that one tries to pick the rare cases out of a sea of regular cases. The same idea could be implemented as one-class classification, the idea being that the model can identify all the common cases (junk) and anything else is potentially valuable.

Finally it's certainly worth investigating in the data if the valuable cars can actually be found from the features: would a human expert with only the information in the features be able to correctly classify a car? I could imagine that for instance if a particular car is valuable because it was used in a famous movie, it doesn't help to just know its model and mileage.  It would also be useful to check the relation between how common a car model is and its junk/valuable status, this could be an important indicator to take into account in the model via a feature. In the most simple case, it might even be possible to detect potentially valuable cars just by looking at this indicator... in which case there's no need for ML at all.


Does white balance correction improve the performance of visual object detection?

I don't know about any papers, though making your objects more distinct should definitely help your model to learn proper filters. One thing to keep in mind though is your production data. Will you be also feeding images with poor exposure, white balance, contrast? If so, you need to train your model on such data. As far as I know, this repository produces by far most possible augmentations to images.


mathematical accurate definition of the binary independence model

It's better to talk about x, q and R as (random) events - set of outcomes of the random experiment. x and q will be one element sets but R is an event denoting x is relevant to q and thus it is a subset of cartesian product X x Q (pair of a document and a query).

Comma is then set conjunction and P(A,B) = P(A ^ B) which equals to P(A) * P(B) when A is independent of B.

The wiki statement 'by using Bayesian rule' is a bit of a shortcut since you need to apply it several times. To derive the above formula for P(R|x,q), I would start with conditional probability definition (which is the root of Bayesian rule):

P(A|B) = P(A ^ B) / P(B)

Then:

P(R|x,q) = P(R ^ x ^ q) / P(x,q) = P(x|R,q) * P(R,q) / [ P(x|q) * P(q) ] =

= P(x|R,q) * P(R|q) / P(q) / [ P(x|q) * P(q) ]

When you divide nominator and denominator by P(q), you obtain

P(R|x,q) = P(x|R,q) * P(R|q) / P(x|q)


Context / big picture:

Two events are independent if they have no influence on each other's outcomes. For example, if event A is "I go get coffee" and event B is "it's raining outside", then events A and B are independent, because I'm a coffee fiend and I don't care whether it's raining - I'm getting that coffee anyway!

Logistic regression example:

Say you have blood pressure samples from 50 different individuals at a hospital. You want to classify these as high or low blood pressure. In this example, your X (input variable) is the blood pressure from the 50 individuals, and your Y (binary response) is yes/no to the question 'is this blood pressure high?'. Each person's blood pressure is independent (if we assume they are unrelated strangers).

If you were to sample the same person's blood pressure twice, like before and after physical exercise, then you would no longer have independent samples.

Why:

We need independent samples in logistic regression because otherwise the degrees of freedom of the model are not what we expect them to be, and this affects further calculations. For example, in your dataset, you have 50 independent samples. You use 2 to estimate your intercept and slope, which leaves you with 48 degrees of freedom to work with. If you didn't have independent samples, then you would not have 48 degrees of freedom.

EDIT: a bit more detail on why independent samples are important.

The assumption of independence ensures that each sample contributes the same amount of information to the experiment. With independent samples, this is the case. A blood pressure measurement from one individual is not dependent on the blood pressure of another individual - they're independent. However, if you measure the same person's blood pressure before and after exercise, a different amount of information is added. You can no longer ask the same questions as you could with independent samples.

Let's pick a simpler model to describe what's going on. We will follow standard hypothesis testing principles to illustrate the practical difference between independent and dependent samples. I will round to two decimal places for ease of reading.

Suppose we have two groups of people: those with blood type A and those with blood type O. (Aside: notice people cannot have both blood types at once). We want to investigate whether those with blood type A have a higher resting heart rate than those with blood type O. We have 12 people with blood type A and 10 people with blood type O. To answer this question, we'll conduct a two-sample hypothesis test.

Null hypothesis: The mean resting heart rate of those with blood type A is the same as the mean resting heart rate of those with blood type O

Alternate hypothesis: The mean resting heart rate of those with blood type A is different from the mean resting heart rate of those with blood type O.

We measure each person's resting heart rate, and we get the following results:

Type A: 60, 65, 70, 62, 55, 80, 70, 72, 66, 81, 77, 78

Type O: 61, 64, 70, 72, 59, 62, 66, 78, 69, 75

We then calculate the sample mean and sample standard deviation of each group:

mean(Type A) = 69.67; standard deviation(Type A) = 8.35

mean(Type B) = 67.6; standard deviation(type B) = 6.28

We can then calculate a test statistic, which will follow a t-distribution. We may decide to use either Welch's method or a pooled variance test - in this case, I'll show Welch's method, which doesn't require the population variances to be equal (unlike the pooled variance test).

The test statistic is calculated via:

standard error: sqrt((8.35^2/12) + (6.28^2/12)) = 3.02

test statistic = (69.67 - 67.6)/3.02 = 0.69

The degrees of freedom here are difficult to calculate, but a generally accepted approximation is "the lesser of the degrees of freedom in the two groups", that is min(df_1, df_2). In this case the degrees of freedom of the first group is 12 - 1 = 11 and the degrees of freedom of the second group is 10 - 1 = 9. We subtract 1 here from each sample size, because we have estimated one parameter for each group (the sample mean).

If we wanted to continue this example, we could look at a t-distribution on 9 degrees of freedom and decide based on the test statistic whether to reject or not reject the null hypothesis. In this case, we're more interested in the degrees of freedom than the result, so we'll move on.

Dependent samples

Suppose we have a that same group of people who all have blood type A. Instead of measuring their resting heart rate just once, we’ll measure each Type A person’s resting heart rate twice. This introduces dependence between the samples.

Null hypothesis: the mean resting heart rate of people with blood type A is the same as the mean resting heart rate of all blood types

Alternative hypothesis: the mean resting heart rate of people with blood type A is different from the mean resting heart rate of all blood types

We measure each person's resting heart rate before exercise, and we get the following results:

Before exercise: 60, 65, 70, 62, 55, 80, 70, 72, 66, 81, 77, 78

Then we measure their resting heart rate after exercise, and we get the following results (notice that person 1 in the previous list of heart rates is the same as person 1 in this list):

After exercise: 90, 85, 70, 72, 76, 88, 89, 92, 96, 93, 84, 85

Now we have two observations for each individual. We can’t ask the same question, unless we want to throw out half of our data! Instead, we’ll ask whether there is a difference in the resting heart rate before and after exercise. We’ll use a paired-difference t-test.

To conduct this test, we have to take both observations from each individual into account. We calculate the differences between each person’s before and after measurement, and treat this set of differences as our actual data set:

60-90, 65-85,70-70, 62-72, …

Then we can calculate a sample mean, a sample standard deviation, etc. Our degrees of freedom would be 12 -1 = 11.

What would have happened if we had assumed that those dependent samples were independent?

Notice that when we had dependent samples, our degrees of freedom was 11; when we had independent samples, our degrees of freedom was 9. If we had assumed that the dependent samples were in fact two independent samples, then we could have used the wrong degrees of freedom.

** What if we had had only 10 measurements instead of 12 in the ‘dependent samples’ section? Wouldn’t we have had the same degrees of freedom? **

Yes, but it still wouldn’t have meant the same thing. We would have had:

Before exercise: 60, 65, 70, 62, 55, 80, 70, 72, 66, 81, 77, 78

After exercise: 90, 85, 70, 72, 76, 88, 89, 92, 96, 93

And we would have had to ‘throw away’ the last two ‘Before exercises’ measurements and use only the first ten, as the last two had no partner measurement. The meaning has still changed, though. With dependent samples, you cannot ask the same kinds of questions as with independent samples.


How does one define the possibility space of valid priors (models)?

From a theoretical perspective, I think this question is a variant of "given a problem, is there a way to determine the absolute best learning algorithm for it?". Why? Because as OP correctly suggests, if there was a way to represent all the possible models/priors, then assuming that an infinite stream of data is provided we would be able to eventually determine the absolute best learning algorithm. Unfortunately the No Free Lunch theorem is a theoretical result which states that there can be no such "absolute best" learning method. (to be honest personally I'm out of my depth with this kind of thing, but Wikipedia gives a nice short explanation of the implications of the theorem for machine learning.)

From a practical perspective, this is a matter of design, i.e. how one chooses to represent the problem. It's easy to overlook how much simplification is made when one represents a problem. In the example proposed by OP, one could imagine an infinity of "alternative worlds": maybe the word "father" doesn't have its usual meaning; maybe it happens in a future where people can have any number of parents; etc. My point is: whenever a problem is formally stated, a vast number of assumptions are made anyway, most of them unconsciously. It's not only a matter of cost (it's important, of course), it's a matter of "targeting" the exact problem: if we tried to leave the space of possibilities completely open, then we cannot even start to solve the problem, in the same way that people have to agree on a common language in order to understand each other. Thus choosing a specific space of priors/models for a particular problem is akin to fixing the language of the problem, and when doing it it's important to take into account the limitations/assumptions associated with it.


The reason is the wider range of your output variable. Consider the following two cases,

Real value was 99, prediction is 101
Real value was 5520, prediction is 5522

In both cases, the absolute error is 2, but relative error in first case is much larger (2% - 2/101) than second case (0.035% 2/5520). Absolute and relative metrics are measuring different aspects of the prediction. So one model is not better than the other in absolute sense (pun intended).

Which metric to value depends on your application. When the outcome range is wide (probably your case) and skewed, relative error measurements are better than absolute error measurements.


oversampling plus down sampling using smote not working on random forests

In my experience, giving weights to observations (if the algorithm in use supports it) generally works better for highly imbalanced  classification problems. Since, your are using RandomForests I would suggest you to try that.


How to find the relationship between categorical variables?

Is Outcome also a boolean variable?  If so, a simple prop.test will do.

Here's a toy dataset where a judge from the same country is less likely to give a guilty verdict.

library(tidyverse)
n<-1000
dataset<-tibble(country_judge = sample(c(TRUE,FALSE), n,
                                           replace=T, prob=c(0.2,0.8))) %>%
  mutate(outcome = ifelse(country_judge,
                          sample(c("Guilty", "Innocent"), n,
                                     replace=T, prob=c(0.4,0.6)),
                          sample(c("Guilty", "Innocent"), n,
                                     replace=T, prob=c(0.5,0.5))))

dataset %>%
  group_by(country_judge) %>%
  summarise(p_guilty=mean(outcome=="Guilty"))

This will give something like:

# A tibble: 2 x 2
  country_judge  p_guilty
          <lgl>     <dbl>
1         FALSE 0.5108835
2          TRUE 0.3698630

Now, pull out vectors of trials, and "successes", and feed those into prop.test.

trials <- dataset %>%
  group_by(country_judge) %>%
  count() %>%
  pull(n)

successes <- dataset %>%
  filter(outcome=="Guilty") %>%
  group_by(country_judge) %>%
  count() %>%
  pull(n)

prop.test(successes, trials)

Which gives something like:

    2-sample test for equality of proportions with continuity correction

data:  successes out of trials
X-squared = 13.068, df = 1, p-value = 0.0003003
alternative hypothesis: two.sided
95 percent confidence interval:
 0.06517776 0.21686317
sample estimates:
   prop 1    prop 2
0.5108835 0.3698630


How to summarize multiple time series like dataset

That depends what you want to show and how much information fits in the graph. Typically you can think of:

Simply using a different colour for every dataset, but 20 is probably too many and that will make the graph hard to read.
Using boxplots (or violin plots) at every instant, where the boxplot represents the 20 values of the dataset.


Isolation Forest height limit absent in SkLearn implementation

Unfortunately, it seems like there is no hlim parameter incorporated into sklearn.ensemble.IsolationForest. Calculation of anomaly score is just based on the depth each point settles to and by the average path length. The only way to tune in a bit is by using contamination which calculates the threshold needed to set for anomaly score.

To achieve the granularity that was given in the original paper using hlim=6 to detect a cluster of small points, using a lot of estimators may solve the problem (still depends heavily on how you sample data from smaller cluster into a lot of estimators). But if that small cluster of data is very less in number, I don't think this idea works and there is nothing much we can do from the current implementation in sklearn. Hope this helps.


The last step for production models is typically to train with the entire set (train + validation) after tuning the hyperparameters using the validation set(s). The difference is typically not too drastic as the validation set should only be a fraction of the dataset but more data is always helpful especially for DL-based models.
I'm not familiar with the library you are using but perhaps you could just set the validation data to 0 and ignore the evaluation it prints.


Feature and the Gaussian Distribution (classification)

It completely depends on the type of model. Some models need to represent the features with parameters: for example Naive Bayes with numerical features needs to have a way to calculate the probability based on the value, and the most common case is to assume that the features follow a normal distribution. On the other hand whether a feature is normally distributed or not doesn't matter at all for Decision Trees.
Yes, it can be very informative in order to know whether this feature is a good indicator or not: the more different the distributions, the more easily the algorithm can distinguish the classes using this feature.
No, don't change the distribution of a feature (unless you have a specific reason to do so, e.g. based on expert-knowledge for this particular data). Any way you would do that would certainly alter the overall distribution of the data and/or the way the features are related within an instance, so the model would not learn from the true distribution and therefore its predictions on real data would likely go wrong.


The loss functions are only simple convex functions with respect to the weight parameters (and specific data) when there is a single layer. More exactly, they can proven to be always convex with respect to the weights in the simple models (linear or logistic regression), but not with respect to weights of deeper networks.

You can prove that there must be more than one minimum in a network with 2 or more layers - and thus the loss function cannot be convex - by considering swapping the weights around when you have found a minimum value. Unlike with a single layer network, it is possible to swap the weights around that feed into the hidden layer whilst maintaining the same output. For example, you can swap the weights between input and hidden layer so that values of neuron output 1 and neuron output 2 are reversed. Then you can also swap the weights feeding out of those neurons to the output so that the network still outputs the same value. The network would have a different set of weights, but generate the same outputs, and so this new permutation of weights is also at a minimum for the loss function. It is the "same" network, but the weight matrices are different. It is clear that there must be very many fully equivalent solutions all at the true minimum.

Here's a worked example. If you have a network with 2 inputs, 2 neurons in the hidden layer, and a single output, and you found that the following weight matrices were a minimum:

$W^{(1)} = \begin{bmatrix} -1.5 & 2.0 \\ 1.7 & 0.4 \end{bmatrix}$

$W^{(2)} = \begin{bmatrix} 2.3 & 0.8 \end{bmatrix}$

Then the following matrices provide the same solution (the network outputs the same values for all inputs):

$W^{(1)} = \begin{bmatrix} 1.7 & 0.4 \\ -1.5 & 2.0 \end{bmatrix}$

$W^{(2)} = \begin{bmatrix} 0.8 & 2.3 \end{bmatrix}$

As we said the first set of 6 parameters was a solution/minimum, then the
second set of 6 parameters must also be a solution (because it outputs the same). The loss function therefore has 2 minima with respect to the weights. In general for a MLP with one hidden layer containing $n$ neurons, there are $n!$ permutations of weights that produce identical outputs. That means that there are at least $n!$ minima.

Although this does not prove that there are worse local minima, it definitely shows that the loss surface must be much more complex than a simple convex function.


How to combine two different random forest models into one in R?

This question has already been answered here. It addresses the problem in hand.


R: Error when using Aggregate function to compile monthly means into yearly means

You can use the group_by() and summarize function from the dplyr package to achieve the above easily:

library(dplyr)
import.list <- ldply(filenames, read_csv_filename)

# No need to create b1, b2 and b3
#by1 <- import.list$Source
#by2 <- import.list$Result
#by3 <- import.list$Year

# here you want to group by the data based on 'Year' as you mentioned you want the yearly mean and you want to calculate mean of 'Result' right?.

import.list %>%
  group_by(Year) %>%
  summarize(Mean = mean(Result, na.rm=TRUE))->Final_Output

The above will group_by the data based on Year and calculate mean of Result column.


A Machine learning model that has a undefined input size but a fixed output?

For Deep Learning (neural networks), there techniques like Adaptive Average Pooling and Adaptive Max Pooling which outputs a fixed number of values irrespective of the size of the input to these layers.


Dimension problem in keras neural networks

Ok I was able to recreate the error. The TimeDistributed layer applies the dense layer to each of the time slices. In this model you will have 100 different time slices as defined by sum_txt_length and the RepeatVector layer. These time slices will be propagated through the network all the way to the TimeDistributed layer. Thus, your $y$ vector must have dimensions $(None, 100, 10000)$.

You can use model.summary() to get the expected dimensions after each layer.

You will need to fix the output layer to match the label dimensions. I believe you will need to add a Flatten() layer before the densely connected layer, and do not use the TimeDistributed layer at the output. That way your output will be of size $vocab_size$, thus predicting the next vocabulary word given an input.


Why I would use TF-IDF after Bag-of-Words (CountVectorizer)?

This is the standard TF-IDF feature extraction: you transform the document counts.  It just looks odd to separate the two steps like this.  sklearn provides both TfidfTransformer and TfidfVectorizer; note the documentation of the latter:

Equivalent to CountVectorizer followed by TfidfTransformer.


Dimensionality reduction based on value of a variable

Actually the first plot is still better! The first point to mention is that in 2d you lose a vast amount of information i.e. none of these plots are necessarily telling you something. If the 2d plot is good you can be happy but if not, you should not necessarily get disappointed! I strongly recommend you to see the performance of your dimensionality reduction by evaluating your regression model.

For 100 dimensions I would first try to find correlation/dependence of features and target (try not to use linear correlation without visualization!). Then, out of all features remove non informative ones. On the remaining features try different dimensionality reduction algorithms (not only PCA, for example NMF if values are all non-negative and/or LLE) with more dimensions. Evaluate your results on your validation set.


k-means classifies one data point as a group

Two ideas come to mind, which could be combined or not.

Try to identify the single point as an outlier, and remove it from consideration for the clustering.
Allow $k$ to vary a little.

Using both and allowing $k\in\{2,3\}$ allows you to find only two groups in the main set of points, plus the outlier.  Using just (2) with $k\in\{3,4\}$ could find clusters Low/Med/Large/Outlier...that has the nicety that outlier detection is done by the k-means algorithm rather than another preprocessing step, but runs the risk of finding four honest clusters when you only wanted three.


I think you are too quick to throw AI at this problem. (As you already noted, keep alternatives to neural networks open: maybe factor analysis or principal component analysis are more suitable, or support vector machines)

First, you have to do more work in the problem domain, to find out that different pathologies may result in different models.

My recommendation would be to start small with one pathology. Use domain knowledge to determine if (and to what extent) the pathology is present in some samples, and how that could be observed from the audio samples, in a limited number of parameters.

For some pathologies, speech recognition would help. You could count differences in the text spoken and the text recognized. The count (and maybe a pattern in those counts) could be an input for the model (not necessary a neural network as you already noted).

Some other pathologies would need audio spectral or temporal observations in the audio samples (unnatural pitch, unnatural pauses).

To give an example: stuttering.

I would look in the audio samples and spectra for differences in the patterns. E.g. a spectral plot of stuttering vs. regular speech could look like:

So typically you'd want to detect/measure relative power in some frequency regions as a measure for stuttering and learn your model with such data.

An alternative that might also work is to run a stuttering sample (and a natural sample) through speech recognition, to see if differences can be observed in a countable/measurable way.


LSTM -RNN : How to get continuous range output instead of categorical?

Using Sigmoid Activation function (standard, no need for customization), you will get outputs in the range [0,1]. If you pre-scale your labels accordingly to fit in [0,1], the network will output values which you can manually re-scale to match the original amplitude. Hope it helps :)


What is the appropriate evaluation metric for RandomForest with probability in R?

Group the entires by the probability score you assigned them (e.g. 100-95%, 95-90%,....) and then measure the likelihood of their purchases from ground truth.  You can then perform a Chi-squared calculation between your predicted and measured outcomes.


Would Topic Modelling be classified as NLP or NLU?

Maybe I'm having trouble formulating the inherent difference between NLP and NLU, when do we draw the line between the two?

There is a confusion here: NLP is the whole domain of AI which deals with natural language. It includes virtually any task related to processing language data (usually mostly written data, but that's not the point). Topic modeling is one of these tasks.

NLU is the problem of Natural Language Understanding, which is usually considered as one of the main goals of NLP. If anything, NLU is a problem that NLP tries to solve, i.e. a sub-topic in the large area of NLP.

Also notice that using words embeddings can improve things, but it doesn't solve all the difficulties related to semantics, far from it.

[edit] The scope of NLU is not strictly defined: in the broadest possible definition, it would include anything vaguely related to extracting meaning from text, and in this very generous sense topic modeling would have a connection to it with or without embeddings (and so would a lot of other NLP tasks). Wikipedia says:

  The umbrella term "natural-language understanding" can be applied to a diverse set of computer applications, ranging from small, relatively simple tasks such as short commands issued to robots, to highly complex endeavors such as the full comprehension of newspaper articles or poetry passages. Many real world applications fall between the two extremes, for instance text classification for the automatic analysis of emails and their routing to a suitable department in a corporation does not require in depth understanding of the text.

But the most commonly accepted definition of NLU is stricter, it would only consider tasks which directly involve the interpretation of text in a quite complex setting. The typical example is the "virtual assistant" such as Amazon Alexa, OK Google, Apple's Siri. In this sense topic modeling is simply a completely different task, no matter the "degree of understanding".


It is a common problem that - with unbalanced classes - some model tends to predict mostly the majority class. You could try to oversample the minority classes. In addition, RF tends to perform weak here. Boosting or NN are often able to recover more details, which can be important to predict the minority classes.

Edit: Okay, now that you clarified that you face a balanced problem, I guess your problem is the classifier. You could try multinominal logit with lasso regulation to „shrink“ irrelevant features (words). You should also consider Boosting or NN (maybe LSTM sequential).


The technique you have mentioned is called "Depth-wise Separable Convolutions" and it relies on the idea that spatial and depth-wise information can be decoupled. Its main advantage is that compared with standard convolution, since it doesn't need to perform convolution across all channels, it has considerably less connections, (so less parameters) which results in a lighter model.

There is an architecture named "Xception" ("Extreme version of Inception") by Google that uses a version of such convolutional layers and produces close to state-of-the-art results. There is a really good review of it at Towards Data Science.

Personally I don't like the idea of decoupling the channels, because I think it gets us a less "sophisticated" model, but it seems to work so who am I to judge?


Comparing TFIDF vectors of different shapes

Your thinking is correct, it is unefficient to recalculate TFIDF on the whole collection every time a document is added.
There are two parts in TF-IDF:

$TF(w,d)$ is the Term Frequency of a word $w$ in a particular document $d$, it's independent from other documents. It's just defined as the proportion of $w$ in $d$.
$IDF(w)$ is the Inverse Document Frequency: it is defined as the log of the inverse of the document frequency, where the document frequency of a word is the proportion of documents in the collection which contain this word:

$$IDF(w)=log\left(\frac{|D|}{| \{d \in D \text{ such that } w\in d \} |}\right)$$
$IDF(w)$ depends on all the documents so it's costly to re-calculate. However updating IDF whenever there is a new documents can be implemented efficiently:

Maintain a global map of the document frequency (not the IDF) for every word. This map is easy to update: for every distinct word $w$ in the new document, increment $m[w]$.
Whenever a TF-IDF weight is needed, collect the required document frequency and apply the IDF formula.

I don't know whether this can be done with scikit API or not, but it's not too big a task to implement it yourself (and it's a good little exercise to understand TFIDF ;) )


Merging two datasets with unclear identifier in Excel

I typically proceed the following way,

convert to the same case with LOWER() or UPPER();
remove all non-alphanumeric characters with several SUBSTITUTE(), e.g. SUBSTITUTE(A1, ".", "");
trim white spaces TRIM();
extract the first word with LEFT(A1, FIND(" ", A1) - 1);
trim white spaces again;
perform a join on the key that you created.

Often this gives you a decent match.


What is "Missing" in output of plot_tree API of XGBoost

The "missing" at the different nodes are the observations for which the feature on which the split is made is missing. E.g. if the value for TotalSubmissionGuidelines is below 0.5 or missing, then the observation follows the left path of the decision tree. XGBoost supports missing values by default and learns the branch direction during training, see the documentation.


Properly using activation functions of neural network

How is data predicted from activation functions? (considering that it returns constants on weighted input sum).

You should consider the fact that the label of your input data is going to be predicted by the network. Moreover, the outputs of the network usually represent the probability of belonging to each class. The label is not predicted by the activation function, it is predicted by the network. Take a network as a mapper which can map the inputs to outputs. The activations are used to add non-linearity in order to approximate complicated non-linear mappings.

  What's the point of multiple hidden layers?

As you can read here, the purpose of adding those is that you can learn different complicated regions. Another interpretation can be this that by adding more layers you can learn more complicated mappings which have non-linear behaviour.

About the global minimum, because you can not see the cost function which is the error based on the weights, due to the fact that it is not possible to visualise it because it has so many dimensions, you should use gradient-descent based algorithms.


Converting dates to appropriate form to train machine learning model

It depends on whether the date is the single source of features data X or if the date is a feature of X, along with other features.

If the date is the only independent variable you have time-series models, such as the autoregressive (AR), integrated (I) or moving average (MA) models. Their combination results in the ARMA or in the ARIMA models. Check this tutorial on using ARIMA with Python

If the date is a feature among others, you can convert the original date into multiple date features. Check this thread for feature engineering with time data.


Pytorch : Loss function for binary classification

You are right about the fact that cross entropy is computed between 2 distributions, however, in the case of the y_tensor values, we know for sure which class the example should actually belong to which is the ground truth. So, you can think of the binary values as probability distributions over possible classes in which case the loss function is absolutely correct and the way to go for the problem. Hope that helps.


This is going to be a situation where there will be no fixed rule. One important factor is how meaningful colour differences are to the other parts of your problem. If colour has low correlation/impact in a supervised learning/prediction scenario for example, and the dataset is noisy, then you will want to merge more colours (at a higher fraction of total number) to reduce sampling bias effects that might otherwise assign importance to the colour and increase error rates in test and production.

The safest approach is to treat the colour combination threshold as a hyper-parameter to the model building process, and test to see what differences it makes. If there is little or no impact to model effectiveness, then a higher threshold could be useful purely to reduce number of parameters in the model - decreasing resources used to train and run it.

If that seems time-consuming, then picking something by feel (e.g. your idea of picking count less than 50 in the dataset) is not usually too bad, at least to start with. You can go back and re-evaluate your choice if you have problems with the model.

One other possibility for feature engineering is to use the rareness of the specific colour as an additional feature. So in addition to categories for the popular colours and an "other colour" category, add a real value "colour frequency" = the observed ratio of that colour in the training set. Whether or not this is useful will depend on the problem, but it may help address some of the lost information when merging categories with a wider range of rareness values, assuming that unusual colours indicate anything at all (they may not).


Best anomaly detection algorithm based on two conditions

Since your data is one-dimensional and numeric, I don't think you need any fancy clustering technique.  Clustering is useful when your data points have multiple attributes.  When there is just a single attribute, then all you need is a good definition of "anomaly".

For example, suppose you decide that an anomaly is any point more than two standard deviations from the mean.  It's easy to find such points using pandas:

import numpy as np
import pandas as pd
from scipy import stats

threshold = 2.0  # in standard deviations
input_file = "path/to/my/file.csv"
target_unit = "meters"

# read the file into a pandas DataFrame
df = pandas.read_csv(input_file)
# filter to only include the target unit
df = df[df['Units']==target_unit]
# compute z-scores for the `values` columns
df['values_z'] = np.absolute(stats.zscore(df['values'].values))
# threshold z-score to identify anomalies
is_anomaly = df['values_z'] > threshold
anomalies = df[is_anomaly]

# `anomalies` is now a DataFrame that contains the anomalous points
# you can consume this however seems appropriate
# for example, you can write the anomalies to a separate file:
anomalies.to_csv('path/to/output.csv')

```


Bootstrapping or Randomly Dividing Dataset to reduce variance?

I think the second method will yield less correlated models than the first method. It is particularly true with decision trees which tend to quickly overfitting in the bottom nodes. It will help reduce the variance.

However, by using the second approach, you will end with 10 smaller datasets and so you risk to introduce a variance error due to the too small number of observations. Discussing about decision trees again, it would mean that your trees algorithms will tend to overfitting upper in the tree. And so you will increase your variance error.

In my opinion, for most of datasets, it is still better to use the first approach than the second. I think that very low correlated estimators won't bring a better improvement that the first method.

We can also observe that differences in the two approaches depend also of the number of observations, the number of features, the kind of estimators you are using.. A benchmark would be really interesting !


I don't know if this is what you are looking for, but Andrej Karpathy has a good blog article about his method for training networks in general: A Recipe for Training Neural Networks

I will put the bullet points of his recipe here, but there is way more practical advice in the actual blog.

The recipe

Become one with the data
Set up the end-to-end training/evaluation skeleton + get dumb baselines
Overfit
Regularize
Tune
Squeeze out the juice

I think the advice that best fit your situation is from step 2:

  At this stage it is best to pick some simple model that you couldn’t
  possibly have screwed up somehow - e.g. a linear classifier, or a very
  tiny ConvNet. We’ll want to train it, visualize the losses, any other
  metrics (e.g. accuracy), model predictions, and perform a series of
  ablation experiments with explicit hypotheses along the way.

You usually want to start with something simple that will not allow you to have issues with overfitting like you have. Then you will progressively work your way into a more complex model that can overfit in step 3.


Are all classifiers linear in some high dimensional space?

The answer is yes!

One can use Cover's theorem to prove it.

Remark 1: by "linear in some high dimensional space" we assume here, for any classifier, an existence of an equivalent linear classifier in higher dimensions (i.e., a decision hyperplane that yeilds identical results to the original classifier for any possible input).

Remark 2: it is definitely true for any classifier on a countable domain (e.g., all possible 8bit RGB 256x256 images is quite large but still a countable domain).

For classifiers on uncountable domains (like fields of real numbers), the answer would depend if Cover's theorem holds for infinitely large number of points.
However, I'd rather not be worried about that, since all real-world classifiers implemented on classical computers are discrete :-)

Remark 3: Of course, classifier must be deterministic...


Use MinMaxScaler , label encoder, one hot encoder , keras prediction file for later prediction

You could use pickle to store your encoders/scalers/etc. It is a common way of storing Python objects

from sklearn.preprocessing import LabelEncoder
import pickle

# Fit a label encoder
le = LabelEncoder()
X = le.fit_transform(X)

# Pickle the encoder for later use
with open('path/and/name', 'wb') as f:
    pickle.dump(le, f)

Then when you have it stored it can be used again by loading the pickle

# Read the pickle from file
with open('path/and/name', 'rb') as f:
    le = pickle.load(f)

# Use the already fitted encoder to transform new data
X = le.transform(X)

Then repeat the process for your all your preprocessing objects


To be useful, doesn't a test set often become a second dev set?

You're absolutely right and yes, it is actually possible to overfit to your validation data if you're not careful. Some researchers at google published an interesting article about this problem and a way to address it called the Reusable Holdout. The general idea is that you only access the test set through a special intermediary algorithm. Obviously this isn't how most people work though. In practice, I think a common approach is to use several holdouts: use one for most of your prototyping, and then once you're satisfied you can extend your evaluation to one or more of your additional holdouts.


Each clustering algorithm in my knwledge is based on distance, which is calculated as pairwise differences. So I'm not sure if it is easy to find a general more elegant approach.

In you case you can calculate the distance and than override it based on your constraint.

Here an simple example reduced to one numerical dimension with distance threshold 30.

 nlst <- c(1,3,5, 36,39,42, 1001,1003,1005, 1036,1039,1042)

 d <- dist(nlst, method="euclidean")
 dr <- as.dist(ifelse(as.matrix(d) > 30, NA, 0))

     1  2  3  4  5  6  7  8  9 10 11
 2   0
 3   0  0
 4  NA NA NA
 5  NA NA NA  0
 6  NA NA NA  0  0
 7  NA NA NA NA NA NA
 8  NA NA NA NA NA NA  0
 9  NA NA NA NA NA NA  0  0
 10 NA NA NA NA NA NA NA NA NA
 11 NA NA NA NA NA NA NA NA NA  0
 12 NA NA NA NA NA NA NA NA NA  0  0

A scan through the points collecting those with distance zero will give the result clustering.

This should be always possible if your constraints are local (i.e. depending only on the two points for which you calculate the distance).


Classification problem approach with Python

You are looking at a Classification problem.

Logistic regression, Decision trees, SVM. Any of the above can solve the job for you. But selecting the best model depends upon how good it is able to predict the test set. Use cross-validations and see which model can give you better accuracy. So split you test and train set accordingly. You don't expect the model to predict, if you haven't trained it. See stratified sampling for starters.

Though your predictors might be categorical or ordinals, they can be treated as numericals.

You have built in functions like predict_proba to find class probability. The references for this can be found in above links. But go through how they work and what is the policy for selecting best plane in any classification, so that you don't feel like you are using a black-box. Since you are saying that you are a beginner, pandas will be a very useful module for reading data into dataframes and mending it as you like.

Hope this clears something.


Can I use cosine similarity as a distance metric in a KNN algorithm

Short answer: Cosine distance is not the overall best performing distance metric out there

Although similarity measures are often expressed using a distance metric, it is in fact a more flexible measure as it is not required to be symmetric or fulfill the triangle inequality. Nevertheless, it is very common to use a proper distance metric like the Euclidian or Manhattan distance when applying nearest neighbour methods due to their proven performance on real world datasets. They will therefore be often mentioned in discussions of KNN.

You might find this review from 2017 informative, it attempts to answer the question "which distance measures to be used for the KNN classifier among a large number of distance and similarity measures?" They also consider inner-product metrics like the cosine distance.

In short, they conclude that (no surprise) no optimal distance metric can be used for all types of datasets, as the results show that each dataset favors a specific distance metric, and this result complies with the no-free-lunch theorem. It is clear that, among the metrics tested, the cosine distance isn't the overall best performing metric and even performs among the worst (lowest precision) in most noise levels. It does however  outperform other tested distances in 3/28 datasets.

So can I use cosine similarity as a distance metric in a KNN algorithm? Yes, and for some datasets, like Iris, it should even yield better performance (p.30) compared as to Euclidian.


Error is occurring setting an array element with a sequence

I managed to make it work, by combining the city column with the venue categories column into a 2D (numpy) array which can be used by the RandomForestClassifier of sklearn.
Example code:
import pandas as pd
from sklearn.preprocessing import LabelEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
import numpy as np

def make_data(df, target_column='Venue', categories_column='Venue Categories'):
    categories = df[categories_column].values
    other_features = []
    for col in df.columns:
        if col in [categories_column, target_column]: continue
        other_features.append(df[col].values);
    return np.array([[col[i] for col in other_features]+categories[i] for i in range(len(categories))]), np.array(df[target_column].values)

labelencoder = LabelEncoder()

dff['Venue'] = labelencoder.fit_transform(dff['Venue'])
dff['city'] = labelencoder.fit_transform(dff['city'])

data, labels = make_data(dff, 'Venue', 'Venue Categories')

train_data,test_data,train_labels,test_labels = train_test_split(data,labels,test_size=0.20)

model = RandomForestClassifier()
model.fit(train_data,train_labels)

Note make sure that Venue Categories column has same number of elements for each row of data, else a new problem will arise again. If needed fill with dummy values


Choose lr of optimizer something very small. It might be because of exploding gradient.

In self.weight use nn.Parameter() then pass your torch.zeros() to make it a model parameter.


Ggplot2 is great for simple visualisations but doesn't handle huge datasets very well, as you found out. Running out of memory is also often an issue with huge sets, and calculating prime divisibility on 30 million integers definitely counts as huge.

If you have access to more powerful systems or cloud platforms that should work a lot better. Some of the AWS and Azure etc. solutions aren't that expensive so you could try one of them.


In theory it's of course possible to reach perfect performance: if the algorithm can find what it needs in the features to correctly distinguish between classes (or clusters), then it will perform perfectly.

In reality however it's very rare that performance is perfect, because:

Text data is noisy and extremely diverse
Most of the time when there is a way to obtain perfect performance, there is also a simple heuristic which can do the same job more efficiently than using ML. Basically ML is used precisely because the task is hard and/or the data is complex, so it's not surprising that there are errors.

In the case of your problem, I notice that you have labels for your data but you use an unsupervised topic modeling method, right? If this is the case you might want to try using a supervised method, since the system would have more clues to find the right answer. Also you use accuracy for evaluation, so be careful: accuracy can be misleading since it doesn't give any detail about the different classes.


How to train ML model with multiple variables?

This is very little data so doing much with it is very difficult. especially considering you expect to get 'next year' estimates without having any historical data to base yourself from.

However, if you want to be able to estimate a cost given your input features which are city, store, units_sold and num_employees then this problem setup is very standard in machine learning.

First I will put your data into a pandas DataFrame and I will encode your categorical features using numerical values. I did this randomly, but I think you can get better results by getting the latitude and longitude of the stores, as well as some data about the neighborhood (density, wealth, etc.).

import pandas as pd

data = {'city': ['New York', 'New York', 'New York', 'New York', 'London', 'London', 'London', 'Paris', 'Paris'],
        'store': ['A', 'B', 'C', 'D', 'A', 'B', 'C', 'A', 'B'],
        'units_sold': [10, 12, 14, 17, 23, 27, 22, 4, 7],
        'num_employees': [4,4,5,6,5,6,3,2,3],
        'cost': [11000, 11890, 15260, 17340, 22770, 25650, 21450, 5200, 9560]}

df = pd.DataFrame(data)

df['store'] =df['store'].astype('category').cat.codes
df['city'] =df['city'].astype('category').cat.codes

When I am working with very little data I always like to pull some visualizations to give me some intuition about the data. Let's first see how the features are correlated.

import matplotlib.pyplot as plt

plt.matshow(df.corr())
plt.xticks(np.arange(5), df.columns, rotation=90)
plt.yticks(np.arange(5), df.columns, rotation=0)
plt.colorbar()
plt.show()

We can see that the cost is associated directly with units_sold. That is not much of a surprise I guess. But it also correlates with the num_employees and is strongly inversely correlated with the city.

If we plot the num_employees and units_sold we can see that the correlations observed above more clearly.

A predictive model

Now we want to be able to give a model our inputs and get an estimation of the cost.

Let's first put our data into a training and testing set. This is very problematic with such a small dataset because you have a very high probability of ending up with a training set that omits a city, or a store type. This will significantly affect the abiltiy of the model to predict an output for data it has never seen.

The labels of the data are the cost.

import numpy as np
from sklearn.model_selection import train_test_split

X = np.asarray(df[['city', 'num_employees', 'store', 'units_sold']])
Y = np.asarray(df['cost'])

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)

Let's start with a standard linear regression

from sklearn.linear_model import LinearRegression

from sklearn.linear_model import LinearRegression

lineReg = LinearRegression()
lineReg.fit(X_train, y_train)
print('Score: ', lineReg.score(X_test, y_test))
print('Weights: ', lineReg.coef_)

plt.plot(lineReg.predict(X_test))
plt.plot(y_test)
plt.show()

  Score:  0.963554136721
  Weights:  [ 506.87136393  -15.48157725  376.79379444  920.01939237]

A better alternative is to use ridge regression.

from sklearn import linear_model
reg = linear_model.Ridge (alpha = .5)
reg.fit(X_train, y_train)
print('Score: ', reg.score(X_test, y_test))
print('Weights: ', reg.coef_)

plt.plot(reg.predict(X_test))
plt.plot(y_test)
plt.show()

  Score:  0.971197683706 Weights:  [ 129.78467277    2.034588
  97.11724313  877.73906409]

If you do another split and rerun both these models you will see that the performance varies greatly for both of them as we expected due to the limited data. However, the effect is much more pronounced for the simple linear regression.

To get a better idea of the actual results, let's do something a bit naughty. We will split the data over and over again to get different models and get the average of their scores.

scores = []
coefs = []
for i in range(1000):
    X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)
    lineReg = LinearRegression()
    lineReg.fit(X_train, y_train)
    scores.append(lineReg.score(X_test, y_test))
    coefs.append(lineReg.coef_)
print('Linear Regression')
print(np.mean(scores))
print(np.mean(coefs, axis=0))

scores = []
coefs = []
for i in range(1000):
    X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, shuffle= True)
    lineReg = linear_model.Ridge (alpha = .5)
    lineReg.fit(X_train, y_train)
    scores.append(lineReg.score(X_test, y_test))
    coefs.append(lineReg.coef_)
print('\nRidge Regression')
print(np.mean(scores))
print(np.mean(coefs, axis=0))

  Linear Regression
  -1.43683760609
  [ 1284.47358731  1251.8762943   -706.31897708   846.5465552 ]

  Ridge Regression
  0.900877146134
  [ 228.05312491   95.33306385  123.49517018  873.49803782]

We can see that across 1000 trials the ridge regression was by far the superior model.

Now you can use this model to estimate costs by passing the model a vector with the features in the same order as the dataset as follows

reg.predict([[2,  4,  1,  12]])

The resulting score is

  array([ 12853.2132658])

This is not enough data to do any machine learning regression reliably. However you can get some insight into what factors affect your cost the most.


What is the shape of conv3d and conv3d_transpose?

Although your input data is three dimensional, you have to use Conv2D for your task. I guess Conv3d is used for data with temporal characteristic, yours is just a simple picture. To illustrate why you should Conv2D, suppose your input image is 224 * 224 * 3 and you employ a Conv2D layer with 10 filters. You have to specify stride and padding in order to specify the output shape. You have to specify dimensions to illustrate the height and width of you filters, also known as kernels, filter size will affect the output size if you assign padding to 'valid'. Here, there is a point. Suppose you have specified the filter size a 10 * 10 filter, then if the input shape was 224 * 224 * 1, each filter would be of size 10 * 10 * 1 to fit the input area. Now that the input is of size 224 * 224 * 3 the size of each kernel is 10 * 10 * 3 to fit the input volume. Consider in all cases the output of each convolution operation, better to say cross correlation, is a scalar. For more information take a look at videos here and for your case I encourage you watching Convolution Over Volume.


Binary problems have the most amount of metrics to measure its performance.
They can be classified as accuracy metrics, probabilistic metrics and metrics depending on true/false positives/negatives. You are not expected to implement them from scratch, the last time I implemented an AUC (area under the curve) was in college so you will find these metrics already implemented in most machine learning libraries in all languages. This is a comprehensive list taken from the Keras library, check the links for more information about the metrics or the library.
About the matter of the outputs you can have a true/false 1/0 values or the probabilities of the observation to belong to a class which is a better way to measure the algorithm's performance. Do you need a threshold to evaluate results properly, it all depends on the data you have. For some problems high accuracy like above 99.999% is critical like clinical diagnoses but for others accuracy like 55% can be seen as great like the case of stock predictions. So it depends on your data.
Accuracy metrics:
These metrics, simple in design, are often raw percentages of the correctly labeled classes for one class or many classes.
Raw Accuracy
Binary Accuracy
Categorical Accuracy - Straightforward accuracy for all classes.
Top K Categorical Accuracy - Accuracy for the top K categories
Sparse Top K Categorical Accuracy
True False Positive Negative metrics:
More expressive metrics useful to determine the impact the algorithm has not only in its accuracy but also in the wrongly classified observations.
AUC
Precision
Recall
True Positives
True Negatives
False Positives
False Negatives
Precision At Recall
Sensitivity At Specificity
Specificity At Sensitivity
Probabilistic metrics:
These as often used as the function loss for machine learning algorithms, however, they can also be understood as a benchmark metric specially for a validation set.
Binary Crossentropy
Categorical Crossentropy
Sparse Categorical Crossentropy
KLDivergence


How to Proceed with Tokenized text content which is given in number?

Using numbers for words is a very common way of representing words in a corpus. The thing that is missing here is how you have arrived at these numbers. Generally, if you have a corpus, you get these numbers by getting the index of these words in the vocabulary. For example, if you have two sentences :
1. John went to London
2. John went to London with Mary

Based on the order in which these numbers come, you can assign the following representation :

John : 1 , went : 2, to : 3, London : 4, with : 5, Mary : 6
Further, your sentence will be
1. 1 2 3 4
2. 1 2 3 4 5 6


Intuition of "Head" in Attention models (Transformer)?

I like to think about it in the context of progression of attention mechanisms in neural networks.
Early attention mechanisms were implemented as explicit sliding windows over the encoding sequence.  For example, Graves 2013 implemented it as the average of several sliding Gaussians.  So, it was a local attention mechanism to attend to a learnable window over the data.
Then you have Luong and Bahdanau attention, which implemented similar mechanisms, but separate from the network type and as a softmax function.  This is essentially single-head attention.  Now it's not a window, but a score and weighted average over all of the encoding input.  The idea of the K, Q, V flows naturally from that, but in that case the Q is a hidden state of the decoder, and K/V the hidden state of the encoder.  This is more flexible and can attend to more arbitrary windows over the sequence.
Transformers flow naturally from these mechanisms, but now K, Q, V all come from the same sequence, except for the second layer of the decoder, which acts the same as Luong or Bahdanau attention.  In addition, they first split the K, Q, V along their last dimension (the model parameter dimension) to create several heads that can attend to different parts of the sequence.  This is much better at attending to long term and varied parts of the sequence to concatenate an overall context for each Query.  This, of course, is very advantageous for natural language translation, but as we are seeing, it is also great for covering the image space in vision problems.  Intuitively, this makes sense as I view the Transformer as almost a superset of a convolution network, where the filters for each partition of an image attend to various parts of the image and this is learned.
Obviously, there are many papers I'm sure I haven't read and steps in between these.


I would suggest you, other than simple histograms, to visualize how variables are associated with each other using a pairplot from seaborn.pairplot(). This will let you check how correlated your explanatory variables are with each other. Multicollinearity can be a problem that you can solve using dimensionality reduction, for example.

Outliers might not be a problem, but you can't say before running any model. On that, I suggest you to run the same model more than once, with and without outliers. Also, always normalize your data, this might affect the "outlierness" of an observation.


Traversing trees in SQL: JOINs vs imperative algorithm

There are several models for holding hierarchical data in a relational database. There's a good summary here. In summary they are

Adjacency list: each row contains a foreign key to its parent. Fast for structure changes. Requires recursive processing to find the base ancestor.
Path enumeration: each node stores its ancestry. An update requires all children to be updated. For a known node reaching the root requires no further work.
Nested sets: additional columns hold the ancestry. The cost of structure changes depends on how big the moving subtree is. The root node is fast to find.
Closure table: store all ancestor-descendant pairs. Structure changes can be expensive. Root node expensive to identify.

From the given information a path enumeration sounds like your best option.


Why does Feature Importance change with each iteration of a Decision Tree Classifier?

From sklearn.tree.DecisionTreeClassifier help:

  The features are always randomly permuted at each split. Therefore, the best found split may vary, even with the same training data and max_features=n_features, if the improvement of the criterion is identical for several splits enumerated during the search of the best split. To obtain a deterministic behaviour during fitting, random_state has to be fixed.

Also, you might want to have a look at my critique on feature importance.


How is the loss between actual and predicted labels being calculated?

The per example loss is calculated by multiplying the one hot encoded result which will be 1 for the true class and 0 for the others with the softmax output which outputs a probability for each class.
Essentially we are picking the prediction for the relevant class.
In the next step we reduce the negative of this value which essentially is maximizing the probability of the class of interest.

The above approach should work fine for multi label as well. However using sigmoid instead of softmax to arrive at the probabilities for the multiple classes doesn't clip probabilities of other classes


What is the best approach to perform information extraction from tourist reviews using NLP, DL?

I'm currently reading Jurafsky and Martin's Speech and Language Processing on NLP. I suggest you to take a look specifically at Chapter 17: Information Extraction. All models for this task are devoted to the extraction of lexico-syntactic patterns. They describe supervised, semi-supervised and unsupervised approaches.