Let's check for continuity for the new proposed function. For convenience, let me call it $L_{\delta, c}$.

$$\lim_{a \to \delta^+}L_{\delta, c}(a)=c\delta$$ and $$\lim_{a \to \delta^-}L_{\delta, c}(a)=\frac{\delta^2}2$$

If we want the function to be continuous, we would want $c$ to take value $\frac{\delta}2$.

Now, let's check for diffentiability of $L_{\delta, \frac{\delta}2}$.

$$\lim_{h \to 0^+} \frac{L_{\delta, \frac{\delta}2}(\delta+h)-L_{\delta, \frac{\delta}2}(\delta)}{h}=\lim_{h \to 0^+} \frac{\frac{\delta(\delta+h)}2-\frac{\delta^2}2}{h}=\frac{\delta}2$$

$$\lim_{h \to 0^-} \frac{L_{\delta, \frac{\delta}2}(\delta+h)-L_{\delta, \frac{\delta}2}(\delta)}{h}=\lim_{h \to 0^-} \frac{\frac{(\delta+h)^2}2-\frac{\delta^2}2}{h}=\delta$$

which shows that we can't have $L_{\delta,c}$ that is differentiable.

Now, let's examine the differentiable of the original huber loss at $\delta$.

$$\lim_{a \to \delta^+}L_\delta (a)=\delta(\delta-\frac{\delta}2)=\lim_{a \to \delta^-}L_{\delta}(a)=L_{\delta}(\delta)$$

Also,

$$\lim_{h \to 0^+} \frac{L_{\delta}(\delta+h)-L_{\delta}(\delta)}{h}=\lim_{h \to 0^+} \frac{\delta(\frac\delta2+h)-\frac{\delta^2}2}{h}=\delta=\lim_{h \to 0^-} \frac{L_{\delta}(\delta+h)-L_{\delta}(\delta)}{h}.$$

The huber loss function is constructed such that it is differentiable.


How to calculate joint feature contribution for XGBoost Classifier in python?

I did some research and learnt about xgbfir package. It gives the joint contributions into an excel file. You can set the level of interaction with this. I wrote some code around it to generate a plot that solves the purpose.

If the package is not installed

pip install xgbfir

After the installation:

import xgbfir
from matplotlib import pyplot as plt

xgbfir.saveXgbFI(model, feature_names=X.columns, OutputXlsxFile='FI.xlsx')

joint_contrib = pd.read_excel('FI.xlsx')

xls = pd.ExcelFile('FI.xlsx')
df1 = pd.read_excel(xls, 'Interaction Depth 0')
df2 = pd.read_excel(xls, 'Interaction Depth 1')
df3 = pd.read_excel(xls, 'Interaction Depth 2')

frames = [df1, df2, df3]
joint_contrib = pd.concat(frames)

joint_contrib=joint_contrib.sort_values(by='Gain', ascending=True)
joint_contrib=joint_contrib.head(20)

height = joint_contrib['Gain']
bars = joint_contrib['Interaction']
y_pos = np.arange(len(bars))

plt.barh(y_pos, height)
plt.yticks(y_pos, bars)
plt.show()

This will give the top 20 feature interactions in terms of gain.

Thanks to Philip Cho who introduced me to xgbfir.

Follow the link for more information regarding xgbfir


Solution for TF-IDF Vectorization in Angular project?

Angular has nothing to do with TF-idf vectorization. Its a web framework, so if you python/java etc code is modularised and has the right workflow it will work.

Just google how to structure it in angular, its a HUGE topic to give an answer.


There is an Amazon Customer Review dataset which you can read more about here - the data is stored in S3 and freely available for anyone to use.

It contains over 160 million customer reviews (the README suggests 130 million, but more data must have been added since that was written) made available by Amazon - you can access them in either TSV or Parquet format.

There is also a blog post on using this data set in sentiment analysis here, which links to another post giving more technical detail. In summary, the blog looks at how sentiment correlates with the star rating in the reviews over time, across product categories, etc.


how to remove columns in Transformer function in Pipeline

I succeeded to get a satisfying solution. I post an entire working script. What do you think about it ? Especially the creation of an attribute (self.lstRemCols) not initialized in the init function ?

import pandas as pd
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.datasets import make_blobs

class myTransfo(object) :
  def __init__(self, colname):
    self.colname = colname

  def transform(self, x) :
    dat = x.copy()
    squared = dat.loc[:,self.colname]**2
    squared.name = "%s_sqre"%self.colname
    dat.loc[:,squared.name] = squared
    dat.loc[:, self.colname+'_2'] = dat[self.colname]
    return dat

  def fit(self, dat, y=None) :
    return self

class removeSameCols(object) :
  def __init__(self) :
    pass

  def _delSameCols(self, df) :
    cols = []
    for i in range(df.shape[1]) :
      for j in range(i+1, df.shape[1]) :
        if (df.iloc[:,i].dtype!='O') | (df.iloc[:,j].dtype!='O') :
          if np.array_equal(df.iloc[:,i],df.iloc[:,j]) :
            cols.append(df.columns[j])
    cols = list(set(cols))
    print( u'      - %s features to be removed'%len(cols) )
    return cols

  def transform(self, x) :
    dat = x.copy()
    lstcols = list(set(dat.columns) - set(self.lstRemCols))
    return dat.loc[:, lstcols]

  def fit(self, x, y=None) :
    dat = x.copy()
    self.lstRemCols = self._delSameCols(dat)
    return self

x, y = make_blobs(n_samples=300, n_features=5)
x_train = pd.DataFrame(x[:150,:], columns=['x1','x2','x3','x4','x5'])
x_test = pd.DataFrame(x[150:,:], columns=['x1','x2','x3','x4','x5'])

makePipe2 = Pipeline([('makeCols', myTransfo(colname="x2")),
                      ('remCols', removeSameCols())])
makePipe2.fit(x_train)
x_1 = makePipe2.transform(x_train)
# test if only same columns in x_train are removed.
x_test.x4 = x_test.x5
x_2 = makePipe2.transform(x_test)


Can I feed only images to simple CNN model without using label data?

To train the model, you need an objective to minimize. Very often, labels are used for this, for example, for an object recognition problem, to tell the model the kind of object in an image. You then train the CNN to encode features in the intermediate layers that are useful to solve this problem. The objective to minimize here has something to do with how often you identify the right object.

You can also train a model on an objective without labels, for example in an auto-encoder, where you use a CNN to find a representation of the input that has a lower dimension than the input, and then you typically use the same architecture but backwards to decode it back to the original input. The objective here is the similarity between the input and the output image.

A CNN trained this way, can be useful as a compression algorithm, because it encodes the image to some smaller representation in the middle layer of the auto-encoder, in such a way that you can rebuild the image to some level of accuracy.

So it's possible, but there is no reason to think that a CNN trained for an auto-encoder would be useful in an object recognition problem.


Loss value going down while accuracy remains constant?

Everything is fine, loss is going down, your model is learning. The problem is that you are using accuracy on a regression problem.

Accuracy should only be used for classification. It is a very common mistake to make when starting out. If you want another metric then you could use mean_absolute_error which, as the name suggest, tells you how large error your model makes on average.

The reason accuracy does not work is because it only works with exact matches where you have y_true == y_pred. In regression you are working with continuous values which makes this quite rare and will not have much to do with the performance of your model.

The reason you get any accuracy at all is likely because Keras does y_true == round(y_pred), rounding the model prediction. Otherwise accuracy would almost always be zero since the model will never get the same decimals. This question has a good answer by Esmailian that goes a bit more into details on this.


A 1 dimensional data set is "linearly separable" at any set of points between the data points. Kmeans isn't a nearest-neighbour type of clustering algorithm, it just divides the space so that each point belongs to its nearest cluster centre, so your data could have huge gaps between the source clusters, but the algorithm would be "wrong" if the clusters had different ranges.

For example, if you generate points from two clusters on the intervals (0,1) and (2,20), even though there's a gap between 1 and 2, the clustering algorithm will still assign a lot of the points from the higher source cluster to the lower cluster because they are nearer that cluster centre. Its distance to cluster centre for each points that counts, not distance to neighbours. Use another clustering algorithm (like nearest-neighbours) if you are expecting gaps between clusters.


Expected Lifetime: Churn Formula vs. Experience Data

The average lifetime is calculated simply by using the geometric distribution's mean formula. If probability of churn in any month (or simply percent churned in a month) is 0.2, then the expected life time of a customer is 1/0.2 = 5 months.
Here you are filtering the unchurned customer at the first step (filter(Cancelled == "TRUE")) and therefore you get the average lifetime of the churned customers in your sample rather than the average lifetime of any customer.


How to implement this CNN architecture in Keras

In this kind of situation, you can not use sequential API which is generally used in the architecture where you to stack layers on each other.

For this kind of problem use the functional API of keras. Based on references image tried to create architecture & as the author has used RESNET architecture in his paper. So, I have tweaked network according to it rather than replicating architecture give in the image.

input_img = keras.Input(shape = (224, 224, 3))

x1 = keras.layers.Conv2D(32, (3, 3), padding = 'same')(input_img)
x1 = keras.layers.BatchNormalization(axis = 3)(x1)
x1 = keras.layers.Activation('relu')(x1)

x2 = keras.layers.Conv2D(32, (3, 3), padding = 'same')(x1)
x2 = keras.layers.BatchNormalization(axis = 3)(x2)
x2 = keras.layers.Activation('relu')(x2)
x2 = keras.layers.Dropout(0.2)(x2)
x2 = keras.layers.Conv2D(32, (3, 3), padding = 'same')(x2)

merge_x2 = keras.layers.Add()([x1, x2])
x3 = keras.layers.BatchNormalization(axis = 3)(merge_x2)
x3 = keras.layers.Activation('relu')(x3)
x3 = keras.layers.Dropout(0.2)(x3)
x3 = keras.layers.Conv2D(64, (3, 3), padding = 'same')(x3)
x3 = keras.layers.BatchNormalization(axis = 3)(x3)
x3 = keras.layers.Activation('relu')(x3)
x3 = keras.layers.Dropout(0.2)(x3)
x3 = keras.layers.Conv2D(64, (3, 3), padding = 'same')(x3)

conv_merge_x2 = keras.layers.Conv2D(64, (3, 3), padding = 'same')(merge_x2)
merge_x3 = keras.layers.Add()([conv_merge_x2, x3])
x4 = keras.layers.BatchNormalization(axis = 3)(merge_x3)
x4 = keras.layers.Activation('relu')(x4)
x4 = keras.layers.Flatten()(x4)
x4 = keras.layers.Dense(2)(x4)
final = keras.layers.Activation('softmax')(x4)

model = keras.models.Model(input_img, final)
model.compile(loss = "categorical_crossentropy", optimizer = "adam", metrics = ["accuracy"])


In reference to Gamma Gamma spend model, is this in any way related to Pareto/NBD, BG/NBD or BG/BB model?

The Gamma-Gamma spend model assumes that the expenditure of a client is independent of the transaction behavior.

Therefore, we can use this in conjunction with any CLV model. Models like Pareto/NBD and BG/NBD only predict the transaction behavior and client death (stops being a customer) independent of the spend value.


Difference between class_weight and loss_weights in Keras

From Keras Team at GitHub:

  loss_weights parameter on compile is used to define how much each of
  your model output loss contributes to the final loss value ie. it
  weighs the model output losses. You could have a model with 2 outputs
  where one is the primary output and the other auxiliary. eg. 1. *
  primary + 0.3 * auxiliary. The default values for loss weights is 1.

  class_weight parameter on fit is used to weigh the importance of each
  sample based on the class they belong to, during training. This is
  typically used when you have an uneven distribution of samples per
  class.


How to assure the best cluster number for the given instances?

No, you can't exclude that there are some clusters in this data.

Maybe they:
- can not scored by CH and Silhouette
- can not be found by k-means (but by other algorithms)
- require better preprocessing
- were destroyed by the preprocessing you applied
- exist only in subspaces

But I would agree that none of the k-means results results you got was good on Silhouette. Which is not surprising given 128 variables.


As you and @gunes pointed out in this post, your formula are correct, but hyperparameters $\alpha$ and $iterations$ were not well adjusted.


Problem of finding best combination of features when desired feature is feature some_feature_A/some_feature_B

In theory, I think a deep neural net might be able to find features that are the product of two other columns.  There exist some nice mathematical results which guarantee the ability of a neural net (with certain activation functions) to approximate any function, so there's no theoretical reason why a neural net couldn't compute the division function $f(x, y) = \frac{x}{y}$.  That being said, it may be difficult to achieve in practice without a little preprocessing.

If you want to give it a try, I would suggest adding additional features obtained by taking the logarithm of any numerical features in the dataset.  This might make it easier for the network for learn features that are products of other features (since $\log{(x * y)} = \log{x} + \log{y}$ and $\log{x / y} = \log{x} - \log{y}$).


Multi-Class Neural Networks | different features

It is essential for all input patterns to have the same number of features. The reason is that each input feature is connected to specified neurons and they have specified trained weights. A typical solution is to resize your input by reserving its aspect ratio. You should consider that if you want to have good accuracy, you have to have such an operation while training the network too. Even in convolutional networks, it is essential to have the same input size. Although the convolutional layers won't bother you, the connection of fully connected layers and flattened layers used after convolutional layers will have a problem if you don't use correct input shape; the dimensions won't match.

About recent studies, I have not seen yet, but a typical solution can be employing PCA and using let's say its top-10 features as the input of a fully connected network, although for input patterns with a huge difference in the number of input dimensions I guess it is not logical.


Work with a Embedding Matrix. The Embedding Matrix Dimension = all of your unique Tokens x Vector Dimension.

Your Embedding Layer should have the Dimension WordVektor x length of your Text.

Watch this: watch this video Part 16 and Part 17. Easy explanation and short videos.


PCA ? after the transformed data, are they still same with original data, (if maintain same dimensional)

As you can see from your pics there is a change.
From an algebra point of view, PCA is a base change. The interest of the algo is how the target base is designed: it is designed as to explain variance in the dataset. Dimension reduction is achieved trough removing some base vector with the least associated variance. In other words : PCA is a base change to an interesting base in term of variance explained, which might be followed by a projection on a sub-base.
When you remove the dimension reduction component you still have a base change. Which in the general case amount to some rotation, translastion, scaling, but no significant change in the underlying data. It seems to be what happen in your second graph : a rotation by 45° plus some scaling.


Splitting image dataset with few subjects but many data

I think you're hitting on the fact that by training and testing on the same subjects your model is not going to be able to generalize to new subjects very well. If you're only interested predicting emotions for these subjects, you are taking the right approach. However, if you want to generalize your model to new, unseen subjects, you should split your training and testing sets such that subjects in the training set are not in the test set, and vice versa to get a more accurate test score. Most likely it will not perform as well and you should consider collecting more data on new subjects if possible.

Here is a similar question on Stack Exchange.


.fit --> You can give Train data and Validation data as well so Train data is used for Training and Validation data is used for validating the Trained model at the end of each epoch.
.predict --> It takes the only X and gives the output of the model.
.evaluate --> It will take X, Y and will evaluate similar to an above evaluation in .fit but after training total model we will use this.
You can write
.evaluate = loss_function(.predict, y_true)
score(.predict, y_true)


How to handle data collecting bias in machine model training

In the world of Big data, the data is already there and there are more than enough ML models currently but constantly there are improvements and applications, the main reason for that is how the features are defined from the raw data.

In your case, introduce a new feature as penalty for the bias present in the data. For ex. in the presentation bias, high penalty is given to the top items and low penalty is given to the items farther down. similarly, introduce weight feature with less weight for men products when most of the products are male products and vice versa.

ML still needs human intuition in the form of features ;-)


Why does joint embedding of word and images work?

The model is more complex than point-wise multiplication of word and image embedding. It is a single neural network model thus backpropagation can improve the weights throughout the entire model. The training signal will adjust all layers to do better at the task, in this case the task appears to be question answering.
The figure is only showing the forward pass.


Calculating derivative of error at point x with respects to weight w_j

Based on chain rule we have:

$$\frac{\partial \mbox{log}(f(x))}{\partial x}=\frac{\partial \mbox{log}(z)}{\partial z}\Bigr|_{z=f(x)}\frac{\partial f(x)}{\partial x}=\frac{1}{z}\Bigr|_{z=f(x)}\frac{\partial f(x)}{\partial x}=\frac{1}{f(x)}\frac{\partial f(x)}{\partial x}$$

More close to the notations used in the question:

$$\frac{\partial \mbox{log}(f_{\mathbf{w}}(\mathbf{x}))}{\partial w_j}=\frac{\partial \mbox{log}(z)}{\partial z}\Bigr|_{z=f_{\mathbf{w}}(\mathbf{x})}\frac{\partial f_{\mathbf{w}}(\mathbf{x})}{\partial w_j}=\frac{1}{z}\Bigr|_{z=f_{\mathbf{w}}(\mathbf{x})}\frac{\partial f_{\mathbf{w}}(\mathbf{x})}{\partial w_j}=\frac{1}{f_{\mathbf{w}}(\mathbf{x})}\frac{\partial f_{\mathbf{w}}(\mathbf{x})}{\partial w_j}$$

Now by setting $f_{\mathbf{w}}(\mathbf{x})=\hat{y}$, or $f_\mathbf{w}(\mathbf{x})=1-\hat{y}$, line 2 to 3 follows. For example for $f_\mathbf{w}(\mathbf{x})=1-\hat{y}$:

$$\frac{\partial \mbox{log}(1-\hat{y})}{\partial w_j}
=\frac{\partial \mbox{log}(z)}{\partial z}\Bigr|_{z=1-\hat{y}}
\frac{\partial (1-\hat{y})}{\partial w_j}=\frac{1}{z}\Bigr|_{z=1-\hat{y}}\frac{\partial (1-\hat{y})}{\partial w_j}=\frac{1}{1-\hat{y}}\frac{\partial (1-\hat{y})}{\partial w_j}$$

Note that input $\mathbf{x}$ to the network is constant w.r.t. changes in  $\mathbf{w}$. Also $x|_{x=y}$ means replace $x$ with $y$.


In order to show a correlation between X and Y they should be aggregated over the same time period. Otherwise, there there may be seasonality or other features in Y that may have been averaged out in X. Is it possible re-index X to get monthly aggregates? I would not sub-sample Y. Find a common time period for both X and Y to make your comparisons.


finding optimal solution $w$ and classification accuracy

The best that we can do is to classify $5$ points correctly and sacrifice one point.

We want to classify $-1,0,1$ as negative and $-3,-2$ as positivie.  (we have to sacrifice $3$).

The boundary with maximum margin would be in the middle of $-2$ and $-1$. That is $-\frac32$.

$$sign(-(2x+3))=sign(-2x-3).$$


How many images are generated when ImageDataGenerator is used, and when data augmentation is included as a part of the model?

In one epoch - It's the number of images in your Directory or the DataFrame
 In case of a custom Generator. It will be batch_size * steps_per_epoch
You may check this with any of these approaches -

Check the shape of prediction on train model.predict(traindata).shape
Save the images into a dir by using save_to_dir='/content/train_data'
Write a callback for batch_end
Get the steps_per_epoch in the training print statement i.e.
125/125 [==============================] - 50s 392ms/step - loss: 0.6637 - accuracy: 0.6000


How to avoid resampling part of pipeline on test data (imblearn package, SMOTE)

Checking the code on GitHub, it seems that a 'transform' performing the resampling of data is not applied (cf : 'if hasattr(transform, "fit_sample"'): pass). So, from my understanding only modifications that do not resample the test set will be applied.


Could principle component decomposed coordinates value be correlated to each other?

PCs are calculated based on eigenvectors (e1 nad e2 in your case) of correlation or covariance matrix. Eigenvectors are orthogonal, so PCs are uncorrelated. If you would like to check the proof, you can find it on page 432 in Applied Multivariate Statistical Analysis 6th edition written by Richard A. Johnson and Dean W. Wichern.


Predicting hardware failures with limited data

Can you use a proxy for hardware failure? Something like a state in which failure becomes likely? Then you could try to predict this state which should be more common and use a fudge factor to get to your failures.


Reducing features to 2 by PCA confusion Matlab

After obtaining the principal component, you should compute $XW$ rather than $X^TXW$ where $W$ is the principal component.
Remark about computation efficiency: As you sort the eigenvalues, you could have obtained the corresponding permutation indices.


As Konstantin has pointed, R performs all its computation in the system's memory i.e. RAM. Hence, RAM capacity is a very important constraint for computation intensive operations in R. Overcoming this constraint, data is being stored these days in HDFS systems, where data isn't loaded onto memory and program is run instead, program goes to the data and performs the operations, thus overcoming the memory constraints.  RHadoop (https://github.com/RevolutionAnalytics/RHadoop/wiki) is the connector you are looking for.

Coming to the impact on algorithms which are computation intensive, Random Forests/Decision Trees/Ensemble methods on a considerable amount of data (minimum 50,000 observations in my experience) take up a lot of memory and are considerably slow. To speed up the process, parallelization is the way to go and parallelization is inherently available in Hadoop! That's where, Hadoop is really efficient.

So, if you are going for ensemble methods which are compute intensive and are slow, you would want to try out on the HDFS system which gives a considerable performance improvement.


How to implement hierarchical labeling classification?

For this problem, you need to develop three models:
Model 1- for two main categories
Model 2- for sub-category A
Model 3- For sub-category B

So when you want to predict the result for an unseen data, first you use the Model 1, to find the main category. Based on the prediction and by using an if-else statement, you decide to perform another prediction using Model 2 or Model 3.
Consequently, the Model 1 is a binary classification task, but Model 2&3 are mutli-class classification task. Your networks may look like this:
Model 1:
model1 = Sequential()
model1.add(Dense(60, input_dim=X.shape[1], activation='relu'))
...
model1.add(Dense(1, activation='sigmoid'))
model1.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

Model 2 & 3:
model2 = Sequential()
model2.add(Dense(8, input_dim=X.shape[1], activation='relu'))
...
model2.add(Dense(n_subcats, activation='softmax'))
model2.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Just don't forget to use correct y (label) for main and sub-categories for each model.


how can i compare two classification algorithms?

Performance of a model can be judged along several dimensions:

Accuracy - Is the train-test validation performing well?

Overfitting - Is the difference between training score and the validation score minimal?

Efficiency - Is the model light-weight, does it compute and calculate fast?

Complexity - Is the model easy to explain, does it use minimal transformations?

Deployability - Will I be able to deploy and use the model easily?

Rather than from a pure statistics perspective (which is just dimension 1 & 2) judge the models from all perspecties. If they still perform equally well than they are in fact equal (not surprising for two similiar models performed on the same data set).


In the general case this is impossible because standard learning algorithms require all the instances at once in order to compute the model. They usually work by making calculations based on the set of instances, then only the results of these calculations are stored as a model (for instance a linear regression classifier stores only the final weights which are needed to make a prediction).

It is however possible to train a model incrementally:

Very often this is done by simply adding the new instances to the original training set, then re-train a new model on the full set of instances (new and old ones together).
There are also actual incremental learning algorithms, including adaptations of some common learning methods. However this is not very common, I'm not aware of these being implemented in the most standard ML libraries that I know of.


Does random forest re-use features at each node when generating a decision tree?

If the feature is discrete/categorial/binary, it won't use the already used feature to split at further levels in any given branch of the tree. But key thing to note here is that the same feature can be used in a different branch of the tree at a different level.

In the above diagram, we can see that same 'size' feature has been used at two levels 'level 1' and 'level 2', but in different branches of the tree.

On the other hand, if the variable is a continuous value, it uses threshold splits at each level and in this case, same feature can be used multiple times in any given branch of the decision tree.

Since random forests is an ensemble of decision trees, the same theory applies to Random Forests as well.

Reference: http://www.cse.msu.edu/~cse802/DecisionTrees.pdf


Should I rescale losses before combining them for multitask learning?

Check this.

Under the heading Multi-tasks losses they have mentioned how they average losses from two different tasks. They do a weighted average depending on their use case.


Fitting a model as a significance testing alternative

It's probably a quite subjective matter, here is my opinion:

I would say that this method makes a lot of sense if the ML model uses a number of features, something which would be hard to factor properly in a standard statistical test (especially if some kind of feature selection is involved).
Also in favor of the method, statistical significance tests are usually interpreted in a binary way (typically p-value higher or lower than 0.05) even though the underlying p-value is continuous: there's very little different between a p-value of 0.049 and 0.51 but the two will be interpreted in a completely opposite way.
However I agree with you that statistical significance tests are much more grounded in statistical theory, with a clear interpretation especially about levels of confidence.

Overall I can see an interest of using such a method from a practical viewpoint, but it is indeed limited in terms of theoretical interpretation and reliability.

Disclaimer: I'm not a statistician, I might be biased towards practical ML methods ;)


Evaluating the result of topic modeling in a way that time matters

I think the issues comes from the fact that the item you are looking at (the word "sleeping") is a rare event, so the probability that you observe one is about 0. Technically, it is called a Poisson process.

One of the way to circumvent it is what you did: aggregate on a period (in your case 6 month) so that the number of events get significant.

You do not need to cut you observation time in 6 month period though, you may use moving averages: period 1-6, 2-7, 3-8, 4-9,... If there is a temporal pattern it will be more visible.

An other way is to use cumulative data: the number of words up-to time t. If you get a logistic "S" shaped curve, then you are on an important thing.

Basically you want to see if a certain word (sleeping) is more frequent in the CI sub-population than the nonCI sub-population. You can use the t-test (unpaired observations, unequal sample size, equal variance), to check if the count of the word is significantly different in the two sub-populations.
And you can do this over time.

More adapted tests can be found in this paper treating of the similar problem of testing if documents have the same author. Here again, nothing prevent to make a test per time period, cumulative or not.


How to get a output of a hidden layer of a single-layer LSTM

You'll definitely want to name the layer you want to observe first (otherwise you'll be doing guesswork with the sequentially generated layer names):

model <- keras_model_sequential() %>%
  layer_lstm(units = 64,
             dropout = 0.2,
             input_shape = c(seqlength, length(chars)),
             name = "lstm") %>%
  ...

The rest is pretty straightforward:

lstm_layer_model <- keras_model(
  inputs = model$input,
  outputs = get_layer(model, "lstm")$output
)
lstm_output <- predict(lstm_layer_model, new_data)

So if you want to know the LSTM neuron values for a given case, define that case's features as new_data and run the last line.


train NER using NLTK with custom corpora (non-english) must use StanfordNER?

It's true that the nltk book doesn't seem clear about this. Traditionally NER models are trained with Conditional Random Fields so I searched for "nltk crf" and found this SO question which points to this detailed example for NER.
To answer your questions:

nltk itself doesn't appear to propose a CRF model, the example above relies on an interface with CRFSuite (as mentioned in the SO question). It's probably possible to use other interfaces such as StanfordNER.
The tricky part is to define your own features, i.e. the conditions that the model uses as features at every step in the sequence. This is how you can specify any kind of specific "grammar rule" you want the model to use.

The example above looks complete but I didn't test anything.


This paper might be useful....

https://arxiv.org/abs/1801.06146


print a specific column with a condition using pandas

In Pandas slice notation one must first indicate the condition to filter on and only eventually the column to select: in particular for the example at hand we have:

df[df['CLASS']==1]['CONTENT']


Seaborn uses inter-quartile range to detect the outliers. What you need to do is to reproduce the same function in the column you want to drop the outliers. It's quite easy to do in Pandas.

If we assume that your dataframe is called df and the column you want to filter based AVG, then

Q1 = df['AVG'].quantile(0.25)
Q3 = df['AVG'].quantile(0.75)
IQR = Q3 - Q1    #IQR is interquartile range.

filter = (df['AVG'] >= Q1 - 1.5 * IQR) & (df['AVG'] <= Q3 + 1.5 *IQR)
df.loc[filter]


The question mixes two different notions: models (or algorithms) and accuracy. Let me clarify them.

Model (or Algorithm) is a classification technique and 'Accuracy' is one of the ways to evaluate the performance of the models.

You can choose any models(Naive bayes, SVM or other deep learning techniques) to implement your classifier. They are independent of 'Accuracy' or 'F1' or any other measures by which you want to test the performance.

At first, you shall pre-process the text (remove stopwords, punctuation, etc.) , and pre-processing is a choice that says how the data should look like before getting into the model. They do influence the model's performance, but not to a great extent when done right. Usually, pre-processing is applied on both the train and test set.

Model performance: Once you implement you model, you may want to see how well it generalises (performamce on unseen data). So, you shall split the dataset into two halves: training and test set. (Usually most of the authors split into 3 portions: training set, validation set(to avoid over-fitting) and test set). You shall train the model with training set, and the test set is used to evaluate the performance of the model.

Model evaluation: Once the model is trained on training data, predict the labels on the test set. So, you have two set of labels on test set: 1: ground truth (the actual labels indicated by the test set) and 2: predicted labels (the labels predicted by the model).
Now, use the evaluation metric of your choice (Let's assume that you want to choose 'accuracy' as evaluation metric).
Accuracy can simply be calculates as: (#Number of correctly predicted samples / #Total number of samples) * 100 . Where #Number of correctly predicted samples is the count of samples for which the ground truth label and predicted labels are the same.


Collaborative filtering when multiple items are rated multiple times by same user

I don't think there is any academic work on the subject, at least that I know of.

One simple way of using that data would be to use the mean of the ratings or other average like measures such as a moving average, a time weighted average, the median, etc.

But this approach is probably not exactly what you're looking for.

Try to look at collaborative filtering approaches with temporal dynamics, there might be something interesting for you.


How to generate abbrevations from shortend words in medical records

First a warning: is your data anonymized? Even if it is, be extremely careful because medical history is super-sensitive personal information. There are legal requirements about how to handle this kind of data and what you can or can't do with it.

As far as I know there's no python API for cTakes or Metamap. Anyway I guess that such an API would boil down to a system call, so it wouldn't be very good.

As an alternative you could implement your own system using the UMLS Metathesaurus, which is a massive list of medical terms grouped by concept (MetaMap extracts UMLS terms/concepts).


What is the meaning of spherical dataset?

In this case, a picture is a worth a thousand words. They literally mean data whose distribution on X,Y is roughly a sphere. Different clustering algorithms work better on different distributions. For example, K means does poorly on the arrangement in the first two rows but OK on the last row.


Is TensorFlow a complete Machine Learning Library?

This is a big oversimplification, but there are essentially two types of machine learning libraries available today:

Deep learning (CNN,RNN, fully connected nets, linear models)
Everything else (SVM, GBMs, Random Forests, Naive Bayes, K-NN, etc)

The reason for this is that deep learning is much more computationally intensive than other more traditional training methods, and therefore requires intense specialization of the library (e.g., using a GPU and distributed capabilities). If you're using Python and are looking for a package with the greatest breadth of algorithms, try scikit-learn. In reality, if you want to use deep learning and more traditional methods you'll need to use more than one library. There is no "complete" package.


When/how should I use the validation set for hyper-parameter sweeps for neural networks?

Let's take a step back and look at why we make these splits:

  Model selection: estimating the performance of different models in order
  to choose the best one.

  Model assessment: having chosen a final model, estimating its predic-
  tion error (generalization error) on new data.

(Source: "The Elements of Statistical Learning - Data Mining, Inference, and Prediction", Hastie et al)

For model selection you use the validation set and for model assessment you use the test set.

Accordingly, a straight forward approach would be this:

Split data into train/valid/test sets
Train models on train dataset
Compare models on valid dataset
Repeat steps 2 and 3 until you meet your personal stopping criterion (e.g. performance sufficiently)
Select your final model and retrain it on the train and valid dataset
Evaluate the performance of your selected and retrained model on the test dataset

Here is an example with an SVM taken from "Introduction to Machine Learning with Python" by Mueller and Guido:

from sklearn.svm import SVC

# split data into train+validation set and test set
X_trainval, X_test, y_trainval, y_test = train_test_split(
iris.data, iris.target, random_state=0)

# split train+validation set into training and validation sets
X_train, X_valid, y_train, y_valid = train_test_split(
X_trainval, y_trainval, random_state=1)
print("Size of training set: {} size of validation set: {} size of test set:"
" {}\n".format(X_train.shape[0], X_valid.shape[0], X_test.shape[0]))
best_score = 0
for gamma in [0.001, 0.01, 0.1, 1, 10, 100]:
for C in [0.001, 0.01, 0.1, 1, 10, 100]:

# for each combination of parameters, train an SVC
svm = SVC(gamma=gamma, C=C)
svm.fit(X_train, y_train)

# evaluate the SVC on the test set
score = svm.score(X_valid, y_valid)

# if we got a better score, store the score and parameters
if score > best_score:
best_score = score
best_parameters = {'C': C, 'gamma': gamma}

# rebuild a model on the combined training and validation set,
# and evaluate it on the test set
svm = SVC(**best_parameters)
svm.fit(X_trainval, y_trainval)
test_score = svm.score(X_test, y_test)
print("Best score on validation set: {:.2f}".format(best_score))
print("Best parameters: ", best_parameters)
print("Test set score with best parameters: {:.2f}".format(test_score))


How can I get probabilities of next word with ELMO?

ELMO is not a language model. As specified in the paper:

  We introduce a new type of deep contextualized word representation that models both (1)
  complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses
  vary across linguistic contexts (i.e., to model
  polysemy).

ELMO representations (embeddings) are indeed calculated by looking at the context of the word. But the output is a fixed size vector that contains/represents the contextualised meaning of that word. This can be compared to other word embeddings such as word2vec, glove or fasttext but with the novelty of having different representations of the same word based on their context within a sentence.

I suggest you do some reading about the differences about language models and word representations.


PCA on MNIST dataset results in very poor performance

There are a few thing that you may be doing, but worth mentioning.

Did you "whiten" your data---mean center and normalize by the standard deviation? This step may be implementation-dependent.
http://deeplearning.stanford.edu/wiki/index.php/PCA

Did you check to see how much variablility was explained by the number of chosen parameters? For images +80% is common.

Make sure you understand the implementation of the PCA you are using. It always confuses me by the terminology different implementations uses, e.g. score, rotation, etc. I suggest following a tutorial and matching their output with your implementation to get a deeper understanding.
http://sebastianraschka.com/Articles/2015_pca_in_3_steps.html

Your model is now built on transformed data. To predict make sure you are using these same transformations on the new input data.


How do I find a repeating pattern of unknown length and start within a string

From your description, it sounds like autocorrelation could be a good initial step to identify the frequency. Autocorrelation is basically the product of the sequence with itself shifted at 0, 1, 2, 3, and so forth. If a client tends to pay monthly, you will see the first positive bump around 4, because that is where each payment aligns with the payment after it. Note that this will work irrespective of the start of the sequence, and is even resilient to some noise. Even Excel seems to have an autocorrelation function.

Of course, you could refine this further with other steps, and/or try this on different windows (in case the patter changes over time).


I think you have a few options:

If you have a pre-set rule to exclude outliers, such as a hard-threshold at 100 which you know the data shouldn't exceed, then something as simple as x = [e for e in x if e < 100] will do.
If you have a parametric belief, such as any observation that falls beyond so many standard deviations from mean, or quartiles, are outliers; then you can implement the other answers that have been mentioned.
Else, you can go for a clustering approach. Here I believe your first shot should be a k-means clustering. This is super easy to build and interpret. See my code below.

x = [0,14,0,6,102,0,0]
from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=2).fit(np.array(x).reshape(-1, 1))

#First cluster:
np.array(x)[np.where(kmeans.labels_ == 0)]

#Second cluster (outliers):
np.array(x)[np.where(kmeans.labels_ == 1)]
K-means is known to be sensitive to outliers, hence a more robust method such as MeanShift, which you tried, is a good rival to k-means. I would run both, and stick with the result that makes better sense to me.

Hope this helps!


How flexible is the link between objective function and output layer activation function?

It is not so much which activation function which you use which determines which loss funtion you should use, but rather what the interpretation you have of the output is.

If the output is supposed to be a probability, then log-loss is the way to go.

If the output is a generic value then mean squared error is default way to go. So for example, if your output was a grey scale pixel with grey-scale labelled by a number from 0 to 1, it might make sense to use a sigmoid activation function with a mean squared error objective function.


Look back, I don't know look back as an hyper parameter, but in LSTM when you trying to predict the next step you need to arrange your data by "looking back" certain time steps to prepare the data set for training, for example, suppose you want to estimate the next value of an episode that happens every time t. You need to re-arrange you data in a shape like:
{t1, t2, t3} -> t4
{t2, t3, t4] -> t5
{t3, t4, t5} -> t6
The net will learn this and will be able to predict tx based on previous time steps.

Batch size (is not referred to LSTM only), roughly is how much samples will be trained per single step, as bigger the batch size is the faster the training is but more memory is needed.
In a GPU is better to have bigger batch sizes because copying the values from GPU to memory is slow.

LSTM units, refers to how much "smart" neurons you will have. This is highly dependent on your dataset, usually you determine this depending on your vector dimensions.

No. of Epochs, how much times the algorithm will run to approximate the observations. Usually to much epochs will overfit your model and to little will end up in an under fitted one.


Validation loss is less than training loss by 5 units. How this result is interpreted?

The key here is dropout. When a model trains with dropout, only a percentage of the total weights (in your case 50%) are used in predictions, which tends to lower the prediction accuracy. When the validation data is tested, dropout is turned off, and the validation loss tends to be a lot lower. see this answer to a similar question for more info.


Understanding the effect of num_words of Tokenizer in Keras

The problem is with the way things are documented. Check this link:
https://stackoverflow.com/questions/46202519/keras-tokenizer-num-words-doesnt-seem-to-work


Detect the time at which deviation occurs in time series data

AnomalyDetection is an open-source R package to detect anomalies
  which is robust, from a statistical standpoint, in the presence of
  seasonality and an underlying trend.

A blogpost that introduces the package can be found here and more formal paper can be found here.


Difference between a log scale and linear scale : np.random.rand()

Let us assume you want to do hyperparameteroptimization with a hyperparameter $h\in[0,1]$. Let us additionally assume that we want to test hundred possible values for the parameter.

If you choose a linear scale your parameter values will be uniformly selected form $[0,10^4]$. It will be very unlikely that you will small parameter values smaller than $10^{-4}$, because it is very unlikely to choose a number from this interval. The probability for randomly choosing a number from this interval is given by the following fraction

$$\dfrac{10^{-4}}{10^4}=10^{-8}.$$

Now if we choose our random numbers from the intervall $[-5,5]$ and use this as the power of the number $10$ we will stretch the interval onto an exponential scale which will make it more likely that we have numbers in $[0,10^{-4}]$ because the chance of choosing  a negative number for the exponent is much more likely. The probability will be $1/10$, because the wanted interval from $[-5,-4]$ has a width of $1$ and the whole interval from $[-5,5]$ has the width $10$.


Filtering Pandas Dataframe Results without saving to a variable first

If you want to filter Pandas Series "on the fly", you can use .loc[] indexer in conjunction with the callable method (for example using lambda function).

Demo:

In [8]: df.var()
Out[8]:
A    2.5
B    0.0
dtype: float64

In [9]: df.var().loc[lambda ser: ser!=0]
Out[9]:
A    2.5
dtype: float64

In [10]: df.var().loc[lambda ser: ser>3]
Out[10]: Series([], dtype: float64)

If you want to filter a DataFrame, then you can use DF.query(...) method:

In [11]: df
Out[11]:
   A  B
0  1  0
1  2  0
2  3  0
3  4  0
4  5  0

In [12]: df.query("A >= 3")
Out[12]:
   A  B
2  3  0
3  4  0
4  5  0


clustering with heterogeneous (quantitative and qualitative)data?

K-means algorithm cannot be directly used for data with both numerical and categorical values because of its objective function. K-means uses Euclidean distance, which cannot be defined for categorical data.

However, there is a simple method which combines Euclidean distance and Hamming distance for the finding the similarity between instances that include both numerical and categorical features. In this method Euclidean distance is used as a metric for computing the similarity between numerical data and hamming distance for computing the similarity between categorical data. Both of them with the appropirate weights form an objective function that can handle mixed data.

For more information check the relevant paper

https://grid.cs.gsu.edu/~wkim/index_files/papers/kprototype.pdf


Integration of Sentiment analysis in CRM

After a lot of searching, I got one paper on the integration of opinion mining in CRM, which helps to get an idea about the process involved.


Algorithm selection for controlling a model vehicle

Please, do not use ANN, because what you are looking for actually is SLAM method. It is a set of algorithms working together to localize your ship in an environment and mapping it simultaneously.

You can find an introduction on the Wikipedia:

https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping

This is what autonomous cars are using.


How to define a distance measure between two IP addresses?

If I understood them correctly, both Jeremy and Edmund's (first) solutions are the same, namely, plain euclidean distance in a 4-dimensional space of IP addresses.BTW, I think a very fast alternative to euclidean distance would be to calculate a hamming distance bit-wise.

Edmund's first update would be better than his second. The reason is simple to state: his 2nd update tries to define a distance measure by considering a non-linear function of the coordinates of a 4D vector. That however will most likely destroy the key properties that it needs to satisfy in order to be a metric, namely

Injectivity: $d(IP_1,IP_2)=0 \iff IP_1=IP_2$,
Symmetry: $d(IP_1,IP_2)=d(IP_2,IP_1)$, and
Triangular inequality: $d(IP_1,IP_2)\leq d(IP_1,IP_3)+d(IP_3,IP_2)\,\forall IP_3$.

The latter is key for later interpreting small distances as close points in IP space. One would need a linear (in the coordinates) distance function. However, simple euclidean distance is not enough as you saw.

Physics (well, differential geometry actually) could lead to a nice solution to this problem: define a metric tensor $g$. In plain english, give weights to each pair of coordinates, take each pair difference, square it and multiply it by its weight, and then add those products. Take the square root of that sum and define it as your distance.

For the sake of simplicity, one could start trying with a diagonal metric tensor.

  Example: Say you take $g=\begin{pmatrix}1000 &0 &0 &0 \\0 &100&0&0\\0&0&10&0\\0&0&0&1\end{pmatrix}$ $IP_1=(x_1,x_2,x_3,x_4)$ and
  $IP_2=(y_1,y_2,y_3,y_4)$. Then the square of the distance is given by
  $$d(IP_1,IP_2)^2=1000*(x_1-y_1)^2+100*(x_2-y_2)^2+\\ \,+10*(x_3-y_3)^2+1*(x_4-y_4)^2$$
  For $IP_1=192.168.1.1,\,IP_2=192.168.1.2$ the distance is clearly 1.
  However, for $192.168.1.1$ and $191.168.1.1$ the distance is
  $\sqrt{1000}\approx 32$

Eventually you could play around with different weights and set a kind of normalization where you could fix the value of the maximal distance $d(0.0.0.0,FF.FF.FF.FF)$.

Furthermore, this set up allows for more complex descriptions of your data where the relevant distance would contain "cross-products" of coordinates like say $g_{13}*(x_1-y_1)*(x_3-y_3)$.

EDIT: While this would be a better "weighting" method than using those other I addressed, I realize now it is actualy meaningless: As Anony-Mousse and Phillip mention, IP are indeed 32 dimensional. This means in particular that giving the same weight to all bits in say the 2nd group is in general not sound: One bit could be part of the netmask while the other not. See Anony-Mousse answer for additional objections.


How to plot number of Trees and OOBs score with Grid Search

To plot feature importance using gridsearch use:

x= X_train_v1.columns,y= rf_grid_search_v1.best_estimator_.feature_importances_


Implementing simple linear regression using a neural network

Your code works perfectly. The only problem is that the learning of the parameters is not finished. If you try with 10000 epochs this will works but this is way too much for this problem. As a matter of fact you can see that the loss is diminishing very slowly.

Solution : Increase the learning rate.

I set the batch size to one because this penalize the convergence speed here. Increasing the batch size is useful when you need to avoid overfitting. Here you want to overfit your data. Moreover I choose to use a simpler update rule with the SGD optimizer. With those changes, you will see that only 4 epochs are necessary to fit perfectly your data.

from pylab import *
from keras.models import Sequential
from keras.layers import Dense
from keras import optimizers

#Generate dummy data
data = data = linspace(1,2,100).reshape(-1,1)
y = data*5

#Define the model
def baseline_model():
   model = Sequential()
   model.add(Dense(1, activation = 'linear', input_dim = 1))
   sgd = optimizers.SGD(lr=0.2)
   model.compile(optimizer = sgd, loss = 'mean_squared_error', metrics = ['accuracy'])
   return model

#Use the model
regr = baseline_model()
regr.fit(data,y,epochs = 4,batch_size = 1)
plot(data, regr.predict(data), 'b', data,y, 'k.')


99% validation accuracy but 0% prediction results (UNET Architecture)

There is no "mismatch" of accuracy. Your problem is that you have an image segmentation problem where 99% of the pixels should be zero. So getting 99% accuracy is trivially easy. A model that predicts just blank output images would score roughly the same as your network has so far. Your accuracy metric is not meaningful.

The low Dice coefficient score gives you a better idea of what is going on. That covers positive case matches only, and a good score would be close to 1.0. The low score shows that the network is not focussing on getting positive pixels correct. Instead the network has learned to predict close to zero everywhere, because that minimises the loss metric to a first approximation very well.

How to fix things?

First, stop reporting accuracy. This metric is misleading you, and you need to find another so that you can assess your model fairly. As you already have Dice, you could just drop accuracy and use Dice instead. Alternatively, you could use a weighted accuracy, inversely proportional to mean number of pixels in each class of your training data. Looking at Keras metrics options, you will probably want to add another custom metric for this. You have already done this with the dice_coef function, so that should not be a problem.

Second, train your network as it is for longer. Your example has only one epoch. Depending on how many training examples you have, this is probably too low for an image processing network. Try increasing the number in a geometric advancement - 3, 10, 30, 100, 300, 1000 - using the new metric to see if you get any improvement during training.

You could also try altering the cost function to weight it in favour of positive pixels. Keras' fit function has a class_weight parameter for this purpose, you could set it e.g. to class_weight = {0:1, 1:20}


Attention using Context Vector: Hierarchical Attention Networks for Document Classification

I can not completely agree with this previous answer.

The context vector $u_w$ is not computed by the Eq. 5 and 6.
As stated in the paper, $u_w$ is randomly initialised and learned during the training.
Instead, it is $u_{it}$ that is computed by Eq. 5 and 6, using the (again) random initialised  $W_w$ and $b_w$.


Using Discretization from Training Set on Test Set in R

You need to save the breaks/endpoints when you perform the bucketing/discretization on the training set as a (named) vector. The same breaks/endpoints can be then re-used on test set.

The I am giving below sample code using cut function in base R to keep the answer more generally applicable (rather than giving answer specific to bnlearn package). Note the way output of quantile function is saved in vector breaks_to_use in line 3 below and re-used while applying cut on test data. You can (hopefully) do the same using the breaks argument in discretize function.

training_data <- runif(100)
test_data <- runif(100)
breaks_to_use <- quantile(training_data, seq(0, 1, 0.25))
discretized_training_data <- cut(training_data, breaks = breaks_to_use)
discretized_test_data <- cut(test_data, breaks = breaks_to_use)


How can generative models be used in machine learning classification applications?

You will find an explanation on Wikipedia. But let us sum up:

Given your Data $D$ you are interested in target values $Y$, your classifications. Discriminative models are, like your said, the straight forward way of modelling the correspondence of your target given your data, $P(Y|D)$. The generative model, on the other hand, calculates $P(D|Y)$.

Let us consider the example of e-mail spam filtering. We have a set of reference e-mails and a label for each mail, which indicates whether it is spam. If we now, for example, look at Naive Bayes, we can see it utilizes the Bayes Formula to calculate the posterior estimate $P(Y|X) = \frac{P(X|Y)P(X)}{P(Y)}$. Opposed to other Bayesian Inference methods, Naive Bayes does not hold $P(X)$ as a model, but $P(X|Y)$, which can be modelled by our reference e-mail set. In this domain, all e-mails can be modelled as equally distributed, thus making $P(X)$ uninteresting. $P(Y)$ can also be approximated by the reference set (only looking at the labels).

This way, we can generate probabilities for our label $Y$, which is binary in our case. Now we can apply the maximum a posteriori approach: We calculate $P(Y=true|X)$ and $P(Y=false|X)$ and see which of both is more likely. The most likely option will be selected as our classification.


If I was in your situation, I would approach it this way:

Create a "preprocessing"/"data" module

That module can either be a simple data access layer that can be shared across notebooks, or it can also include preprocessing steps that you can add. This enforces a common data access layers across notebooks without necessarily duplicating code.
So you could do something like this:
from data_layer import Data, Preprocessor

preprocessor = Preprocessor(**kwargs)
data = Data(preprocessor = preprocessor, **kwargs)

In an ideal world, you should be able to define your data access layer as data itself. For instance, a JSON document containing the data source, and the preprocessing options.

Create a different notebook per model

And in this case, if your models share components, I'd also create modules they can all import from. The same way you'd use sklearn or torch component.


CNN Back Propagation without Sigmoid Derivative

error is defined just as (outputs - target)

This is the correct gradient for cross-entropy loss function with Sigmoid as the last layer.

For squared (quadratic) loss $$(y-f(x))^2,$$ the gradient is, as you said, $$(y-f(x))f'(x)$$ (constant $2$ is removed), but for binary cross-entropy loss $$y\text{log}f(x) + (1-y)\text{log}(1-f(x)),$$the gradient is $$yf'(x)/f(x) - (1-y)f'(x)/(1-f(x)),$$
since for Sigmoid we have $f'(x)=f(x)(1-f(x))$, by substitution the gradient becomes
$$y(1-f(x)) - (1-y)f(x)=y-f(x)$$
 To distinguish between these two gradients, author sets cnn.CalcLastLayerActDerivative = 0 to be checked later in an if statement in bpcnn.m file as follows (comments don't exist in the original code):

...
else
  % error = (f(x) - y)
  er = ( cnn.layers{cnn.no_of_layers}.outputs - yy);
...
if cnn.CalcLastLayerActDerivative ==1
    % change the error from (f(x) - y) to f'(x)(f(x) - y)
    er =applyactfunccnn(cnn.layers{cnn.no_of_layers}.outputs,cnn.layers{cnn.no_of_layers}.act_func, 1, er);
end

which means gradient is $(y-f(x))f'(x)$ for quad and $(y-f(x))$ for cros (bad variable name!).

As a side note, author only allows Sigmoid for cross entropy which means only binary classifier is supported (multi-class classifier requires SoftMax).

error('cross entropy is implemented only when last layer is sigmoid');

EDIT

Thanks to @Edison for pointing out that error and gradient were not handled  the same as loss values in the code, which substantially changed the final answer.


Training loss stuck in the starting epochs but then starts decreasing. What could be the reason for it?

It could be any of those factors. The root cause can be found through experimentation. Hold everything constant and change a single factor. Then systematically change each factor while holding all other factors constant.


Do I need to buy a NVIDIA graphic card to run deep learning algorithm?

I would recommend familiarizing yourself with AWS spot instances. It's the most practical solution I can think of for your problem, and it works your computer too. So, no you don't have to buy an Nvidia card, but as of today you will want to use one since almost all the solutions rely on them.


How the "def match(self, example)" method is automatically calling and how the example arguments is working here?

Typing

Question(0,'Green')

should not print out anything. However if you type

print(Question(0,'Green'))

then, the $\_\_$repr$\_\_$ method is called.


nltk's stopwords returns "TypeError: argument of type 'LazyCorpusLoader' is not iterable"

There are a couple of items that could be improved in your code:

nltk.corpus.stopwords is a nltk.corpus.util.LazyCorpusLoader. You might want stopwords.words('english'), a list of stop English words.
It can cause bugs to update a variable as you iterate through it, for example sentance in your code.
In your code preprocessed_reviews is not being updated.
You might want to tokenize instead of str.split().

Here is a revision version:

import nltk
from nltk.corpus import stopwords
from tqdm import tqdm

reviews_raw = ('The quick brown fox jumps over the lazy dog', "I have been doing what I should not been doing")
reviews_processed = []
for sentence in tqdm(reviews_raw):
    reviews_processed.append(' '.join(token.lower() for token in nltk.word_tokenize(sentence) if token.lower() not in stopwords.words('english')))

assert reviews_processed == ['quick brown fox jumps lazy dog', '']


Decision tree regression: Polynomials unnecessary?

Since CARTs (Classification And Regression Tree) are a non-parametric algorithm, they should be able to find interactions between variables and non-linear behaviors.

Nevertheless, building polynomials can help them have a better performance.


scikit-learn: High / low value for C in SVM

The loss function in SVM consists of both Classification error and Margin error.

Now C is constant that attaches itself to the classification error as given below:

SVM_Error = C*Classfication Error + Margin Error

So, the high value of C means that classification error will be more than the marginal error and therefore model will focus on classifying all the data points accurately. Conversely, if the low value of C means that marginal error will be more than the classification error and therefore model will focus creating the larger margin for the hyperplane


No. Xgboost is more like traditional ML algorithm. It doesn't need too much data and it also perform better than almost all ML algorithm


How can we create an label, value detector?

Here is a paper doing exactly what you want with detailed explanation of architecture.

There is not much sense of me doing a TL;DR here because they have some special labeling of data etc... but the gist of it is LSTM if you do some pre-processing steps and get your data in right format.


Answer
Thought I'd elaborate a little on my comments with a demo of different curves.
The main thing is that short of fitting a curve backed by theory (which we've discussed and I know isn't an option), I don't think there's any one method which is more 'valid' than the others.  With just a single value at each wavelength, using a notion such as error doesn't help, as we can just choose splines so that they pass through the original data points.  You could try to use theory to suggest the curve should be smooth (if that is appropriate) but each of the interpolation methods I demonstrate below provide smooth curves, so this is unlikely to be fruitful as you would need to argue some degree of smoothness and it does not sound like that argument can sensibly be made.
Illustration of Interpolation Curves
To illustrate some of the options for interpolating this data, I've taken the first few data points provided in the question and added a few estimated from the provided plot.
Using that data, I fitted a smoothing spline (smooth.spline), loess regression (loess), cubic spline (spline) and polynomial (lm with poly).
The result can be seen in this plot:

I haven't tried to tune any of the methods but you can still see the main differences.  The splines are all quite similar.  Obviously, they have the advantage of passing through the given data points.  You could argue that the 'anchoring' of the spline curves would make interpolating at values close to the original wavelengths more likely to be closer to the 'true' value (but there is no guarantee that would be the case unless some theory can be applied to explain why the curve should be smooth like that).  On the other hand, you could consider the polynomial to give some degree of 'regularisation' and so may postulate that, were many data points to be taken, the true values may 'average out' to a smoother curve of this shape.  Without more data, it really is a case of choosing the assumptions that you wish to make rather than choosing an objectively better way to interpolate this data.
It sounds like gathering more data (repeat readings to average and/or intermediate values) would be the only way to really hone in on a method but I realise that may well not be practicable.
Below is the R code I used to generate the plot, in case it's of use.
library(ggplot2)
library(magrittr)
library(tidyr)
theme_set(theme_light())

wavelen <- seq(380, 440, 10)
coeff <- c(0.982, 0.564, 0.378, 0.220, 0.175, 0.160, 0.150)

dat <- data.frame(wavelen = wavelen, coeff = coeff)

# Create a new vector of wavelengths with a finer resolution to view the results
new_wavelen <- seq(380, 440, 1)

# Smoothing spline
smooth_spline <- smooth.spline(dat$wavelen, dat$coeff)
spline_pred <- predict(smooth_spline, new_wavelen)
spline_pred <- as.data.frame(spline_pred)

# Loess Regression
loess_reg <- loess(coeff ~ wavelen, dat)
loess_pred <- predict(loess_reg, new_wavelen)

# Third degree polynomial
poly_mod <- lm(coeff ~ poly(wavelen, 3, raw=TRUE), dat)
poly_pred <- predict(poly_mod, data.frame(wavelen = new_wavelen))

# Cubic spline
cubic_spline <- spline(x = dat$wavelen, y = dat$coeff, method = "fmm", xout = new_wavelen)

# Plot results
plot_dat <- cbind(new_wavelen, spline_pred$y, loess_pred, poly_pred, cubic_spline$y)
colnames(plot_dat) <- c("wavelen", "Smooth", "Loess", "Poly", "Cubic")

plot_dat <- plot_dat %>%
  as.data.frame() %>%
  pivot_longer(-wavelen, names_to = "Method", values_to = "Coeff")

dat$Method <- "Original_Data" # hacky way to get the original data in the plot and legend.
fig <- ggplot(data = plot_dat, aes(x = wavelen, y = Coeff, colour = Method, linetype = Method)) +
  geom_line()
fig <- fig + geom_point(data = dat, mapping = aes(x = wavelen, y = coeff))
fig <- fig +
  labs(title = "Demo of Different Curve Fitting Methods",
       x = "Wavelength")
fig


SelectKBest and Correlation returns me excatly same feature selection. How?

Likely what you are seeing is that those features that are highly correlated with each other are overweighting the results, so that if one of the features describes the target well, then the other highly correlated (or uncorrelated) features also match the target just as well.


Dueling DQN - why should we decompose and then combine them back into?

The Dueling DQN is a good strategy to avoid your network be overoptimistic. Otherwise it will cause actual DQN and to get as optimistic as it explodes. So the Q-values become larger and larger over time on some games and sometimes they never get back.
https://arxiv.org/pdf/1509.06461.pdf
A couple words about Double Q-Learning
$$Q_1^{\hat{}}(s_t,a_t) = r_t + \gamma*Q_2(s_{t+1}, argmaxQ_1(s_{t+1}, a^*))$$
$$Q_2^{\hat{}}(s_t,a_t) = r_t + \gamma*Q_1(s_{t+1}, argmaxQ_1(s_{t+1}, a^*))$$
If one Q function is not enough to make an accurate approximation, Double Q-learning suggest to learn two networks and train one another. So you implement Q1 and Q2 - two independent estimates of the actual value function. Q1 and Q2 in a DQN case are networks with a different set of weights.
If take an action which is optimal in the first Q function, there won't be any connection between this overoptimism for selected action in Q1, and the same overoptimism in Q2. In fact, the same action in Q2, is more or less independent. It can be overoptimistic or overpessimistic and it can be exactly the true value.
The idea here is that, the noise in Q2 is independent of the noise in Q1. And if you update them that way, then you take the maximization which will take account of the sampling error.
Since they two networks are more or less decorrelated, they have different kinds of noises, then the overoptimism disappears.
The drawback of this method - training two networks separately will double the convergence time.
But a DQN algorithm that can serve as an alternative source of q-values, aside from the main q-network - we take the older snapshot of your network as the source of independent randomness as the other Q network
$$Q^{\hat{}}(s_t,a_t) = r_t + \gamma*Q_{old}(s_{t+1}, argmaxQ(s_{t+1}, a^*))$$
Another huge problem is that another huge problem with DQN is that it actually tries to approximate a set of values that are very interrelated. And double Q-learning can't solve this problem easily.
Dueling DQN
The Dueling DQN  can solve the overoptimistic problem as well
$$ Q(s,a) = V(s) - A(s,a)$$
The intuition behind the advantage is how much you're action value differs from the state value.
For example, if you have a state in which you have two actions, the first brings you returns of + 100 and the second is + 1. In this case, both action values are positive because you get positive reward. The advantages on the contrary are different. If you take this suboptimal action, then you get the action value of (+ 1) minus the state value of (+ 100), which is -99. Basically, it tells you that you have just lost 99 potential units of rewards in this case.
To predict advantages, we have to constrain them. In case of $V^*$, the maximum possible advantage value is zero because you can never get action value which is larger than maximum over all possible action values from the state.
So you you train two halves together and then just add them up to get your action values.


suggestion for good online source according to Syllabus

A good deal of the topics, excluding the more advanced ones like autoencoders or manifold learning, can be learned in the excellent course of andrew ng https://www.coursera.org/learn/machine-learning
There's also https://www.coursera.org/specializations/deep-learning?utm_source=deeplearningai&utm_medium=institutions&utm_campaign=WebsiteCoursesDLSBottomButton that goes deeper into aspects of deep learning
For generative models I don't have specific recommendations...


learning rate very low 1e-5 for Adam optimizer good practice?

I think that for the most part, the ends justify the means when it comes to learning rates. If the network is training well and you're confident that you're evaluating its generalization properly, use what works.

With that said, overfitting isn't usually caused by high learning rate.  It's hard to say with the information you've supplied here, but I suspect that your regularization isn't set up well enough, so using a high learning rate overfits the poorly regularized network rapidly, while training with a low learning rate doesn't get close to a local optimum because it can't move fast enough, but it's still heading towards an overfit optimum.

I'd check in on your regularization. What are you using?


How do I split number string with digit pattern?

I just did this-

> library(stringr)
> x <- c("-26755.22-50150.60")
> matc = "\\-[0-9]*\\.[0-9]{2}"
> as.numeric(unlist(str_match_all(x, matc)))
[1] -26755.22 -50150.60

After this, you can select the numbers by using respective indexes.


Is the learning rate related to the shape of the error gradient, as
it dictates the rate of descent?

In plain SGD, the answer is no. A global learning rate is used which is indifferent to the error gradient. However, the intuition you are getting at has inspired various modifications of the SGD update rule.

If so, how do you use this information to inform your decision about a value?

Adagrad is the most widely known of these and scales a global learning rate η on each dimension based on l2 norm of the history of the error gradient gt on each dimension:

Adadelta is another such training algorithm which uses both the error gradient history like adagrad and the weight update history and has the advantage of not having to set a learning rate at all.

If it's not what sort of values should I choose, and how should I choose them?

Setting learning rates for plain SGD in neural nets is usually a
process of starting with a sane value such as 0.01 and then doing cross-validation
to find an optimal value. Typical values range over a few orders of
magnitude from 0.0001 up to 1.

It seems like you would want small values to avoid overshooting, but
how do you choose one such that you don't get stuck in local minima
or take too long to descend? Does it make sense to have a constant learning rate, or should I use some metric to alter its value as I get nearer a minimum in the gradient?

Usually, the value that's best is near the highest stable learning
rate and learning rate decay/annealing (either linear or
exponentially) is used over the course of training. The reason behind this is that early on there is a clear learning signal so aggressive updates encourage exploration while later on the smaller learning rates allow for more delicate exploitation of local error surface.


Classify Customers based on 2 features AND a Time series of events

I know it has been a while since the question has been posted but for future readers I propose here another solution for the problem of classifying time-series of proportional data i.e., vectors of proportions.

Hidden Markov Models (HMMs) are broadly used for time-series classification. The original implementations were developed for discrete and Gaussian data but since then, they have been extended to more probability distributions.

Among them, the Dirichlet, the generalized Dirichlet, and the Beta-Liouville distributions which are perfectly suited to work with proportional data.

Usually, a model is trained for each category ("Tag" in this question) from time-series for which the category is known. When a new time-series has to be classified, its likelihood with respect to each model is computed. The new time-series is usually assigned to the category leading to the highest likelihood.

Here are the reference papers for these models presenting them more deeply and providing all the equations needed for implementation:

Dynamical Dirichlet Mixture Model, Research Report from IDIAP, L. Chen, D. Barber, J.-M. Odobez, April 2007
Proportional data modeling with hidden Markov models based on generalized Dirichlet and Beta-Liouville mixtures applied to anomaly detection in public areas, E. Epaillard, N. Bouguila, Pattern Recognition 55, pp 125-136, 2016

For the additional two features mentioned in the question, HMMs for mixed data have also been developed in the following research paper:
Hybrid hidden Markov model for mixed continuous/continuous and discrete/continuous data modeling, E. Epaillard, N. Bouguila, MMSP, pp. 1-6, 2015.

These models are heavier to deploys than regression models but have the advantage of being discriminative and generative models and to be able to generalize well when some variability is present in the processed data.


K-Fold cross validation and data leakage

A reproducible example with no data leakage:
In there I'm scaling the data only with the train data on the k-fold stage
import numpy as np

from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer

X, y = load_iris(return_X_y= True)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 42)

model = Pipeline([("imputing", SimpleImputer(strategy= "mean")),("scaling", StandardScaler()), ("modeling", LogisticRegression(random_state= 42, class_weight= "balanced"))]).fit(X_train, y_train)

cv_scores = cross_val_score(estimator= model, X = X_train, y = y_train, scoring= "accuracy")

print(f"Mean accuracy cv: {np.mean(cv_scores)}")

model.score(X_test, y_test)

Note that in this case it is easy to apply all the pipeline because all the feature are the same data type, but imagine you have categorical and continuous features, so you need to apply different preprocessing and imputing to each.
In that case a combination of ColumnTransformer and Pipeline would do the job.
For reference check:
https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html


Classifying potential triggers for events (many-to-many relationship): Which algorithm to use?

I did not find an existing algorithm, but I have since published an algorithm for "set cover rule mining" myself. It is called GIMO: https://doi.org/10.1016/j.eswax.2020.100040


Feature Importance - How to choose the number of best features?

This is an important problem, since many feature selection methods return feature scores/importances rather than a finite feature set. I currently know three approaches:

choose the k best attributes (fixed number defined)
choose the best k * 100% of attributes (relative number of features defined)
make a cutoff at the biggest difference in feature scores: all features are ordered according to their score and a split is made at the largest difference between one score and the next lower (biggest loss in importance)

You find a nice implementation doing this in R with the FSelector package.

I think, there are some research works tackling exactly this problem and may suggest better approaches, but I hadn't had the time to go deeper into this.


How to fake data based on the condition and weight

There are many Python packages to generate synthetic data. One example is Faker.


Why might a neural network consistently underestimate its target?

Consistently underestimating target could be due to the distribution of the target variable. If the target distribution has a negative-skew (i.e., a long tail towards lower values), then the neural network is just pattern matching to minimize those errors. For example, if the network has a squared loss function then large estimation errors in the tail are weighted more heavily than small estimation errors in the head, even though there are fewer data points in the tail. The model is trying to minimize the overall loss function and is "cheating to win" by consistently underestimating.


Extracting events with attributes from unstructured text

I think the closest standard NLP task would be relationship extraction. In general it's a quite complex task which involves NER, syntactic analysis and semantic role labeling.
Note that there are various works using the term "event extraction" (for example this), but as far as I know there is no clear definition of the task. It's often related to putting events on a timeline, this would be quite different from your goal but possibly related.
A basic approach would be to treat the problem as a sequence labeling task like NER: given some annotated "events" in a training corpus, the model might be able to learn the patterns and detect any new "event" in a text.


Can I continue a machine learning process after shutdown?

In order to continue the learning process, you will need to use checkpoints.

# source: https://stackoverflow.com/questions/36356004/continue-training-from-a-specific-epoch
weight_save_callback = ModelCheckpoint('/path/to/weights.{epoch:02d}-{val_loss:.2f}.hdf5', monitor='val_loss', verbose=0, save_best_only=False, mode='auto')
model.fit(X_train,y_train,batch_size=batch_size,nb_epoch=nb_epoch,callbacks=[weight_save_callback])

# load model
model = Sequential()
model.add(...)
model.load('path/to/weights.hf5')

# use model.fit to continue training.


You can't fit X_train into y_train without encoding.
Try something like this for features:

from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(handle_unknown='ignore')
enc.fit(X)

and label encoding for labels.


Effect of adding gaussian noise to the input layer in a CNN

Adding noise in the input data is equivalent to adding a regularization term to the objective function.

See Training with Noise is Equivalent to Tikhonov Regularization and Adding noise to the input of a model trained with a regularized objective.


How do I find the relevant features out of 11,000+ possibilities?

Your solution will depend on a couple of factors. One is what type of model you are using. If you are using something that automatically calculates feature importances then you could simply look at these (or take a more balanced look using permutation importances).

While you could look at feature importances, with ~11,000 possibilities this is going to be pretty difficult. The main focus should be to cut down these features into something more manageable, do you really need one hot encoding? Without knowing more about the dataset I can't provide much more advice.


I intend to do classification modelling, but my target variable has only one value

Assuming I can’t use external datasets, what’s the solution to this problem? How can I best understand those who have defaulted if I don’t have something to compare them to?

If you wanna use this only this data as classification task, you can't perform this task. You could way this around, by generated fake data with label 0(you think about which value of parameters comes to default).

But you could try clustering algorithms(like k-means) and find the groups of specific clients. Perhaps one of these groups its group of default.


Given that you only four data points (i.e., 970, 1985, 2000, 2015), there is no way to apply machine learning to make an accurate prediction.

If you get more data points, you can start with a sign prediction of surface area. You predict if the shape is increasing or decreasing in size.


What is the best way (cheapest / fastest option) to train an model on massive dataset (400GB+, 100m rows x 200 columns)?

I don't know many boosting packages but I've been using XGboost for a while now and the biggest tabular dataset I've had was more than 40 times smaller than yours. The training took 2-3 days.
In my experience training time is worse than linear with the size of the data even though it highly depends on the data itself and the hyperparameters you chose. My guess is your training would be very (too) long.
If you really want to use XGboost, you should train on GPU, it seems to me that you are looking at cloud providers, I know google offers managed training of XGboost on GPU, others surely do as well.
With that amount of data I think you should consider using deep learning. You could maybe use tabnet which is a great model developed for tabular data by Google AI. It is easy to try using pytorch for instance.


How to use fresh data when target prediction period is long?

I am confronted with the same problem and I am afraid the answer is quite generally no. You'll have to wait the length of the time horizon to use new data in your calibration process. Similarly, for validating your prediction you will have to wait the time horizon.
Somehow building a target from some information you have would not yield interesting results. Because you will build the target based on some features, once you put your features trough a ML model, said ML model will just learn what rule you used to build the target.
As you mentionned the main solution is to build a shorter horizon target. A shorter horizon might help your main problem, but if it too short you will get stability problems. (And generally speaking a bad target, if for exemple, the sale usually happen 3 months after, using a 1 month horizon target won't help you learn the relationship between feature and the sale). Maybe you should try other target horizon between 1 month and a year.
Note that the time horizon you want to use might also depend on which frequency you observe your instances. Using a time horizon for a target that is longer than the time interval between your observation will create correlation between targets. This might get really bad for separating independent subsets (like for train and test set splitting or using cross validation).


Calculation of Output in LSTM Many-to-One Architecture

While training, a set of training examples will be provided in a batch. At end of each batch, weights for all layers are updated (Dense and LSTM).

https://adventuresinmachinelearning.com/keras-lstm-tutorial/


What are the applications of Solr in Big Data?

Solr is a highly scalable, fault tolerant search server.

So, you can store files in the form of JSON, XML, CSV or binary over HTTP.
And you can query it via GET and receive JSON, XML, CSV and binary results. It also has a distributed architecture for the purpose, which is the primary and most important concept in Big Data and it's handling.

So, as it is highly scalable and fault tolerant for high traffic and huge documents, it is a choice for a reliable search engine for huge documents, or in short Big Data.


How to use random forest with large number of categorical features and categories?

You have a few options on your hand:
Option 1 - Augment you dataset with additional Supplier-related features
Do you have additional features about the supplier that makes sense to augment into your dataset ? For e.g. you might be able to replace the Supplier field with something like "Years in Business" , "Primary Industry", "Reputation Score" etc.
That way you convert the categorical variable into something more meaningful that can be compared across the universe of Suppliers.
Go this route if and only if you have meaningful and easily accessible features for your problem domain.
Option 2 - Encode each Supplier with a unique integer
I know this sounds stupid - but it really works for Random Forest models (wouldn't work on simple Linear Regression for sure).
Random Forest models are scale invariant. They won't get biased towards suppliers with a larger numeric Supplier Id. They'll figure out the best way to split your Supplier Ids to meet your objective function.
I would recommend Option 2 to start with since that is the simplest.


When to stop calculating values of each cell in the grid in Reinforcement Learning(dynamic programming) applied on gridworld

1- You should set a threshold (a hyper-param) that will allow you to quit the loop.

Let V the values for all state s and V' the new values after value iteration.

if $\sum_s|V(s) - V’(s)| \le threshold$, quit

2 - V is a function for every cell in the grid yes because you need to update every cell.

Hope it helps.


What could explain a much higher F1 score in comparision to accuracy score?

You have imbalanced classes. Notice that your accuracy is very close to your precision, and quite dissimilar to your recall. This means that your precision (accuracy of positive predictions) is dominating the overall accuracy measure - nearly all of the cases in your data are classified as positive, so the accuracy among predicted positives is almost equivalent to the accuracy among all cases.

The F1 score is the harmonic mean of precision and recall, so it's a class-balanced accuracy measure. You have better performance on the minority class than the majority class, which is evidenced by the nearly equivalent accuracy and precision, and much higher recall.


What is considered short and long text in NLP (document similarity)

As Erwan said in the comments, it depends. In my experience, it depends specifically on two things:
Tokenization method: The length of a document in number of tokens will vary considerably depending on how you split it up. Splitting your text into individual characters will result in a longer document than splitting it into sub-word units (e.g. WordPiece), which will still be longer than splitting on white space.
Model: Vanishing gradients aside, an RNN doesn't care how long the input text is, it will just keep chugging along. Transformers, however, are limited. BERT can realistically handle sequences of up to 512 WordPiece units, while the LongFormer claims to handle sequences of up to 32k units (given sufficient compute resources). Thus your documents of 10 - 600 tokens would be long for BERT but short for the LongFormer.
Whether you should treat documents of length 10 differently from those of length 600 is not something I can answer without knowing the details of your specific task. Intuitively, I doubt a very short document would ever be very similar to a much longer one, simply because it likely contains less content.


Why encode pitch as one-hot encoding instead of ordinal encoder?

Because average of encoded pitches is not the encoded class.

Look at this example:

LabelEncoder can turn [dog,cat,dog,mouse,cat] into [1,2,1,3,2], but then the imposed ordinality means that the average of dog and mouse is cat. Still there are algorithms like decision trees and random forests that can work with categorical variables just fine and LabelEncoder can be used to store values using less disk space.


Using LeakyRelu as activation function in CNN and best alpha for it

The code,

model.add(Conv2D(32, kernel_size=(3, 3), input_shape=(380,380,1))
model.add(LeakyReLU(alpha=0.01))

will definitely transform the outputs from the Conv2D using the LeakyReLU activation given parameter alpha ( negative slope of ReLU ).

  Further, I want to know what is the best alpha? I couldn't find any
  resources analyzing it.

In ReLU, we simply set the activation to 0 for negative values. This causes the dying ReLU problem which leds to overfitting. Hence, we return $x \alpha$ instead of 0, so that the unit does not become non-functional.

If we look at TensorFlow's tf.nn.leaky_relu method, we will find that the alpha is 0.2.

Whereas in Keras' layers.LeakyReLU class, you will find the alpha is 0.3.

So you can clearly get an idea of what the parameter's value should be. It's basically a hyperparameter which you need to adjust through trial-error observations.


Testing independence of random variables in Python

As is discussed in the link to Cross-Validated SO from Mephy, this is isn't an easy thing to do.

If they are independent, you might expect a correlation between pairs of variables to be close to zero. That would mean that knowing anything about one of the two variables doesn't give you any insight as to the behaviour of the second. To this end, there is a nice answer here, which shows how to compute pairwise pearson correlation (with corresponding p-values) for all columns in a Pandas DataFrame.

The Pearson correlation does assume your random variables to be normally distributed, so keep that in mind when interpreting results. Alternatively, you could swap out that pearsonr function for the Spearman Rank correlation function: spearmanr, which does not assume normality of your variables.

Another (perhaps simpler) way using just a Pandas DataFrame is to use the built in method corr: This takes a keyword method, which allows you to specify one of three:

  method : {‘pearson’, ‘kendall’, ‘spearman’}

If you random variables are time-series (you didn't mention it), another possible tool to look at would be Granger Causality. This could also be performed pairwise (or batch-wise) across variables. It tests to see if a the future value of variable can be better predicted when historic values of the a different variable are included in the model. For example, if the price of StockA can be predicted with an accuracy of 52% using its own prices of the previous 5 days, the Granger test would have a null hypothesis that including some lags from StockB would not improve the accuracy. So if the accuracy does indeed jump up to 53% when including lagged prices of StockB, (and the test is significant), the null hypothesis is rejected and we say that StockB Granger-causes StockA.

This is implemented in the vars package in R (there are others too). As a bonus, this version can also perform the Wald test for correlation in error processes of predictor and target variables.


TF tells me to use Keras optimizer, tells me the opposite when I change it?

The error is telling you to use the actual keras optimizer, not the tensorflow.keras class. Import Adam directly from keras library and it runs ok:

from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.optimizers import Adam

model = Sequential()
model.add(Dense(32, input_dim=9, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(1, activation='relu'))

learning_rate = 0.001
optimizer = Adam(lr=learning_rate, decay=1e-6)

model.compile(loss='mean_squared_error', optimizer=optimizer, metrics=['mse'])


Scikit-learn: train/test split to include have same representation of two different types of values in a column

Welcome to Data Science at StackExchange,
One way to accomplish this is to use the stratify option in train_test_split, since you are already using that function (this will also work for ensuring your labels are equally distributed, very useful in modelling an unbalanced dataset):
Train,Test = train_test_split(df, test_size=0.50, stratify=df['B'])
In my example, you can see that there are 2 values in column B, and they are equally distributed between the 2 datasets. In your case, B would be the column indicating those customers with an account and those without.


If the set of all possible states changes each time, how can Q-learning "learn" anything?

if the length and height of the rectangle are random, as well as the starting position and the location of the Treasure, how can the bot apply the knowledge acquired to the new problem?

You have two possible approaches here, depending on how the problems are being presented to you:

If the agent has time to learn/plan on each environment separately, then you need an agent that has capability to learn each environment. A simple tabular Q-learning agent already has this capability, up to a certain size of problem (where the number of states and actions would fit in memory and can be iterated over in simulation enough times). Beyond that size, provided you can come up with a fixed feature set capable of representing any shape and size of problem that the agent could be presented with, and used e.g. DQN or other approximation technique, then you still have a generalised learning bot.
A bot that is generalised during training to attempt to solve new instances with variation needs to be trained with many variations and more state data. If shape, size and layout of the maze can change between episodes, then this data must become part of the state. This can expand the state space a lot, and requires different representations. A simple representation of grid spaces would be actual map of the grid as a rectangular "image", assuming the agent start, walls and goal position could be almost anywhere within the space. If the maze has lots of objects, then you can put each object type into a separate "channel" and use a Convolutional Neural Network as part of the Q function approximation. If the area is more sparse with just a few objects (e.g. just the agent, a single blocker and the treasure) then it would be easier to use a direct vector representation of positions of the objects and use a fully connected neural network.

For a toy problem where key positions can change between episodes, yet this still can be solved easily by a tabular agent, see Open AI's Taxi-V2 which is an implementation of a classic control problem where the locations of a sub-goal and goal are randomised on each episode.


Machine learning methods for panel (longitudinal) data

Panel data = multi-object time series.
in other words you have time series problem (time) for different objects (obj) that you are trying to predcit (Y).
If I were you I would just dissect this problem and start thinking in terms of time-series + another discriminative column called obj. What approaches do you know there? Here is really cool and MODERN time-series tutorial, check it out.
Regarding NN, why are you trying to squeeze it in so hard? Let the data tell you what algo can model it. Personally given these 3 features NN is just too much and you can achieve similiar results with less complexity using some simpler/less expensive approaches.


If you go to the source page for this method you can find the function:
def center_of_mass(input, labels=None, index=None):
    """
    Calculate the center of mass of the values of an array at labels.
    Parameters
    ----------
    input : ndarray
        Data from which to calculate center-of-mass.
    labels : ndarray, optional
        Labels for objects in `input`, as generated by `ndimage.label`.
        Only used with `index`.  Dimensions must be the same as `input`.
    index : int or sequence of ints, optional
        Labels for which to calculate centers-of-mass. If not specified,
        all labels greater than zero are used.  Only used with `labels`.
    Returns
    -------
    center_of_mass : tuple, or list of tuples
        Coordinates of centers-of-mass.
    """
    normalizer = sum(input, labels, index)
    grids = numpy.ogrid[[slice(0, i) for i in input.shape]]

    results = [sum(input * grids[dir].astype(float), labels, index) / normalizer
               for dir in range(input.ndim)]

    if numpy.isscalar(results[0]):
        return tuple(results)

    return [tuple(v) for v in numpy.array(results).T]

Whereby sum is (sum - centre)^2 of the input labels


t.ppf is calculating a 1-tail inverse cdf. It looks like you're trying to look up the t-value for p=.95, but the value you are referring to in the table is a 2-tailed value of .95, meaning the one tailed value is 1 - (1-.95)/2 = .975. So

In [104]: st.t.ppf(0.975, df=9)
Out[104]: 2.2621571627409915