Different strategies for dealing with features with multiple values per sample in python machine learning models

You should make the "abdomCirc" as Multiple separate features (e.g. one for each month).
 Then handle NaN as it should be i.e.

Remove columns that are having more NaN than a Threshold
Try to fill NaN for cases where only a few are missing.

If we can't accommodate the above solution because of NaN counts, then we should accept that we don't have enough data and ignore the feature-set (i.e. abdomCirc) altogether.
Otherwise you may be at risk of finding a pattern that doesn't exist because of very small amount of available data points for abdomCirc.

I initially thought about using a python list for these features. However, I do not know if a machine learning model can handle this data type

None of the Model will accept this data i.e. a List


Calculating saliency maps for text classification

I've been working on this for the last few days, and I think I've answered my own question. Calculating individual word saliency is not possible with the model structure as is. This is because of the GlobalAveragePooling layer of the model. This averages the embedding matrix in the 'word' dimension, removing the ability to distinguish the effect of an individual word on the classification. This is the code I used to convince myself of what was happening, left here in the hope that the next soul to try this finds this answer.

outputTensor = model.output

embeddingTensor = model.layers[1].input
gradientsEmbedding = tf.gradients(outputTensor, embeddingTensor)

globalAverageTensor = model.layers[2].input
gradientsAverage = tf.gradients(outputTensor, globalAverageTensor)

idx = 1

sess = keras.backend.get_session()

embedding = sess.run(embeddingTensor, feed_dict={model.input:train_data[(idx-1):idx,:]})
globalAverage = sess.run(model.layers[2].input, feed_dict={model.input:train_data[(idx-1):idx,:]})

print(np.mean(embedding, 1))
print(globalAverage)

gradientMatrixEmbedding = sess.run(gradientsEmbedding, feed_dict={embeddingTensor:embedding})
gradientMatrixAverage = sess.run(gradientsAverage, feed_dict={globalAverageTensor:globalAverage})

print(np.sum(gradientMatrixEmbedding, 2))
print(gradientMatrixAverage)


You can see in the following link examples in Numpy of the following optimisers

Stochastic Gradient Descent
Stochastic Gradient Descent + momentum
Stochastic Gradient Descent + Nesterov momentum
Adagrad
Adadelta
RMSProp
Adam
AdaMax
Nadam

Just clone the repository and open the Report.ipyb with Jupyter Notebooks.

https://github.com/ivallesp/awesome-optimizers


Entity Recognition in Stanford NLP using Python

The Stanford CoreNLP released by the NLP research group at Stanford University. It offers Java-based modules for the solution of a range of basic NLP tasks like

POS tagging (parts of speech tagging)
NER (Name Entity Recognition)
Dependency Parsing, Sentiment Analysis etc.

Before doing all above task you should first setup stanford CoreNlp.

If you want to have clear picture about stanford coreNlp starting from setup core nlp for python, NER , POS to sentiment, you can have a look at below link.

Advanced Natural Language Processing with Stanford CoreNLP


How to use ColumnTransformer and FunctionTransformer to apply the same function to many columns, but separately?

I solved the problem by creating a wrapper around pd.cut, which then applies pd.cut using the apply method of DataFrame:

if isinstance(x, pd.Series):
    return pd.cut(x, bins_final, labels=labels, **kwargs)
elif isinstance(x, pd.DataFrame):
    return x.apply(pd.cut, args=(bins_final,), axis=0, labels=labels, **kwargs)


Outlier detection and removal using scatter-plots and histograms

Yes. This code does nothing but to plot the points for two chosen columns and assign different color and markers to them. It does not do any outlier selection/removal.
Column 3 and 4 (which are features in your dataset) are reasonably predictive of different classes.
Plot a histogram. See if there is any clear disjoint in among the bars. Investigate a little time to find a suitable bin size.

In addition, the title of your question does not match the content. Can you please update the question?


model.predict() accuracy extremely low on training dataset

The problem with the way the prediction is done in the code found in the question is that the generator is shuffling the data (even though the shuffle=True is commented out, True is the default). Additionally, the batch_size in the code is set to 16. Setting batch_size = 1 produced the correct output for me, and the resulting accuracy increased to what I was expecting.
Oh, and don't use the same generator to test the predict function!
Hope this helps someone! :)


Understanding

In order to calculate the loss function for each of the observations in a multiclass SVM we utilize Hinge loss that can be accessed through the following function, before that:

The point here is finding the best and most optimal w for all the observations, hence we need to compare the scores of each category for each observation. Imagine, we have an observation j which we know its label. the score of its actual label is $s_(y_j)$ or based upon what you wrote $f(x, w)_(y_i)$.

We also know the scores of other classes for this observation, in fact, we have a vector of scores, namely, $f(x, w)_j$ What do we need to do now? calculate the differences between the actual class score and other class scores (the score vector $f(x, w)_j$) obviously not for the actual score ($j != y_i$) and sum them up.

Ultimately, we would have $L_i$ that is the loss value for each observation. now if the difference between the actual class's score and the other scores is high enough, then the prediction is ok, yeah? so we put 0 as the loss value, consider that we put a safety margin like 1 or your $\delta$ and if otherwise, the actual class's score is not higher or equal than the incorrect class's score plus that $\delta$ we put their difference as loss value, meaning $s_j - s_(y_i) + 1$

then we take the mean of loss values ($L_i$)s for all of the observations and set it as Loss value, L.

The second part of your formula is a $L_2$ regularization value in order to work better for the test set by impeding the model to be too complex.

If the score for the actual class is high enough, higher than the score for other classes plus the safety margin, the loss value for that observation is 0, hence we have a better classification. In fact, we want to have the weights in such a way that for each observation, the score of it's actual class is enough more than the scores of other classes.

Formula

(i is each of the observations in the dataset)

$$Li = \sum_{j != y_i} something$$

and this something is $0$ if $s_(y_i) >= s_j + 1$ and $s_j - s_(y_i) + 1$ otherwise.

in fact, $L_i$ can be written as:

$$Li = \sum_{j != y_i} max(0 , s_j - s_(y_i) + 1)$$

and then, in order to find out the general cost function for all of the observations in the dataset, we need to calculate their average, hence:

$$L = \frac{1}{N} \sum_{i}^n Li $$

Plot

This Hinge name comes from the plot:

(the values and slope might differ)

Here the $x-axis$ is the actual class's score and the $y-axis$ is the Loss value. As seen in the plot, we set a sample $s_j$ for a class, for this $s_j$, if the real class's score is something more than 0, then since $s_j - s_(y_i) + 1$ is negative, the loss value would be 0 and so on.

Inspired by Justin Johnson's lecture in Stanford Machine Learning course.


Dealing with multiple distinct-value categorical variables

Have you heard of CatBoostClassifier?

https://tech.yandex.com/catboost/doc/dg/concepts/python-reference_catboostclassifier-docpage/

It is type of Boosting classifier developed to deal specifically with categorical features. It has achieved state of the art results and the package developed by the authors have excellent support and even GPU portability. Take a look, this can be your solution.


One option would be to do something more similar to bootstrapping since that would be re-sampling existing data.
Another option would be to generate extra samples then prune based on the constraints.


Your code has two typos:

1. Error when selecting data for the target variables.

In the second line, you are using [:, :-1] which means that you select all rows, and all columns except the last one.

In the third line, you are using [:, 1] which means that you select all rows, and only the first column instead of the last.
Instead, you want to select the columns of your target variable, i.e. the last one:

y=dataset.iloc[:, -1]

2. Error when fitting the model

You are fitting your model using the X training set and the Y validation set. They have different length. Use this instead:

regressor.fit(x_train, y_train)


Time Series prediction using LSTMs: Importance of making time series stationary

In general time series are not really different from other machine learning problems - you want your test set to 'look like' your training set, because you want the model you learned on your training set to still be appropriate for your test set.  That's the important underlying concept regarding stationarity.  Time series have the additional complexity that there may be long term structure in your data that your model may not be sophisticated enough to learn.  For example, when using an autoregressive lag of N, we can't learn dependencies over intervals longer than N. Hence, when using simple models like ARIMA, we want data to also be locally stationary.

As you said, stationary just means the model's statistics don't change over time ('locally' stationary).  ARIMA models are essentially regression models where you use the past N values as input to linear regression to prediction the N+1st value. (At least, that's what the AR part does).  When you learn the model you're learning the regression coefficients.  If you have a time series where you learn the relationship between the past N points and the next point, and then you apply that to a different set of N points to predict the next value, you are implicitly assuming that the same relationship holds between the N predictor points and the following N+1st point you're trying to predict.  That's stationarity.  If you separated your training set into two intervals and trained on them separately, and got two very different models - what would you conclude from that? Do you think you would feel confident applying those models to predict new data? Which one would you use? These issues arise if the data is 'non-stationary'.
My take on RNNs is this - you are still learning a pattern from one segment of a time series, and you still want to apply it to another part of the time series to get predictions.  The model learns a simplified representation of the time series - and if that representation applies on the training set but not in the test set, it won't perform well.  However, unlike ARIMA, RNNs are capable of learning nonlinearities, and specialized nodes like LSTM nodes are even better at this.  In particular, LSTMs and GRUs are very good at learning long-term dependencies.  See for example this blog post.  Effectively this means that what is meant by  'stationarity' is less brittle with RNNs, so it's somewhat less of a concern. To be able to learn long term dependencies, however, you need LOTS of data to train on.

Ultimately the proof is in the pudding. That is, do model validation like you would with any other machine learning project.  If your model predicts well for hold-out data, you can feel somewhat confident in using it.  But like any other ML project - if your test data is ever significantly different than your training data, your model will not perform well.


Which algorithm should I choose and why?

You can choose the optimal method using cross-validation. If your sample size is relatively small, use leave-one-out cross-validation... I would not be surprised if $K = 5$ worked well. Linear discriminant analysis (LDA) will not work here because it implies linear decision boundaries. Unless you enlarge the set of predictors with non-linear transformations.
Also, the picture above is a classic case where support vector machines (SVM) with a Gaussian kernel could be of use. R has a friendly implementation of SVM in the "kernlab" package.


Importing Excel format data into R/R Studio and using glmnet package?

After soliciting assistance from many colleagues, I found the answer to my problem.  And, I thought it be worth sharing.  glmnet package requires that you format your data as matrix.

What I had already done is import and transform the data as matrix as follows:

data.matrix(data)

But, that is not enough.  You still have to convert the X and Y arrays as matrix as follows:

a) First do the x matrix:
x = as.matrix(data,...);
b) Second do the y matrix: y = as.matrix(data,...).

Once I did the above change the glmnet package seems to work just fine.

I am unclear why the glmnet package is so much more sensitive to how much you matricize your data.  But, that is the way it is.


Which process step in KDD or CRISP-DM includes labeling of the data?

Data Labeling is a very trivial process as you have mentioned.

As far as I know, it falls under Data Understanding(Exploration Analysis). When you don't know anything about data then you do exploratory analysis to understand and derive some insights. If you don't know the target variable then the problem falls under Unsupervised Learning, as you have mentioned in the question that your problem is unsupervised. So, you don't know your Target Variable, you are trying to make new feature/dimension to get some good insight irrespective of the factor whether you derive your Target Variable or any other feature it falls under Data Preparation(any new derived variables), which ever we think are important for our Analysis


Elasticsearch is the right tool to use if you don't want to code this yourself. Indeed, you need an indexing algorithm that is able to efficiently retrieve pieces of texts in a big database, and SQL isn't particularly good at it. Moreover, Elasticsearch is quite user friendly, so it won't be an overkill to actually install it and use it. You might discover in the process that finding most similar articles isn't that easy and that Elasticsearch is of a great.

Here is the documentation for a Python client:


Count the max number of consecutive 1 and 0 in Pandas Dataframe

To check if a value has changed, you can use .diff and check if it's non-zero with .ne(0) (the NaN in the top will be considered different than zero), and then count the changes with .cumsum, like this:
df['counter'] = df.diff().ne(0).cumsum()

Afterward, you can create a second dataframe, where the indices are the groups of consecutive values, and the column values are the value (0 or 1, in your case) and length (which is what you ultimately want):
df2 = df.groupby('counter')['column1'].min().to_frame(name='value').join(
df.groupby('counter')['column1'].count().rename('number'))

The resulting max_consec0, max_consec1 are just the values in the [number] column, filtered by the [value] column:
max_consec0 = df2[df2['value']==0]['number'].tolist()
max_consec1 = df2[df2['value']==1]['number'].tolist()

You can verify that the result is [2, 1, 2, 1] and [1, 1, 2, 3], as desired.


Confidence Intervals for Multi-Categorical Votes

I found a concise description of how to do this.

(But note that this doesn't correct for sentence length, so it's not a complete solution bcs longer sentence will have more "votes".)


Predicting which drug is most appropriate for which patient returns an accuracy of almost 0

I can't say exactly why you get null accuracy, but I have some comments that may help:

you are mixing continous data and categorical data. You might already be aware:

when standardizing your data: you are although standardizing your categorical data (sex, disease_type), just be sure that it makes sense (depending on the classifier you're using)
depending on the classifier, mixing continous and binary data may lead to undesirable results. This post explains it in the case of KNN

Otherwise I don't see anything suspicous after a first look. Could you share more data so we can reproduce your result ?


Is it possible to update data and retrain just one of several data series in bigquery model

After implementing and training this model and understanding the details better, it seems the short answer is NO, I have to remake and train the entire model.
It takes about 7 minutes to train about 3,500 series with about 100 datapoints each from a seed dataset. This may grow as the incremental ETL adds data indefinitely.


Speaking as a Graph/Complex Networks guy I'd recommend Networkx package in Python. This is the main library I used for my master thesis and my research during last 2 years. As long as your graph is not gigantic (millions of nodes) you can handle it using Networkx.

But what you need is not only the library but a philosophy to convert your data into a graph. What is your data exactly? If you have some addresses, what would be the node and what would define an edge? This is important actually. Graph is not constructed from your addresses unless you have a definition for vertices and edges.

When you convert your data into a graph Networkx provides you all fancy algorithms for everything (shortest path, community detection, statistical analysis, etc).

If you provide info in the comment here I'll answer your question in details.


Finding the equation for a multiple and nonlinear regression model?

This question clearly is the basics of linear regression.

If you are looking for a way to implement linear regression in R, use the lm method.

fit <- lm(y~X1+X2, data = datasetname)

summary(fit)

This should give the coefficients and the p value for each variable.


In RNNs why do networks always use the last output vs the last input?

That would work if the Markov Assumption was in place here (i.e. the last state was enough to determine the new state).

However, that is exactly what RNNs try to break. RNNs try to encode the whole history of inputs and predictions to determine the next output.

So in the formula, the $h_{t-1}$ is the last output, but it is a hidden state which encodes all the previous inputs, outputs, and calculations. So when you have new information coming in (i.e. $x_t$), your model can make use of all the history and the new input to determine the new input.

The same applies to LSTM, you are working with a sequence that the new output depends not only on the last input but on more things from the past, and this information is encoded in $h_{t-1}$.


permutation_importance is considering the top-level features. It is permuting each one sequentially and learning the importance.
So, the inner encoding i.e. OHE/tfid is not visible to it.
To get the importance of components of the top-level feature, you should encode it separately and then pass the encoded data to the permutation_importance

Get the pre-processed data using preprocessing_cv.fit_transform(X_train)
Call your permutation_importance code on the above data and any model of your choice

Edit
Adding the snippet. I am excluding ColumnTranformer as it is causing some issue.
data = {'Number':[1,2,3], 'Text':['pen is table', 'sky is blue','Sun is kool'], 'Cat':['A','B', 'C']}
df = pd.DataFrame(data)

categorical_preprocessing = OneHotEncoder(handle_unknown='ignore')
numeric_preprocessing = SimpleImputer(strategy='mean')
text_preprocessing_cv =  CountVectorizer()

text_tfid = text_preprocessing_cv.fit_transform(df['Text']).toarray()
num = numeric_preprocessing.fit_transform(df['Number'].values.reshape(-1, 1))
cat = categorical_preprocessing.fit_transform(df['Cat'].values.reshape(-1, 1)).toarray()

data = np.concatenate((cat,num,text_tfid), axis=1)
cols =  np.concatenate((categorical_preprocessing.get_feature_names(), text_preprocessing_cv.get_feature_names(), ['Num'])) # New cols name

df = pd.DataFrame(data, columns=cols) # Encoded DataFrame with col name

from sklearn.inspection import permutation_importance
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier

clf = DecisionTreeClassifier().fit(df, [1,0,1])

result = permutation_importance(clf, df, [1,0,1], n_repeats=2, random_state=42)
sorted_idx = result.importances_mean.argsort()

plt.boxplot(result.importances[sorted_idx].T,
            vert=False, labels=df.columns[sorted_idx]);


Apparently the paper introducing the dataset mentions a list of "online appendices" (table 1 p 363) which seems to contain details about the categories. However I wasn't able to find these appendices in the additional material.

[edit] Since it appears that there is no existing source for the original list of topics, I think your best bet is to rely on an acronym dictionary. Based on Acronym Finder, given that the data is mostly about economic news, I think the most likely meaning for the three topics you mention are:

DLR: dollar
GDP: Gross Domestic Product
BOP: Balance Of Payments

You could check whether these are correct by looking at a few documents labelled with these topics in the dataset.


What to do when facial recognition fails to find a face?

I guess you have a learning problem that has low number of training data. I recommend you a two-step solution.

First you need to do error analysis. Find the images that your classifier does not recognize and try to compare with those which your classifier correctly classified them. find the features that your classifier can not do well on them and try to add features that your classifier can discriminate those, I mean try to find yourself what prompts your classifier to make mistake on those images and why you don't make mistake during recognizing them. Try to add those features that you care but your classifier does not and add them as input.
Try to augment your data using data augmentation for both you mistake and those which you don't. There is an important fact here. don't change the distribution of your data. I mean if one class has two times bigger population than the other class, after augmentation try to keep this ratio. The reason is that maybe after equalizing the number of population of classes you reduce the Bayes error but at borders of classes the answers would be random.


Which variables matter most for prediction of another variable?

Let's use the following notation : $(x_1, ...x_9)$ are the first 9 variables out of which you try to predict $y$, your tenth variable.

I will try to address what looks like a PCA misunderstanding, then give you some ways to predict which variables $ x_i $ matter most to predict $y$

Principal Component Analysis

The returned features you get from the PCA are not the original one!

The PCA transforms $(x_1, ..., x_9)$ into $(x'_1, ..., x'_9)$ where each $x'_i$ is a particular combination of the $(x_1, ..., x_9)$ such that for any $i, i'$, $x_i$ and $x_i'$ are linearly uncorrelated (ie orthogonal). Moreover, it is done such that each $x_i$ explains a certain part of the training dataset variance. In fact, most of the implementation sort the $x'_1, ..., x'_9$ such that the first one is the one explaining the variance the most, then the second, ...

Thus you end up having your first variable $x'_1$ explaining a lot your prediction $y$. But this $x'_i$ is a particular combinaison of $(x_1, ..., x_9)$.

Getting the feature importance

As to assert which variable $x_i$ explains $y$ the best, one might use different ways. Just to mention that the feature importance is not absolute ; it relies on the technique (or estimator) you use to address this question. Here is a non exhaustive list of possibilities :

The linear correlation is a good start
With the RandomForestClassifier (or RandomForestRegressor depending on $y$) of sklearn.ensemble, you can use feature_importances_ method to get which one is used the most at tree nodes. Note : it works with any decision tree estimator.
With standard linear or logistic regression (same with Lasso, Ridge, ...), you can check which variable has the higher coefficient (do not forget to normalize your input variables $(x_1, ..., x_9)$ )

I don't know how it is done using neural networks but would be glad if anyone has an hint.


Converting similarity matrix before inputting to t-sne

You need to scale the values by some constant factor so the sum of every entry in the matrix results in 1.0.

You can achieve this by using mat /= mat.sum(), where mat is your matrix.


Meaning of curved line shape distribution in t-SNE plot

From my own experience, if you have curved lines in your t-SNE plots, it usually implies your original data are scattered similarly in lines as well. One thing you can confirm whether this hypothesis is correct is that you can do principle component analysis (PCA) on your original data and then plot the first two dimensions. Usually, this plot will also be in curved lines.

Your explanation is plausible. However, you suggest you plot your first two PCA components to see what is really going on in your data.


How do I allow user to run a R script without making a website or web application?

If you work on Linux, you may create a script that loads data and runs R inside and give the users the access to execute the main script. I do not know the details but this may be a possible solution.


Printing all files' names from a folder (Kaggle kernel)

Usually a simple

import os
print(os.listdir("../input"))

works pretty well. Modify as required


What is the effect of KL divergence between two Gaussian distributions as a loss function in neural networks?

It's too strong of an assumption (I am answering generally, I am sure you know. Coming to VAE later in post), that they are Gaussian.

You can not claim that distribution is X if Moments are certain values. I can bring them all to the same values using this.

Hence if you can not make this assumption it is cheaper to estimate KL metric

BUT with VAE you do have information about distributions, encoders distribution is $q(z|x)=\mathcal{N}(z|\mu(x),\Sigma(x))$ where $\Sigma=\text{diag}(\sigma_1,\ldots,\sigma_n)$, while the latent prior is given by $p(z)=\mathcal{N}(0,I)$. Both are multivariate Gaussians of dimension $n$, for which in general the KL divergence is:
$$
\mathfrak{D}_\text{KL}[p_1\mid\mid p_2] =
\frac{1}{2}\left[\log\frac{|\Sigma_2|}{|\Sigma_1|} - n + \text{tr} \{ \Sigma_2^{-1}\Sigma_1 \} + (\mu_2 - \mu_1)^T \Sigma_2^{-1}(\mu_2 - \mu_1)\right]
$$
where $p_1 = \mathcal{N}(\mu_1,\Sigma_1)$ and $p_2 = \mathcal{N}(\mu_2,\Sigma_2)$.

In the VAE case, $p_1 = q(z|x)$ and $p_2=p(z)$, so $\mu_1=\mu$, $\Sigma_1 = \Sigma$, $\mu_2=\vec{0}$, $\Sigma_2=I$. Thus:
\begin{align}
\mathfrak{D}_\text{KL}[q(z|x)\mid\mid p(z)]
&=
\frac{1}{2}\left[\log\frac{|\Sigma_2|}{|\Sigma_1|} - n + \text{tr} \{ \Sigma_2^{-1}\Sigma_1 \} + (\mu_2 - \mu_1)^T \Sigma_2^{-1}(\mu_2 - \mu_1)\right]\\
&= \frac{1}{2}\left[\log\frac{|I|}{|\Sigma|} - n + \text{tr} \{ I^{-1}\Sigma \} + (\vec{0} - \mu)^T I^{-1}(\vec{0} - \mu)\right]\\
&= \frac{1}{2}\left[-\log{|\Sigma|} - n + \text{tr} \{ \Sigma \} + \mu^T \mu\right]\\
&= \frac{1}{2}\left[-\log\prod_i\sigma_i^2 - n + \sum_i\sigma_i^2 + \sum_i\mu^2_i\right]\\
&= \frac{1}{2}\left[-\sum_i\log\sigma_i^2 - n + \sum_i\sigma_i^2 + \sum_i\mu^2_i\right]\\
&= \frac{1}{2}\left[-\sum_i\left(\log\sigma_i^2 + 1\right) + \sum_i\sigma_i^2 + \sum_i\mu^2_i\right]\\
\end{align}

You see that the mean minimisation is the same so the only cost factor is between these stds and the covariance matrix values. We can see that its more expensive to evaluate these integrals in covariance matrix and then minimise them, then just to minimise these std.

TL;DR cheaper, but you are right it could be done. Good question!


Minimal example: Keras functional API & multi-input/multi-output regression

Here is a possible solution:

import numpy as np
import pandas as pd
from keras.datasets import boston_housing
(train_data, train_targets), (test_data, test_targets) =  boston_housing.load_data()

# Standardise data
mean = train_data.mean(axis=0)
train_data -= mean
std = train_data.std(axis=0)
train_data /= std
test_data -= mean
test_data /= std

# Add an additional target (just add some random noise to the original one)
import random
train_targets2 = train_targets + random.uniform(0, 0.1)
test_targets2   = test_targets + random.uniform(0, 0.1)

# https://keras.io/models/model/
from keras import models
from keras import layers
from keras.layers import Input, Dense
from keras.models import Model
from keras import regularizers
from keras.layers.normalization import BatchNormalization

# Input and model architecture
Input_1=Input(shape=(13, ))
x = Dense(1024, activation='relu', kernel_regularizer=regularizers.l2(0.05))(Input_1)
x = Dense(512, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(256, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(8, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)

# Outputs
out1 = Dense(1)(x)
out2 = Dense(1)(x)

# Compile/fit the model
model = Model(inputs=Input_1, outputs=[out1,out2])
model.compile(optimizer = "rmsprop", loss = 'mse')
# Add actual data here in the fit statement
model.fit(train_data, [train_targets,train_targets2], epochs=500, batch_size=4, verbose=0, validation_split=0.8)

# Predict / check type and shape
preds = np.array(model.predict(test_data))
#print(type(preds), preds.shape)
# is a 3D numpy array

# get first part of prediction (column/row/3D layer)
preds0 = preds[0,:,0]
# second part
preds1 = preds[1,:,0]

# Check MAE
from sklearn.metrics import mean_absolute_error
print(mean_absolute_error(test_targets, preds0))
print(mean_absolute_error(test_targets2, preds1))


How many minima does the residual sum of squares have for the logistic curve?

Sigmoid itself is not a convex function (see this) and square loss based on sigmoid, such as $ \left( A - \frac{1}{1 + e^{-z}} \right)^2 $ is not convex.

Simply plotting the square loss function with A = 5 shows that it's convex for x > 0 and concave, otherwise

If you want mathematical proof, take the second derivative and you'll see that it not strictly positive nor negative.


The original papers written by Thomas Mikolov are word2vec and doc2vec.

Lecture notes given by Richard Socher is simple to understand.

word2vec explained gives an intuition how the negative sampling in word embedding works. Not the whole algorithm though.

Nevertheless, there are implementations in various languages and libraries explaining how it works in their docs.

Python - gensim, Tensorflow

java - deeplearning4j


Making outputs of sub-models the inputs of other models LSTM

Try removing the Input layer

hidden_1_2 = LSTM(300,return_sequences = True)(merge)
hidden_1_3 = Dropout(0.5)(hidden_1_2)
hidden_1_4 = LSTM(200,return_sequences = False)(hidden_1_3)
hidden_1_5 = Dropout(0.5)(hidden_1_4)
hidden_1_6 = Dense(100,kernel_initializer='uniform',activation='relu')(hidden_1_5)
hidden_1_7 = Dense(1,kernel_initializer='uniform',activation='relu')(hidden_1_6)


Spacy Returns Nonidentical Results for Doc. Examples?

As I commented on your post - I think you have a problem with your environment. If you cannot solve that, you should file a bug with the spacy team on their GitHub.

I tested the same code and could only see a difference in results compared to the website when using language models of different sizes. If you installed the English model using the default command: python -m spacy download en, then you get the smallest model by default.  You should be using the medium-sized model to produce their results. I compare the two below

The script

import spacy

# Small setup
nlp_small = spacy.load('en')
tokens_s = nlp_small(u'dog cat banana')

# Medium setup
nlp_medium = spacy.load('en_core_web_md')
tokens_m = nlp_medium(u'dog cat banana')

# large setup would use 'en_core_web_lg' ...

print('Results using small model:\n')
for token1 in tokens_s:
    for token2 in tokens_s:
        print(token1.text, token2.text, token1.similarity(token2))

print('\nResults using medium model:\n')
for token1 in tokens_m:
    for token2 in tokens_m:
        print(token1.text, token2.text, token1.similarity(token2))

The output

Results using small model:

dog dog 1.0
dog cat 0.53906965
dog banana 0.28761008
cat dog 0.53906965
cat cat 1.0
cat banana 0.4875216
banana dog 0.28761008
banana cat 0.4875216
banana banana 1.0

Results using medium model:

dog dog 1.0
dog cat 0.8016855
dog banana 0.24327648
cat dog 0.8016855
cat cat 1.0
cat banana 0.28154367
banana dog 0.24327648
banana cat 0.28154367
banana banana 1.0

The results are clearly better when using the medium sized model - and they match the results on the website.

Here is the spacy version as used within a conda environment to produce these results:

n1k31t4@n1k31t4~$ conda list | grep spacy
spacy                     2.0.12                    <pip>

And the Python version information:

n1k31t4@n1k31t4:~$ ipython
Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:53:17)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.

Although I ran the script on Linux (OP used MacOSX) and my Python version is slightly newer... I wouldn't expect these things to explain the difference in results.


Is there a good alternative to XGBoost for learn to rank?

You asked for alternatives - also consider: LightGBM, CatBoost or RankLib.
How big is your dataset?  If you are hitting memory problems just creating the DMatrix its possible the other libraries will have similar problems.
What are you loading your dataset from?  Perhaps creating the DMatrix from a numpy array is trying to fit two copies of your data in memory at the same time?  Perhaps you could divide your dataset into smaller chunks and append them to the DMatrix to avoid having them all in memory at once?


I also have this issue, but didn't have the answer with a customized Model.
There is a workaround as follows:
 model = Model(inputs=inputs,
                  outputs={'ctr_output': ctr_pred, 'ctcvr_pred': ctcvr_pred, 'cvr_output': cvr_pred})


How to print a Confusion matrix from Random Forests in Python

From the code and task as your present it, a confusion matrix wouldn't make sense. This is because it shows how well a model is classifying samples i.e. saying which category they belong to. Your problem (as the author in your link states) is a regression problem, because you are predicting a continuous variable (temperature). Have a look here for more information.

In general, if you do have a classification task, printing the confusion matrix is a simple as using the sklearn.metrics.confusion_matrix function.

As input it takes your predictions and the correct values:

from sklearn.metrics import confusion_matrix

conf_mat = confusion_matrix(labels, predictions)
print(conf_mat)

You could consider altering your task to make it be a classification problem, for example by grouping the temperatures in to classes of a given range.

You could say transform the target temperature to be a new_target_class, then change your code to use the [RandomForestClassifier][3].

I have done a quick and dirty conversion on the same data linked in that article, check it out here. I basically use the minimum and maximum values of the target variable to set a range, then aim for 10 different classes of temperature and create a new column in the table which assign that class to each row. The top it looks like this (click on picture to enlarge):

If you can get those predictions going using the RandomForestClassifier, you can then run the confusion matrix code above on the results.


When I train the Network the Number of examples used for training vs. Cross-entropy, the performance of the Network it's not improving when adding new examples.

You are assuming improving new samples will always improve the performance. This is not true in many cases. Did you plot a learning curve to see how the performance is improving with respect to increasing number of samples? You probably should try to do this before delving deep into sample correlation.


What exactly is a dummy trap? Is dropping one dummy feature really a good practice?

There are two main problems -

You have one Feature which is correlated (multi-collinearity) to all the others.

If you are trying to solve using "closed-form solution", the following will happen

$y = w_0 + w_1X_1 + w_2X_2 + w_3X_3$
$w_0$ is the $y$ intercept and to complete the matrix form 1=$X_0$. Hence,
$y = w_0X_0 + w_1X_1 + w_2X_2 + w_3X_3$
Solution for $w$ is $(X^{T}X)^{-1}X^{T}y$
So, X must be an Invertible Matrix. But,
If the model contains dummy variables for all values, then the encoded columns would add up (row-wise) to the intercept ($X_0$ here)(See below table) and this linear combination would prevent the matrix inverse from being computed (as it is singular).
\begin{array} {|r|r|}
\hline
X_0 & X1 &X2 &X3 \\
\hline
1 &1 &0 &0 \\
\hline
1 &0 &1 &0 \\
\hline
1 &0 &0 &1 \\
\hline
\end{array}

why my model got successfully trained at all since if the article is to be believed, the transposition error should have prevented it from being successfully trained.

Valid question! Aurelien Geron(Author of "Hands-On Machine Learning" has answered Here.
- The LinearRegression(Scikit-Learn) class actually performs SVD decomposition, it does not directly try to compute the inverse of X.T.dot(X). The singular values of X are available in the singular_ instance variable, and the rank of X is available as rank_
On Performance

In a practically large dataset, a closed-form solution is not preferred. May use
an Iterative approach algorithm i.e. Gradient-Descent
Multi-collinearity too, will not impact the performance but Interpretability of Features.
Coeff changes depending upon which dummy is removed - This is obvious as each dummy is now a Feature with a different level of contribution (based on data). The only thing that is sure is their effect together and one-less is the same.
The inconsistent result you are getting should be due to some other issue.

Dummy-variable Trap

I have never heard of this term except "Udemy course A-Z ML". So I don't think that there is any special meaning of the word "trap" if you understand the points(i.e. Singularity, Multi-collinearity, and Interpretability) separately
References -
www.feat.engineering - Sec#5.1
Sebastian Raschka
stats.stackexchange


How do I deploy a model when using Stratified K fold?

k-fold CV is meant to evaluate the model. Once the evaluation is done and one is ready to move to deployment, there's no point using CV anymore: the method has been tested and validated, so one can reasonably assume that from now on applying the same method to the same kind of data will lead to the same level of performance. Thus the usual process is:

Train a final model on the full dataset (no CV, no testing)
Apply the model to new instances


Is it my Sales growth is depend on Commisions/Discounts?

Multiple regression sounds appropriate in this case. Real question is what variables to use. Definitely, you should include indicator variables commisions and discounts (and possibly their combination - commisions_and_discounts - as a separate variable). Date and time information may play important role as well, though treating them is a little bit harder. At the very least, it's worth to include year (because there may be a global trend), month and day_of_week (many patterns are repeated periodically).

Product type is a little bit more complicated. Obviously, some products will always have higher sales than others, so first idea is to include a set of product_type_x dummy variables. But product type may affect sales not additively, but instead multiplicatively, i.e. not as

sales ~ beta_0 + beta_1*product_type_x + ...

but instead as

sales ~ beta_1 * prodcut_type_x * (beta_0 + ...)

In this case better solution would be to create separate regression models for each product type.


Generating normally distributed data frame with 3 columns

A possibility in the following code snippet.

import pandas as pd
import numpy as np
df = pd.DataFrame()
mean = [0,1,2]
scale = [1,2,3]
max_val = [2,3,4]
min_val = [-2,-1,0]
for n in range(3):
    df[n] =  np.random.normal(loc=mean[n],scale=scale[n], size=100)
    df.loc[df[n] < min_val[n],n]= min_val[n]
    df.loc[df[n] > max_val[n],n]= max_val[n]
print('The Dataframes mean is: ',df.mean().mean())


This is no longer the case; as of sklearn 0.20.0, missing values are ignored in such preprocessors' fit and silently passed along in their transform:
https://scikit-learn.org/stable/whats_new/v0.20.html#id37 (fourth bullet)
https://github.com/scikit-learn/scikit-learn/issues/10404


why trainable parameters are not considered right?

In your code by default training=False, set training=True it will work right.
Also, since you are already subclassing keras.Model you don't need to again call keras.Model. Remove the model() method, pass the input directly to call() method by setting training=True.


Is it possible to have differential weights as per input importance for a Neural Network

Yes! It is totally possible, generally weights are never same, they differ for different inputs.This is the first reason why they are called weights, as they associate weight to every input.

Preparing neural network parameters (weights and bias) using TensorFlow Variables in python:(assuming you have 3 inputs and want to assign them with .3,.1 and .8 weight respectively.)

weights = tensorflow.Variable(initial_value=[[.3],[.1],[.8]],dtype=tensorflow.float32)
bias = tensorflow.Variable(initial_value=[[1]], dtype=tensorflow.float32)

Happy to answer.


If you have a dataframe df1 with two columns Id and CategoryId you can chain get_dummies and groupby, e.g.

>>> df2 = df1['CategoryId'].str.get_dummies().groupby(df1['Id']).max()
>>> df2
    A  B  E  F
Id
1   1  1  0  0
2   1  0  1  1

It's not quite the format you wanted but it avoids the lookup.


Data Extraction from images using NLP and ML

You need to perform OCR (Optical Character Recognition).
However, this can lead to bad results if your images are of very bad quality, or they are skewed and distorted).
One tool you can use is tesseract tesseract from Google. You can train it for your purpose to recognize the text in your images.


How curvature information in second order optimization methods helps

You need to consider two steps of your first order optimisation process to see why a second order method can be usefull. (For more clarity we'll work in one dimension).
First step : calculate the derivative and move your evaluation point accordingly.
Then, second step : calculate the derivative and move your evaluation point accordingly.
If the derivative in the second step is bigger / lower than in the first step, you'll move less/more in the second step than in the first step. So the rate of change of the derivative along your path impact the change of your step size. So if you had information on the rate of change of the derivative in the frist step it may have been better to incorporate it directly in your first step calculation. In the end it is rather intuitive that the second derivative may able to find better suited step sizes.
In practice it is a bit more complex as the actual gain in performance depends on how the second order derivative is calculated / approximated. (See here)


External cluster evaluation for a varying number of cluster

Most measures such as (Adjusted,Normalized) Rand index / normalized mutual information do perform an optimal n:m-like comparison by design.

They avoid having to make the assumption that cluster A in one clustering is cluster X in the other; instead they look at the resulting object pairs instead. Methods that rely on "set matching" are very uncommon because of this reason. It appears to mostly be  classification people that think you should be using the precision and recall of a 1:1 matching, because that is the measure they always use.

In the Rand index, Mutual Information, etc. moving a single element from a cluster into a small second cluster reduces the quality roughly by 1/n initially; and once the resulting cluster starts becoming larger even less than that (until eventually when most elements have moved, the index starts I proving again). That is not what I would call inappropriately worse, nor "not very good at comparing clusters of different sizes". I think you should provide examples for your claim.

I am not convinced by the Rezaei paper. On one hand, they seem to assume that behavior must be linear in their synthetic data change (Rand index would often be linear; it's the desirable adjustment for chance that produces the non-linearity, and his can we'll be used as an argument that the measure should not be linear). Second, they seem to require that every cluster must have equal weight, so increasing the size of a "perfectly found" cluster must not affect the outcome. Here, again, I beg to disagree. Cluster influence should depend on the number of objects in the cluster. From the "cluster size imbalance" plot my take is that once you add a lot of 1-element clusters, the proposed PSI measure starts looking much worse, because it cannot deal with such noise.


The answer given so far is wrong.
1.) In general, adding more and more layers does not lead to better training accuracy.
2.) The issue is certainly not overfitting, as Andrew Ng is talking only about the training set.
The answer becomes true if you add layers, where each additional layer $l$ is represented by a mapping $f^{(l)}_{w}: \mathbb{R}^{n_{l}} \rightarrow \mathbb{R}^{n_{l}}$ for which weights $w$ exist, such that $f^{(l)}_{w}$ is the identity map. In such a case, you can add arbitrary many layers to an existing neural network architecture and in theory optimizing this new network cannot be worse than without the additional layers (since all additional layers can become the identity map). Potentially these new layers could improve the training accuracy.
In practice this is not observed (see the ResNet paper). The reasons are mainly
1.) The employed solvers do not deliver the global optimal solution.
2.) Adding more layers makes the gradient computation more prone to numerical errors due to the involved chain rule, e.g. have a look here.


R: Revalue multiple special characters in a data.frame

That seems pretty straight forward to me. I'm pretty new too but in general I'm not sure if you can get better than one command. Though you could have combined all that:

> newValueVector <- c("+"="5", "?"="2", "K"="3", "k"="3")
> data$Multiplier <-  revalue(data$Multiplier, newValueVector)


Generalization bound (single hypothesis) in "Foundations of Machine Learning"

You are right. The relaxed inequality
$$R(h) \le  \hat{R}_S(h)+ \epsilon.$$
can be replaced with the complete inequality
$$\left |\hat{R}_S(h) - R(h) \right| \le \epsilon.$$
Actually, authors use this complete inequality for the follow up examples in the book. Again in Theorem $2.13$, they write the relaxed inequality, but prove for the complete inequality.

We could say that the relaxed inequality is written for the sake of readability and/or convention.

On the relation of inequalities

Let us denote:
$$A:=\hat{R}_S(h) - R(h) \le \epsilon$$
$$B:=\hat{R}_S(h) - R(h) \ge -\epsilon$$
thus,
$$\left| \hat{R}_S(h) - R(h) \right| \le \epsilon = A \text{ and } B$$
Equation $(2.16)$ states:
$$\begin{align*}
& {\Bbb P}(\left| \hat{R}_S(h) - R(h) \right| \ge \epsilon) \le \delta \\
& \Rightarrow {\Bbb P}(\left| \hat{R}_S(h) - R(h) \right| \le \epsilon) \ge 1 - \delta \\
& \Rightarrow {\Bbb P}(A \text{ and } B) \ge 1 - \delta \\
\end{align*}$$
knowing that ${\Bbb P}(B) \ge {\Bbb P}(A \text{ and } B)$,
$$\begin{align*}
& {\Bbb P}(B) \ge {\Bbb P}(A \text{ and } B) \ge 1 - \delta \\
& \Rightarrow {\Bbb P}(\hat{R}_S(h) - R(h) \ge -\epsilon) \ge 1 - \delta
\end{align*}$$
which is equivalent to
$$R(h) \le \hat{R}_S(h) + \epsilon$$
with probability at least $1-\delta$, i.e. equation $(2.17)$.


Integer programming formulation: which algorithms

You may reparameterize $x_i$ as $x_i = sigmoid(\alpha \cdot z_i), z_i \in \mathbb{R}$, with $\alpha$ being a constant such that $\alpha \gg 1$, and then proceed with your favorite optimization method for finding $z_i$, and from there $x_i$.

This, of course, may present problems, as $x_i$ are no longer bounded in $\{0, 1\}$, but may suffice your use case.


I don't understand RidgeCV's fit_intercept, and how to use it for my data

The weight on the constant column was also penalized by Ridge, whereas the intercept fitted directly is not.

To your second question: Centering shouldn't be necessary (except maybe for computational reasons), though scaling is still recommended so that coefficient penalties are applied more equally.


The explanation in the documentation of the Huggingface Transformers library seems more approachable:

Unigram is a subword tokenization algorithm introduced in Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (Kudo, 2018). In contrast to BPE or WordPiece, Unigram initializes its base vocabulary to a large number of symbols and progressively trims down each symbol to obtain a smaller vocabulary. The base vocabulary could for instance correspond to all pre-tokenized words and the most common substrings. Unigram is not used directly for any of the models in the transformers, but it’s used in conjunction with SentencePiece.
At each training step, the Unigram algorithm defines a loss (often defined as the log-likelihood) over the training data given the current vocabulary and a unigram language model. Then, for each symbol in the vocabulary, the algorithm computes how much the overall loss would increase if the symbol was to be removed from the vocabulary. Unigram then removes p (with p usually being 10% or 20%) percent of the symbols whose loss increase is the lowest, i.e. those symbols that least affect the overall loss over the training data. This process is repeated until the vocabulary has reached the desired size. The Unigram algorithm always keeps the base characters so that any word can be tokenized.
Because Unigram is not based on merge rules (in contrast to BPE and WordPiece), the algorithm has several ways of tokenizing new text after training. As an example, if a trained Unigram tokenizer exhibits the vocabulary:
["b", "g", "h", "n", "p", "s", "u", "ug", "un", "hug"],
"hugs" could be tokenized both as ["hug", "s"], ["h", "ug", "s"] or ["h", "u", "g", "s"]. So which one to choose? Unigram saves the probability of each token in the training corpus on top of saving the vocabulary so that the probability of each possible tokenization can be computed after training. The algorithm simply picks the most likely tokenization in practice, but also offers the possibility to sample a possible tokenization according to their probabilities.
Those probabilities are defined by the loss the tokenizer is trained on. Assuming that the training data consists of the words 1,…, and that the set of all possible tokenizations for a word  is defined as (), then the overall loss is defined as
$\mathcal{L} = -\sum_{i=1}^{N} \log \left ( \sum_{x \in S(x_{i})} p(x) \right )$

There are some parts that are not very detailed, though, for instance, how it initializes the base (seed) vocabulary to a large number of symbols".
This part is more clearly explained in the original article by the end of section 3.2:

There are several ways to prepare the seed vocabulary. The natural choice is to use the union of all characters and the most frequent substrings in the corpus. Frequent substrings can be enumerated in $O(T)$ time and $O(20T)$ space with the Enhanced Suffix Array algorithm (Nong et al., 2009), where T is the size of the corpus.

About the details of the expectation maximization algorithm used to compute probabilities, this is what happens:

[Expectation] Estimate each subword probability by the corresponding frequency counts in the vocabulary
[Maximization] Use the Viterbi algorithm to segment the corpus, returning the optimal segments.

You can check the details, together with practical examples, in this tutorial.


What is the difference between fit() and fit_generator() in Keras?

In keras, fit() is much similar to sklearn's fit method, where you pass array of features as x values and target as y values. You pass your whole dataset at once in fit method. Also, use it if you can load whole data into your memory (small dataset).

In fit_generator(), you don't pass the x and y directly, instead they come from a generator. As it is written in keras documentation, generator is used when you want to avoid duplicate data when using multiprocessing. This is for practical purpose, when you have large dataset.

Here is a link to understand more about this-

A thing you should know about Keras if you plan to train a deep learning model on a large dataset

For reference you can check this book- https://github.com/hktxt/bookshelf/blob/master/Computer%20Science/Deep%20Learning%20with%20Python%2C%20Fran%C3%A7ois%20Chollet.pdf


Technique/Algorithm for product categorization Machine Learning

As Sean Owen said, it's a wide problem so I can't be generic on this one. I'll just give you an example on how you could tackle this:

Let say you sell some products on a website. If you have 100 different products, you can represent each sale by a 100 dimensional vector, where each component is the count of ordered product. Example: sale1 = (2,0,1,0,...,0) means 2 item1 and 1 item3 sold. We use the same technique with words, and it is called bag of words. If you stop here, you only can perform statistical counts on individual product sales. So let's use matrix factorization.

If you sold products to 1000 people, then you can concat all your sales in a matrix of dimension 1000 X 100 where each row represent one sale and each column represent one product. Using decomposition algorithm such as SVD, you can find out singular values in your data that will show you latent factors in your sales (basically trends or connections between your products).


Churn Prediction Model based on Customer Activities

First of all, you need to see your churn rate to see how rare it is. If your churn rate is really low ( below < 5% of total labels ), then you could treat this as an anomaly detection system, meaning you train your model on non-churn users ( if churn is the label you're predicting then the non-churns would have 0 as label ), and flag any activity different from what you trained your model on as an anomaly ( in case you don't have enough examples of activities that led to churn ), or if you have enough 'churners' to train your model on, treat this as a classification problem. I'll suggest some indicators for you even though i haven't worked on a churn model before.

Assuming you only have email activity ( following your statement ) you could :

create a feature count on the email activity to see if there's a pattern to churn after certain numbers of non-reads,non-clicks,non-opens.
try to collect more features if there are more because 3 is a low feature count for a predictive model ( maybe not only the activity but the user's personal information like age / type of profession etc.. )
you can also see this article discussing user churn.

Hope this helps you.


How to deal with analyzing optional survey data

Your best bet is to change the way you ask question C and the way you measure it. Giving a yes / no response for C will not let you gauge the actual spiciness so you need to adjust it to a scale and allow individuals to rank it. While all of your questions are fairly subjective, C is the most so as tolerance to spice plays a big role in if one thinks it spicy or not.

I suggest using a 1-5 scale, as you have for the others, and give comparative suggestions for each level of the scale e.g. 1 is eating bread while 5 is equivalent to eating a habnero pepper.


Calculate number of parameters for ConvLSTM2D layer

i=512, input dimension or input channel

h=128, output dimension or output channel

k=3, kernel size

number of parameters = 4 * h * ( k**2 * (i+h) + 1 )  = 2949632

if k == 1, ConvLSTM is just a LSTM.

You can check the original paper for detail.
https://arxiv.org/abs/1506.04214


Train error vs. Test error in linear regression by samples analysis

It occurs because your data is noisy. Suppose that your model is
$$ y = X\beta + noise. $$
Suppose that $\beta$ is recovered exactly. Since the noise is present, our model will NOT predict accurately, it will be always some noise. In other words, think that any time we predict the values, there is some random perturbation. The amount of perturbation is of the same order each time. When we compute the error, we square these perturbations, sum them up, and then take the square root, i.e., compute the norm
In your example, the size of the training set is growing, and, therefore, the test error WITHOUT averaging is growing as well -- you sum up squares of the perturbation. The size of the test set is steady, so the norm of error is about the same there.
Solution: use appropriate averaging to both train and test set.
train_loss = (1/np.sqrt(i) * tf.tensordot(tf.transpose(h(X2[:i], weights) - Y[:i]), (h(X2[:i], weights) - Y[:i]), axes=1))[0][0]
test_loss = (1/(2 * X2.shape[0]) * tf.tensordot(tf.transpose(h(X2, weights) - testY), (h(X2, weights) - testY), axes=1))[0][0]

P.S. X2 in my formulas are as defined in your cell, accordingly. I would suggest choosing a different name for the variable.


How to properly predict data with LSTM using train/test?

You seek to augment the external validity of your model.

The most common way of doing so is by applying k-fold cross-validation to verify that your model generalizes well on unseen data.

  In k-fold cross-validation, the original sample is randomly
  partitioned into k equal sized subsamples. Of the k subsamples, a
  single subsample is retained as the validation data for testing the
  model, and the remaining k − 1 subsamples are used as training data.
  The cross-validation process is then repeated k times, with each of
  the k subsamples used exactly once as the validation data. The k
  results can then be averaged to produce a single estimation.

This will reduce the variance of your model and will reduce its error on unseen data.


why do transformers mask at every layer instead of just at the input layer?

If the masking were only applied in the first layer, the self-attention in the subsequent layers would bring to each position information from future tokens.
Let's break it down with numbers:

At layer $i$, if causal masking is applied, the output at position $t$ contains information about layer $i-1$ at positions $1..t-1$, that is, $L_{i,t} = f_i(L_{i-1,1},...,L_{i-1,t-1})$.

If no causal masking is applied, then the output at position $t$ contains information about layer $i-1$ at all positions in the sequence of length $T$, that is, positions $1..T$ $L_{i,t} = f_i(L_{i-1,1},...,L_{i-1,T})$

If causal masking is applied at layer 1 (the first layer) but not at layer 2 or 3, we obtain that for position t at layer 3 we would have: $L_{3,t} = f_3(L_{2,1},...,L_{2,T}) = f_3(L_{2,1},...,f_1(L_{1,1},...,L_{1,T}))$, which means that position $t$ contains information from future tokens, as $T > t$.

Note: The original answer was wrong and was completely edited. To check the original answer, refer to the post timeline.


If deep-learning learns features, aren't we saying it can learn association rules?

Finding valuable association rules is defined base on their support and confidence. ANN is not proper for these kinds of specifications.
Hence, if we use ANN for learning association rules, we can't guarantee the specified support and confidence are satisfied.

Anyhow, there are other machine learning methods to learn association rules such as decision tree. Although, that methods not such that proper to find rules with specified support and confidence. You can know more about this in this article.


What does the “numDecimalPlaces” in J48 classifier do in WEKA?

The short answer is: nothing. The numDecimalPlaces option, like debug and doNotCheckCapabilities, is part of the base class that all classifiers in WEKA inherit from. However, it is up to the implementation of the actual classifier to use this value to change the way the model is printed. There are no mentions of numDecimalPlaces anywhere in the source code for J48, which suggests to me that it does nothing in this case.

It looks like the only classifier in WEKA that does anything with the value entered into numDecimalPlaces is LinearRegression.


Is the distance in Nearest Neighbors a good measure of similarity?

This is counter-intuitive, because one would expect [0,2,0,0] to be more similar to [0,1,0,0] than [0,1,1,0].

No this is expected, since the two points are exactly at the same distance in the Euclidean space. To see it take a simplified 2D version of your points:

A (1,0)
B (2,0)
C (1,1)

Both B and C are exactly at distance 1 from A.

  But for my dataset Jaccard metric makes the kNN queries taking very long time, perhaps it is more suited for binary features.

Jaccard actually assumes binary features, it will consider all non-zero values the same way. Its result is based on how many non-zero dimensions the two points have in common. I'm assuming that the implementation follows the original definition, there might be variants. Normally it's a very simple measure which doesn't require any heavy computation so it's surprising that it takes a long time.

  When I checked that other one I could not see ANY similarities, in fact most of the features where very dissimilar by value and/or by sign. A change of sign has no particular importance in the euclidean space, it's the distance which matters.

Well the only way to really check would be to calculate the euclidean distance, with 52 dimensions just looking at the values is not going to give a good indication.

  I could get the same distance with a made-up sample of the first row from the training set where any one feature was increased by 0.02.

I'm not sure I understand this part but again this sounds perfectly correct: changing any feature by 0.02 will move the data point in the space by a distance of 0.02 so... it's going to be at distance of 0.02 from the original point.

  I am wondering now how to overcome this problem and if there is an easy way (ie. by tweaking parameters of NearestNeighbors) or a hacky way (ie. custom metrics, feature weights etc.)

If there is any data-specific way to measure the distance between two points, you should definitely define a custom metric. It's not a hack, as you can see NN relies entirely on the metric. The default euclidean distance is a generic measure, it might not be fit for your purpose.


Where can Batch Normalization be used? CNNs or everywhere?

A Batch Normalization layer essentially is normalizing outputs of your hidden units, so their outputs would always have a similar scale. By eliminating such internal covariate shift in outputs of hidden layers, your deeper layers become more independent from their previous layers.

Nothing here is CNN specific - Batch Normalization may be applied to Fully Connected and Recurrent neural networks as well, but they are more useful with deep neural networks, which tend to accumulate this shift with each layer activated during Forward Propagation.


Why infinite sampling is not realisitc assumpition in most real applications

It is not a realistic assumption because you don't have infinite time or decimal precision to find the absolutely correct value function, but you don't need that anyway since a rough estimate of it will be enough to improve the policy.

If your question is why you can't let the agent learn indefinitely in real applications, I'm guessing it is because it may be potentially expensive or dangerous to let it explore randomly in a real scenario, so you want to deploy it with an optimal or near-optimal, deterministic, policy.


I have the same problem, I was told to avoid Excel as it did not conform to the ISO 8601 standard of 2016-12-27.

Then I transferred the data to Google Sheets as illustrated in Orange tutorial YouTube videos.

The dates were altered to match the required format however the results remain exactly the same.


Do we need the Preprocessing step on both Test and Train data sets?

There are a few things that you need to be careful with here.

You can do certain things when preprocessing data or performing data augmentation that can be applied across an entire dataset (train and validation). The main idea is not to allow the model to gain insight from the test data.

Time-series example

Missing data can be managed in many ways, such as simple imputation (filling the gaps). This is very common in time-series data. In your training data, you can fill the gaps using the previous value, the following value, the average of the data or something like the moving average. Where you must be careful is with violating the information flow through time. For example, in your test data, you should not fill gaps using a method that looks at data points in front of the emtpy time slot. This is because, at that point in time, you will not be able to do the same as you shouldn't know the future values.

Image data example

Looking instead at image data, there are data-preprocessing steps such as normalisation. This means just scaling the image pixel values to a range like $[-1, 1]$. To do this, you must compute the population mean and variance, which you then use to perform the scaling. When computing these two statistics, it is important not to include the test data. The reason is that you would be leaking information the dataset that is then used to train a model. Your technically knows things that it shouldn't; in this case, clues regarding the mean and variance of the target distribution.

People might also consider "missing data" to include imbalanced datasets; i.e. there are cases that you know of, but just don't appear in your dataset very often. There are some tricks to help with this, such as stratified sampling or cross-validation. The optimal solution would, of course, be to gather a dataset that more closely represents the problem at hand.


Does the test set has to be in [0,1] range?

Standardization centers the values around a mean of $0$ with standard deviation $1$. Therefore, having values smaller than $0$ or greater than $1$ is to be expected. If you want to make sure values are between $0$ and $1$ you need to normalize the data instead.
Here is an example of the two procedures taken from the book "Python Machine Learning" by Raschka:

Be aware though to apply the procedure to your test data with parameters obtained from the training data (in case of standardization: mean and std. dev. of train data).
Sklearn has methods for standardization and normalization which you might want to have a look at.


is there an adequate number of levels of categorical variables?

No, there's no "adequate" number of levels for a categorical variable.
The choice to simplify the data by discarding some levels (for example by using a default category, as you propose) depends on what the goal is (and also number of instances, etc.). Very often this choice is made experimentally, that is by trying different methods (e.g. different thresholds) and observing which one gives the best performance: here you could do a program which tries the different proportions as threshold, then train and test the resulting model for every value, and finally plot the performance for every value.


Correct learning algorithm for grouped examples

Let $M$ be the size of each group. You want to learn the function $f$ that takes $M$ vectors $v_{i}$, $i \in [1,M]$  and yields a vector $\hat{v}$ that can be 0 or one of the $v_{i}$.
You can train a DNN to learn $f$ by organizing your training data as follows:

Input              Label
[batchSize, M, 1]  [batchSize,1]

Optimize on the reduced RMS error between the predicted vectors and the label vectors. The group information will then be learned by your DNN.


Preparing data for a live prediction engine

You'll have to do some feature engineering on your data. Beneficiary and Customer ID has no meaning for machine learning, so you first have to join all other data, taking care of date types (split hour, day of week, month) and should be ordereed by the ones you will not use for training (beneficiary, customerID). than you can create new features like for each transaction the time spent since the last one for each customer, count of transcactions by day/day of wee/month, min, max, median amout spent and son on. It'll be a binary classification (SVM, random forest, logistic regression, etc.).


Use Machine Learning/Artificial Intelligence to predict next number (n+1) in a given sequence of random increasing integers

You are solving a problem which is not designed for a ANN, neural network has difficulties when you deal with univariate, few data.

Because they will have to learn a solution based on few examples.

However, a possible solution is achievable:

import numpy as np

# Init sequence
data = [[0, 1.0], [1, 3.0], [2, 7.0], [3, 8.0],
    [4, 21.0], [5, 49.0], [6, 76.0], [7, 224.0]]

X = np.matrix(data)[:, 0]
y = np.matrix(data)[:, 1]

Reg=neural_network.MLPRegressor(solver='lbfgs',random_state=4,hidden_layer_sizes=(100,20,2),learning_rate='adaptive',verbose=True)

y2 = np.ravel(y)
F=Reg.fit(X=X,y=y2,)
F.predict(8)

Note that the random_state parameter has a very large influence when I use a Neural Network. If you move this parameter (integer) you will find that the solution has a really large range.


Collaborative filtering with human-adjusted latent factors

What you are looking for in this case is matrix factorization with side information. And yes, there are quite a lot of works in the area. I would recommend you this paper as a start: Relational learning via collective matrix factorization.


Data augmentation for the inputs of CNNs to identify flowers

As you've mentioned your task is classification and due to using images, it is better using convolutional neural networks. First I have a suggestion, try to find an appropriate size with the same dimensions for all the images and feed them to your network. you can keep aspect ratio or not depending on the environment you are going to test your model. You can also take a look at Why do we scale down images before feeding them to the network.

  Which will be better: to put whole images into training set or to divide each image into 4 parts (cutting in midline horizontally and vertically)?

I guess you attempt to do data augmentation. If so, it depends. If you do that, you may have images that do not contain the flower or they may contain a part of a flower which is common among different classes. Consequently, it may increase the Bayes error. If you are sure you do not have these problems, you can use it albeit I do not think so.

  Will it help if I rotate/tilt these images and put them also in training set?

Yes, it is better than the previous technique. You have to be aware that this is dangerous in some cases. Basically, you should train your network using the training set which is given i.i.dly from the real distribution that your test data has. Suppose that while testing your model, your test data all are given from flowers which are vertically placed in the scene. In such cases, if you train your network with just rotated versions, you may not have a good test result.

  Will it help if I blur these images and put them also in training set?

Again like the previous answer, it depends on your test data. If it is something that happens while testing, it is legitimate.

I didn't notice to the images. Based on the comment of our friend, I update the answer. Your classes can be one these things. They can be exhaustive or not and they can be mutually exclusive or not. if the former is satisfied, it means that the input should at least belong to one class. The latter means that if you have mutually exclusive classes, the inputs should contain only one class. If they are not mutually exclusive, you can have multiple classes in a single input.

To add an update for the answers above you should consider that data augmentation with reduction can be difficult because it needs an expert to label the inputs by hand which is time consuming.


What is the difference of R-squared and adjusted R-squared?

A google search for r-squared adjusted yielded several easy to follow explanations.  I am going to paste a few directly from such results.

Meaning of Adjusted R2
Both R2 and the adjusted R2 give you an idea of how many data points fall within the line of the regression equation. However, there is one main difference between R2 and the adjusted R2: R2 assumes that every single variable explains the variation in the dependent variable. The adjusted R2 tells you the percentage of variation explained by only the independent variables that actually affect the dependent variable.

What Is the Adjusted R-squared?
The adjusted R-squared compares the explanatory power of regression models that contain different numbers of predictors.
Suppose you compare a five-predictor model with a higher R-squared to a one-predictor model. Does the five predictor model have a higher R-squared because it’s better? Or is the R-squared higher because it has more predictors? Simply compare the adjusted R-squared values to find out!
The adjusted R-squared is a modified version of R-squared that has been adjusted for the number of predictors in the model. The adjusted R-squared increases only if the new term improves the model more than would be expected by chance. It decreases when a predictor improves the model by less than expected by chance. The adjusted R-squared can be negative, but it’s usually not.  It is always lower than the R-squared.


Image Segmentation with a challenging background

It would be appreciated if you could explain precisely what your goal is:

you want to identify what animal is in your picture ?
you want to count the number of animals ?
you want to get the position of each animal in the picture ?

In any case, I know that you can get some already trained neural nets from google or anywhere else. This neural net can be used with caffe as it is the case in this google deepdream stuff on github (look at ):

https://github.com/google/deepdream/blob/master/dream.ipynb

Then, if you want to highlight or identify the positions of your animals, you'll find this article inspiring:

http://www.matthewzeiler.com/pubs/cvpr2010/cvpr2010.pdf

It explain how to reverse convolutional networks to identify what part of your image helped to recognize what is inside. The found projection get you something similar to your second picture (called a mask), but depending on the neural net you use, you can get better results.


K-medoids of course works for purely categorical data.

But there also is k-modes, easy to find with a little bit of research... It's much cheaper to run, it is O(n), not O(n²)

It doesn't solve the problem that categorical data usually will not cluster in a meaningful way...


Maybe a little bit late to the party, but this might help you in the future:
Neural networks in general are heavily overparameterised functions that are fitted to match the training data. This is true for simple MLPs or more complex RNNs. These functions never encapsulate the "true" function, but only approximate it.
In you data, the spikes are simply outliers (meaning that they cannot be explained by the given, underlying input data). This means that to approximate those outliers, a function of your given inputs is not enough, since you are "averaging out" or ignoring these spkies as outliers during training (I assume you have a large dataset to train on).
See it that way: the given input data to your approximated function is not enough to explain the full "world" that your time series exists in.


Difference between learning_curve and validation_curve

Both curves show the training and validation scores of an estimator on the y-axis.

A learning curve plots the score over varying numbers of training samples, while a validation curve plots the score over a varying hyper parameter.

The learning curve is a tool for finding out if an estimator would benefit from more data, or if the model is too simple (biased).

Above example shows the training curve for a classifier where training and validation scores converge to a low value. This classifier would hardly benefit from adding more training data; a more expressive model may be more appropriate.

The validation curve is a tool for finding good hyper parameter settings. Some hyper parameters (number of neurons in a neural network, maximum tree depth in a decision tree, amount of regularization, etc.) control the complexity of a model. We want the model to be complex enough to capture relevant information in the training data but not too complex to avoid overfitting.

Above example shows the validation curve over a support vector machine's gamma parameter. A too low value of gamma restricts the model too much; both, training and validation scores are very low. A high value of gamma causes overfitting: very good training score but low validation score. The optimal value is somewhere in the middle, where the curves do not diverge too much.

Image source: scikit-learn documentation


For logistic regression, Predict.glm() outputs $p$ or $ln(p/1-p)$?

It returns the log odds. You can see that with this basic example,

# Create a perfect correlated data.
data <- data.frame(x = c(1,0,1,0), y = c(T, F, T, F))

# Estimate the model.
model <- glm(y ~ x, data)

# Show predictions.
predict(model)
##        1         2         3         4
## 23.56607 -23.56607  23.56607 -23.56607

# Compute the inverse of the log odd to come to your prediction.
boot::inv.logit(predict(model))
##           1            2            3            4
##1.000000e+00 5.826215e-11 1.000000e+00 5.826215e-11

If logit would return probabilities you could not take the inverse to come to your predicted values.


What are the default values of nodes and internal layers in Neural Network models?

There are websites that explain these pretty well.

Deciding on the number of neurons in the hidden layer(s)

From https://www.r-bloggers.com/selecting-the-number-of-neurons-in-the-hidden-layer-of-a-neural-network/:

  The most common rule of thumb is to choose a number of hidden neurons between 1 and the number of input variables.

Deciding on the number of layers of hidden layers

From https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw:

  For most problems, one could probably get decent performance (even without a second optimization step) by setting the hidden layer configuration using just two rules: (i) number of hidden layers equals one; and (ii) the number of neurons in that layer is the mean of the neurons in the input and output layers.

Hope that answers your question!


Topic modeling for short length sentences

A possible approach might be to use the most predictable and predictive word(s) in each cluster as its name(s). The following is inspired by Fisher's category utility used by the COBWEB algorithm.

An attribute is highly predictable in a cluster $C_l$ if most elements in that cluster have the same value $a$ for it, thus a word represented by attribute $A_i$ has a predictability of $P(A_i=a|C_l)$

An attribute is highly predictive for a cluster $C_l$ if knowing it's value $a$ implies you can say with high certainty to which cluster it belongs, expressed as $P(C_l|A_i=a)$

Assume now you process each label org.java.somepackage.validateLogin as a sentence: "org java somepackage validate Login" and apply one-hot encoding to all your sentences in the dataset. The occurrence of a word is now represented by a value of 1 for it's corresponding attribute.

The task of representing each cluster by a word can now be formulated as finding for each cluster $C_l$ the word represented by attribute $A_i$ that equals 1 and has the highest predictability and predictiveness, weighted by the total probability of this word being in a sentence.

$$ P(A_i=1|C_l)\times P(C_l|A_i=1) \times P(A_i=1)$$

Which can be calculated by counting occurrence of a word per cluster and cluster per word.


How regularization helps to get rid of outliers?

You don't get rid of the actual outliers (no data reduction). But statistical methods can be robust against outliers.

  Robust statistics are statistics with good performance for data drawn
  from a wide range of probability distributions, especially for
  distributions that are not normal. Robust statistical methods have
  been developed for many common problems, such as estimating location,
  scale, and regression parameters. One motivation is to produce
  statistical methods that are not unduly affected by outliers.

Source: wikipedia.

So, L-1 regularization is robust against outliers as it uses the absolute value between the estimated outlier and the penalization term.

Whereas, L2-regularization is not robust against outliers as the squared terms blow up the differences between estimation and penalization.


Looking for a good package for anomaly detection in time series

I know I'm bit late here, but yes there is a package for anomaly detection along with outlier combination-frameworks.
The package is in Python and its name is pyod. It is published in JMLR.
It has multiple algorithms for following individual approaches:

Linear Models for Outlier Detection (PCA,vMCD,vOne-Class, and SVM)
Proximity-Based Outlier Detection Models (LOF, CBLOF, HBOS, KNN, AverageKNN, and MedianKNN)
Probabilistic Models for Outlier Detection (ABOD and FastABOD)
Outlier Ensembles and Combination Frameworks(IsolationForest and FeatureBagging)
Neural Networks and Deep Learning Models (Auto-encoder with fully connected Neural Network)

Finally, if you're looking specifically for time-series per se, then this github link will be useful.
It has the following list packages for timeseries outlier detection:
datastream.io
skyline
banpei
AnomalyDetection


Which is the "BEST" deep learning model for "Custom" object detection for images & real time. YOLO v3, v4, v5, EfficientDet?

There are many subtle differences between yolov4 and v5 other than speed like YOLOv4 exceeds YOLOv5's performance on the COCO benchmark.this link will help you explain.
YOLOv4 VS YOLOv5.
I dont know much about EfficientDet.
Here's YOLOv3 Versus EfficientDet for State-of-the-Art Object Detection
.
It explains quite nicely difference between the two.


Image features (spectral bands) other than RGB for image analysis

I think what you are looking for is called a colour space. There are multiple available, and each of them was created to solve a certain problem.

The RGB colour space is the most common I guess, since that is close to the way the computer displays show information (although they do apply some transformations before lighting the pixels). It is also similar to the way we see light (we have red, green and blue receptors).

Light is additive. If you keep adding colours, you get to white. Paint is subtractive. That leads to some differences into how colours mix and lead to the creation of the CMYK standard for printing. It represents colours in terms of cyan, magenta, yellow and black.

Perhaps more important for you is the HSB color space. It is rather nicely shaped, as a 3D cylinder. H stands for hue, the colour nuance, S stands for saturation, how pronounced is the colour and B stands for brightness. HSL is a similar color space, where the vertical axis is lightness instead of brightness. I find these two color spaces quite useful in analysis.

The RG cromacity space is used to analyse the color itself, and discard the brightness information. You basically represent the amounts of red and green from a normalized color (a color where the sum of r g b is 1). This way, you operate in a simpler, 2D space, and analyze the color information. See [1], [2].

The CIE colorspace is based on measurements of our color perception. It tries to mimic the way our photoreceptors react to color. The original CIE colorspace, known as CIEXYZ, was published in 1931. It has been revised in 1976 to form the CIELab colorspace, which is perceptually uniform. This means that it accounts for our non-liniar perception of color, making sure that the same increase produces the same perceptual difference across the color space. See [3], [4], [5].

The TSL colorspace (tint, saturation, lightness) was proposed in [6] for face detection.

There are many other colorspaces, but these are the ones I found most useful. There are usually conversions across different color spaces.

A publication by Logvinenko, 2014 might help with understanding the geometry of a color space and choosing a colorspace that helps your analysis [7].
Also, have a look at [8].


After extracting C50.tgz file, give execution rights to Makefile.

chmod +x Makefile

Then install csh

sudo apt install csh

Run below command to check if csh is install and check csh version (if installed)

dpkg -l csh

It should show output like below.

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                           Version              Architecture         Description
+++-==============================-====================-====================-==================================================================
ii  csh                            20110502-3           amd64                Shell with C-like syntax

Now run below command.

./Makefile

It should give output like below. (Ignore the warning messages).

./Makefile: line 9: CC: command not found
./Makefile: line 10: CFLAGS: command not found
./Makefile: line 11: S: command not found
./Makefile: line 11: LFLAGS: command not found
./Makefile: line 12: SHELL: command not found
./Makefile: line 19: src: command not found
./Makefile: line 48: obj: command not found
./Makefile: line 59: all:: command not found
cat defns.i global.c c50.c construct.c formtree.c info.c discr.c contin.c subset.c prune.c p-thresh.c trees.c siftrules.c ruletree.c rules.c g     etdata.c implicitatt.c mcost.c confmat.c sort.c update.c attwinnow.c classify.c formrules.c getnames.c modelfiles.c utility.c xval.c\
        | egrep -v 'defns.i|extern.i' >c50gt.c
gcc -ffloat-store  -O3 -o c5.0 c50gt.c -lm
c50gt.c: In function ‘ListAttsUsed’:
c50gt.c:14025:12: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
      Att = (Attribute) DefSVal(D[e]);
            ^
c50gt.c: In function ‘Error’:
c50gt.c:15561:17: warning: format not a string literal and no format arguments [-Wformat-security]
     fprintf(Of, Buffer);
                 ^~~~~~
strip c5.0
rm c50gt.c
./Makefile: line 61: CC: command not found
./Makefile: line 61: LFLAGS: command not found
./Makefile: line 61: -o: command not found
./Makefile: line 67: obj: command not found
./Makefile: line 67: c5.0dbg:: command not found
./Makefile: line 68: CC: command not found
./Makefile: line 68: obj: command not found
./Makefile: line 68: -g: command not found
./Makefile: line 74: src: command not found
./Makefile: line 74: c5.0:: command not found
./Makefile: line 75: src: command not found
./Makefile: line 77: CC: command not found
./Makefile: line 77: LFLAGS: command not found
./Makefile: line 77: -O3: command not found
./Makefile: line 82: obj: command not found
./Makefile: line 85: .c.o:: command not found
./Makefile: line 86: syntax error near unexpected token `newline'
./Makefile: line 86: `  $(CC) $(CFLAGS) -c $<'

You will see a new file c5.0*
Check command line options with below command.

./c5.0 -h

it should show below output.

C5.0 [Release 2.07 GPL Edition]         Fri Mar 15 17:43:39 2019
-------------------------------

    Options:
        -f <filestem>   application filestem
        -r              use rule-based classifiers
        -u <bands>      order rules by utility in bands
        -w              invoke attribute winnowing
        -b              invoke boosting
        -t <trials>     number of boosting trials
        -p              use soft thresholds
        -e              focus on errors (ignore costs file)
        -s              find subset tests for discrete atts
        -g              do not use global tree pruning
        -m <cases>      restrict allowable splits
        -c <percent>    confidence level (CF) for pruning
        -S <percent>    training sample percentage
        -X <folds>      cross-validate
        -I <integer>    random seed for sampling and cross-validation
        -h              print this message

Now C50 is ready to be used. Here is a sample command(assuming sampledata.data and sampledata.names files exists in the same folder).

./c5.0 -r -f sampledata

After successful execution of the command, sampledata.rules will be generated in the same folder.

See full tutorial from here


Convert Lat /Lon of User input to Lat/Lon of Open Data

What's wrong with what you've suggested?  It seems fine.

You've supplied four of possible points, essentially:

(latmin, lonmin)
(latmin, lonmax)
(latmax, lonmin)
(latmax, lonmax)

Clearly one of them is the closest grid point to the user input. Then just use whatever distance calculation you like (e.g. Haversine distance, or better measurements if you have access to them) and pick which of the four points is closest to what the user specified.  It'll be accurate and performant.


You're trying to do Extractive Text Summarization.

You're first approach is the right one, using BERT + CLassifier.

From NLP Progress, the best method so far is Presumm.


Whenever you train the network using batch means that you have chosen to train using batch gradient descent. There are three variants for gradient descent algorithm:

Gradient Descent
Stochastic Gradient Descent
Batch Gradient Descent

The first one passes the whole data through the network and finds the error rate for all of them and finds the gradients with respect to all the data samples and updates the weights after passing the whole data-set. That means for each epoch, passing the whole data-set through the network, one update occurs. This update is accurate toward descending gradient.

The second one, updates the weights after passing each data which means if your data sample has one thousand samples, one thousand updates will happen whilst the previous method updates the weights one time per the whole data-sample. This method is not accurate but is so much faster than the previous one.

The last one tries to find a trade-off between the above approaches. You specify a batch size and you will update the weights after passing the data samples in each batch, means the gradients are calculated after passing each batch. Suppose you have one thousand data sample and you have specified a batch size with one hundred data sample. You will have 10 weight update for each epoch. This method is more accurate than the second approach and is more faster than the first approach.

  Do I back propagate after each batch has been presented to network or after each image?

Your method is the last one. Consequently, after passing the entire batch, you would update the weights.

Based on the comments of one of our friends, the above approaches are named as follows, respectively:

[Batch] Gradient Descent (batch size = all training samples)
True SGD (batch size = 1 - weights update for each training sample)
Mini-batch SGD (batch size = m out of n training samples).


Kaggle Titanic Survival Table an example of Naive Bayes?

Naive Bayes is just one of the several approaches that you may apply in order to solve the Titanic's problem. The aim of the Kaggle's Titanic problem is to build a classification system that is able to predict one outcome (whether one person survived or not) given some input data. The survival table is a training dataset, that is, a table containing a set of examples to train your system with.

As I mentioned before, you could apply Naive Bayes to build your classification system to solve the Titanic problem. Naive Bayes is one of the simplest classification algorithms out there. It assumes that the data in your dataset has a very specific structure. Sometimes Naive Bayes can provide you with results that are good enough. Even if that is not the case, Naive Bayes may be useful as a first step; the information you obtain by analyzing Naive Bayes' results, and by further data analysis, will help you to choose which classification algorithm you could try next. Other examples of classification methods are k-nearest neighbours, neural networks, and logistic regression, but this is just a short list.

If you are new to Machine Learning, I recommend you to take a look to this course from Stanford: https://www.coursera.org/course/ml


If your objects move slow enough you can just find the nearest object on the other frame. In your case the objects move too fast. If you significantly increase the frame rate you will be able to see a one pixel movement. Then you just find the closest object on the previous frame and this identify the same object. If you want to have a movement on the scale you showed you should track the object across multiple frames until you reach the need distance. And you will not only have a movement vector, but also a full path from one point to the other.

If you cannot increase the frame rate the task becomes complicated even for humans. I had trouble identifying the same objects. It means that the task will be really hard for an algorithm. You can still try the nearest neighbor, but it will have much worse results than with a high frame rate. Any additional information to differentiate the objects will significantly improve matching, like color and shape.

After nearest neighbor you can also run an algorithm to minimize a loss function. Where your loss is a distance between the matches and a penalty for non matched objects and a huge penalty for same matching.  You minimize by something like evolutionary algorithm, or empirical hill descent. The variables are the matched object number from the other frame for each object on this frame.


You can try padding the inputs with zero that is of lesser in length.


Convert a csv file so that all attributes will become columns with corresponding element

Here's two different ways to do it. One reading through the file one line at a time to generate a list of dictionaries and one using pure pandas;

import pandas as pd

with open('datafile.csv', 'r') as f:
    data = []
    current_university = {}
    for line in f:
        line = [x.lstrip().strip() for x in line.split(',')]
        if (line[0] == 'University'):
            if current_university:
                data.append(current_university)
            current_university = {line[0]: line[1]}
        else:
            current_university[line[0]] = line[1]
    data.append(current_university)
df = pd.DataFrame(data)
df.set_index('University', inplace=True)

And then pandas;

df = pd.read_csv('datafile.csv', header=None)
df = df.set_index(0).T
df = df.groupby(df.columns.values, axis=1).agg(lambda x: x.values.tolist()).sum().apply(pd.Series).T.sort_values('University')
df.set_index('University', inplace=True)

Personally I think it's much easier to see what's going on in the first option but just wanted to show that it can be done using only pandas as well.


Model for classifying time-series data with distinct features?

This specific problem looks at the pattern across the whole data I.e. pattern will not show up from time < -3 or time > 3 for a given curvature.

You can try two models :

Simple feed-forward Network with number of inputs = number of time steps (Maybe scale / shift the data so that it always has the same number of time steps )

This should be able to detect some patterns for classification (Like f(0) must be less that f(4))

Univariate LSTM with different sizes of time steps in each sample

This should be able to learn that f(x) should stay near constant, reduce and then increase and return to constant

Both networks will have a sigmoid in output layer since it is a binary classification problem.

Code exmaple for LSTM : https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/


converting array to a true/false matrices

You can try this one line solution:
d = pd.json_normalize([{
    "label":"bad",
    "id": "0009",
    "signals":["high_debt_ratio", "no_job"]
},

{
    "label":"good",
     "id": "0002",
    "signals":["high_debt_ratio", "great_credit", "no_id_match"]
},

{
    "label":"good",
    "id": "0003",
    "signals":["low_debt_ratio", "great_credit"]
},

{
    "label":"bad",
    "id": "0001",
    "signals":["high_risk_loc", "high_debt_ratio", "no_job", "no_id_match"]
}])

d.merge(d.signals.apply(lambda x: "|".join(x)).str.get_dummies(sep = "|"), left_index = True, right_index= True)

Outputs:


The "ch" sound is called a phoneme. Automatic speech segmentation on the phoneme level (as opposed to word or sentence) will allow you to extract each phoneme from the speech samples. This is sometimes referred to as phoneme segmentation. Several papers for this are available in the literature.

Once you have the phonemes extracted, you can compute distances/similarities between them (in some feature space). An example feature representation could be MFCC.


Why is my Python Altair chart printing a blank line?

The issue was enabling the vega module/extension in the notebook. Installing it is only half the battle, then you must enable it. :)

sudo jupyter-nbextension enable vega --py --user


When is it useful to measure the Frobenius norm of a matrix?

It is basically a similarity measure for matrices.
There are multiple uses for Frobenius Norm, some examples could be classifying covariance matrices for action recognition and low-rank bilinear pooling.


Anomaly Detection from available sensor data set?

You are talking about anomaly detection, and there are many approaches. If you can create a training set, one-class SVM is a place to start, but even simple control charts can be useful, particularly with live streaming data.