What’s the best way to save many pandas dataframes together?

If you want to get quite involved and be able to specify names for each of the panels you create, you could look at the h5 file format.

This allows you to group datasets in named containers. You can then read them from disk later on one by one i.e. you don't need to read the whole dataset into memory.

Here is an example of a function that would save such a dataset:

def save_h5(h5_filename, data, labels, descr=None,
            data_dtype='float32', label_dtype='float32'):
    """Create a compressed .h5 file containing:
    data    : numpy array
    labels  : numpy array
    descr   : text description ofthe data contained (must be a string)
    """

    if os.path.exists(h5_filename):
        # prevent overwriting a file
        sys.exit('File already exists!')

    h5_fout = h5py.File(h5_filename)

    h5_fout.create_dataset(
        name='data',
        data=data,
        compression='gzip', compression_opts=4,
        dtype=data_dtype)

    h5_fout.create_dataset(
        name='labels',
        data=labels,
        compression='gzip', compression_opts=4,
        dtype=label_dtype)

    if descr is not None:
        h5_fout.create_dataset(
            'description', data=descr)

    h5_fout.close()

For the meaning of the parameters, head over the documentation.

You can write a similar function to make accessing the saved h5 file. This really is a flexible way to save data, and it can be compressed with one of the best known (widely-spread) algorithms in the open-source world: gzip! There are also other possibilities implemented.

On a side note, if you want to minimise the possibility of corruption, you could consider saving each panel/DataFrame (whichever method you go for) into separate files, and then make copies/backups.

Additionally, you said:

  I tried using pickle and csv, but once a line was corrupted I lost the whole file.

... the beauty of a simple csv file is that you can actually open it in notepad or a spreadsheet and usually find the line which is "broken" and fix/delete it. Pickle, on the other hand, is a little more complicated to debug.


What is the best way to calculate score based on input features?

I assume that currently you're training a binary classification model, right?
You could try training a regression model which predicts the score between 0 and 100. It would be better to have some examples in your training data which are between 10 and 90, because that would make the model learn the distribution of the scores. But even if you don't have this kind of instances, it's possible (not sure though) that the model will predict instances in this range in some cases.


It's $F(x)$; the difference between the mapping $H(x)$ and its input $x$. It's a common term in mathematics (DE).


Bert for QuestionAnswering input exceeds 512

The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not possible for the model to index the positional embedding for positions greater than the maximum.
This limitation, nevertheless, is not arbitrary, but has a deeper justification: in a vanilla transformer, the memory requirements are quadratic on the input length. Therefore, limiting such a length is necessary for the model not to need too much memory.
There are variants of the Transformer architecture that are designed to overcome the quadratic memory problem, like Reformer, Linformer, Longformer, BigBird. They, however, are not "compatible" with the Transformer weights, so someone would need to train one of those models in a masked language modeling task (potentially with multitask learning on the next sequence prediction task) in order to reuse their weights in your context.


What is the best strategy to use on data with many classification labels?

There are many factors which influence choice of classifier algorithm. Number of target classes does not generally have an influence, compared to the nature of input features.

As one example, if your input data is natural audio or image, then regardless of the number of classes, a deep convolutional neural network is very likely going to have the best performance.

  What algorithms generally make the best predictions with the smallest amount of data at each of these tiers?

There is no a priori best approach based on number of output classes. The "best predictions" vs "smallest amount of data" is also a trade-off, where simpler models will perform better than complex ones on small amounts of data, but the more complex models will cope better with larger amounts of data, and will then give you better predictions. At some point you might have enough data so that sampling more will not improve your trained models, but you need to establish that empirically.

Most algorithms allow you to explore the trade-off within them by varying hyper-parameters to make the model simpler for smaller data sets and more complex when there is more training data.

  I know at some point it makes more sense to use regression rather than a classifier. How many potential class labels would this be?

That is not strictly true. In general, the distinction between classifying and regression is a hard line. If you are classifying hand-written symbols into an alphabet for instance, it doesn't really matter if you are doing this for 10, 100 or 1000 classes, there is not a practical point at which the symbols turn from being a set of objects into a continuous space to perform regression over.

It could be true if your target class represents a range within some continuous variable (e.g. classifying an event by some of its properties into which year it occurred). But in that case the problem is inherently a regression problem to start with. In fact you may be better off training a regression algorithm in this case even for small number of target classes, and simply binning the predictions into relevant classes.

It could also be true that your target class represents a rank or sequence within an ordered set. In which case this does look more like a regression problem when you have a longer sequence. In general, if you can arrange your target classes into a meaningful sequence, then you might be able to perform some kind of ordinal regression which could be a better choice than using a classifier. However, classifying symbols/alphabets does not work this way, because the sequence in those is arbitrary.

Finally, you might be facing such a large number of classes that a single classifier model is overwhelmed and you need to approach the problem differently.

For an example of this last case, consider a classifier for images of pets. If it had three classes (cats, dogs, rabbits), then you'd clearly use standard classification approach. Even when classifying by breed - 100s of classes - then this approach still works well enough, as seen in ImageNet competitions. However, once you decide to try and detect the identity of each pet (still technically a class) you hit a problem using simple classifier techniques - in that case the structure of the solution needs more thought. One possible solution is a regression algorithm trained to extract biometric data from the image (nose length, distance between eyes, angle subtended between centre of jaw and ears) and move the classification stage into KNN based on a database of biometric data for observed individuals. This is how some face identifying algorithms work, by mapping images of faces into an easy-to-classify continuous space first (typically using a deep CNN), then using a simpler classifier that scales well across that space.


Probability of similarity of two clusters

You can consider that the clusters are defined by a multinomial distribution with 4 classes. Thus you would identify the cluster as a vector of 4 probabilities, one for each class, and the sum of probabilities being 1.

In this form there are some tests and distance functions which can prove to be useful for you.

You can ask questions like "is the cluster 1 is different than the cluster 2?". In order to do that you can employ hypothesis testing, and specifically chi square independence test (wikipedia page here). You select from your table only the first 2 columns, compute the marginal totals, compute the estimated counts as $E_{ij} = rowsum_i * colsum_j / grandtotal$. Denoting with $O_{ij}$ the cell counts you can compute a test statistic with
formula:

$$chi = \sum_{i=1}^4\sum_{j=1}^2\frac{(E_{ij}-O_{ij})^2}{E_{ij}}$$

This is known to has a chi square distribution with $(rows-1)(cols-1)=3$ degrees of freedom. Thus using the cumulative distribution of chi square you can compute the p-value as $1-cdf_{\chi_{3}^2}(chi)$. Now if p-value is smaller than any critical value you choose before test (like 0.05, 0.01 or so), you have a statistical proof that the clusters have a different distribution (this follows from the fact that your procedure tests for the independence of cluster and object distributions, which with short derivation is the same as saying the clusters have the same distribution).

To answer to the same type of question you can also employ G-test (wikipedia page here) or Fisher exact test (wikipedia page here).

Note also that all those tests can't prove that two clusters are the same. The logic behind the tests can be formulated as "we consider the distributions the same, as a natural state, accepted without other proofs, and if we see some exceptionally obvious differences, we can bring that as a proof that they are different indeed".

A different approach would be to build a distance between those two distributions. The advantage of a distance function would be that it could be used for comparisons and eventually some further algorithms based on distances. The drawback of a distance function is that it gives you no hint to answer the first type of questions like "are the differences between 2 distributions small enough in order to be explained by the natural noise in data?". Thus I have two candidates in mind.

First distance is the chi square distance. This type of distance gives a measure to the distance between two normalized histograms (or, put it simply, between two vectors of probabilities of the same size). Thus, if we denote with $cluster_i = (n_{i1}/n_i, n_{i2}/n_i, .., n_{ik}/n_i] = (p_{i1},p_{i2},..p_{ik})$, the distance between cluster i and j would be:

$$d(i,j) = \frac{1}{2}\sum_{k=1}^{4}\frac{(p_{ik}-p_{jk})^2}{p_{ik}-p_{jk}}$$

This distance function has the advantage that it is symmetric regarding $i$ and $j$. To be honest I rarely used it since I did not fully understand it's meaning. In general I use another function, described below.

The second distance function is mutual information (see wikipedia page here). The idea is similar with Kulback-Leibler divergence, however the KL distance is an oriented measure (measures how a distribution can be expressed through another one). The MI distance is a measure of mutual independence between two distributions. In your case the two distributions are two clusters, and the MI measures how dependent they are. I give formula just to illustrate the idea behind it.
$$MI(x,y) = \sum_{x,y}p(x,y)log(\frac{p(x,y)}{p(x)p(y)})$$
You can see from the formula that when the two variables are independent, the quantity inside the logarithm is $1$, and as a consequence, the MI is $0$. As the difference grows, the function also grows. I will not derive what p means, see what I explained already for chi square test, since is very similar.


There is a very cool active Python package called pandas-profiling, is exactly what you want. With a simple pandas_profiling.ProfileReport(df) it returns a lot of important statistical information about your data, the official documentation says:

  For each column the following statistics - if relevant for the column
  type - are presented in an interactive HTML report:

  Essentials: type, unique values, missing values
  Quantile statistics like minimum value, Q1, median, Q3, maximum, range, interquartile    range
  Descriptive statistics like mean, mode, standard deviation,    sum, median absolute deviation, coefficient of variation, kurtosis,
  skewness Most frequent values
  Histogram
  Correlations highlighting of highly correlated variables, Spearman and Pearson matrixes

Look at their demo here.
I personally used it few times and it quite nice, BUT it dataset is large (mostly I mean in variable space) it takes a long time to give the statistics. I think there should be an option to even return all these statistics for a subset of features (columns) in case you do not want for all and it is very high dimensional!


How can I get variable values from a plot?

It seems what you are looking for is a function of your data, not of matplotlib. I would think of this as a second derivative problem -- you care about differences in slopes of successive lines.

Sort a dataframe [x,y] by x values, in increasing order.
Calculate the first discrete derivative delta1 = (y[i+1] - y[i]) / (x[i+1] - x[i]). This tells you the slope of each line.
Now you can do one of two things, depending on what you want to identify.

Calculate the second discrete derivative delta2 = (delta1[i+1] - delta1[i]) / (x[i+1] - x[i]).
Calculate the difference of successive slopes diff = delta1[i+1] - delta1[i]

Select all is for which absolute value of {diff or delta2} is greater than a certain threshold.

Selecting all x values for which np.abs(x[i]) > threshold will give you the x-locations of these large changes in slope.

Here's an example I just coded up,for identifying the jump in a jump function $x / |x|$.

import numpy as np
import pandas as pd

n_data = 100

x = np.random.uniform(-10,10,n_data)
y = x / np.abs(x)

delta = [0] * n_data
diff = [0] * n_data

data = pd.DataFrame({'x':x, 'y':y})
data = data.sort_values('x')

for i in range(n_data - 1):
    a, fa = data.iloc[i]
    b, fb = data.iloc[i+1]
    diff_quot = (fb - fa) / (b-a)
    delta[i] = diff_quot

data['delta'] = delta

for i in range(n_data - 1):
    a, fa, da = data.iloc[i]
    b, fb, db = data.iloc[i+1]
    dd = db - da
    diff[i] = np.abs(dd)

data['diff'] = diff

plt.plot(data['x'],data['y'])
plt.plot(data['x'],data['diff'])

threshold = 1
jumps = [dd > threshold for dd in diff]

Here is the jump function, whose jump we might want to locate

Here is a graph of the differences in slopes, where we see the jump is located near x = 0


How to customize word division in CountVectorizer?

It's possible if you define CountVectorizer's token_pattern argument.

If you're new to regular expressions, Python's documentation goes over how it deals with regular expressions using the re module (and scikit-learn uses this under the hood) and I recommend using an online regex tester like this one, which gives you immediate feedback on whether your pattern captures precisely what you want.

token_pattern expects a regular expression to define what you want the vectorizer to consider a word. An example for the string you're attempting to match would be this pattern, modified from the default regular expression that token_pattern uses:

(?u)\b\w\w+\-\@\@\-\w+\b

Applied to your example, you would do this

vectorizer = CountVectorizer(token_pattern=r'(?u)\b\w\w+\-\@\@\-\w+\b')
corpus1 = ['abc-@@-123','cde-@@-true','jhg-@@-hud']
xtrain = vectorizer.fit_transform(corpus1)
xtraindf = pd.DataFrame(xtrain.toarray())
xtraindf.columns = vectorizer.get_feature_names()

Which returns

Index(['abc-@@-123', 'cde-@@-true', 'jhg-@@-hud'], dtype='object')

An important note here is that it will always expect your words to have -@@-  nested in your tokens. For instance:

corpus2 = ['abc-@@-123','cde-@@-true','jhg-@@-hud', 'unexpected']
xtrain = vectorizer.fit_transform(corpus2)
xtraindf = pd.DataFrame(xtrain.toarray())
xtraindf.columns = vectorizer.get_feature_names()
print(xtraindf.columns)

Would give you

Index(['abc-@@-123', 'cde-@@-true', 'jhg-@@-hud'], dtype='object')

If you need to match words that don't have that exact special character structure, you can wrap the string of special characters in a group and use the non-matching group modifier ?:

more_robust_vec = CountVectorizer(token_pattern=r'(?u)\b\w\w+(?:\-\@\@\-)?\w+\b')
xtrain = more_robust_vec.fit_transform(corpus2)
xtraindf = pd.DataFrame(xtrain.toarray())
xtraindf.columns = more_robust_vec.get_feature_names()
print(xtraindf.columns)

Which prints

Index(['abc-@@-123', 'cde-@@-true', 'jhg-@@-hud', 'unexpected'], dtype='object')

I hope this helps!


Keep word2vexc/fasttext model loaded in memory without using API

Surely, I will good idea to write embedding on some database like SQLite,Postgres or RocksDB. I highly appreciate using RocksDB as it is store data in the form of a key-value pair which is the best suit for your use case. Maybe it will take little more time but your main issue will get resolve through it. If u directly deal with python than there is support of SQLite and Postgres. If u work with java or .NET than RocksDB or LevelDB is supercool. I hope this explanation will help you. Please have review my answer for better reach.


func = @(xtrain, ytrain, xtest, ytest) sum(ytest ~= predict(Md1,xtest));

The function "func" is called by sequentialfs, which does not pass "Md1" to the "predict" function. Therefore, Md1 is unknown to that "cost" function. You should include "Md1=fitcdiscr(xtrain,ytrain);" within your "func" function.

Beside, "sequentialfs" does a 10-fold CV. Why are you trying to pass only 299 samples instead of all 300?


Need help with how to diagnose training and validation metrics

Generally, the fact that your training and validation performance are improving at the same rate is a good thing- this (usually) means that the algorithm is learning generalizable features of your problem space rather than overfitting to the noise of your training set.

Reaching a plateau in performance is also to be expected- it's very rare that a real-world machine learning problem can be perfectly solved, and a perfectly solved problem would be the only type that didn't reach a plateau in testing and validation performance (before it eventually did reach a plateau at 100% accuracy).

Think of the plateau as the maximum performance that can be achieved given the particular parameter values, features, and architecture of the solution. In order to achieve performance beyond your plateau values, one of these considerations will need to be adjusted.


I would say that it is really up to you here. I don't think that the grid lines are distracting in any way, but can be helpful if you are trying to show more exact results. For example, it if it important that the scatter plot shows exact points then keep them. However, if you are demonstrating more of a general trend, then you could go without them. I'm not sure if these are your finished graphs or not, but make sure to label the color differences in the lines and provide a title. Also, is there a reason why your x-axis is on the top of the graph? If you want to keep it there and use the grid lines, try to have the numbers be above the graph instead of being slightly blocked by them.


You should not use the categorical cross-entropy loss, but the binary cross-entropy.
The categorical cross-entropy is meant for categorical probability distribution. In your scenario, this means that only one label would be active at the same time. If multiple labels can be active at the same time, then they are independent, and you need to compute the binary cross-entropy.


recurrent weights on LSTM Data-gate? (the one with tanh)

There's more than two or three variants with regards to LSTM.

A paper that explores these variants can be read here:

LSTM:A Search Space Odyssey


How many vectors does paragraph vector generate for each paragraph?

It would make no sense of the word embeddings to change within a document. It would be as if the spellings of the words changed; how would that help?

When you use a document embedding, you find its numerical representation in a way that loosely captures its meaning. If you want to capture the meaning of each paragraph separately, find their embeddings separately. If you want to capture meaning of the entire document, feed the entire document.

For example, if your document covers a range of topics and you want to allow users to pinpoint where a particular topic is covered, you can find embeddings of each section (paragraph, page, etc.), then find the section that's nearest to your query's embedding.

One use case for dynamic word embeddings is to identify temporal dynamics; how the meaning of a word changes over time, as in this paper: The Visualization of Change in Word Meaning over Time using Temporal Word Embeddings.


Random selection of variables in each run of python sklearn decision tree (regressio )

The documentation says:

random_state : int, RandomState instance, default=None
Controls the randomness of the estimator. The features are always randomly permuted at each split, even if splitter is set to "best". When max_features < n_features, the algorithm will select max_features at random at each split before finding the best split among them. But the best found split may vary across different runs, even if max_features=n_features. That is the case, if the improvement of the criterion is identical for several splits and one split has to be selected at random. To obtain a deterministic behaviour during fitting, random_state has to be fixed to an integer.

So:

it takes different variables to build tree each time?

The randomness comes from picking different variables when building the tree, indeed.

Shouldn't there be only few top variables which should be used to split and should throw me similar trees everytime?

Not necessarily, this depends on the parameters and the data:

If an int is given as value for the parameter random_state, a particular random state is used every time so the tree is fixed, even though it's not necessarily the only one possible. Let's assume that the default value None is provided.
If the parameter max_features is anything other than None or auto, then a random subset of the features is selected before every split. Of course, this means that a particular variable might not be selected, so this might cause differences at every run. Let's assume that the default value None is provided.
If all the features can be used at every run, there's no randomness anymore in the selection before the split. But even if the default value best is provided for splitter, it would be wrong to assume that there is a single best variable
at each node: it's possible that two (or more) different variables are tied, i.e. they would provide an equal improvement in the criterion chosen (by default mse).

Conclusion: assuming you used the default None for max_features, the fact that you obtain different trees at every run means that there are frequently ties between variables with your data. If there are only minor differences in the nodes at the bottom of the tree then it's not significant, but if there are many changes including close to the root of the tree then it means that the model is unstable. The latter case is usually a bad sign, possibly none of the features is really helpful to predict the value and the model overfits.

Also, if I use integer for random_state and run the decision tree, it gives me a different tree for each random_state setting. Which tree should be selected in case of so many trees?

The possibility to select a random state is not intended to select a particular tree, it's intended to make the experiment reproducible. So in my opinion you shouldn't select one of the trees like this. Instead of selecting an arbitrary tree, you could try to investigate why the model is unstable and fix it if possible. The first step is to actually evaluate the model(s) on some test data: if there is a problem such as overfitting, it will be visible in the performance.


Is GridSearchCV in combination with ImageDataGenerator possible and recommendable?

As promised, here you can find an example of how you could apply kfold cross validation for a defined convolutional neural network model, applied to an augmented dataset. You can find the code as a simple gist here
It is done as follows:

for a subset of the CIFAR10 images dataset, generate 3 augmented images (by applying horizontal_flip) per original image, so we should finally have as the number of final images in the augmented dataset: 'number of images in the original dataset' * 3.

check that indeed the built augmented dataset has the new expected number of images. We have just created the augmented dataset, not the fit step yet

apply kfold cross validation on the augmented dataset for several hiperparameters combinations; in this example, 3 pairs of 'learning rate-momentum' have been tried. It is made via the usual 'fit' method:

display the results in a dataframe

This way, we have applied hyperparametrization via kfold cross validation; not a full grid search but only with 3 pairs of hiperparams, but the idea would be the same, not depending on the fit_generator method but making yourself your k folds cross validation on the generated augmented dataset. We could also include other data augmentation strategies in this cross validation.


There are a few steps you can take to choose features for linear regression:

1 - Exclude variables that are highly correlated with each other. If variables are highly correlated you are essentially inputting the same information multiple times which can cause over-fitting and does not satisfy the properties no multi-collinearity for linear regression. You can create a Pearson correlation matrix and decided which variables are too highly correlated using some chosen threshold i.e only keep variables with a correlation coefficient of < 0.3

2 - If you have many variables you could perform principal component analysis (PCA) to reduce the dimensions of the data and use those as your linear regression features. The idea of PCA is reduce dimensions while holding all of the information. Each component from PCA are uncorrelated, satisfying the no multi-collinearity property.

3 - There is also a method known as stepwise linear regression. You allow all variables to enter the model and it will iteratively remove and add variables until the model with the highest R-squared (or whatever your chosen model metric is) is produced. You do have to be cautious using the stepwise  method as it can lead to overfitting, but it can give an indication on what features to use. Here's some info on stepwise: https://en.wikipedia.org/wiki/Stepwise_regression

4 - If you are using R, there is a brilliant package called "caret" that can help with feature selection. Here is a fantastic link to use as a guide: https://machinelearningmastery.com/feature-selection-with-the-caret-r-package/

Hope this helps out as a starting point


What is the best to identify the proper hierarchy of this data?

I don't think your clustering makes sense in the first place. That you actually get some output is more of a coincidence than a result.
Just invoking some functions in a way they don't fail is not enough to get a trustworthy result...

Your matrix is vaguely resembling a similarity matrix, except that it is not symmetric, and the diagonald are not necessarily the maximum values.

Now Ward usually needs a squared Euclidean distance matrix, something very different. The reason why your program doesn't fail supposedly is because it now computes such a distance matrix based on the rows from your similarity matrix. That will usually give somewhat okay looking results (as similar items should have similar rows) but causes subtly problematic bias based on the number of product types.

Later on you can evaluate each hierarchy level based on the agreement with some known label (e.g., manufacturer or product class) and you then can argue which agreement comes earlier. But on a result that isn't valid this doesn't make much sense.


It is definitely possible!

You just need thousands of labeled pictures, a team of ML engineers, a few months, and some tens of thousands dollars to invest.

Jokes aside, it is not a simple task. Also, the labeling of the images will be a pain if they are not already labeled. You will need a team for that.


Composing phrases into a grammatically correct sentence?

There are few modules implemented using TensorFlow which intended to do the same.
Example:
https://github.com/atpaino/deep-text-corrector

This deep-learning models trained to check spellings and correct small grammatical errors.

Thanks!


How to classify and cluster this time series data

I have shared a number of resources on time series classification and clustering in one of my recent answers here on Data Science StackExchange: https://datascience.stackexchange.com/a/3764/2452. Hopefully, you will find them relevant to this question and useful.


Algorithm to construct similarity structure from hash lookup table

The algorithm you are asking about is very straightforward.

What you do, you are looking for connected components in a certain graph where edges are determined by matching hash values. You can implement this with a modification of the disjoint-set data structure.

Your particular variation would be that apart from keeping track of vertices in each component, you have to keep track of m (number of rows) sets of hash values that have been found for each component.


Expected 2D array, got scalar array instead: array=11

Your code has a potential error

x = veriler.iloc[:,2:3]
y = veriler.iloc[:,5:]
X=x.values.reshape(-1,1)
Y=y.values.reshape(-1,1)

As you can see, you x has ONE column (2:3), but your y may have more than one (5: means 6th column to the end). As a result, when you do the reshape, then len(Y) might be two  times (or more) than len(X).

Please post few records of you data to help you more; but generally, X can have multiple columns and Y has one column. However, if you are going to do the regression for 1 column X and Y, the possible fix for your code is as below (remove the : for your y)

y = veriler.iloc[:,5]


There has been an answer on this question, please check it :

Test for bimodal distribution


Feature Selection on Aggregated Targetdata

It seems to me that there's a bit of confusion between these two steps:

Designing the problem and preparing the data accordingly
Applying ML methods to the data (feature selection, etc.)

The two steps must be distinct: any aggregation must take place during step 1 so that at the end of step 1 the dataset is fixed.

In your example in step 1 you decided to predict the total volume of sale across  stores given the week and price per unit. Note that you could also choose other options:

average volume of sale by store as target
keep one row for each store: depending on the goal, sometimes it's ok to have inconsistencies in the data, regression can deal with these.
you could decide that an instance for a store spans over the N past weeks and provide the sales for these weeks, the target being the week after that (that would probably help btw).
...

And of course you can do this as many times as needed for studying different problems, each time obtaining a different version of the data which represents a particular problem. It's only once you have formally defined your problem and formatted your data for it that the ML part starts. For instance stepwise regression (or any other technique) cannot work if the data is modified/aggregated during the steps.


How cross validation works for regression?

The idea behind cross validation is to understand the performance of some measure of your model's performance on unseen data. This can be applied to loads of different statistics, not just ones relevant to classification like accuracy. Common measures for holdout performance of regression models include MSE and MAPE, either (or both) of which can be cross-validated over.


Its definitely not regularization since default values are 0 (check here)

n_estimators is the number of decision trees you take into bagging. These decision trees take random number of rows AND columns (again depending on the parameters) so it could be that you took some unfortunate combination which should level out with higher number of n_estimators.

Suggestion: use different criteria for feat.importance for example Permutation importance, and then compare.

Final thought, hyperparams can be irrelevant. Have a look at the effect of parameters on certain tasks, with different algorithms. Conclusion: it can be an overkill (sometimes its important!)


As the data augmentation must only be used for the training data, I will have very few images from the 4 classes for the validation set, would it be a problem?

You may first perform data augmentation on the image dataset and then split it into train/test or train/test/validation. Just make sure that the images present in the test split are not included in the training split or vice-versa. See this.

Is it needed to split it in training/validation/test, or is training/validation enough ?

The validation dataset is particularly to fine-tune the model's hyperparameters i.e for hyperparameter optimization. If you want to simply skip that procedure, just make two splits, train, and test. See this.
See this answer to know the difference between test and validation dataset.

Another problem is the unbalanced data with such a difference
between the number of images in each class, would oversampling or
undersampling be a good solution ?

I would encourage you to augment the images for a specific class until you are left with a near-balanced dataset.

For transfer learning and fine tuning, should I freeze all the convolutional layers or should I only train a FC layer?

As a common practice, you may try unfreezing some initial layers ( layers that usually have lesser filters ). Refer to this blog. Deeper layers have more filters and extract high-level features.


Frequency bin of the positive and negative frequency

The tutorial you are looking at is attempting to model only the seasonality of a time-series.  They first look at power of different frequencies, in the second figure in the tutorial.  They find a lot of power for the 1/12 frequency, as well as a few other harmonics, and also the 1/72 frequency.  But they decide that since the 1/12 frequency has the most power, and most of the others are harmonics of this, they will only model the 1/12 frequency.  You can see this from this quote:

  Based on the spectral analysis of the data, fit a simple linear
  regression model using a cosine and sine term with a frequency of the
  most signficant component: 1 cycle/year (1 cycle/12 months).

So you therefore only need one frequency bin, and its negative, since you are only modeling one frequency.  In other words, in this part, they are just using essentially the equivalent of a single sine wave (at the 1/12 frequency) as their model to fit the data.  All the frequency bin variable does is to act as an index and select the component of the fft() that contains the 1/12 frequency.

They then analyze their result (the autocorrelation plots) and find that they have indeed missed some of the cyclical patterns at other frequencies they did not model.  So then they add three more frequencies to the model.  Rather than using frequency bins this time to do the indexing, they simply ask for the 6 highest in power (6 because each of the 3 needs both positive and negative frequencies) components of the fft() with these lines:

[Y,I] = sort(abs(tsdft),'descend');
indices = I(1:6);
tsfit2dft(indices) = tsdft(indices);

You can see the resulting plot that follows now better models the time series since it includes more of the frequencies that showed increased power.


migrating to python from R: specific questions

1) Anaconda Spyder, maybe

2) sklearn, random forest has an option to select the number of jobs, and it will take care of parallelizing

3) i don't think so, but you can pickle objects and load them up. you can probably do something like create a variable_name - value dictionary and just pickle that

4) never tried

5) according to https://stackoverflow.com/questions/21288133/loading-rdata-files-into-python, I don't think so


How can I extract the top words from a string in my dataframe column?

If the words are all short and like the one you mentioned, I think you're pretty done with just removing the stopwords. It is a simple text cleaning of the most frequent words that don't bring additional meaning to the text.
Let me know.


Neural network q learning for tic tac toe - how to use the threshold

You are effectively implementing $\epsilon$-greedy action selection.

The usual way to represent this in RL, at least that I am familiar with, is not as a "threshold" for probability of choosing the best estimated action, but as a small probability, $\epsilon$, of not choosing the best estimated action.

For consistency with RL literature that I know, I will use the $\epsilon$-greedy form, so instead of considering what happens as your threshold rises from 0 to 1, I will consider what happens when $\epsilon$ drops from 1 to 0. It is the same thing. I hope you can either adjust to using $\epsilon$ or mentally convert the rest of this answer so it is about your threshold . . .

When monitoring Q-Learning, you have to be careful how you measure success. Monitoring the behaviour on the learning games will give you slightly off feedback. The agent will make exploratory moves (with probability $\epsilon$), and the results from a learning game might involve the agent losing even though it already has a policy good enough to not lose from the position where it started exploring. If you want to measure how well the agent has learned the game, you have to stop the training stage and play some games with $\epsilon$ set to $0$. I suspect this could be one problem - that you are measuring results from behaviour during training (note this would work with SARSA)

In addition, choosing values that are too high or low for your problem will reduce the speed of learning. High values interfere with Q-learning because it has to reject some of data from exploratory moves, and the agent will rarely see a full game played using its preferred policy. Low values stifle learning because the agent does not explore different options enough, just repeating the same game play when there might be better moves that it has not tried. For Tic Tac Toe and Q-learning I would suggest picking a value of $\epsilon$ between $0.01$ and $0.2$

In fact, with Q-learning there is no need to change the value of $\epsilon$. You should be able to pick a value, say $0.1$, and stick with it. The agent will still learn an optimal policy, because Q-learning is an off-policy algorithm.


I don't know the specific tools that you mentioned. But generally speaking, to create a chatbot, you would need to connect an API from Slack to your own API layer that handles the NLP side of it. So, if you're just starting out, your focus should be on creating an NLP algorithm that handles the types of questions that you want your bot to answer and converse with. You need to create code that both recognizes new inputs and understands how those inputs fit to everything else (i.e. an ongoing conversation). From there, you can create an API that you consume in any number of programs an you can practice with it (even in a terminal screen) for your own testing & algorithm training purposes. At that point, you should be able to integrate that into any number of platforms, slack being only one option.


Few activation functions handling various problems - neural networks

Image classification and other task can be expressed as function approximations and, in theory, neural networks can approximate (almost) any function (given few assumptions on the activation function (see Universal Approx. Theorem)).

However, in practice, not all functions fulfilling these assumptions work equally well. Popular activation functions usually share some properties that allow neural networks to learn efficiently in practice, e.g. they are continuously differentiable (for gradient descent), close to the identity near zero (accelerates initial learning from small random weights) and so on.

Activation functions like the sigmoid function are not directly related to image classification or any other tasks. Rather, they allow for efficient training of neural networks, which, in turn, can represent a wide variety of tasks using different architectures and cost functions.


Which algorithm can be used to reduce dimension of multiple time series?

So, to be clear, you observe $N$ samples with $P$ measurements at $T$ times (where $P = 6$ and $T = 12$).

I would first determine whether or not you expect the joint distribution of your $P$ features to change over time. For example, if these features are $X$, does $\mathcal{P}(X) = \mathcal{P}(X | T)$ for all values of $T$? If so, you could do something like stack your data over time (so, you'd have $NT$ samples of $P$ features) and perform PCA, then use the top $k$ loadings to rotate the measurements at individual timepoints, leaving you with $N$ $k$-dimensional time series of length $T$. You could choose $k = 1$ if necessary, and use it to visually identify outliers or set a simple cutoff. If not, you'd want to be more careful with this technique - especially if the covariance changes. If it's the variance that's changing but correlation is remaining the same, perhaps some procedures can be done (e.g. perform PCA on the correlation matrix/standardized data) and this technique would still work.

What form do you expect your outliers to take? If it's a temporary deviance from the rest of the sample (e.g. only at one or two measurement times), your technique of PCA on the concatenated data will not be very sensitive. The technique I just described would be more sensitive.


Predicting probability of occurrence using Historical Time Series Data

Without having more context this is how I would approach this -
Assuming there is not meaning to being 'close' in the prediction. I.E, if real 1001 number was 42, then predicting 41 and 99 is just as wrong.
If we assume no stochastic structure to the series, I.E previous number dosnt affect current number I would use the top 5 most frequent numbers as predictions.
If we assume there is a stochastic structure to the series, under our assumption that difference between numbers is not a factor. We can treat this problem as a sequence prediction problem. This is common in text analysis and there are various models that can be used.
Assuming there is a meaning of being 'close' to the prediction.
IF no stochastic nature, use the 5 percentiles based on 1000 values series as predictions.
If yes stochastic nature, use time series modelling. AR(IMA) modelling perhaps.


Distribution of error values in linear regression vs logistic regression

Lets clarify with a simpler question

  Why points on a circle must be equally distanced from center, but
  not points on a square?

Because of how circle and square are "defined". The same goes for linear and logistic regression, we cannot pose a "why?" question when errors are "defined" in a certain way (or a better word here "assumed"). We may design a new version of linear regression by replacing Normal distribution with some other distribution, and then proceed to derive a formula or algorithm for estimating the parameters. Then use it in a real-world scenario to see how it works, and so on.

Also here is a list of good posts on stats.stackexchange.com related to characteristics of those errors:

Error distribution for linear and logistic regression,
Logistic Regression - Error Term and its Distribution.


A useful introductory reference is Forecasting: Principles and Practice by Hyndman and Athanasopoulos. Depending on your data, you might be able to build a simple additive or multiplicative model with seasonal, trend, and cyclic components.

Besides exponential smoothing, you can look at ARIMA models. Exponential smoothing is based on trend and seasonality while ARIMA is based on autocorrelations in your data.


How to calculate ideal Decision Tree depth without overfitting?

No! the best score on validation set means you are not in overfitting zone. As explained in my previous answer to your question, overfitting is about high score on training data but low score on validation. So to avoid overfitting you need to check your score on Validation Set and then you are fine. There is no theoretical calculation of the best depth of a decision tree to the best of my knowledge.

So here is what you do:

Choose a number of tree depths to start a for loop (try to cover whole area so try small ones and very big ones as well)
Inside a for loop divide your dataset to train/validation (e.g. 70%/30%)
Each time train your decision tree with that depth on training data and test it on the validation set, then keep the validation error (you can also keep the training error)
Plot the validation error (you can combine it with evolution of training error to have a prettier plot for understanding!)
Find the global minimum of validation error.
Then you can narrow your search in a new for loop according to the value you found to reach a more precise value


Mapping a set of corrupted strings to the correct ones

That problem is called approximate string matching or fuzzy string searching. It has been well studied in computer science.
If there is a limited, known collection of valid strings (aka, a dictionary), then the problem can be framed as spell correction.


Mutation on demand in genetic algorithms

If the breeding at one point doesn't result into "new" children/chromosomes, the algorithm will run very inefficiently. Assume that all individuals in the population have the same chromosome, then crossovers will result into the same individuals until a mutation occurs in one of the chromosomes

This is correct, but in theory this case should happen only when the algorithm has already converged, meaning that the chromosome is actually optimal. If this happens by chance before reaching an optimal solution, then it's probably because the size of the population is too low and should be higher in order to maintain diversity and allow mutations from time to time.

For example, if the population size is 100 and the mutation rate is 0.02 then there are in average two mutations for a particular gene at every generation. This maintains a bit of diversity, as the mutated gene is likely to be reproduced at the next generation if it turns out to be beneficial.

My experience with genetic algorithms is that many variants are possible and imho it's a good idea to explore various variants with a dataset, including non-standard parameters such as this idea of "automatically mutating", because why not? The danger I can imagine with that is that this could cause too much variation and prevent or slow down the convergence, but testing is the best way to see what happens.


To answer simply, Euclidean distance is generally used:

$$ d = |\mathbf{x}-\mathbf{y}|=\sqrt{\sum_{i=1}^n|x_i-y_i|^2}$$

Read more about k-means over here:

Introduction to K-means
k-means clustering algorithm

Using kmeans() in R:

K Means Clustering in R


Why Orange "Predictions" and "Test & Score" produce different results on the same data?

Perhaps I can elaborate a bit on this.

Test&Score is used for evaluating a model. You provide T&S with training data and the learner (e.g. Random Forest) and then the widget performs 10-fold cross-validation on training data, leaving out 1/10 of a data for testing. Results of all 10 fold are then combined into a single evaluation result.

Predictions, however, does not perform cross-validation. It doesn't test the data provided. Instead, Random Forest passes a model to Prediction, not a Learner. A model is built on train data. Predictions widget does not consider the real class, it uses the model from RF and applies it to the (new) data.

These are two different procedures, thus you are getting different results.


Leaky ReLU inside of a Simple Python Neural Net

I assume the task you're working on is a binary classification task since y = np.array([[0,1,0,1]]).T. And as an error function you use self.errors = self.y - self.l1. Now compare this to the curve of a leaky ReLU function:

The leaky ReLU is an unbounded function. How is your network supposed to model a binary classification task where output values are elements of $\{0,1\}$ using this function? And what is the result of applying the absolute difference as an error function to your labels $y \in \{0,1\}$ and your outputs $\hat{y} \in (-\infty,\infty)$?

What you would need is a translation of the lReLU outputs to your classes, e.g. something like prediction = 1.0 if activation >= 0.0 else 0.0.

Which is one reason why it is more common to apply last layer activation functions to classification problems which can be interpreted as class probabilities, e.g. Softmax or Sigmoid:

You still need a threshold here to move from real valued outputs (which are interpreted as probabilities) to binary class labels but that is fairly straightforward since often you just set it to a probability of $0.5$.

I suggest to go follow a tutorial for your from scratch implementation. If you are willing to take a step back here is one for a simple percepton (i.e. just a single neuron).

Images taken from this article


"Invalid value" in RMSprop implementation from scratch in Python

The regularization term (reg_term) is sometimes negative due negatative parameters. Hence S[f"dW{l}"] contains some negative values. I realize the reg_term has to be added before taking the sqrt, like this:
S[f"dW{l}"] = beta2 * S[f"dW{l}"] + (1 - beta2) * np.square(gradients[f"dW{l}"] + reg_term)


What metrics must i use in my data(unstructured) preprocessing research?

The corpus quality can be determined using the zipf law ,vocabulary growth curve and the pareto distribution. An interesting thesis which explains this can be found at

http://ilk.uvt.nl/downloads/pub/papers/hait/camp2008.pdf


How to train a single Artificial Neuron to predict f(x), where f(x)=x

This will not work.
because, for this problem x=99, and when finding gradients, the results will be large numbers and updating weight and bias using learning rate 0.01 will take huge steps, and so the bias and weight will cross the solutions.
using learning rate less than 0.001 will solve this problem.
I tried learning rate 0.0001 and it worked.


Why can distributed deep learning provide higher accuracy (lower error) than non-distributed one with the following cases?

About the accuracy: Going with the strongest reason; memory problems will diminish due to the distribution of the computation. That will allow you to increase your training batch size which will reduce the gradient noise due to small mini-batch sizes. The steeper gradient moves will be towards the minima, with less noise.

You can refer to this video for deeper understanding: https://www.youtube.com/watch?v=-_4Zi8fCZO4&list=PLkDaE6sCZn6Hn0vK8co82zjQtt3T2Nkqc&index=16

About the speed: It is more obvious I think. You distribute your gradient descent computations to multiple machines or CPUs/GPUs/TPUs, so a faster training speed you acquire as a result.


How to work with Sensitive (personal) Data, without actually seeing it?

You can use the following techniques to mask sensitive data:

Substitution cipher -  any character of plain text from the given fixed set of characters is substituted by some other character from the same set depending on a key.

Tokenization masking - mask source string data based on criteria that you specify in an algorithm.

Principal Components Analysis (PCA) or other dimension-reducing techniques - combine several features and then carry out ML training only on the resulting PCA vectors.

Data Coarsening - decrease the precision or granularity of data to make it more difficult to identify sensitive data within the dataset.

Also, read  Data Masking on Wikipedia.


Can a recommendation system be used as a binary classifier?

Would this approach work?

I don't know, you should try, it very well could.

Can recommender systems be used as classifiers?

Absolutely! But first, let's clarify a couple of things. Recommender systems are systems. They are the output you are trying to achieve. This can theoretically be achieved without machine learning but in general, building recommenders happens via a learning algorithm (i.e. collaborative filtering, neural nets, etc.).

What that learning algorithm does is learning to provide a "score" to each item given a specific context (i.e. user X wanted to watch a movie on a Tuesday night) such that the highest scores are given to items that are worth recommending.

This can directly be applied to a classification problem where your learning algorithm learns to give high scores to the correct class.

So in summary, you would still be building a classifier system (what it does) by using a recommendation learning algorithm (how it does it).


Can dropout and batch normalization be applied to convolution layers

In short, yes.

Batch Normalization Batch Normalization layer can be used in between two convolution layers, or between two dense layers, or even between a convolution and a dense layer. The important question is Does it help? Well, it is recommended to use BN layer as it shows improvement generally but the amount of improvement you will get is more problem dependent.
Dropout: Convolution layers, in general, are not prone to overfitting but it doesn't mean that you shouldn't use dropout. You can, but again this is problem dependent. For example, I was trying to build a network where I used Dropout in between conv blocks and my model got better with it. It is better if you apply dropout after pooling layer.


What is a visual bag of words and how is it implemented?

The necessary information can be found on wikipedia.
"when we use more technical features such as colour histograms"
Judging from this sentence I guess you need to understand the "Codebook" generation.
First step is to extract features of patches in the image. For efficiency you only want to take patches which are interesting and calculate discriminative features on them. SIFT is one Method which performs both steps for you. It takes care of finding good spots and it calculates features on this spot.
Now you can generate your codebook. A codebook will map every possible featurevector ( after all they're just numeric vectors ) to a certain output codeword. One possibility to do this, is to use k-means for codebook generation. After you built your codebook, a vector is mapped to a code by finding the minimal distance to all the entries ( since you used k-means you can use euclidean distance ).
Now you have a complete realization of the bag of words model. You can now dive into using it for classification.
The required algorithms can be implemented using the libraries you mentioned.


Document clustering to merge common labels

I might misunderstand something but it looks to me like you're trying to find a complex method for a simple problem: if there are many strings which occur multiple times in the list, you should deduplicate the list before comparing all the pairs. You could use a set, but since you will need to count how frequent each string is you should probably directly create a map (dictionary) which stores the frequency for every string (just iterate over the list of strings, then increment the frequency of this string (key) in the map).
Depending how many distinct strings you have, this simple step might be enough to allow you to compare all the pairs of strings efficiently.
Then you could for example decide on a threshold for frequency, for instance keep only the strings which appear at least 10 times. For any string which appear less than 10 times, replace with the frequent string (more than 10 times) which has the highest similarity with it.


where can i find the algorithm of these papers?

You can email the authors to ask them if they could share their code with you, but maybe they can't for IP reasons or don't want to share it.

Papers like these are not unusual in experimental research. In theory you should be able to reproduce their system following the explanations in the paper.

However there are other tools available for biomedical NER: MetaMap, cTakes.


Categorical features preprocessing for clustering

Using Gower distance for the clustering. You can find different article about this measure in this and this article.

  Gower Distance is a distance measure that can be used to calculate distance between two entity whose attribute has a mixed of categorical and numerical values.

  Gower distance is computed as the average of partial dissimilarities across individuals. Each partial dissimilarity (and thus Gower distance) ranges in [0 1].
  $$d(i,j) = \frac{1}{p}\sum_{i=1}^{p}d_{ij}^{(f)}$$

  Partial dissimilarities ($d_{ij}^{(f)}$) computation depend on the type of variable being evaluated. This implies that a particular standardization will be applied to each feature, and the distance between two individuals is the average of all feature-specific distances.

  For a numerical feature $f$, partial dissimilarity is the ratio between 1) absolute difference of observations $x_i$ and $x_j$ and 2) maximum range observed from all individuals: $d_{ij}^{(f)} = \frac{|x_i - x_j|}{|(\max_N(x) - \min_N(x))|}$ , $N$ being the number of individuals in the dataset.
  Partial dissimilarity computation for numerical features ($R_f$ = maximal range observed)
  For a qualitative feature $f$ partial dissimilarity equals 1 only if observations $y_i$ and $y_j$ have different value. Zero otherwise.


Can I classify a healthy population into clusters found in an injured population?

The dataset (population) that you used to define the clusters was a different one from the one you are looking at now (healthy population). That means you can’t use your clusters to categorize healthy people just like that, but your research question is not what cluster each person belongs to.

If your question is to see whether people are consistent in their movement strategy then you can use the information from the clusters you have, or any other clusters you just made up. The cluster assignment is a proxy (a simplified representation) of each measurement, and you’re trying to see if they are consistent from the cluster numbers of each measurement.

This seems to me to lead to results that are difficult to interpret. What if healthy people are consistently very close to a border between two clusters? Consistent, but jittering between the clusters.

I would say you need to drop the idea of using a cluster as a proxy. Let’s say your measurement is how fast people can run a mile, time after time. To know if they’re consistent, you would just look at the standard deviation for each person.

I guess your problem is that you have many measurements in each example, and you’re trying to simplify by using a cluster number rather than the actual measurement. That’s dimensionality reduction, to a single dimension, which is extreme, and I wouldn’t use clustering for it. Is it really necessary to reduce the dimensions, or would it be possible to define a consistency metric on the original data? If it’s not and you need to reduce the dimensions, I would first turn to PCA and if that’s not enough, maybe an embedding from an auto-encoder.


Cannot make user directory on a new CDH5 installation (Hadoop)

The simplest way to setup a user directory is to login to Hue as an admin and create a user account. This will create a directory in HDFS located at /user/yourusername with the correct permissions to work in.


Any difference between adding epochs and duplicating data for neural nets?

The most obvious difference is that by adding epochs, you (still) never get the same observation twice in a same batch, which isn't the case when you duplicate data.

I'm no expert in NNs, but it seems that if you do not introduce any noise in the duplicated data, then you might as well just add epochs.

Of course, if you fit two models over a single epoch,

model 1: using your n=1000 observations

model 2: using your n*10 = 10,000 observations (with duplications)

your second model will most probably return a lower loss value. However, a fairer comparison would be to train model 1 with 10 epochs so that in both cases you expose your net to your observations the same number of times in total.


How to find the best fitting parametric distribution for an empirical dataset (stock returns)?

First of all, if you want to find the best distribution that fits your data you just iteratively fit your data to the longlist of distributions. Scipy supports most of them. After fitting, you can either use KS-test to find which distribution fitted best or you can use fit error to decide. This solution does what you want, also other solutions in that post are very good approaches to your problem.
Also, if you are sure about your distribution's being Johnson-SU distribution, to find the parameters, use Scipy's fit function, which will return you the parameters that best represent your data.
a, b, loc, scale = scipy.stats.johnsonsu.fit(data)


How to model a Machine learning problem considering links between features

Welcome to the site!

What you are describing above is known as an interaction.

You should consider the algorithm you wish to use and whether it allows for interactions between predictors. Some techniques - like generalised linear models - will require interactions to be stated explicitly, while tree-based algorithms will capture interactions automatically.


How can I change not only the value of the custom field but the what the field shows itself?

I'm by no means a PowerBI expert, so there may be options I'm not aware of. I don't think that there is a tidy solution to this in PowerBI, as this is not quite what BI tools are usually intended to do. However, it can be done-- you'll just have to accept a degree of hack/kludge character to your projects.

The most straightforward approach I can think of is to make a new table for your presentation values, with something like a column for each company and then another column for "meaningful data to present". That latter column should be set to something like text, so that it will display different values with the correct conventions (dollar signs, comma placement, etc.). Then, when drawing on data to populate the report, set the card (or whatever structure you're using) to draw from that presentation table. Since field1 will always contain the correct data to display for a given company, and that field is formatted as text, you won't have to worry about mismatched data types.

This approach will require some preprocessing of your data, so that the presentation table is populated correctly for each distinct company whose data you are presenting, but it's nothing that a SQL query can't produce pretty easily. This can be done within PowerBI or externally, with the presentation table imported. Importantly, you will not have the option of using aggregations or similar operations (that would be the case anyways, with this type of report output, but it's worth mentioning explicitly). This will be, essentially, a display-only report.

The key is that field1 (using the naming convention in the question) is representing something unstable in the data itself (different data types in the same field), but relevant in an "above the data" sense.

As a final note, I'll encourage you to consider whether or not this data really belongs in a single, conglomerated report. That might legitimately be the case, or it may be that this structure is demanded by a superior (whether it makes sense or not). But maintaining a single, kludge-filled report like this might ultimately be more work (and less valuable) than multiple reports which fit better with PowerBI's design favoring strongly-typed data fields.


Word embedding vectors for keyphrase extraction

Word embeddings are just a way to represent tokens (often words, but could be characters) in a way that it inherently carries semantic meaning (i.e. as opposed to simply one-hot encoding), but it is not a keyphrase extraction technique.

Word embeddings can help you extract key phrases better because they will make your input more meaningful, but unless you have a technique to extract such phrases, they won't help you.


How is the 'feature_importance_' value calculated in sklearn random forest regressor?

Decision trees usually base feature importance on the impurity reduction achieved by splitting on the features. In classification a usual choice is gini impurity, while regression trees typically use the mean squared error or node sample variance. This is also the case in scikit learn.

For a given (binary) node $m$ with left and right child nodes the impurity reduction $Gain_{m}$ is calculated as

$$ Gain_{m} = impurity_{m} - (weight_{left} \cdot impurity_{left} + weight_{right} \cdot impurity_{right})$$

with the weights being defined as the share of the parents examples in a child node (e.g. $weight_{left} = N_{left} / N_{m}$ where $N$ is the number of examples in a node or leaf).

Now, to derive the total impurity reduction of a given feature $f$ in tree $t$ you need to sum across all nodes $m \in M_f^{(t)} $ which perform a split on that feature $f$ and divide it by the total impurity reduction number of all nodes of that tree:

$$Importance_f^{(t)} = \frac{\sum_{m \in M_f^{(t)}} Gain_m}{\sum_f\sum_{m \in M_f^{(t)}} Gain_m}$$

(Note that due to this normalization step your feature importances sum up to $1$)

Eventually, the total importance of a feature $f$ is calculated across all trees $t$ in your random forest with a total number of trees $T$:
$$Importance_f = \frac{1}{T} \sum_{t=1}^TImportance_f^{(t)}$$


What is the best approach for classifying non-English text

As far as I know, the best way would be to use pretrained embedder. Embedder encodes your text into language-agnostic latent space. You input your text and you get fixed-length numerical vector as an output. You can use latent space encodings as a feature vector, which you can use to train discriminative models. They also suit well for resampling like SMOTE or ADASYN.

Some time ago Facebook released a model called LASER. You can read about it here. It supports Sinhala and Tamil as well. Here is a github repository. There is also unofficial distribution on pypi. It substitutes internal tools for tokenization and BPE encoding. For sake of convenience I've been working mostly with this distribution and I can confirm it works just fine. Here is a repository.

I'd also suggest to consider embedder abuse. It covers a lot of languages, which means you can train your models on e.g. English and prediction will work for Tamil out of the box!

Natural language is hyper-dimensional space and most seminal models are using encoders. To my knowledge it is the go-to approach for any language.


NoSQL engine/service recommendation for geolocation data

One approach can be to use a no-SQL database running on top of a distributed store (like Cassandra or Hbase). Add an external index which support spatial indexing (say elastic) for fast search. This makes your solution scalable (due to distributed store) and fast enough for spatial search queries.


How to know if my Decision tree model is good or bad?

It appears that you are training your model and generating predictions on the same dataset (X_new). You should not attempt to evaluate your model's performance using this output - because you are applying the model to the same data you trained it on, your evaluation will be over-optimistic. You need to set a portion of your dataset aside as test data, train the model on the remainder, and then apply the model to the independent test data. You can then calculate performance metrics like MSE or MAE as mentioned in the other answers, which will give you an unbiased estimate of how your model performs on unseen data. Cross-validation is another way to get an unbiased estimate of performance, which is essentially repeating train-test splits in a way to leverage all the available data.


Based on transformer, how to improve the text generation results?

If you have a lot of data available to train, you should apply the techniques used in large transformer models, like GPT-2: very deep models (48 layers for the 1.5B parameters), modified initialization, pre-normalization,
and reversible tokenization. You could also apply GPT-3's locally banded sparse
attention patterns.
If you have very small training data, you can apply the "unwritten" aggressive techniques described in this tweet, namely data augmentation, discrete embedding dropout, normal dropout and weight decay, and lots of patient training time.
Update: I feel like the tweet thread I referred to is important, so here are the most relevant tweets:

How can you successfully train transformers on small datasets like PTB and WikiText-2? Are LSTMs better on small datasets? I ran 339 experiments worth 568 GPU hours and came up with some answers. I do not have time to write a blog post, so here a twitter thread instead.

To give a bit background: All this came about by my past frustration with replicating Transformer-XL results on PTB and having very poor results on WikiText-2 (WT2). On WT2, my best model after 200+ experiments was 90ish ppl which is far from standard LSTM baselines (65.8 ppl).

...

The key insight is the following: In the small dataset regime, it is all about dataset augmentation. The analog in computer vision is that you get much better results, particularly on small datasets, if you do certain dataset augmentations. This also regularizes the model.

The most dramatic performance gain comes from discrete embedding dropout: You embed as usual, but now with a probability p you zero the entire word vector. This is akin to masked language modeling but the goal is not to predict the mask — just regular LM with uncertain context.

The second most important factor is regular input dropout: You take the embeddings and dropout elements with probability p. This also has a data augmentation effect very similar to dropping out random pixels for images. What is a good way to think about this? 1/2

Remember that we can do King-man+woman=Queen? Now imagine input dropout removes the "man" component of "King". This forces the model to distribute specific information (gender in this case) into multiple dimensions to improve generalization making it more robust. 2/2

Otherwise, it is a game of further regularization (more dropout + weight decay) and of patience. I can train a good model without these tricks in 15 minutes and get 97 ppl. If I apply all these dropouts the model underfits after 7h of training to 63.4 ppl (better than LSTM).

You can also apply these data augmentation recipes to large datasets, but nobody would like to train for months on WT-103 for a couple of ppl points. In my opinion, techniques that require so much extra compute are more harmful to the community than useful. 1/2

Here the code changes to the public Transformer-XL repo that my results are based on: https://github.com/TimDettmers/transformer-xl/tree/wikitext2

With my changes to the public Transformer-XL repo, you can run this script to get down to 63.4 ppl on WT2: https://github.com/TimDettmers/transformer-xl/blob/wikitext2/pytorch/replicate_wt2.sh


Welcome to the site. I would encourage you to think about your problem in a different way. You are focused on "what sold today" whereas you should be focused on "who bought what over a historical timeline".

What you're looking for is known as a recommender system and there are (generally speaking) two types:

Content-based - what should you recommend based on attributes of products. The algorithm is basically saying, "You bought breakfast cereal, here are other products that might go with your cereal . . ."
Community-based - what should you recommend based on attributes of people who bought products. The algorithm is basically saying, "You are a female, under 30, with no kids. Other females, under 30 with no kids also liked these products . . ."

I will assume that you don't have info on your customers so let's focus on the content-based recommenders. You are on the right track by thinking about the attributes of products, but you should be thinking about them both (1) over a longer timeline than just yesterday and (2) how the products and their attributes relate to each other. The people who need attribute X might also need attribute Y and that is (most likely) across multiple products and will generate higher demand for those products.

Start researching content-based recommender systems in your language/tool of choice and you will end up with the desired algorithm. From there you can also think about user data collection and then move into a community-based recommender over the long term.


Why does my minimal CNN example show strongly fluctuating validation loss?

There is nothing fundamentally wrong with your code, but maybe your model is not right for your current toy-problem.

In general, this is typical behavior when training in deep learning. Think about it, your target loss is the training loss, so it is directly affected by the training process and as you said "improving quickly". The validation loss is only affected indirectly, so naturally it will be more volatile in comparison.

When you are training, the model is attempting to estimate the real distribution of the data, however all it got is the distribution of the training dataset to rely on (which is similar but not the same).

Suggestions:

I think that your model is an over-kill for a 256x256 grayscale dataset with just 5 classes (Resnet was designed for ImageNet which contains RGB images from 1000 categories). In current state, the model finds its very easy to memorize the training set and overfit. You should look for models that are meant to be used for MNIST, or at most CIFAR10.

You can insist on this model and attempt to increase regularization, but I'm not sure if it will be enough to prevent the model from overfitting in this case.


Reference about social network data-mining

I'm going to pursue the following series of online courses on the Coursera:
Become a Social Scientist: Methods and Statistics by University of Amsterdam. The good news - it is free, or you can get a nice-looking certificate for $49 or so. The bad news - the nearest enrollment is Aug 31st 2015. You will have opportunity to get a lot of information in condensed way during a short time frame and you will be enforced to actually apply the knowledge in exercises, quizzes and project assignments. You will also have opportunity to discuss lessons/projects on the forum with many other students and lecturers.

[update] I apologize, I just remembered there is another course Introduction to Statistics for the Social Sciences by University of Zurich - just started April 28th, 2015. If you want to pursue it - do not forget about deadlines for quizzes and homeworks. Good luck!


samples for different objects with unique labels

Have run into this problem, when we wanted to run with patient-based deep learning models or individual observations (the same patient could have come several times) based deep learning models.

In your case, they could be analogous to Animal-based model vs general-one-for-all model which has all the animals.

Pros of animal-based:

If you are going to build an animal-level classifier, it certainly is going to generalize better when compared to one-model for every animal.

Cons of animal-based:

If you don't have enough data for one particular animal, you can't help out much, for that case. This is slightly advantageous as well, I can't say for veterinarian examples, but if you are going to use a general-one-for-all model it might misclassify the prediction as well (because of lack of data for that particular example).

I would suggest you try both. There is nothing certain in this world. Only hunches and guesses from past experiences.

Hope this helps.


What's a good F1-score in (not) extremely imbalanced dataset?

I would say you data in not imbalanced. 33:67 is not a bad ratio but Try using under sampling of majority class.
As another option you can try difference algorithms like randomforest. You can also try boosting.


This seems to fail only when you're using categorical_features, which was deprecated at the same time the encoder was extended to strings.  Using the now-recommended ColumnTransformer to specify which columns to encode works with strings (as does applying the encoder to the entire frame, though that's not what you want, with features like bmi).

E.g.,

onehot = OneHotEncoder(...)
cat_cols = [1,4,5]
preproc = ColumnTransformer(transformers=[('onehot', onehot, cat_cols)],
                            remainder='passthrough')
preproc.fit_transform(X)


How can I tell whether my Random-Forest model is overfitting?

I think you are probably overfitting.

The issue is that while you have performed a train/test split, you are selecting your hyperparameters based on the whole dataset! This way you are feeding information to the model, about the test set, through your hyperparameter selection. To be honest I haven't seen this in such small grid searches, but to be sure you aren't overfitting you need to change:

gc = GridSearchCV(rfc, params, cv=3).fit(X, y)

to

gc = GridSearchCV(rfc, params, cv=3).fit(X_train, y_train)

Another detail I'd like to point out is that MAE isn't the proper metric for classification; it is more suited for regression problems. For example is an example belongs to class "1", predicting the class "3" isn't twice as worse than predicting class "2". They are both simply misclassifications.

If after these you are still getting high performance, then I'd say its safe to assume you aren't overfitting. If you are I'd like to point you to this answer.


Because there is no more column "NOX"

you changed it in the previous mean aggregation. Try "avg_Nox" or whatever column  name you get after applying your first aggregation function

bh_df.groupby('CAT.MEDV').agg(
  avg_Nox=('NOX', 'mean'))


Is this a valid way of training a model ? training loss in 1.1921e-07?

Loss is just a sum of errors that your model makes, so lower values are desirable. In your case, the training accuracy is 1.00, meaning that your model is making perfect predictions during training. No mistakes - no loss, so this is to be expected.

However, having 100% accuracy in training is usually an indication that the model has become too familiar with the training data, effectively memorizing it instead of learning to generalize. Such a model will most likely fail to make adequate predictions on data it hasn't seen before. Unless your validation accuracy is also around 100%, then you're severely overfitting. Great explanations that will get you started on interpreting loss and accuracy can be found here.


Algorithm for multiple input single output ML

Since you believe the output can be predicted by a linear combination of the inputs, a reasonable approach to try is Linear Regression, specifically Multiple Regression since you have more than one input variable.

Linear regression will attempt to fit the best parameters $\beta_0$ and $\beta_1$ to model your output as a weighted sum of your inputs, ie $\beta_0*input_1 + \beta_1*input_2$. This is exactly the same as the expression you gave, but it's more standard to call the weights $\beta_i$s instead of $x$ and $y$.

The most standard form of linear regression using Ordinary Least Squares will find $\beta_0$ and $\beta_1$ that minimize the sum of the squared errors over your dataset, which are the differences between the actual values of output and the predicted values generated by computing $\beta_0*input_1 + \beta_1*input_2$ for each row.

EDIT: To answer your question in the comments:

It is always reasonable to try linear model first since it is simple and efficient, and it will give you a good baseline.

However, if you suspect there is a non-linear relationship between your inputs and outputs you can also try more flexible models such as gradient boosting regression trees or a neural network.

You do not need to know what the exact relationship is to use these models - they will learn it for you. In theory a neural network can fit any function.

As you use more complex models, however, you should be increasingly wary of overfitting.


Creating labels for Text classification using keras

You can build the text classification application with CNN algorithm by Keras library.
Please take a look at this git repository. Here

As you can see, you need to create training and testing data by loading polarity data from files, splitting the data into words, generating labels and returning split sentences and labels.
And you can create the convolutional neural network with Dense, Embedding, Conv2D, MaxPool2D of keras.

Here is the final model training snippet.

from keras.layers import Input, Dense, Embedding, Conv2D, MaxPool2D
from keras.layers import Reshape, Flatten, Dropout, Concatenate
from keras.callbacks import ModelCheckpoint
from keras.optimizers import Adam
from keras.models import Model
from sklearn.model_selection import train_test_split
from data_helpers import load_data

print('Loading data')
x, y, vocabulary, vocabulary_inv = load_data()

# x.shape -> (10662, 56)
# y.shape -> (10662, 2)
# len(vocabulary) -> 18765
# len(vocabulary_inv) -> 18765

X_train, X_test, y_train, y_test = train_test_split( x, y,     test_size=0.2, random_state=42)

# X_train.shape -> (8529, 56)
# y_train.shape -> (8529, 2)
# X_test.shape -> (2133, 56)
# y_test.shape -> (2133, 2)

sequence_length = x.shape[1] # 56
vocabulary_size = len(vocabulary_inv) # 18765
embedding_dim = 256
filter_sizes = [3,4,5]
num_filters = 512
drop = 0.5

epochs = 100
batch_size = 30

# this returns a tensor
print("Creating Model...")
inputs = Input(shape=(sequence_length,), dtype='int32')
embedding = Embedding(input_dim=vocabulary_size,     output_dim=embedding_dim, input_length=sequence_length)(inputs)
reshape = Reshape((sequence_length,embedding_dim,1))(embedding)

conv_0 = Conv2D(num_filters, kernel_size=(filter_sizes[0],     embedding_dim), padding='valid', kernel_initializer='normal',     activation='relu')(reshape)
conv_1 = Conv2D(num_filters, kernel_size=(filter_sizes[1],     embedding_dim), padding='valid', kernel_initializer='normal',     activation='relu')(reshape)
conv_2 = Conv2D(num_filters, kernel_size=(filter_sizes[2],     embedding_dim), padding='valid', kernel_initializer='normal', activation='relu')(reshape)

maxpool_0 = MaxPool2D(pool_size=(sequence_length - filter_sizes[0] + 1, 1), strides=(1,1), padding='valid')(conv_0)
maxpool_1 = MaxPool2D(pool_size=(sequence_length - filter_sizes[1] + 1, 1), strides=(1,1), padding='valid')(conv_1)
maxpool_2 = MaxPool2D(pool_size=(sequence_length - filter_sizes[2] + 1, 1), strides=(1,1), padding='valid')(conv_2)

concatenated_tensor = Concatenate(axis=1)([maxpool_0, maxpool_1, maxpool_2])
flatten = Flatten()(concatenated_tensor)
dropout = Dropout(drop)(flatten)
output = Dense(units=2, activation='softmax')(dropout)

# this creates a model that includes
model = Model(inputs=inputs, outputs=output)

checkpoint = ModelCheckpoint('weights.{epoch:03d}-{val_acc:.4f}.hdf5',     monitor='val_acc', verbose=1, save_best_only=True, mode='auto')
adam = Adam(lr=1e-4, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)

model.compile(optimizer=adam, loss='binary_crossentropy', metrics=['accuracy'])
print("Traning Model...")
model.fit(X_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1, callbacks=[checkpoint], validation_data=(X_test, y_test))  #     starts training

By running this code, you will get the trained model with the format of hd5. Finally, you can use your model for prediction.


which NN should I use for Time-series dataset, whose pattern change as time goes

Do you know when does the new pattern start? You could reset the hidden state of the RNN each time this happens during training. A more detailed explanation in this paper: https://arxiv.org/pdf/1706.04148.pdf

In that paper, they try to predict the next item in a user's session. They use a GRU to create a user-level representation (which in your case could be the pattern representation) and another GRU to predict the next item based on the current session information.

To do that, they reset the hidden state of the second GRU every time the session changes and reset the hidden state of the first GRU every time the user changes, and I think you could reset the hidden state every time the pattern changes. It is difficult to explain it all right here, I recommend you to read the paper I mentioned before.

EDIT:

To know where are the pattern change points you could train another network, maybe using as labels points where your GRU's accuracy decreases substantially, and then use this network's predictions to reset the hidden state on these points at training time as I mentioned before.


Explaining what a neural network is learning

That's a popular question, since many use cases require to understand what's going on in the model.

I suggest to do just some simulation with the resulting NN. Like in your example you could have a test-dataset and den modify the 'studying_hours' by +5%, +10%, +15%, ... and then plot the result over this modification factor. This way you can see by an simulation what the NN is thinking of the 'studying_hours' and how it influences the result.


Here's the question: is rewarding the agent for going into the right direction a "correct approach" in Reinforcement Learning?

It depends on what you are hoping the agent is capable of learning by itself. This is an issue for you here, because you have a "toy" problem where you can control a lot more of the environment and alter the meaning of what it means to win.

In general, then yes this is "cheating", at least in terms of claiming to have written an RL agent that solves the game. The academically ideal basic RL agent is rewarded by the gain of something meaningful in the context of the problem being solved, and is not helped by interim rewards. In a game of snake, and any other arcade-style game, it should really be the official points scored in the game and nothing else.

  Is passing the coordinates of the food as state an other way of "cheating"?

Again it depends on what you expect the agent to learn from. If, in your target production environment, this data was easy to obtain, and you intended to use it to write a game bot working from the trained policy, then this is fine. There is no requirement that you do one thing or another if you have a practical problem to solve.

However, learning from a pixel-only state, as in the DQN original papers, is of academic interest, because that is a generic state representation that applies to many problems, whilst the distance from the snake to food is a specific feature that you have engineered that makes learning easier in a smaller set of games.

The main issue here is again that your goal is not really to put a "snake bot" into a production system, but to learn how RL works. RL is tricky, and often doesn't work as well as you expect - or at all, for many combinations of algorithm and problem.

It is worth reading this article: Deep Reinforcement Learning Doesn't Work Yet - it may put disappointing results from basic DQN into perspective.

I would encourage you to strip back your Snake problem to remove "helpful" rewards and state, and instead look into extensions to the core DQN algorithm, or different learning agents such as A3C.


Problem understanding probabilistic generative models for classification

My understanding is:

The first line is OK - it derives from Bayes Rule

Assume that this probability follows a logistic function, that is that

$$P(C_1|x) = \frac{1}{1+exp(-a)}$$

Then if

$$ y = \frac{1}{1+exp(-z)}$$
then
$$ z = ln(\frac{y}{1-y})$$
(some lines below but with $a$ and $\sigma$)

Therefore:

$$ a = ln(\frac{P(C_1|x)}{1-P(C_1|x)})$$

If there are only two classes then $1-P(C_1|x)= P(C_2|x)$.

Then

$$a = ln(\frac{P(C_1|x)}{P(C_2|x)}$$

Using Bayes on $P(C_1|x)$ and $P(C_2|x)$

$$a = ln(\frac{P(x|C_1) P(C_1)}{P(x|C_2)P(C_2)}$$

The whole thing now is to model $P(x|C_i)$. He assumes that it is a normal distribution. And from there you should the second set of equations. The important part (as far as I remember) is that the two covariance matrix (for $C_1$ and for $C_2$) are the same. I do not have access to the book now but I will look it up tomorrow and see if there is any difficult step in the derivations.

tldr: The whole point is he assumes that the probability as a logistic, then he gets the formula for the $a$, then he assumes a normal distribution for $P(x|C_i)$, then he assumes same covariance matrix.


Clustering method for 2-D data that self-detects number of clusters and takes care of outliers

DBSCAN is the algorithm of choice for this task.
This a density based algorithm which will look for clusters according to two main parameters, epsilon and min samples.
It will also identify those samples that do not form any cluster according to prior parameters as outliers.
From Scikit-learn documentation:

The DBSCAN algorithm views clusters as areas of high density separated by areas of low density. The central component to the DBSCAN is the concept of core samples, which are samples that are in areas of high density. A cluster is therefore a set of core samples, each close to each other (measured by some distance measure) and a set of non-core samples that are close to a core sample (but are not themselves core samples). There are two parameters to the algorithm, min_samples and eps, which define formally what we mean when we say dense. Higher min_samples or lower eps indicate higher density necessary to form a cluster.

Additional material worth to be checked:
https://www.naftaliharris.com/blog/visualizing-dbscan-clustering/


Why are character level models considered less effective than word level models?

You are absolutely right about vocabulary size. I am actually conducting research on making character-level more effective.

Here is why word-level tokens are often favoured despite, characters requiring a much smaller vocabulary size.

Bag-of-word

In a bag-of-words scenario, it is pretty obvious. First, the name. Second, if you receive a word cloud of most common words, you may be able to know what the document is about. If you receive a character cloud, you would likely be completely lost. So would your computer.

Character tokens contain much less information than word tokens.

Sequential approaches

Whether you use RNNs or Transformers, dealing with text as a sequence is often where the difference between words and characters is less obvious.

Words will create sequences containing larger vectors since each vector needs to encode more information. However, it is quite rare to use a one-hot encoding approach, which would require a vector to be the size of the vocabulary (usually 30,000). Instead, word embedding are used, which are usually less than 1024-dimensional, often between 100 and 300. So, practically, those vectors are not huge.

Characters can require much smaller embeddings. Based on my research, embeddings can be as low as 32-dimensional. But, on average, at least for western languages, each word contains 7 characters. Meaning that your models will need to deal with a sequence that is 7 times bigger than if you were dealing with words.

So in terms of actual size, if you use 32-dimensional embeddings for characters, it would take as much "space" to use 224-dimensional word embeddings.

Now, where is the difference? Ultimately it comes down to this:

Characters allow you to virtually not have any "out-of-vocabulary characters" (~99.99% using a vocabulary of 300 tokens) whereas words usually only cover 50% of a language if you use 30,000 tokens. Preprocessing techniques (stemming, amortization) are there to help reduce vocabulary size, but one can argue that it removes valuable information from the input.
Words are more informative than characters and word embeddings have shown to embed information really well. Basically, performance is often good enough, and computational constraints not too big for people to worry.

In summary, when using character-level tokens, your model has to do more work. The input itself is more raw and contains less information. Plus, sequential models have shown to struggle with long-range dependencies.


I am not sure this type of model is a good use case for the particular task. More specifically, citing Chollet from Deep Learning with Python book,

Always remember that when it comes to markets, past performance is not a good predictor of future returns—looking in the rear-view mirror is a bad way to drive. Machine learning, on the other hand, is applicable to datasets where the past is a good predictor of the future.

— Deep learning with python, Francois Chollet
What is essentially being argued here is that, stock historical data is not a phenomenon that repeats itself based on its own underlying distribution.


Summary statistics by category using Python


Having $(X1,y1), (X2,y2), (X2,y2)$  saved in a dictionary:

dictionary = {X1:y1, X2:y2, X3:y3}

accuracies = []
for k,v in dictionary.items():
    X_train, X_test, y_train, y_test = train_test_split(k, v,
    test_size=0.25, shuffle=False)
    logreg = LogisticRegression()
    logreg.fit(X_train,y_train)
    y_pred = logreg.predict(X_test)
    accuracies.append(metrics.accuracy_score(y_test, y_pred))

But it's not really a good way especially when the datasets are too big.

So:

accuracies = []
for i in range(1,4):
    X = 'X{}'.format(i)
    y = 'y{}'.format(i)
    X_train, X_test, y_train, y_test = train_test_split(vars()[X], vars()[y],
    test_size=0.25, shuffle=False)
    logreg = LogisticRegression()
    logreg.fit(X_train,y_train)
    y_pred = logreg.predict(X_test)
    accuracies.append(metrics.accuracy_score(y_test, y_pred))

Note:
in order to access the varibale names which are different in a small part, like here for X1,X2,..., we can use the mentioned method:

import numpy as np
X1= np.arange(1,10)
y1=[i**2 for i in X1]
X2= np.arange(-5,5)
y2=[i**2 for i in X2]
for i in range(1,3):
    X = 'X{}'.format(i)
    y = 'y{}'.format(i)
    print('X_{}'.format(i) , vars()[X])
    print('y_{}'.format(i) , vars()[y])

Output:

X_1 [1 2 3 4 5 6 7 8 9]
y_1 [1, 4, 9, 16, 25, 36, 49, 64, 81]
X_2 [-5 -4 -3 -2 -1  0  1  2  3  4]
y_2 [25, 16, 9, 4, 1, 0, 1, 4, 9, 16]


In machine learning how to find feature interdepencies?

For me it seems you are looking for Association Rule Learning. A quite simple example for an algorithm you could use is the Apriori Algorithm. You could try that one first and see if it does what you want to achieve. Then you can take a look into more sophisticated algorithms.

Generally these algorithms try to find associations between your features. E.g. they are able to find the implication, that many people who buy Coca Cola & Beer, will buy a pack of chips - if supported by the data:

$(Coca Cola, Beer)\implies(Chips)$.


Basically, we add a regularization term in order to prevent the coefficients to fit so perfectly to overfit.

The difference between L1 and L2 is L1 is the sum of weights and L2 is just the sum of the square of weights.

L1 cannot be used in gradient-based approaches since it is not-differentiable unlike L2

L1  helps perform feature selection in sparse feature spaces.Feature selection is to know which features are helpful and which are redundant.

The difference between their properties can be summarized as :


Confusion matrix - classes get confused with each other

The confusion matrix as the name suggest helps to identify how many of the predicted classes are being confused with true classes. For instance, in this case, the rows indicate the predicted classes and the columns indicates the correct classes.

So if we consider the diagonal values, the predicted and the true class is the same. Hence, the diagonal values do not indicate any confusion. When we check the upper triangular part of this matrix,  the value 11 in your example indicates that 11 examples have been predicted as A when the correct values are actually B. Also, looking at (2,1)(the first value of second row) the value is 10. This implies that 10 values predicted as B actually are A. This shows that A and B are confused.

When two classes are closely related, it becomes harder to predict. For instance, an example I come across in my work is this. When I want to classify customer reviews, Speed of App and Speed of delivery classes seem to get confused because they have almost similar keywords in many of their examples. Since they both refer to speed, they are closely related and this leads to the confusion


Are both the candidates and values to be estimated? Or is it only the candidates.
If it's only the candidates, then try using a classification model. You can then use the predicted class probabilities as weights on the candidate  values to arrive at a final values for your entities.


Number of parameters in Neural Network Language Model

NNLM has following sets of parameters ).
Using $V$ to denote the number of words in the vocabulary:

The matrix which creates the embedding for each word in the context . This distributed embedding is in $\mathbb{R}^{m}$ space. This matrix is $C :{V \times m}$
The matrix which transforms the concatenated list of word embeddings (active in the current context of size $n-1$) to the hidden layer (of size $h$). This matrix is $H:{(n-1)m\times h}$
The matrix which maps the hidden layer to unmormalized probabilities for each word in the vocabulary. This matrix is $V :{V \times h}$
The matrix connecting the context word embeddings to the output layer. This connection is optional and has dimensions $W:{V \times (n-1)m}$
Note: while training using SGD, for  a single example only $n-1$ words in the context are active out of words $V$ in the vocabulary. I have also omitted the parameter vectors for the bias terms $b:V \times 1$   when computing unnormalized outputs and when computing the hidden layer ($d: h \times 1$).

Hence my current understanding is that  number of parameters in NNLM is:
$$dim(C)+ dim(H)+ dim(V) + dim(W)= (n-1)m \times V + h \times (n-1)m+ V \times h + V \times (n-1)m$$


In my opinion you're looking into the wrong methods to solve your problem.

You have strictly no numeric data.

Statistics based machine learning has a very hard time with such problems.
Your problem sounds more like one of the problems which should be solved with rule based systems. My first instinct would be to try to understand the rules and put them into code leading to a classification.

There are however methods for learning such rule based systems based on logic. They are only quite unfashionable for machine learning today. https://en.wikipedia.org/wiki/Rule-based_system


Understanding the XLNet model for a concrete case

There's no way to do that, XLNet is a pretrained model so the only thing we can do is tune it.