Detecting person on the picture if we know it is there

What you are trying to do is called object localisation. If you have multiple of that object in the scene it is called object detection. There are numerous things that you can try that I refer to some of them.

YOLO paper which may be the perfect thing for your task which has the state of the art performance these days.
There is another algorithm which is old and is not efficient but is a simple way of detecting objects. It is not that much accurate. It is called Sliding Window.
The previous algorithm is time-consuming due to the fact that it calculates repetitive calculations. In OverFeat: Integrated Recognition, Localization and ... paper they have tried to overcome that mistake.

There are other approaches too because object detection is an important task in computer vision and every day someone makes another approach. You can see the citation of the mentioned papers which means there are abundant studies about the problem.


k-means in R, usage of nstart parameter?

nstart option attempts multiple initial configurations and reports on the best one. For example, adding nstart=25 will generate 25 initial random centroids and choose the best one for the algorithm. Hope this helps!

You can read more here...


Help framing a sequence prediction problem

If I understand your meaning: you are modelling by proxy. In other words, it could be seen as using a common autoregressive, simply with the autoregressive part itself removed. I won't get into the details of RNNs and Keras, but let me first explain with a common model used for time-series analysis, ARIMA:

ARIMAX stands for:

Autoregressive - model a variable using its own historic values
Integrated - use differences between values over timesteps (not the raw values)
Moving Average - include historic values of the actual forecast errors
X - common final letter to show we use exogenous variables, variable apart from our target

ARIMA is characterised by three parameters, which tell us how many previous timesteps of the first three terms above are used: ARIMA(p, d, q). p is for AR terms, d for I terms and q for MA.

Within this framework, your problem would be equivalent to setting p = 0. We can see on the Wikipedia site linked above, or here, that doing so makes the ARIMA model essentially an exponential smoothing model!

You could start by analysing how correlated your variables X1 and X2 are with the target variable X3, in order to assess how well (or badly) they might explain the target.

If you follow some tutorials on setting up an experiment for ARIMA, it will only be technicalities to feed the same data into an RNN, or anything else you might build using Keras. You need to feed blocks of data into the model, leaving out previous timesteps of the target variable itself... although if you have them and can use them for validation/testing, I would wonder why you aren't using them to improve the model!


How to run R programs on multicore using doParallel package?

I would add a comment but I do not have enough reputation points. I might suggest Using "R revolution open". It is a Build of R that includes a lot of native support for multi-core processing. I have not used it much as my computer is very old, but it is defiantly worth looking at. Plus it is free.


How to plot data points and not centroids using sklearn k-means?

Scatter plots work well for 2 dimensional, continuous data. Like the toy examples you see everywhere.

But the data produced by tfidfvectorizer is high-dimensional and sparse.

It is to be expected that almost all points will have a 0 value in any two dimensions you plot. One does not simply plot test as 2d coordinates in a meaningful way. And regular k-means doesn't work that well either. The means just move close together, and all correspond to average stopword vectors - not very useful.


What is a reasonable way to compare the improvement in accuracy?

In high accuracy regimes (>0.9), I look at the error rate -- the complement of the accuracy -- or its reciprocal; the mean time between errors. This captures the intuition that an accuracy of 0.99 is ten times as "good" as an accuracy of 0.9.


Directly proportional Trend in training and cross validation curves

These learning curves seem unusual - the training accuracy should start high and get lower as more samples are added.

I suspect that your data is sequential (it has some kind of time dependency). sklearn's learning_curve function does not seem to shuffle the data (should it?), so the training accuracy can change/increase once new structures appear in the data over time.

Here's a notebook trying to reproduce the effect:
https://gist.github.com/stmax82/79b744877b0a482f8739d372c4777e0d

Two images from the notebook:

The learning curves on the shuffled data look like that (as expected):

While the learning curves on the original / sequential data look like that (unusual because training accuracy is rising over time):

That's just my try at explaining your learning curves. It might be something completely different... try shuffling your data to find out.


How to convert a text to lower case using tm package?

This seems like an encoding error. Try adding the line

Encoding(movie_Clean)  <- "UTF-8"

before you lowercase the data. Check out this answer for a little context: https://stackoverflow.com/a/28340080/4539807


Having difficult interpreting the eigenvectors for a simple 3x2 matrix

The eigenvectors can be used to transform your data into a coordinate system in which no covariance is there. Assume we have a $p$ dimensional multivariate normal distribution with

$$\mathcal{N}(\boldsymbol{x}|\boldsymbol{\mu}=\boldsymbol{0},\boldsymbol{\Sigma})=\dfrac{1}{\sqrt{(2\pi)^p\det\boldsymbol{\Sigma}}}\exp\left[-\dfrac{1}{2}\boldsymbol{x}^T\boldsymbol{\Sigma}\boldsymbol{x} \right].\quad (*)$$

As the covariance matrix is real and symmetric we know it is diagonalizable and that we can scale the eigenvectors to represent an orthonormal basis by the set of all eigenvectors. The eigenvalue equation for the covariance matrix is given by

$$\boldsymbol{\Sigma}\boldsymbol{v}_i=\lambda_i\boldsymbol{v}_i \quad \forall i=1,...,p.$$

We can combine all equations into a single matrix equation

$$\boldsymbol{\Sigma}[\boldsymbol{v}_1,\ldots,\boldsymbol{v}_p]=[\boldsymbol{v}_1,\ldots,\boldsymbol{v}_p]\text{diag}\left[\lambda_1,\ldots,\lambda_p \right].$$

If we call $\boldsymbol{V}=[\boldsymbol{v}_1,\ldots,\boldsymbol{v}_p]$ and $\boldsymbol{\Lambda}=\left[\lambda_1,\ldots,\lambda_p \right].$ With these definitions in hand we can write the eigenvalue equation as

$$\boldsymbol{\Sigma}\boldsymbol{V}=\boldsymbol{V}\boldsymbol{\Lambda}$$
$$\implies \boldsymbol{\Sigma}=\boldsymbol{V}\boldsymbol{\Lambda}\boldsymbol{V}^{-1}.$$

as $\boldsymbol{V}$ is orthogonal (consists of orthonormal vectors) we can write $\boldsymbol{V}^{-1}=\boldsymbol{V}^T.$ This implies

$$\boldsymbol{\Sigma}=\boldsymbol{V}\boldsymbol{\Lambda}\boldsymbol{V}^{T}.$$

We plug this into $(*)$ and introduce the new variable $\boldsymbol{z}=\boldsymbol{V}^T\boldsymbol{x}$.

$$\mathcal{N}(\boldsymbol{x}|\boldsymbol{\mu}=\boldsymbol{0},\boldsymbol{\Sigma})=\dfrac{1}{\sqrt{(2\pi)^p\det\boldsymbol{V}\boldsymbol{\Lambda}\boldsymbol{V}^T}}\exp\left[-\dfrac{1}{2}\boldsymbol{x}^T\boldsymbol{V}\boldsymbol{\Lambda}\boldsymbol{V}^{T}\boldsymbol{x} \right]$$
$$=\dfrac{1}{\sqrt{(2\pi)^p\det\boldsymbol{V}\boldsymbol{\Lambda}\boldsymbol{V}^T}}\exp\left[-\dfrac{1}{2}\boldsymbol{z}\boldsymbol{\Lambda}\boldsymbol{z} \right]$$
$$\implies \mathcal{N}(\boldsymbol{z}|\boldsymbol{\mu}=\boldsymbol{0},\boldsymbol{\Lambda})=\dfrac{1}{\sqrt{(2\pi)^p\det\boldsymbol{\Lambda}}}\exp\left[-\dfrac{1}{2}\boldsymbol{z}\boldsymbol{\Lambda}\boldsymbol{z} \right].$$

In the last step I used $\det \boldsymbol{ABC}=\det\boldsymbol{A}\det\boldsymbol{B}\det\boldsymbol{C}$, $\det\boldsymbol{A}=\det\boldsymbol{A}^T$ for orthogonal matrices and $|\det\boldsymbol{A}|=1$ for orthogonal matrices. Hence, we proved that we can use the eigenvectors to linearly transform our variables to obtain a new coordinate system which has a diagonal covariance matrix $\boldsymbol{\Lambda}$ (which is diagonal). This implies that we do not have any covariance anymore.

Remark 1: This diagonalization is what the principal components analysis is also doing under the hood.

Remark 2: There is one suboptimal part of your code. You should explicitly determine the sample_size = data.shape[0] and then calculate cov = 1 / (sample_size - 1) * np.dot(difference.T, difference).


Compare the MAE Vs R^2 output for Machine learning 10-fold CV

I am not sure about the scenario when the data is following the assumption of linearity.
But let's say the data has non-linearity, then this situation is possible considering the fact that squares of bigger values can easily cover-up the absolute sum of too many smaller values.
Let's see this plot below -
If we move from purple to the red line, MAE will shoot-up since we have too many points with increased MAE but SSE might come down because larger errors have been corrected. SSE coming down will improve the R2Score.
$\hspace{3cm}$
 This was a rough calculation to get the intuition. You may extrapolate it.


DBSCAN is O(n) times the cost of a neighbor search.

If you use an index like LSH that could answer neighborhood search in O(1) (assuming a very even data distribution, with O(1) neighbors per point) then DBSCAN can run in O(n) plus the time needed to build such an index.

Yes, minHash indexes could be used if they are appropriate for your data and distance.


Difference between Dueling DQN and Double DQN?

They are separate techniques.
Double uses two networks to avoid over optimistic q-values.
Dueling separates the estimator using two new streams, value and advantage. The two streams are then aggregated.


How to validate regex based Resume parser efficiently

Is there any better way to validate this piece of python code. ?

No, because any automatic method would be equivalent to creating another learner, and there would be no way to know if it's good or bad at the job. So unless you find another annotated dataset (i.e. a set of resumes with labels indicating whether or not they switched jobs in the past year), you must create your own annotated set.


Python Convert indexes starting from 1 to 365 into months names and display them into the x axis without changing the plot structure?

You will have to tweak the xaxis locator
e.g. For your sample data -

Month,Day,Max_Data,Min_Data,MonthDay
1,1,1.1,-13.3,1-1
1,2,3.9,-12.2,1-2
1,3,3.9,-6.7,1-3
1,4,4.4,-8.8,1-4
1,52.8,-15.5,1-5

import pandas as pd, numpy as np, seaborn as sns, matplotlib.pyplot as plt

data_2015 = pd.read_csv("/content/tst.csv")
plt.scatter(data_2015.index, data_2015['Max_Data'], marker='^', color='green',s=40, alpha=1.0,)

import matplotlib.ticker as mticker
plt.gca().xaxis.set_major_locator(mticker.FixedLocator(np.arange(len(data_2015.index))))
plt.gca().xaxis.set_major_formatter(mticker.FixedFormatter(np.array(['Jan','Feb','Mar','Jan','Feb'])))

Output


The target data is the desired output that happens to be in the real word. This information is known to you already(given the problem is supervised learning problem). So, for an 'AND' gate the target value is the vector stored in y on the right hand side as shown below:

x1    x2      y(desired output)
0     0          0
0     1          0
1     0          0
1     1          1

So this vector y = [0,0,0,1] is your target to these input types that you feed into the network as labels corresponding to your above input data.


I invite you to look at the Horovod project on github.
It's the most efficient way to currently execute distributed training with tensorflow. They have tutorials and benchmarks resource available


How to combine categorical and continuous input features for neural network training

There's three main approaches to solving this:

Building two models separately and then training an ensemble algorithm that receives the output of the two models as an input
Concating all the data into a single vector/tensor as a preprocessing step and then train a simple single input NN
The multiple input NN architecture you proposed

The ensemble approach is the most straight forward option and will yield decent results, however, it will not work as well as the option you proposed as because the ensemble network only receives class probabilities from the two networks as an input and will in comparison to your approach miss out on more complex relationships between the data types.

The second approach is in theory not that different from your proposed approach, with the difference being that it assumes that the network will on it's own figure out that the input consists of two types of data (as they are both in the same vector/tensor). It will take a lot of training time for the network to learn that and you might get stuck in a local minima before that even happens.

Based on my personal experience, the network you proposed is the best option as and it's likely to have shortest train time and once you get the architecture right you will find it very easy to train and maintain (retrain) the network in production as well as you will only ever have to retrain one single model.


In a regression framework one would usually not speak about (class) imbalance. In this case it is about the distribution (mostly of the dependent variable). A distribution has several moments. The expected value (1st), variance (2nd), skewness (3rd) and kurtosis (4th). They all together define how a distribution looks like.

Generally, a normal distribution works well with regression. You can also test normality. As mentionned by others, you can plot the (empirical) probability density function. Another useful thing is a boxplot. And you can look at the moments.

R example (skewed distribution):

library(sn)
library(moments)

set.seed(12)
d1=rsn(n=10000, xi=1, omega=5, alpha=100, tau=0.5, dp=NULL)
plot(density(d1))
boxplot(d1)
skewness(d1) # 0 for normal
kurtosis(d1) # 3 for normal

This yields:

> skewness(d1) # 0 for normal
[1] 0.8130714
> kurtosis(d1) # 3 for normal
[1] 3.412369

The probability density function shows a "tail" to the right (positive skewness / right-skewed distribution).

Also the box plot indicates that there are some "unusually high" values in the distribution.

R example (normal distribution):

Same procedure for normal...

d2=rnorm(10000, 3, 2)
plot(density(d2))
boxplot(d2)
skewness(d2) # 0 for normal
kurtosis(d2) # 3 for normal

Density function and boxplot look much more "harmonic" in this case.

> skewness(d2) # 0 for normal
[1] 0.002732919
> kurtosis(d2) # 3 for normal
[1] 2.947005

What can you do with non-normal distributions?

With skewed/non-normal distributions you often get bad predictions from a regression model, especially where "unusually low/high" values are found. See this post for example. The solution often is to do some kind of linear transformation to the data. This is one reason why it often is a good idea to standardize data in predictive modelling. One can also try other linear transformations, e.g. taking logs. However, there is no "silver bullet" to adress the problem.

Another consequence of skewed data in linear regression can be heteroscedasticity, which leads to unreliable standard errors, p-values, and confidence bands. See this post for a discussion.


Deep Learning Network decreasing in accuracy

Possible tricks which can improve the accuracy of semantic segmentation models:

try lowering the learning rate
try variating the batch size
train longer (more epochs)
train the model on larger images
increase the dataset size


Weights in scikit-learn metric functions

Unbalanced classes is definitely one use case, another purpose is cost sensitive learning, where the cost of misclassifying classes is different for each class. In particular scikit-learn give weights for each sample so it allows a greater flexibility.


What is the correct way to call Keras flow_from_directory() method?

Please find a working solution here.

The generators look like:

# Data generators
train_datagen = ImageDataGenerator(
      rescale=1./255,
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest')

# Note that the validation data should not be augmented!
test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        # This is the target directory
        train_dir,
        # All images will be resized to 150x150
        target_size=(150, 150),
        batch_size=batch_size,
        # Since we use categorical_crossentropy loss, we need binary labels
        class_mode='categorical')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size=(150, 150),
        batch_size=batch_size,
        class_mode='categorical')

Make sure you have your images stored in the correct way. E.g.

...images/train/class1/
...images/train/class2/
...images/val/class1/
...images/val/class2/

The generator function does need this structure. So make sure there are subfolders for each class in the train/test directories.

You can also use the data generator function for prediction as described in this post.


Exploratory Data Analysis with Image Datset

Have a look at The HASYv2 dataset. I tried to do as much of the exploratory work as possible to make sure that others can directly try more interesting thing with the dataset.

Image format specific stuff

(Minimum / Median / Mean / Maximum) (width / height / area)
Image formats
Timestamps
Exiff meta data

For this kind of stuff, you might want to have a look at edapy.

Image/ML specific stuff

Things you can do with images:

Compute the mean image

Mean image by class

Eigenfaces (or rather "Eigenimages")
Fisher-Faces

You can compute the correlation of pixels, e.g. Figure 3:

Classification-specific stuff

Plot the distribution of classes.
Behaviour of standard classification algorithms (CNNs, VGG-16)
Confusion Matrix Ordering (page 48 - 52, especially Figure 5.12 and 5.13): Find similar classes


XGBoost becomes unstable when predicting more than ~300 classes

I am affraid that it may not be a problem about tuning the XGB classifier, but about your dataset.

Even with 10M rows, a density of 0.003 over 5000 rows seems far from enough to get interesting results about 300+ classes.

I would suggest to start with a way simpler approach to confirm that the modelling approach is okay. That would mean :

Group your variables significantly more (<100), identify and keep the one that appears significant predictors
Group your classes significantly more (<10)
Start with a simpler model (random forest maybe)

Once you do that and get baseline performance metrics, you can try to add more classes / variables / complexity to your model and see if it actually improve.

A one step calibration process, where you dump all your data into a very complex model is rarely the way to go.


How to find correlations between data over time?

You can read about time varying correlation, you can obtain the correlation at each point in time based on some fixed number of last days, enough days to make it smooth but not many so as to be able to see changes in correlation. You can also use exponential smoothers to give more weight to recent observations. Also the dynamic conditional correlation might be interesting to you.


Recommender System: how to treat different events

I assume you're using an implicit feedback recommender approach like ALS. Otherwise, summing data points generally won't make sense, such as if you're feeding it to a recommender that expects ratings.

The input to implicit ALS is, conceptually, weighted user-item pairs. Therefore it makes sense to perhaps use a sum of user-item clicks as the weight. Summing makes sense.

However, does a purchase and view seem to carry the same weight? obviously not. A purchase is a much stronger association and should be weighted accordingly.

As to how much, I'd suggest weighting purchases simply by price, to start. Then weight clicks by price times purchase-to-click ratio. A $10 item purchase is weight 10; if 1 in 200 clicks results in a purchase, then weight a click 0.2.

This is crude but probably about as close as anything for capturing this info in the context of ALS.


How to adjust/smooth a certain number using constants or rules

This is a good question. The best method will be whatever gives you the best result, so the method you suggest of adding a constant to the denominator might be the right idea. However, I don't think it is ideal. The reason is that you will be systematically underestimating the percentage of coupon uses, which is not ideal. It also might be difficult to interpret or explain.

What I suggest instead is to borrow a play from the Bayesian handbook and make use of a prior. A prior represents your beliefs about the coupon spending of the average person. For example, you can represent your belief about the distribution of percent coupon usage as a beta distribution. All you need to know about this distribution is that it has parameters $\alpha$ and $\beta$ and that $\alpha$ corresponds to the number of products that used coupons while $\beta$ corresponds to the number of products that didn't use coupons. Let's say you believe the average person uses coupons for 10% of items. You might choose $\alpha=1$ and $\beta=9$ for your prior which has a mean of $\tfrac{\alpha}{\alpha+\beta}=0.1$.
 For each person in your dataset, you can estimate how often they use coupons by updating the prior with the new information. For example, if the person used one coupon on two items, you update the beta distribution by adding 1 to $\alpha$ and 2 to $\beta$ which gives a new estimated mean of $\tfrac{2}{2+10}=0.17$. You can see that when the number of purchases for an individual is small, you have little information, so your estimate is going to be close to your prior. Without much information, you assume a person is close to the average. However, when you have a lot of information, the effect the prior will be minimal because you have a lot of information and no longer need to make assumptions.

The simple way to think about this is that you are adding a constant to the numerator and the denominator: (# coupon purchases + $\alpha$) / (# entire purchases + $\alpha$ + $\beta$)

Note: even for a constant proportion like 10%, you can choose different values for $\alpha$ and $\beta$ that represent how confident you are in the prior. For example, you can choose $(\alpha,\beta) = (0.1, 0.9)$ or $(\alpha,\beta)=(10,90)$. This controls how much "smoothing" you apply.


Get values from column that appear more than X times (without pandas or external libraries)

The "no pandas or other external libraries" instruction was included in the homework question? ;-)

You need a group-by the CUST_ID and then select on the count in each group.

Your question doesn't specify, but you tag with Python. Otherwise I would say to load the data into an SQL database or use the great CSV toolkit.

If you need to implement it using Python, the group-by is done by sorting on CUST_ID, and then going through the sorted data (with a for-loop, for example), counting the number of repeated CUST_ID's. When the CUST_ID changes, you check how often you have seen this CUST_ID. If greater than X, then you output this CUST_ID. Otherwise, reset your counter.


How to print feature names in conjunction with feature Importance using Imbalanced-learn library?

The issue is that you are not introspecting properly the feature importances.

for x in bbc.estimators_:
    print(x.named_steps['classifier'].feature_importances_)

[ 0.  0.  0.  0.  1.  0.  0.  0.]
[ 0.  0.  0.  0.  0.  1.  0.  0.]
[ 0.  1.  0.  0.  0.  0.  0.  0.]
[ 0.  0.  1.  0.  0.  0.  0.  0.]
[ 1.  0.  0.  0.  0.  0.  0.  0.]
[ 0.   0.   0.   0.5  0.   0.   0.   0.5]
[ 0.  0.  1.  0.  0.  0.  0.  0.]
[ 0.  0.  0.  1.  0.  0.  0.  0.]
[ 0.  0.  0.  0.  1.  0.  0.  0.]
[ 0.  0.  1.  0.  0.  0.  0.  0.]

As you can see, the XGBoostClassifier reports the feature importances only on 8 features as defined in the parameters list of the BalancedBaggingClassifier. Those 8 features presented to each XGBoostClassifer are in fact randomly selected for each estimator of the ensemble. So you have to first find which features were subsampled and given to each XGBoostClassifier and then indirectly find the real feature importances.

names = np.array(names)
for x, feat_sel in zip(bbc.estimators_, bbc.estimators_features_):
    feat_imp =  np.nonzero(x.named_steps['classifier'].feature_importances_)
    print(names[feat_sel[feat_imp]])

['x11']
['x11']
['x7']
['x11']
['x11']
['x16' 'x3']
['x7']
['x6']
['x7']
['x7']


How to use Chi-square test in dataset with negative values

First and foremost, it does not matter to the chi-square test whether your data is positive, negative, string or any other type, as long as it is discrete (or nicely binned). This is due to the fact that the chi-square test calculations are based on a contingency table and not your raw data. The documentation of sklearn.feature_selection.chi2 and the related usage example are not clear on that at all. Not only that, but the two are not in concord regarding the type of input data (documentation says booleans or frequencies, whereas the example uses the raw iris dataset, which has quantities in centimeters), so this causes even more confusion. The reason why sklearn's chi-squared expects only non-negative features is most likely the implementation: the authors are relying on a row-by-row sum, which means that allowing negative values will produce the wrong result. Some hard-to-understand optimization is happening internally as well, so for the purposes of simple feature selection I would personally go with scipy's implementation.

Since your data is not discrete, you will have to bin every feature into some number of nominal categories in order to perform the chi-squared test. Be aware that information loss takes place during this step regardless of your technique; your aim is to minimize it by finding an approach that best suits your data. You must also understand that the results cannot be taken as the absolute truth since the test is not designed for data of continuous nature. Another massive problem that will definitely mess with your feature selection process in general is that the number of features is larger than the number of observations. I would definitely recommend taking a look at sklearn's decomposition methods such as PCA to reduce the number of features, and if your features come in groups, you can try Multiple Factor Analysis (Python implementation available via prince).

Now that that's out of the way, let's go through an example of simple feature selection using the iris dataset. We will add a useless normally distributed variable to the constructed dataframe for comparison.

import numpy as np
import scipy as sp
import pandas as pd

from sklearn import datasets, preprocessing as prep

iris = datasets.load_iris()

X, y = iris['data'], iris['target']
df = pd.DataFrame(X, columns= iris['feature_names'])
df['useless_feature'] = np.random.normal(0, 5, len(df))

Now we have to bin the data. For value-based and quantile-based binning, you can use pd.cut and pd.qcut, respectively (this great answer explains the difference between the two), but sklearn's KBinsDiscretizer provides even more options. Here I'm using it for one-dimensional k-means clustering to create the bins (separate calculation for each feature):

def bin_by_kmeans(pd_series, n_bins):
    binner = prep.KBinsDiscretizer(n_bins= n_bins, encode= 'ordinal', strategy= 'kmeans')
    binner.fit(pd_series.values.reshape(-1, 1))
    bin_edges = [
        '({:.2f} .. {:.2f})'.format(left_edge, right_edge)
        for left_edge, right_edge in zip(
            binner.bin_edges_[0][:-1],
            binner.bin_edges_[0][1:]
        )
    ]
    return list(map(lambda b: bin_edges[int(b)], binner.transform(pd_series.values.reshape(-1, 1))))

df_binned = df.copy()
for f in df.columns:
    df_binned[f] = bin_by_kmeans(df_binned[f], 5)

A good way to investigate how well your individual features are binned is by counting the number of data points in each bin (df_binned['feature_name_here'].value_counts()) and by printing out a pd.crosstab (contingency table) of the given feature and label columns.

  An often quoted guideline for the validity of this calculation is that the test should be used only if the observed and expected frequencies in each cell are at least 5.

So the more zeroes you see in the contingency table, the less accurate the chi-squared results will be. This will require a bit of manual tuning.

Next comes the function that performs the chi-squared test for independence on two variables (this tutorial has very useful explanations, highly recommended read, code is pulled from there):

def get_chi_squared_results(series_A, series_B):
    contingency_table = pd.crosstab(series_A, series_B)
    chi2_stat, p_value, dof, expected_table = sp.stats.chi2_contingency(contingency_table)
    threshold = sp.stats.chi2.ppf(0.95, dof)
    return chi2_stat, threshold, p_value

The values to focus on are the statistic itself, the threshold, and its p-value. The threshold is obtained from a quantile function. You can use these three to make the final assessment of individual feature-label tests:

print('{:<20} {:>12} {:>12}\t{:<10} {:<3}'.format('Feature', 'Chi2', 'Threshold', 'P-value', 'Is dependent?'))
for f in df.columns:
    chi2_stat, threshold, p_value = get_chi_squared_results(df[f], y)
    is_over_threshold = chi2_stat >= threshold
    is_result_significant = p_value <= 0.05
    print('{:<20} {:>12.2f} {:>12.2f}\t{:<10.2f} {}'.format(
        f, chi2_stat, threshold, p_value, (is_over_threshold and is_result_significant)
    ))

In my case, the output looks like this:

Feature                      Chi2    Threshold  P-value    Is dependent?
sepal length (cm)          156.27        88.25  0.00       True
sepal width (cm)            89.55        60.48  0.00       True
petal length (cm)          271.80       106.39  0.00       True
petal width (cm)           271.75        58.12  0.00       True
useless_feature            300.00       339.26  0.46       False

In order to claim dependence between the two variables, the resulting statistic should be larger than the threshold value and the p-value should be lower than 0.05. You can choose smaller p-values for higher confidence (you'd have to calculate the threshold from sp.stats.chi2.ppf accordingly), but 0.05 is the "largest" value needed for your results to be considered significant. As far as ordering of useful features goes, consider looking at the relative magnitude of the difference between the calculated statistic and the threshold for each feature.


Create new variable in python for each iteration

The operation above is called cross-join. So one way to do it is like this. Let your first dataframe called df1 and the second one called df2.

df1['key'] = 0
df2['key'] = 0
df_joined = df1.merge(df2, how='outer', key='key')
df_joined.drop('key', inplace=True, axis=1)

This should work.


Hyperparameter Tuning in Random Forest Model

You can access the GPU by going to the settings:
Runtime> Change runtime type and select GPU as Hardware accelerator.


How to draw a sample from data set with respect to a given categorical or numerical variable based on given freely chosen distribution? (Python)

Let's assume you want a sample of size $N$ where variable $V$ follows its distribution in the new dataset:

Draw $N$ instances from the new dataset. Let $A= [v_1,..,v_N ]$ the list of $N$  values of $V$ corresponding to these instances.
For every distinct value $v \in A$:

Select the subset $S$ of instances in the old dataset which have $v$ as a value for $V$,
let $\#v$ be the frequency of $v$ in $A$: draw $\#v$ instances from the subset $S$

At the end of this process you have obtained $N$ instance from the old dataset which follow the distribution of $V$ in the new dataset.


What is the term for when a model acts on the thing being modeled and thus changes the concept?

There are three terms from social science that apply to your situation:

Reflexivity - refers to circular relationships between cause and effect.  In particular, you could use the definition of the term adopted by George Soros to refer to reverse causal loop between share prices (i.e. present value of fundamentals) and business fundamentals.  In a way, the share price is a "model" of the fundamental business processes.  Usually, people assume that causality is one-way, from fundamentals to share price.
Performativity - As used by Donald MacKenzie (e.g. here), many economic models are not "cameras" -- taking pictures of economic reality -- but in fact are "engines" -- an integral part of the construction of economic reality. He has a book of that title: An Engine, Not a Camera.
Self-fulfilling Prophecy - a prediction that directly or indirectly causes itself to become true, by the very terms of the prophecy itself, due to positive feedback between belief and behavior. This is the broadest term, and least specific to the situation you describe.

Of the three terms, I suggest that MacKenzie's "performativity" is the best fit to your situation.  He claims, among other things, that the validity of the economic models (e.g. Black-Scholes option pricing) has been improved by its very use by market participants, and therefore how it reflects in options pricing and trading patterns.


Consider differentiating this $$\nabla_\theta\|X\theta -y\|^2=2X^T(X\theta -y)=0$$

Hence solving this, would give us $$X^TX\theta =X^Ty$$

Solving this would give us the optimal solution theoretically. However, numerical stability is an issue and also don't forget computational complexity. The complexity to solve a linear system is cubic.

Also, sometimes, we do not even know even have a closed form, a gradient based approach can be more applicable.


Memory efficient structure for membership checking without false positive

For web crawler scale why not use a distributed database like Apache Cassandra? Lookups on indexes are efficient and no false positives.


What's the of all values above some percentile called? How do I get it in pandas?

I don't think there is a standard library for it.
You can just skip nan value as mean treat them as zero which can mess up the result.
import pandas as pd
df=pd.DataFrame(lst) #For a given list
df[df >= df.quantile(.95)].mean(skipna=True)


Uncertainty in connection to explainability

There is a bit of confusion, I'm afraid:

The definition you propose for uncertainty doesn't really represent the concept of uncertainty: if $p_i$ is the probability that $x$ belongs to category $a_i$, then $1-p_i$ is just the probability that $x$ doesn't belong to category $a_i$.
Yes, if the classifier assigns a very high $p_i$, say 0.99, it is supposed to mean that the classifier is very confident in its prediction. But it's also the case for a very low probability: if $p_i=0.01$, the classifier is very confident that $x$ doesn't belong to $a_i$. According to your definition, the uncertainty in this case would be very high (0.99) even though the classifier is very confident, so your definition is inconsistent.

Now the main problem: whatever you call a confidence measure based on the probability predicted by the classifier, it's not reliable. A prediction is at best an informed decision of the classifier given the data it has seen in the training set and the features of instance. But it could be a random classifier, or a majority-class classifier: in these cases the probability it "predicts" is arbitrary. Imagine you are a teacher and one of your students says "the answer of x=2+2 is x=5", I'm 100% sure". The fact that the student is "100% sure" doesn't make them right, same thing for the classifier. In other words, any reliable measure of uncertainty involves the gold-standard answer, so it's usually part of the evaluation process. That's not to say that the predicted probability is useless, but in general it has no direct link to accuracy, and it would be a mistake to interpret it in this way.
Interpretability (or explainability) is a completely different matter: the general idea is to know whether the answer predicted by a classifier can be understood by a human. Typically traditional models like Naive Bayes or Decision Tree models are more directly interpretable (at least with not too many features) than deep NN models.


NLP: How to group sub-field into fields?

You need an ontology or any form of third-party data which describes the relationships between fields and sub-fields. You could use resources such as Wikipedia categories or standard library classifications for instance. There are probably other options for scientific fields.


This dataset presents transactions that occurred in two days, where we have 492 frauds out of 284,807 transactions. The dataset is highly unbalanced, the positive class (frauds) account for 0.172% of all transactions.

I guess your model learnt nothing at all ;)

You should consider some form of resampling and using metrics that can handle imbalance.
This might be a good starting point. You can find bunch of similar threads here as well.


How are images (roughly) clustered when using KNN

Lets say its a gray scale image. And of size 224*224 than you have 50176 possible pixel values. So you can plot on x axis widt=224 and on y heigth=224.

Lets say they are coloured. Than you have more chanels=dimensions. In that case you perform clustering in higher dimensional space


I suggest to first parse the pdfs to raw text and then retrieve the information in the next step. I assume you use Python.

As commented, you can use PyPDF2 to obtain the raw text and the number of pages of the documents. Alternatively you can use cloud services such as AWS Textract to parse multiple pdfs to raw text.

You may then apply regular expressions, for example re.findall() with specific matching patterns to the parsed pdf documents to match the terms you seek to find.


Uniformity of color and texture in an image

this is pretty simple approach

Firstly, can you evaluate your scripts on more images to get an idea of how well it performs? If you get an acceptable classification accuracy (or e.g. a good F1 score), then there is no need to try out a CNN!

  I have no idea about the false positives

Actually if you cannot evaluate your method like that, then a CNN is also not possible!

If you still want to try a deep learning method, bear in mind that you will generally need a lot of images, let's say at least 1000 in your "master image" training set - then hopefully a good percentage of images to test again (hold-out set / test set).

  I don't know the approach for the problem is the best one

Your current method seems reasonable to me. There probably is "the best" method in general, so don't be too worried.

Somebody has written An Analysis of Deep Neural Networks for Texture classification - maybe that contains some ideas to get you started  :-)

Here is a short video intro to one approch that introduced the Local Binary CNN, which was originally used for image classification, but perhaps could be adjusted to your problem.

Instead of aiming for target classes, you would simply need to map input to the labels, and could even relax the focus on colour/texture - the CNN would extract what it needs to learn the mapping.

NOTE: the downside of an end-to-end system like that is that you would no longer know what your model is using as its features to make its prediction! Colour? Texture? Raw RGB values in the top left corner?

This is actually where the LBCNN could shine, because its sparse (stochastically) binary kernels are much less likely to overfit to your training data, compared to a standard CNN.

While I couldn't find the code of the authors from the video above, here is an LBCNN implementation for face detection.


Problem with basic understanding of polynomial regression

"Linear regression" (aka. "ordinary least squares", OLS) refers to the type of estimator. Linear here means that you minimise the sum of squared residuals for a given (linear additive) regression equation. You can write a simple model:

$$y = \beta_0 + \beta_1 x_1 + u. $$

This would fit a linear function with intercept $\beta_0$ and slope $\beta_1$. So the "fitted" function would be:

$$ f(x) = \beta_0 + \beta_1 x. $$

You can "propose" basically any functional form to be fitted by the linear model, e.g. with a squared term (just add $x^2$ as feature):

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x^2_1 + u. $$

This will give you the (estimated) quadratic function:

$$ f(x) = \beta_0 + \beta_1 x + \beta_2 x^2. $$

In essence, you can add any kind of linear transformation of $x$, e.g. $log(x)$, $exp(x)$ etc.

"Non-linear" models (there is no really clear definition), would be e.g. logistic regresion, non-parametric regression, tree-based models, neural nets etc. While each coefficient $\beta$ directly gives you a "marginal effect" in linear (OLS) regression, this is not the case for the other (non-linear) models.

When you use regularisation in linear models, you simply add a penalty term to the loss function, so you minimise the sum of squared residuals given a penalty term. However, regarding your last question, I'm not quite sure what your actual question is.


Unable to figure out nInputPlane in SpatialConvolution in torch?

The nInputPlane is the depth or the number of the layers of the input image. In case of RGB images, this should be 3 which corresponds to the first number in the input_shape=(3, 256, 256).

The nOutputPlane is the the number of the layers of the volume that the convolution step will produce which is also the number of filters/kernels applied to the input. By convention, there is an output layer for each filter. This corresponds to the first argument of the Convolution2D function.


How do I show my pandas dataframe that I have in my jupyter notebook so that I can ask a question on stack exchange/stack overflow?

df.head().to_dict()

will give something which looks like

  {'x': {0: 1, 1: 200, 2: 4, 3: 5, 4: 6}, 'y': {0: 4, 1: 5, 2: 10, 3: 24, 4: 4}}

you can then add pd.DataFrame() like so

pd.DataFrame({'x': {0: 1, 1: 200, 2: 4, 3: 5, 4: 6},
              'y': {0: 4, 1: 5, 2: 10, 3: 24, 4: 4}})

This will allow anybody else just to copy and paste into their editor and run your code.

Here is a really detailed discussion on what you are describing
https://stackoverflow.com/a/20159305/6646710


Naive Bayes and Support Vector Machine (NBSVM) Classification

you use sklearn "CountVectorizer" and "TfidfVectorizer" to covert the text data into vector

    tfidf = TfidfVectorizer(sublinear_tf=True, min_df=5, norm='l2', encoding='latin-1', ngram_range=(1, 2), stop_words='english')
X_train, X_test, y_train, y_test = train_test_split(df['text'], df['class'], random_state = 0)

# vector representations of the text
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)

# Building a SVM model
svmmodel = LinearSVC().fit(X_train_tfidf, y_train)


In general, if you want to generate data in X, or X, Y pairs, then you should start by training a generative model - as opposed to a discriminative model, which is what most NN classifiers are. There are many types of generative model.

Variational Autoencoders (VAE)and Generative Adversarial Networks (GAN) have been demonstrated recently with interesting results on images - although both take a lot of training data and time, and are limited to relatively small image dimensions (e.g. 128x128). There are many sub-types of these two designs, including a combined VAEGAN which attempts to combine strengths of both.

With a trained CNN, you can generate data - sort of. What you can do is start with some arbitrary value of X and Y, then use back propagation to calculate gradients of a cost function. But instead of using the gradients to update weights, you back propagate all the way to the input, and use the gradients at the input to alter X, repeating the process multiple times.

This is essentially how Deep Dream and Style Transfer work (although in general these don't use a Y value, but selected activation values within layers). There is a major caveat to this approach - your generated X will not be sampled evenly from any distribution of X that the network has been trained with. Instead you will generate a "super stimulus" X for the given Y.

You mention RNNs. One way these can be used to generate X is by sampling from their output and feeding this back into the input. For text sequences this tends to generate grammatically correct nonsense. I am not sure if this would be considered a strictly generative model, since it is not clear to me whether the input X is being sampled evenly. It is likely that you could use the approach to generate images too, although you would have to take care defining what the sequence is (just a sequence of pixels line-by-line will probably not produce any recognisable image).


Simple logistic regression wrong predictions

Because you have a class imbalance and very little data. Your model is essentially working off of a prior probability that disproprtionately favors the positive class, and you didn't give the model enough data to escape this prior in the learning process. Consider the following detailed output from your model:

model.score(X,Y) # 0.6666
model.coef_      # array([[ 0.36566712,  0.36566712]])
model.intercept_ # array([ 0.18517658])

model.predict_proba(X)
# array([[ 0.45383769,  0.54616231],
#        [ 0.36566869,  0.63433131],
#        [ 0.36566869,  0.63433131]])

Your model correctly identifies that [0,0] is more likely to be negative than either of the other observations, but the model isn't able to escape its bias towards the positive class. If you duplicate your dataset a couple of times, the model will eventually have enough "evidence" to escape this bias. Geometrically, two things will happen when we give the model more data:

The discrimination boundary will shift towards the negative class. This will manifest as an increasingly negative intercept term.
The slope of the logistic curve will steepen, having the effect that a unit change in the inputs will have a larger impact on the outcome. If we give the model enough perfectly separable data, the curve will approach a step function and the model will output probabilities of 0 and 1. This manifests in the magnitude of the coefficients. With just three observations, the curve is fairly flat and changing the values of the inputs doesn't have much effect on the outcome probability, so the resulting class assignment is essentially determined by the intercept alone.

Contrast the results from your original model with the what happens when we replicate your dataset 100 times:

model2 = LogisticRegression()
X2 = np.matlib.repmat(X,100,1)
Y2 = np.matlib.repmat(Y,100,1).ravel()
model2.fit(X2, Y2)

model2.score(X,Y) # 1.0
model2.coef_      # array([[ 4.95489068,  4.95489068]])
model2.intercept_ # array([-2.00091433])
model2.predict_proba(X)

# array([[ 0.88089304,  0.11910696],
#        [ 0.04954891,  0.95045109],
#        [ 0.04954891,  0.95045109]])


How to handle multiple datasets of sequential data

Generally, to perform machine learning all data needs to be in a single dataframe. Team name, better yet team id, should be the primary key. Then modeling each year requires selecting the relevant columns (e.g., wins_year_1, loses_year_1, …). Finally, the general model would select all the columns.

It is best practice to transform all data before modeling. This avoids making transformation errors on some of the data which can lead to errors in modeling.

Pandas support this type of combining through the merge command.

Typically, this data is not called sequential data. Sequential data implies a notion of state. The state changes values over time. In most sports, recent performance is not a strong indicator of how likely a team will the next game. This is often referred to as the Hot Hand Fallacy. Thus, current "win/lose" state of a team is not a strong feature in modeling.


In Recommendation systems, Does we need to build each model for each product if we are using Logistic regression?

I can only guess that since it is binary classifier based recommender model, it predicts whether a product is recommendable to a user or not with a probability score which can be set to a threshold.


measuring flip-flop behaviour across several topics

So I have a theory for this. We want to be able to distinguish users who are emotional and biased from those that are unemotional and unbiased.
$$M = emotionality\\
B = Bias\\
t_i = tweet\ number\ i\\
b_i = bad\ words\ in\ t_i\\
p_i = good\ words\ in\ t_i\\
w_i = words\ in\ t_i\\
M = \frac{(b_i+p_i)}{w_i}\\
B = \frac{(b_i-p_i)^2}{w_i}\\
$$

M is the emotionality because it calculates the frequency that word is going to be an emotional one (instead of a descriptive one).

B is the bias because biased tweets will have more good (or bad) words than ones that hold both sides equally.

I don't know if this prediction will hold true, but i reckon if you plot each tweets x as bias and y as emotionality with a hue of good or bad, there will be two clusters (i am eager to be shown to be mistaken this certainly sounds like an interesting problem)!


If you choose zero initial weights, then the perceptron algorithm's learning rate $\eta$ has no influence on a neuron's predicted class label.

This is because the decision function used in the perceptron algorithm depends only on the sign of $z$:
$$
\phi(z) = \begin{cases} 1 &\text{ if } z \ge 0\\ -1 &\text{ otherwise.} \end{cases}
$$

Consider a perceptron algorithm with a single neuron.
(I will write ${\bf x} \cdot {\bf y}$ for the vector product ${\bf x}^T {\bf y}$ to avoid ugly double superscripts).

Pick an initial weight vector ${\bf w}^{(0)}.$
Plug in the first input vector ${\bf x^{(1)}}$ and predict the class label
$$
\hat{y}^{(1)}  = \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)}).
$$
This gives the weight update
$$
\Delta {\bf w}^{(1)} = \eta (y^{(1)} - \hat{y}^{(1)}){\bf x}^{(1)},
$$
where $\eta \in (0, 1)$ is the learning rate and $y^{(1)}$ is the true class label.
The new weights are ${\bf w}^{(1)} = {\bf w}^{(0)} + \Delta {\bf w}^{(1)}.$
Similarly, plug in our the input vector ${{\bf x}^{(2)}}$ and predict the class label
$$
\hat{y}^{(2)} = \phi({\bf w}^{(1)} \cdot {\bf x}^{(2)}).
$$
This gives the weight update
$$
\Delta {\bf w}^{(2)} = \eta (y^{(2)} - \hat{y}^{(2)}){\bf x}^{(2)},
$$
where $y^{(2)}$ is the true class label.

Notice that $\Delta{\bf w}^{(2)}$ implicitly depends on $\Delta{\bf w}^{(1)}$, which in turn implicitly depends on ${\bf w}^{(0)}$.
Let us unravel these dependencies by plugging in:
\begin{align*}
\Delta {\bf w}^{(2)} &= \eta \big(y^{(2)} - \hat{y}^{(2)}\big){\bf x}^{(2)} \\
&= \eta \big(y^{(2)} - \phi({\bf w}^{(1)} \cdot {\bf x}^{(2)}) \big) {\bf x}^{(2)} \\
&= \eta \big(y^{(2)} - \phi\big( ({\bf w}^{(0)} + \Delta {\bf w}^{(1)}) \cdot {\bf x}^{(2)} \big) \big) {\bf x}^{(2)}\\
&=  \eta \Big(y^{(2)} - \phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \hat{y}^{(1)}){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)} \\
&=  \eta \Big(y^{(2)} - \phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)})){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}.
\end{align*}

Now suppose we have initialized with zero weights: ${\bf w}^{(0)} = {\bf 0}.$
Since ${\bf 0} \cdot {\bf x}^{(1)} = 0$ and $\phi(0) = 1,$ the last line in this calculation simplifies to
$$
\Delta {\bf w}^{(2)} = \eta \Big(y^{(2)} - \phi\Big(  \eta (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}
$$

Let us zoom in here and look at $\phi$ and its argument:
$$
\phi\Big(  \eta (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big).
$$
Since $\eta > 0$, it does not change the sign of $(y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)}.$ But the sign is all that matters to the function $\phi$.
So we can simply remove $\eta$ from the function argument without changing the result:
$$
\Delta {\bf w}^{(2)} = \eta \Big(y^{(2)} - \phi\Big( (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}.
$$

The same arguments hold for $\Delta {\bf w}^{(3)}, \Delta {\bf w}^{(4)}, \dots$

It follows that
$$\Delta {\bf w}^{(i)} = \eta * \text{[Something that does not depend on $\eta$]}$$
for every $i = 1, 2, \dots$
This is what your quote means by saying that the parameter $\eta$ "affects only the scale of the weight vector, not the direction."

For nonzero initial weights, this is different. Recall the last line of the long calculation $\Delta {\bf w}^{(2)} = \dots$ above and zoom in again on the function $\phi$ and its argument:
$$
\phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)})){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big).
$$
If ${\bf w}^{(0)}$ is nonzero, then $\eta$ may affect the sign of the function argument, and therefore the predicted class label.


Why categorical cross entropy loss is not correlated with NLP scores?

There may be no mathematical linkage between categorical cross entropy and BLEU. BLEU if i remember correctly is probably a distance based measure of computer output to human judgement - there's no loss variable anywhere. This is unlike the perplexity score which IS derivable from the categorical cross entropy.
Loss function really depends on what you're comparing against your labels. Categorical cross entropy gives a way to compare the distributions of the prediction and truth labels and calculate an error based on the 'distance' between the distributions. Categorical cross entropy really works well to model the distributions of discrete vector labels, common in NLP.


My suggestion is to use Convolutional and Recurrent layers in the same Neural Network.

You'd have to capture a given number of frames of a video (let's say one each 0.5 seconds), and feed arrays of screenshots into the model. Its structure would be:

Conv (and MaxPool) layers to process pixel data - they will extract and process relevant information from each screenshot.
LSTM layers - that will process their sequence, extracting meaning from their flow.
Dense layers at the end to perform classification, with softmax activation at the output layer.

That's how I would do. It's going to be computationally expensive, if you don't have a GPU it won't be easy.


For feature importance, you might want to consider using Shapley values or LIME. There are some examples here.


Feature-to-parameter mapping in neural networks

Yes, at least you can identify what pixels' are contributing most in the prediction.

Tool like Layerwise Relevance Propagation, used for Explainable AI, serves the similar purpose and evaluate the values(weights) during back propagation and evaluate what pixels are contributing most.

Many opensource implementation are available and on similar track, instead of identifying just relevant pixel you can perform activity for each pixel.

I believe that I'm able to answer your question.


Can I use SVC() as a base_estimtor for ensemble methods?

In short: Yes.

Conceptually, bagging and boosting are model-agnostic techniques, meaning that they work regardless of the learner.

Bagging essentially is the following:

create multiple predictors (they can even be hard-coded!)
gather predictions from the learners and come up with a prediction

Boosting can be seen as:

train a predictor
find where the predictor makes mistakes
put more emphasis on these mistakes
repeat until satisfactory

Regarding the specific Sklearn implementations, here are the base learners that you can use:

AdaBoostClassifier()

The documentation says Support for sample weighting is required, as well as proper classes_ and n_classes_ attributes.

This means that you can use all models that can give weight to your samples as part of the learning process (KNN, SVM, etc.)

BaggingClassifier()

This is a simple bagging strategy, so all estimators can be used here.

GradientBoostingClassifier()

This requires that your learners are differentiable so that the gradient can be computed. Generally, this technique is specific for tree learning.


Which machine (or deep) learning methods could suit my text classification problem?

You state that some of the words may occasionally be truncated or concatenated. Thus I would extract n-grams from your Strings and then use that into a bag-of-words vector.

How does this work?

n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...

The segmentation works as follows:

The String: Hello World

2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld"
3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld"
4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"

Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features.

Bag-of-Words builds a dictionary of the words it has seen during the training phase. Then using the word the frequency of each word in the example a vector is created. This can then be used with any standard machine learning technique.

Due to the high number of grams that will result, you will want to do some feature dimensionality reduction. You can use techniques such as PCA and LDA to determine which features (grams) are most pertinent to your decision boundary.


How to group a unique column and find the mean value in a separate column in R

Something like this for instance:

library(plyr)
ddply(data,'region',function(x) {mean(x$age)})


Convert Atypical Date Format for Time Series in Python

Let pandas determine what datetime format you are using automatically.

import pandas as pd
raw_data = pd.DataFrame(data={'BC_DT':['27-MAR-18','28-MAR-18','29-MAR-18']})
raw_data['BC_DT'] =  pd.to_datetime(raw_data['BC_DT'])
print(raw_data)

  BC_DT 0:   2018-03-27 1:  2018-03-28 2:  2018-03-29


how to increase the coverage of wordnet using wikipedia

You should look at Open Multilingual Wordnet (http://compling.hss.ntu.edu.sg/omw/).

They used wiktionary (not wikipedia, because wikipedia lemmas are nouns only) to get translations and synsets. EDIT: In the extended version, there is also a Hindi wordnet with 3,593 synsets, see http://compling.hss.ntu.edu.sg/omw/summx.html

There is a non-open Hindi wordnet at https://catalog.ldc.upenn.edu/LDC2008L02.


Only square matrices have an inverse, and neither $X$ nor $X^T$ here are (necessarily) square. I guess that's the most direct answer. $X$ may have a left-inverse $A$, such that $AX = I$. In that case, if $X\theta = y$ then $AX\theta = Ay$ so $\theta = Ay$.

The direct way to construct that $A$ is just where you started though; it's $(X^TX)^{-1}X^T$. The Gramian matrix $X^TX$ is (square, and) invertible if the features (columns) of $X$ are linearly independent, and, $X^TX$ is more importantly relatively quite small and so it's feasible to invert.

You normally wouldn't compute the inverse anyway, but try to solve the system $X^TX\theta = X^Ty$ with QR decomposition or something, but that's not what you're asking.


Scaling continuous data to discrete range

Update Accordingly: Your question was not clear before, therefore, sorry for the irrelevant solution.
To achieve that, I don't think there is a public library function, however, you can build your solution using some beautiful "ready" functions.
Two solutions come into my mind:

First one's time complexity is O(N*M)

N is your prediction (list a in your case) size, and M is your dictionary (list b in your case) size.
import numpy as np

def findClosest(dictionary, value):
    idx = (np.abs(dictionary - value)).argmin()
    return dictionary[idx]

#[findClosest(b, elem) for elem in a]

print([findClosest(b, elem) for elem in a])

This just subtracts your prediction value from the values in your dictionary and takes the absolute value of them. Then in the resulting array, it looks for the location of value that is smallest.

Second one's time complexity is O(N*log(M))

N and M denote the same thing as the first solution.
from bisect import bisect_left
def findClosestBinary(myList, myNumber):
    pos = bisect_left(myList, myNumber)
    if pos == 0:
        return myList[0]
    if pos == len(myList):
        return myList[-1]
    before = myList[pos - 1]
    after = myList[pos]
    if after - myNumber < myNumber - before:
        return after
    else:
        return before

#[findClosestBinary(b, elem) for elem in a]

print([findClosestBinary(b, elem) for elem in a])

Note: To save time I didn't implement myself but took the findClosestBinary() function from here.
This one is a better algorithmic approach in terms of time complexity. This does the same thing but uses a binary search to efficiently find the closest value in the dictionary.  However, it assumes your dictionary (list b) is sorted. Since your dictionary is a predefined list, you can improve the performance by providing a sorted array.
However, if your dictionary that you will map predictions to is not a very big one, then you can just use the first one. In the case of the dictionary being small, these two functions will behave the same in terms of time.


sentence classification with RNN-LSTM - output layer

When doing a multiclass classification problem, in which the goal is predict exactly one class label for each input, it is standard to use the softmax function (a normalized exponential) as the activation function for the last layer. However, if your problem is a multilabel classification (in which the classes are not mutually exclusive), then using a sigmoid as the activation function would be appropriate. In both situations, you would have as many output units as labels (20 in your case).


How to export PCA to use in another program

Ideally PCA should not be used as a part of pre-processing feature reduction.

Anyhow regarding saving and reusing PCA model, sharing a basic code snippet which is working very fine in my case(as I'm not able to reproduce the error case).

from sklearn.decomposition import PCA
import pickle as pk
pca = PCA(n_components=2)
result = pca.fit_transform(X) # Assume X is having more than 2 dimensions
pk.dump(pca, open("pca.pkl","wb"))
.
.
.
# later reload the pickle file
pca_reload = pk.load(open("pca.pkl",'rb'))
result_new = pca_reload .transform(X)

# result and result_new same in my case


What are the best practises to decide whether a variable is categorical?

The number of categories within a variable does not matter whether a variable is categorical or not.

Categorical variables are mutually exclusive, unordered groups. For example, Christmas and Halloween are different holidays but have no order in the concept of "holiday-ness".

Ordinal variables which are mutually exclusive, ordered groups with no consistent measure of the distance between the ordering. For example, ranking items (e.g., first, second, third, …). There could be big (or small) differences between each of the places.

Numeric variables have consistent differences between values. The difference between 10 and 11 miles is the same as the difference 20 and 21 miles because "mile-ness" is a consistent measure.


I'm not sure irrelevant is the right word, but here is an example where it may not be an easy trend.

You're trying to predict blood pressure.

If you're only given weight, there may be some correlation, but someone who is 150lbs could be 5'10 or 5'4, and that would make a large difference to their overall health.

If you have both, you'll likely have a much stronger correlation to work with.

But as all things in ML you should test it yourself usingany real world examples as theory doesn't mean much if it doesn't hold true with real data.


Instead of mean and standard deviation, you could estimate the median and mean absolute deviation. The median is immune to outliers, and the MAD should be at least more robust than the standard deviation formula.

You will probably have to change your critical value to something other than 5 to get the same kind of coverage. According to Wikipedia, you'll want the new critical value to be $5\sqrt{\frac{\pi}{2}}$ if your data are iid Gaussian.

An alternative that might be more difficult to implement, but is probably more statistically appropriate, is to use trimmed estimators for the mean and standard deviation. With trimmed estimators, you throw away the most extreme values in your data (the proportion of which is specified beforehand), and estimate your statistics on the remaining data.

The estimator for the mean would be the truncated mean, and the Wikipedia page for trimmed estimators mentions how to get a decent estimator for the standard deviation from the interquartile range.

I hope this helps!


The strided deconvolution is expanding the size of the layer (you can think of it as undo-ing the effect of the pooling without doing unpooling).

Check out slide 18 here. That might help.


What is it used for?
Data ingestion into a distributed datastore (e.g. HDFS). See image (I did not make the image and am only including the image for visual aid). There are other tools that will help you with the ingestion of data as well (Storm and Sqoop are mentioned).
At which stage of a BigData analysis is it used?
It is used for data ingestion into your distributed datastore (e.g. HDFS). So for example a webserver is running logging information into /var/logs/webserver.log. Apache Flume can look at that file, grab what it needs out of it and send it to HDFS. Once the data gets put into your datastore you can then utilize other tools to analyze the imported data (e.g. Hive, Pig, MR, etc.)
And what are prerequisites for learning it?
Understanding of how to write scripts, edit configuration settings, and your way around Linux would be the absolute minimum to get started. This set of instructions are old but a starting point would be to look at the hortwonworks tutorial on flume. http://hortonworks.com/hadoop-tutorial/how-to-refine-and-visualize-server-log-data/
If you would like me to elaborate more I would be glad to but I wanted to try and meet your requirement of a simple, short explanation.


If the dim of your data is more than 2, you may want to transform your data with PCA, t-SNE or Custom linear projections.

t-SNE is recommended for clustering result.

For example:

from sklearn.manifold import TSNE
tsne = TSNE(n_components=2, random_state=0)
data = tsne.fit_transform(data)

This may lose some information, but useful for visualization.


Do generative model produce varying outputs for same input

This depends entirely on the specific model. There are generative models, like most Generative Adversarial Networks (GANs) that receive a random number and generate data. There are other generative models that generate a probability distribution over the output space (e.g. text generation models), and therefore whether the model generates data deterministically depends on the inference procedure (e.g. greedy, sampling, beam search).
If you want your model to generate outputs deterministically, you just select a model and inference method that assures that.
In your example, you may have a normal seq2seq model (e.g. Transformer) and use beam search for decoding, and the outputs will be the same given the same input.


What is the meaning of an empty SHAP graph in Explainable AI?

that means shap values were nan.Model was not able to compute shap values.


Weighted degree in Multidimensional networks

I think your weight function should depend on $d$ as well.

The contribution from a single dimension is just the sum of incoming weights. Even for simple graphs this does not seem to have a standard name, although it is quite commonly encountered.
In the context of random walks it is sometimes called simply degree, in the context of empirical networks, I found node strength.

I think this is a simple and natural statistic to look at if you wish to model something as a multidimensional network. However, being a linear sum of rather trivial functionals over simple graphs, I doubt it is of much fundamental interest.


CNN strategy in recognizing spinned images

Learning for a CNN depends on the width and depth of a network. Wider and deeper networks can learn more complex data structures, including data augmented images. Increasing the width and depth of a network increase the capacity of the model to learn features.

Width is generally associated with the number of features. The wider a network is the more features it is able to learn. Imagine a network that is very thin. It would only learn the single feature that is correlated with high performance on the task.

Depth is will increase the capacity of the model better weigh lower-level features. Each successive layer is a combination of the previous layer. The lower layers learn simple features which are combined to create complex features in higher layers. In the case of faces, lower layers learn lines of different orientations. Higher layers learn to combine those lines to form eyes and noses.

Increasing the number of kernels is one way to increase the width of the network.

Increasing the number of layers (i.e., depth) is generally more useful.


YOLO: What is better? multiple networks per label or multiple labels in single network?

It depends specially on data size and model capacity.

Therefore, we cannot give a definite "yes" or "no" prior to any experiment. However, we can enumerate possible cases, causes, and ways to improve, regardless of any specific model or dataset.

If the number of data points per category is insufficient, which is frequently the case in reality, a single model for combined categories performs better, since the effect of data shortage will be attenuated by sharing a model among different but related tasks, i.e. shared knowledge. However, we should not forget about the capacity of model, if model capacity is not enough for the combined task (which depends on the architecture and size of the model), this sharing would backfire as the model spreads its limited capacity across multiple tasks and becomes "jack of all trades, and master of none". Therefore, this approach is favorable if model capacity permits, which is less of a concern for neural networks with flexible size constraints.
If the number of data points per category is sufficient, which is rarely the case, or model capacity is limited (for example, when model performance stops improving when trained on more than 10% of data), single model per category is better since no capacity will be wasted on different (although related) tasks. From a different point of view, although this choice might be less favorable in terms of model accuracy, but it allows parallel development on smaller tasks, i.e.  faster production, which could be a justified benefit for the cost we pay in terms of model accuracy.

Both approaches

We can also combine the two approaches. First we train a model on combined categories, second we feed a part (or parts) of the trained model (such as a layer, the final prediction, etc.) to a specific model that will be trained on one category. This is exactly what a pre-trained model does for us, it is trained on a more general task with more data (possibly even a different but related task), and then we use it for a specific task.

In your case, you can go for three levels:

The pre-trained YOLO which is already built,
Training MyYOLO on all categories (for example, by using one input from the last layer of YOLO, and one input directly from image), and
Training MyYOLO2 on specific categories (for example, by using one input from MyYOLO and one input directly from image). Noting that at each level, weights of previous model remain constant to serve a constant input.


We should not use time series concept here. You have to do a pair plot analysis to find your best predictor. I do not think index column is much useful here. I feel length, width or breadth are features if you classify the vehicles as sedan, suv or small car. Not sure if it be of much help in predicting the price. If you add more breadth to the dataset, make sure you have enough depth or training examples

Do a heat map and find the co-relation between the feature and target variable as well. You can remove those features which does not have good co-relation with the target variable.

Try to start with Linear regression model, since you have to predict the price. then move on to more complex models. Random forest regressor is also a good choice for smaller datasets


Replace data in Pandas dataframe based on condition by locating index and replacing by the column's mode

Your entire code is correct except at the last point where you are equating with df['columnname'].mode(). The value here should have a dtype int or string but this has a dtype object. Just replace it with df['columnname'].mode().values and you are good to go.

Also, I see a lot of stuff that is not required here. Since you are using pd.loc, you can simply write:

df.loc[df['columnname'] == "not known",'columnname'] = df['columnname'].mode().values


Policy Iteration is essentially a two step process:

Evaluate the current policy by calculating $v(s)$ for every non-terminal state.

Internally this requires multiple loops over all the states until the value calculations are accurate enough

Improve the policy by choosing the best action $\pi(s)$ for every non-terminal state

The overall process terminates when it is not possible to improve the policy in the second part. When that happens, then - from the Bellman equations for the optimal value function - the value function and policy must be optimal.

In detail for your example, it might work like this:

Initialise our value function for $[S_0, S_1, S_2]$ as $[0,0,0]$ - note that by definition for the terminal state $v(S_2)=0$ so we never need to recalculate that (this is also consistent with the diagram, which shows the "absorbing" actions) so we could actually calculate it if we want, but it's a waste of time.
Choose a value for accuracy limit $\theta$. To avoid going on forever in the example, let's pick a high value, say $0.01$. However, if this was computer code, I might set it lower e.g. $10^{-6}$
Set our discount factor, $\gamma = 0.1$. Note this is quite low, heavily emphasising immediate rewards over longer-term ones.
Set our current policy $\pi = [B, R]$. We're ready to start.
First we have to do policy evaluation, which is multiple passes through all states, updating $v(s) = \sum_{s',r} p(s',r|s,\pi(s))(r + \gamma v(s'))$. There are other ways of writing this formula, all basically equivalent, the one here is from Sutton & Barto 2nd edition. We also can choose between a "pure" iteration calculating $v_{k+1}(s)$ from $v_{k}(s)$, but I will do the update in place to the same $v$ array, because that is easier and usually faster too.

Note each sum here is over the two possible $(s', r)$ results from each action.
Pass 1 in detail, iterating though states $[S_0, S_1]$:

Set max delta ($\Delta = 0$) to track our accuracy
$v(S_0) = \sum_{s',r} p(s',r|S_0,B)(r + \gamma v(s')) = 0.5(-5 + 0.1 v(S_1)) + 0.5 (-2 + 0.1 v(S_0)) = -3.5$
This sets max delta to 3.5
$v(S_1) = \sum_{s',r} p(s',r|S_1,R)(r + \gamma v(s')) = 0.6(-5 + 0.1 v(S_1)) + 0.4 (0 + 0.1 v(S_2)) = -3$
Max delta is still 3.5
The $v$ table is now $[-3.5, -3, 0]$

Pass 2, because $\Delta \gt 0.01$:

Reset $\Delta = 0$
$v(S_0) = 0.5(-5 + 0.1 v(S_1)) + 0.5 (-2 + 0.1 v(S_0)) = -3.825$
$\Delta = 0.325$
$v(S_1) = 0.6(-5 + 0.1 v(S_1)) + 0.4 (0 + 0.1 v(S_2)) = -3.180$
$\Delta = 0.325 \gt 0.01$
The $v$ table is now $[-3.825, -3.180, 0]$

Pass 3, $v = [-3.850, -3.191, 0]$
Pass 4, $v = [-3.852, -3.191, 0]$, this is small enough change that we'll say evaluation has converged.

Now we need to check whether the policy should be changed. For each non-terminal state we need to work through all possible actions and find the action with the highest expected return, $\pi(s) = \text{argmax}_a \sum_{s',r} p(s',r|s,a)(r + \gamma v(s'))$

For $S_0$:

Action B scores $0.5(-5 + 0.1 v(S_1)) + 0.5 (-2 + 0.1 v(S_0)) = -3.852$ (technically, we already know that because it is the current policy's action)
Action R scores $0.9(-2 + 0.1 v(S_0)) + 0.1 (0 + 0.1 v(S_2)) = -2.147$
R is the best action here, so change $\pi(S_0) = R$

For $S_1$:

Action B scores $0.8(-2 + 0.1 v(S_0)) + 0.2 (-5 + 0.1 v(S_1)) = -2.971$
Action R scores -3.191
B is the best action here, so change $\pi(S_1) = B$

Our policy has changed, so we go around both major stages again, starting with policy evaluation and the current $v = [-3.852, -3.191, 0]$, using the new policy $[R, B]$

I leave it up to you to do the second pass through evaluation and improvement. I suspect it will show that the new policy is optimal, and the policy will remain $[R, B]$ (although $[R,R]$ seems possible without doing the calculation). If the policy stays the same on a complete pass through evaluation and improvement, then you are done.


How can I preprocess multi-page image inputs in a theano/lasagne network?

First point is that convolutional neural networks would be incredibly expensive to train on images that large. The minimum size you could allow would be num_pixels_per_letter x num_letters_per_row_or_col.

It sounds like you want to achieve two tasks: OCR then document classification. Given that you have predominantly text in these documents, you would almost certainly be better served by using a more a traditional method for OCR. If you want or need to use NNs for the OCR piece, you will almost certainly need to use a sliding window. For inspiration on how you might do that, you could look to Recurrent Neural Networks that process images in sequences

Once you have the text there are again much simpler methods to deal with text than NNs, but they can achieve great results. To deal with different document lengths, you can use an RNN architecture like LSTM. Or you can use paragraph vectors, which do most of the heavy lifting for you and give you an N-dimensional representation of your text.


Why there is very large difference between cross validation scores?

Your $R^2$ scores indicate that a linear model does not describe your data well. On top of this, there seems to be a large variability in data. You could try the following:

If the linear model is supposed to describe the data, check for outliers. They might be responsible for the large variation across the CV folds.
Try reducing the number of features if there are many. The model might be fitting noise.
Introducing regularization (lasso or ridge regression) might make the model more robust. This should decrease the variability of the CV errors, but the $R^2$ scores will get even worse.


Wolfram recently released a free Wolfram Engine that can be called in Python (and other languages) so you can use its functions like TimelinePlot.

First get some dateIntervals by using the start and end dates of 2017 "TropicalStorm" hurricanes.

ec = EntityClass["TropicalStorm", "Hurricanes2017"];
timeIntervals = Interval /@ EntityValue[ec, {"StartDate", "EndDate"}];

timeIntervals contains Intervals of DateObjects. The first 3 of these are

timeIntervals[[;; 3]]

A basic TimelinePlot of timeIntervals gives

TimelinePlot[timeIntervals]

However, TimelinePlot has many options to customise it.  For example, take timeIntervals, GatherBy "Month", SortBy the first date of first interval in each set (see Part), and add an Automatic LineLegend using the "MonthShortName"s of each set as legend keys.

With[{
  byMonth =
   SortBy[#[[1, 1, 1]] &]@GatherBy[timeIntervals, DateValue[#[[1, 1]], "Month"] &]
  },
 TimelinePlot[
  byMonth,
  PlotLegends ->
   LineLegend[Automatic, DateValue[#[[1, 1, 1]], "MonthNameShort"] & /@ byMonth]
  ]]

Hope this helps.


Is it good practice to transform some variables and not others?

About the question whether to scale only a subset of features, I would tell you to do it over all the features (at least the continuous numeric ones) since the goal of data-scaling is to put these data on the same "reference scale" to be fairly compared.
Nevertheless, having mixed data types (continuous numerical, categorical...) for your classification problem looks more appropriate for scale-invariant algorithms like the ones based on decision trees. More precisely, you can have a look at XGBoost, where the author explains in this link that you do not actually have to re-scale your data.
Actually, in a recent real use case at my company, we tried re-scaling data VS not re-scaling it applying XGB, and we had better results with the second option.


Installing Orange Package with older Python Version?

The error message shows exactly what you need to do:

Orange requires Python>=3.4

You have to specify a Python>=3.4 version (with orange3 itself installed) while installing orange3-prototypes.
I'm not a Mac user, however, in Windows, Orange3 installer will automatically install a Python 3.4 if there's no compatible Python available.


Binary classification: is there a tradeoff between predicting class 1 vs 0?

To me, an algorithm is considered having an 'edge' for a given class if the accuracy on that class is higher than the proportion of that class in the dataset. E.g : if we have 20% Class A and 80% Class B, I would consider the model being 'performant' if accuracy on class A is >20%, or if accuracy on Class B is >80%.

Consider the following. Your 80% class is 0. You predict a constant 0 for all datapoints. You have 80% accuracy already with no modelling done and 100% accuracy on the majority class. If you fit a model to that data, you'll get an accuracy above 80% on the majority class and lesser accuracy on the minority class, and that's not a good performance which debunks your theory of a good metric.

The case above is from personal experience, seeing your results makes me think if your datapoints are really separable.
It is possible to have accuracies above the distribution marks if your features are good enough for model to use to separate each class from the other which in reality is feasable.

And again the choice of the metric in binary classification will always be a business problem, what is the cost of misclassifying a 1 as a 0 and a 0 as a 1? If the answer to these two questions is not the same, you should have some specific metrics for it.

  Hopefully the meta-ensemble will have a value above 0 for all classes for my made up metric.

It really depends on how diverse your base models are. But in general, meta-learners made out of weak learners outperform a single complex model. So a sort of a yes to your question.


SciKit Learn: Multilayer perceptron early stopping, restore best weights

To restore the best weights you would need a way to monitor a metric of your choosing and keep track of the best weights up to a given point. This has nothing to do with early stopping, you could return the best weights regardless of how many iterations the classifier has performed. Unfortunately, as far as I know, this functionality is not supported by scikit-learn.

In early stopping you assume that the best weights are those that the point you stopped your training. Early stopping isn't as much a convergence speed technique as it is a regularization, one.


Text classification and clustering with complete date imbalance

The question is not very clear but I will give a try anyways.

First of all please note that the major approach to tackle the analysis of co-authorship networks is not what you mentioned, but Network Science techniques as co-authorship is a Social Network.

After this necessary tip let's back to the question itself which is tackling the problem from NLP point of view.

  Unfortunately because my data is all of one class already I have an extreme data imbalance problem. How should go about fixing this?

You don't have a class imbalanced problem. Class imbalanced problem means that you have more than one class in which one or more classes are highly larger than others. But you have only one class! This is not a class imbalanced problem. You only need to define your question precisely.

  My set of papers and authors are assumed to share similar research interests. To build a classification algorithm do I need papers and authors that I know aren’t in the set?

Depends on what you want to do! Classification of what? You can set your question like classification of authors in the same field then you do not need additional papers. You extract text features according to different papers of all individuals and train your classifier:

Concatenate all sentences of the papers of one individual and call it data. the name of individual is the label. Of course if a paper has two authors, the text will be assigned to both of them.
Extract features from your text data. Unsupervised feature extraction does not care about the labels (TF-IDF is an example) and supervised methods do care.
Choose a classifier and start your learning phase as always.

but if you set your question like classification of authors from different field then sure you need data from other fields! In this case instead of each individual being a class, each field (including all authors in that field) becomes a single label.

  Should I look at the paper as a whole or should I look at the papers written by each author for classification and instead of looking at the papers as a group?

It's super difficult to understand what you mean but I guess I answered it above. Depends on your question! If you are going to classify authors inside a community, then all papers of that author is one text data object.

  I suppose I could randomly pick other papers and use those at the examples to one that don’t share research interests. Of course, picking paper at random there is the chance of false negatives.

Again difficult to understand but anyways. In this case you better gather data from another field. Randomly mine some papers from other fields and construct a new class of NOT THIS FIELD. Then go for a binary classification(Logistic Regression works well on TF-IDF features). It should be pointed that as your data is text you can not cover every text which is not in your main class so limit your question to a specific border in domain.

Hope I could help :)

PS: The question needs a surgery. I will edit it soon and please check if my edit is conceptually correct.


Does multi-head attention remove the need for self-attention?

Self-attention means X pays attention to X, as opposed to "normal" attention where X pays attention to Y.
Multi-head attention is as opposed to single-head attention. You can choose to use multi- or single-head attention equally for self-attention and for normal-attention.
Masking X and/or Y is a third independent aspect of a design.
In a Transformer encoder there is only self-attention and feed-forward networks (FFNs). Without the self-attention aspect the layers are reduced to just being FFNs. The FFNs operate only on one item, so without the self-attention they would never discover relationships between items. In a text application that means it would never discover the relationships between words. In a time series analysis it would never discover relationships across time.
The decoder uses both self-attention and normal attention. If there is no relationship across words/time in the output of the model, then maybe you could get rid of the self-attention part. But, if the transformer is a useful model for your data, that is unlikely to be the case.
So if you wanted a clear-cut yes/no answer (to if it removes the need for self-attention): in the case of the encoder, "no", and in the case of the decoder, "probably not".
UPDATE: Having now read the papers (SepFormer, and the two in comments, TSTNN, and DPTNet) I think the above still applies. Those models only use the encoder part of the transformer.
DPTNet and TSTNN change the first part of the FFN with a RNN or GRU respectively, so I suppose theoretically it can learn about other tokens from that; but the motivation for this is as an alternative to needing positional encodings, and I doubt it is worth the trade-off of no longer being able to process the data in parallel.


Program to fine-tune pre-trained word embeddings on my data set

While I am not aware of software specifically for tuning trained word embeddings, perhaps the following open source software might be helpful, if you can figure out what parts can be modified for the fine-tuning part (just an idea off the top of my head - I'm not too familiar with the details):

GloVe: Global Vectors for Word Representation (part of Stanford NLP Group software);
SENNA (its use for word embeddings is mentioned in this blog post);
Code on GitHub for Deep Learning-based word embeddings training;
Neural Probabilistic Language Model Toolkit, mentioned above (also Deep Learning-based).


SRGAN Generator Architecture: Why is it possible to do this elementwise sum?

It's possible by adding 0 as padding to the tensor. What is currently refered to "same padding" (in particular in Keras).


Implementing a neural network: low accuracy

It seems like you just got a bug @ your parse_data. (see my comment: replace X[idx][int(s[7])+j] = 1.0 # a6 to X[i][int(s[7])+j] = 1.0 # a6).

I altered the parameters. You will see that they have a big influence. I guess this problem is easily stuck on a local minima.

Could you run my code please (Only difference is that it is python 3, but should run on python 2)

This is the link of the data saved in data1.txt:
http://mlearn.ics.uci.edu/databases/monks-problems/monks-2.test

import numpy as np
    import os
    import matplotlib.pyplot as plt

abspath = os.path.abspath(__file__)
dname = os.path.dirname(abspath)
os.chdir(dname)

# paramaters
eta = 0.05
hidden_dim = 2
alpha = 0.8
reg_lambda = 0.0
epochs = 100

file = 'data1.txt'

input_dim = 17
output_dim = 1

# read and parse MONK data
def parse_data(text):
    with open(text) as f:
        content = f.readlines()
    t = np.zeros((len(content), output_dim))
    X = np.zeros((len(content), input_dim))
    for i,val in enumerate(content):
        s = val.split(" ")
        t[i] = float(s[1])
        # for idx in range(17):
        #   X[i][idx] = 0.0
        j = -1
        # |0 |1 |2 |3 |4 |5 |6 |7 |8 |9 |10|11|12|13|14|15|16
        # |a1|a1|a1|a2|a2|a2|a3|a3|a4|a4|a4|a5|a5|a5|a5|a6|a6
        X[i][int(s[2])+j] = 1.0 # a1
        j = 2
        X[i][int(s[3])+j] = 1.0 # a2
        j = 5
        X[i][int(s[4])+j] = 1.0 # a3
        j = 7
        X[i][int(s[5])+j] = 1.0 # a4
        j = 10
        X[i][int(s[6])+j] = 1.0 # a5
        j = 14
        X[i][int(s[7])+j] = 1.0 # a6

    return (X, t)

X, t = parse_data(file)

print(np.sum(t)/len(t))

def sigmoid(x):
    output = 1/(1+np.exp(-x))
    return output

def sigmoid_output_to_derivative(output):
    return output*(1-output)

# initialize weights
seed = 1
np.random.seed(seed)
W0 = np.random.normal(0, 1/input_dim, size = (input_dim,hidden_dim))
W1 = np.random.normal(0, 1/hidden_dim, size = (hidden_dim,output_dim))
b0 = np.zeros((1, hidden_dim))
b1 = np.zeros((1, output_dim))
dW0 = 0
dW1 = 0

error_list = []
for j in range(epochs):
    # forward propagation
    l0 = X
    l1 = sigmoid(np.dot(l0,W0) + b0)
    l2 = sigmoid(np.dot(l1,W1) + b1)

    # back propagation
    l2_error = l2 - t
    l2_delta = l2_error*sigmoid_output_to_derivative(l2)

    l1_error = l2_delta.dot(W1.T)
    l1_delta = l1_error*sigmoid_output_to_derivative(l1)

    error = np.mean(np.square(l2_error))
    error_list.append(error)
    if (j% 20) == 0:
        print("Training error after", j, "iterations:",error)

    dW1 = -eta * (l1.T.dot(l2_delta) + reg_lambda * W1) + alpha * dW1
    dW0 = -eta * (l0.T.dot(l1_delta) + reg_lambda * W0) + alpha * dW0
    db1 = -eta*(np.sum(l2_delta, axis=0))
    db0 = -eta*(np.sum(l1_delta, axis=0))

    W1 += dW1
    W0 += dW0
    b1 += db1
    b0 += db0

plt.plot(error_list)
plt.show()


Is it possible that I conduct a supervised machine learning like random forest after creating a target variable through clustering?

Generally speaking it is possible.
Somemore information would be helpful:
How do you plan to test/train? Do you want to use k-fold cross validation, or split the data? Are you going to use exactly the same variables as predictors in the Random Forest, that u used for the clusteranalysis? In this case you propably need to be careful with your interpretation since it is circular by nature.


Popular ways to combine trained models are schema's based on voting: predictions are treated as weighted votes, the class with the majority of votes is selected as ultimate prediction, and/or stacking: predictions are treated as features in a newly trained model.

Depending on the domain at hand one of this approaches might help you. Stacking (depending on the metalearner of choice) introduces more freedom and might introduce a complexity that is not warranted by the concept you are searching for, while voting won't help much when identifying subdomain fitted for perticular problems in your search space.

Also when combining multiple classifiers it might be worth your while to try out some fundamentally different approaches.


Dataset : Handling rows where some column value is not applicable

You can use a model that can handle NaN values. Most commonly this would be something based on decision trees. I recommend LGBM and XGBoost.

Decision trees also work well when these kind of values are set outside of the range for this feature, i.e. setting it to something like -999. This alternative is better if you want to use models from scikit-learn since NaNs are not supported there.


Demand Forecasting for Multiple Products Across Thousands of Stores

I was also in a similiar situation. I had a dataset with number of stores and product categories and I needed to predict weekly sales for each category. You may want to check gradient boosting alghorithm and random forest which give promising results in terms of forecast accuracy. Also, regardless of the method, the best thing to do is to create dummy variables as creating single time series is very time consuming.


The paper you've linked describes a method for predicting an ordinal dependent variable, but it sounds like you're looking for a way to use ordinal independent variables.  That part makes up just the first part of the paper's method, and is relatively easier.  It's somewhat analogous with one-hot encoding: you have indicators $x\leq \alpha$ for each level $\alpha$ (except the maximum one).

For example, if the levels are Low, Medium, High, your new variables will be "Low" and "Low or Medium" (skipping "Low or Medium or High" since that's always true).


Interpret clustering results after variable transformation

Very interesting!
What you did to your data is simply a feature mapping/transformation. So how this affects the clustering results?

Clustering is not a clearly defined problem but at least we know something about it: It's about internal similarities (patterns) so these similarities should be maintained through the feature transformation. In your example if you found clusters in transformed space, it shows that you have had clusters in original space as well. You just couldn't see them according to the algorithm you used in that space!

For instance if you use Kernelized versions of algorithms you easily find that what they do is nothing but what you did as transformation. They first use a kernel to map the data into new space and then use the algorithm in that space (of course with a bit of theoretical differences/constraints).

To summarize, no transformation produces a fake pattern in the data. In worst case it vanishes the original pattern and in the best case it reveals the pattern which was not visible originally (which is your case).

I mentioned Fake Pattern above so let me say a bit more on it. I think there is a fundamental issue concerning your question:

You assume that there is a Right clustering that you got after transformation. Actually there is no right clustering!

We do not have fake pattern! If there is a pattern in a feature space, then that is true! i.e. you found an interesting representation of your data. If it does not match with the labels then either the data is very noisy or wrong features have been chosen to represent classes (maybe more reasons. just these two came to my mind now). If there is no label (your case) be sure there is a correlation between features of those cluster members.


Is Data Science the Same as Data Mining?

@statsRus starts to lay the groundwork for your answer in another question What characterises the difference between data science and statistics?:

Data collection: web scraping and online surveys
Data manipulation: recoding messy data and extracting meaning from linguistic and social network data
Data scale: working with extremely large data sets
Data mining: finding patterns in large, complex data sets, with an emphasis on algorithmic techniques
Data communication: helping turn "machine-readable" data into "human-readable" information via visualization

Definition
data-mining can be seen as one item (or set of skills and applications) in the toolkit of the data scientist.  I like how he separates the definition of mining from collection in a sort of trade-specific jargon.
However, I think that data-mining would be synonymous with data-collection in a US-English colloquial definition.
As to where to go to become proficient?  I think that question is too broad as it is currently stated and would receive answers that are primarily opinion based.  Perhaps if you could refine your question, it might be easier to see what you are asking.


What is the difference between model hyperparameters and model parameters?

Hyperparameters and parameters are often used interchangeably but there is a difference between them. You call something a 'hyperparameter' if it cannot be learned within the estimator directly. However, 'parameters' is  more general term. When you say 'passing the parameters to the model', it generally means a combination of hyperparameters along with some other parameters that are not directly related to your estimator but are required for your model.

For example, suppose your are building an SVM classifier in sklearn:

from sklearn import svm
X = [[0, 0], [1, 1]]
y = [0, 1]
clf = svm.SVC(C =0.01, kernel ='rbf', random_state=33)
clf.fit(X, y)

In the above code an instance of SVM is your estimator for your model for which the hyperparameters, in this case, are C and kernel. But your model has another parameter which is not a hyperparameter and that is random_state.