First of all, I think you are confused with pretrained and finetuned.

BERT is pretrained on a lot of text data. By using this pretrained BERT, you have a model that already have knowledge about text.

BERT can then be finetuned on specific dataset, where BERT learn specific knowledge related to the dataset. That's why a finetuned BERT is bad on other datasets : the knowledge does not apply.

You have a custom dataset. So you don't want to use a finetuned model, but you want to still use a pretrained model (to have access to the general knowledge of BERT).

If you use a BERT model from scratch (not pretrained), you're basically creating an empty model, randomly initialized : it's useless.

Now that we have clarified this, let's see your problem.

What you are doing is calling the constructor of BertModel with the arguments of the forward pass. You first need to create the model, and then use the forward pass :

from transformers import BertModel, BertConfig
config = BertConfig(...)    # Here your parameters to initialize your BERT as you want, such as number of heads, etc...
model = BertModel(config)
hidden_reps, cls_head = model(token_ids, attention_mask=attn_mask, token_type_ids=seg_ids)

BUT, as I explained in the first part, you should not do this : it's completely useless and your results are going to be random. You want to use a pretrained BERT, to have some meaningful results :

from transformers import BertModel
model = BertModel.from_pretrained('bert-base-uncased')
hidden_reps, cls_head = model(token_ids, attention_mask=attn_mask, token_type_ids=seg_ids)

Note : Your results can be improved further if you finetune your BERT model on your custom dataset.

Final point : Don't reinvent the wheel

People already tried to use BERT for word similarity. Instead of implementing this from scratch, using only a pretrained model, potentially adding bug to your own implementation, just use some already existing code !

I'm not sure about NER, but for word similarity, you can take a look at BERT Score. Their model is already finetuned for word similarity, so it's likely that you will get higher score using this approach, and also much easier path to take.


Get the Polynomial Equation with Two Variables in Python

So you want to fit 6-th degree polynomial in python to your data?

Main thing you should note is that it will be still linear regression, its juts that predictors are polynomial (most important is that your weights are still linear (betas in lin.regression))

You can transform your features to polynomial using this sklearn module and then use these features in your linear regression model.

> from sklearn.preprocessing import PolynomialFeatures from sklearn
> import linear_model poly = PolynomialFeatures(degree=6)
> poly_variables = poly.fit_transform(variables_length_wand_rate)
> poly_var_train, poly_var_test, res_train, res_test = train_test_split(poly_variables, results,test_size = 0.3, random_state=4)
> regression = linear_model.LinearRegression()
> model = regression.fit(poly_var_train,res_train)
> score = model.score(poly_var_test, res_test)


sklearn RandomForestRegressor oob_score_ looks wrong?

You are using Random Forest Regressor, but since you have a binary response where a passenger survives or not, the Random Forest Classifier should be used.  If this doesn't fix the issue, please send the link to the tutorial.


Multiclass ROC Curve using DecisionTreeClassifier

The combination of class_weight and indices_test only have 10 data points results in Class label 2 not present.
Since iris dataset is perfectly balanced there is no reason to specify class_weight. Additionally, scikit-learn has train_test_split which automatically makes a split that maintains equal proportions of every class in both training and testing.


Why does discriminiator accuracy falls to 0%, and is there a fix around this?

I figured out my answer a while back. SO here it goes...

Ensure that your training data is normalized. This was what causing the discriminator's accuracy to fall.
You need to decrease your dropouts as it can cause heavy bias if done ineffectively.
Remove any unnecessary layers in your discriminator as they can cause overfitting.
Use batch normalization.
Introduce dropouts in your generator.


Error with pandas dataframe (needs to be 1-dimensional)

It seems to me that this question is better off on stackoverflow.
Nevertheless, X_cal gets generated from X_train and X_train from valid. But this is an atleast 2-dimensional dataframe with new_host and sequence. Like the error says you should only input data that is 1-dimensional.


What do we visualize in showing a VAE latent space?

When people make 2D scatter plots what do they actually plot?

First case: when we want to get an embedding for specific inputs:

We either

Feed a hand-written character "9" to VAE, receive a 20 dimensional "mean" vector, then embed it into 2D dimension using t-SNE, and finally plot it with label "9" or the actual image next to the point, or
We use 2D mean vectors and plot directly without using t-SNE.

Note that "variance" vector is not used for embedding. However, its size can be used to show the degree of uncertainty. For example a clear "9" would have less variance than a hastily written "9" which is close to "0".

Second case: when we want to plot a random sample of z space:

We select random values of z, which effectively bypasses sampling from mean and variance vectors,

sample = Variable(torch.randn(64, ZDIMS))

Then, we feed those z's to decoder, and receive images,

sample = model.decode(sample).cpu()

Finally, we embed z's into 2D dimension using t-SNE, or use 2D dimension for z and plot directly.

Here is an illustration for the second case (drawn by the one and only paint):

As you see, the mean and variances are completely bypassed, we directly give the random z's to decoder.

The referenced article says  the same thing, but less obvious:

  Below you see 64 random samples of a two-dimensional latent space of MNIST digits that I made with the example below, with ZDIMS=2

and

  VAE has learned a 20-dimensional normal distribution for any input digit

ZDIMS = 20
...
self.fc21 = nn.Linear(400, ZDIMS)  # mu layer
self.fc22 = nn.Linear(400, ZDIMS)  # logvariance layer

which means it only refers to the z vector, bypassing mean and variance vectors.


Andrew Ngs Class - Why Did He Change up the Cost Function?

The difference is the "original" is the derivative of a cost function and "new" is the result of substituting in an actual cost function.

When the derivative of a function raised to a power, the exponent becomes a constant. In this case, the derivative of the squared loss function becomes a constant 2, and the 2 in the numerator then cancels with the 2 in the denominator. The result is just dividing by $m$.


How to run two different models in single frame?

Use ray or Numba for Parallel Computation. Question is "are not you able to do transfer learning to detect both in one single model"?


What exactly is A dataset? is a database contain more than 2 tables counted as 1 or 2 datasets?

A dataset might refer to any collection of information about a specific topic. Namely, a file, an excel file, a .txt file, etc.

Usually, we use datasets that are saved in a table format (like .csv files)
a dataset in this form is exactly like a table having columns (fields) for the name of its features and rows that are the records (samples) and an index for each row.
A problem might be solved using multiple datasets (tables) but each dataset is referred to a single table (data frame)

Whatever the format is, for utilizing, we turn the dataset to a table (dataframe)


Similarity Measure between two feature vectors

So, thank you for clarifying the question. Just to confirm that the question is asking how to set an appropriate threshold for face feature vectors (represented a a and b, for example).
What I would recommend is to look at either cosine similarity or euclidean distance, which you have implemented. From here, I would then look the distribution fo the similarity metric over all face vector pairings. Here, this can give you an idea of general level of similarity over all face vector pairings (e.g. if most of the distribution is skewed towards lowers facial similarities, then this could indicate overall lack of similarity between face vector pairings).
If the distribution is skewed to higher similarity values, then you could take, say, the 90th percentile over this distribution as the threshold for determining whether a == b.
Another idea would be to use the idea of softmax in the following way. This method would allow for others to replicate your methodology precisely:
For each face $f_i \in F$, you compute the softmax over the face vector pairings between $f_i$ and $f_j \in F, i \neq j$. Then, from this, you select the index $j$ which has the highest value, which can then be assumed to be such that $f_i$ == $f_j$ (a == b)


You can make sure that command is executed for every terminal (meaning Anaconda will be found) by adding it to your user's bash profile.

Open a terminal and follow these steps:

open the terminal profile: gedit ~/.bashrc
at the end of the file, add: export PATH=~/anaconda3/bin:$PATH
save the file (control+s) and close it
load the changed profile: source ~/.bashrc

Now this terminal window (and any new ones) should find Anaconda! See which version of Python is now the default, using which -a python. First in the list should be something like /home/username/anaconda3/bin/python

The Anaconda setup/installation usually asks you if you want to prepend Anaconda to start of your path, so whoever installed it must have said no or skipped that step.


Keras weird loss and metrics during train

This is because the calculation of the loss and accuracy are done before the first weight update (i.e. the model with the initialized parameters). After the loss is calculated the first time the loss is used to backpropagate the error throughout the network and to update the parameters. The loss is not calculated again during training (since this would just be an added training cost). The adjusted parameters are then used to calculate the loss for the next batch (training or validation). It would therefore make more sense to compare the training statistics of the second batch with the validation statistics of the first batch.


Well, when k = n, the obvious global minimum occurs where $x_n = \mu_k$ for all cluster centers. This falls out from your derivate because $r_{nk} = 1$ iff $n = k$.


How does skewed data affect deep neural networks?

Strictly theoretically it makes no difference on DNN, I answered it today here and I said:
Here is why: We already know mathematically that NN can approximate any function. So lets say that we have Input X. X is highly correlated, than we can apply a decorrelation technique out there. Main Thing is, you get X` that has different numerical representation. Most likely more difficult for NN to learn to map to Outputs y. But still in Theory you can Change the architecure, Train for longer and you can still get the same Approximation, i.e. Accuracy.
Now, Theory and Praxis are same in Theory but different in Praxis, and I suspect that this Adjustments of Architecture etc will be much more costly in reality depending on the dataset.
BUT I want to add another point of view: Convergence speed. Strickly theoretically you dont even need [batch normalization] for performance (you can just adjust weights and bias and you should get same results) but we know that making this transformation has big benefits for NN
To conclude for you: Yeah, I had experience where it made difference, and where it didnt. You cant expect theoretical results that say skewed is bad


It seems like you have a classical bayesian problem. You have some sort of prior distribution, a distribution over years of birth, your prior distribution is bimodal with peaks at the two years,  you can probably use a convolution of two normal distributions to model this variable. Then have it spit out a posterior distribution after you feed in some data.

The real problem that I have with this analysis is it seems your features aren't particularly good. It is true these vars might have information about birth year, for example for the 20th century the average age of first marriage has steadily been increasing. But I suspect that the signal is going to be fairly weak. Essentially, if I tell you that I got married at age 24, had my first child at 26, and that my older brother is 3 years older than me and my younger sister is 2 years younger than me, can you tell me in what year was I born, 1956 or 1989?

I suspect that without additional data this information that I provided would be completely useless, mostly because it is a very noisy signal. That information could apply equally to someone born in 1956 or 1989. It isn't very helpful.

Essentially, what I am saying is that when you update your prior, it isn't going to change very much. (Your posterior distribution would look very similar to the prior distribution.) Instead of doing some mustache twirling over what is the right algorithm to crack this problem, I think a much more fruitful exercise would be to think up some better features.


I also thought about the very same question recently and I think I might have a possible explanation.

Since we need to calculate distances between k-nearest neighbors, we have to provide some synthetic value which denotes the difference between nominal features. As a matter of fact, it could be any value. For example, imagine you have your nominal features one-hot encoded. In this case the difference between two different nominal features would be 1.

However, you would also like to keep the calculated distances more or less on the same scale as if you would have by using only continuous features. I believe that using median of standard deviations of continuous features helps to achieve exactly that and is the reason why it was chosen as a measure of difference between nominal features.


You are effectively trying to classify images. Convolutional neural networks are much better at this task. And Keras is an easy to use library for neural networks.

Check tutorials such as: https://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-python-keras/

and

https://www.pyimagesearch.com/2017/12/11/image-classification-with-keras-and-deep-learning/


How to do Feature Scaling for these ranges [0,1] and [-1,1]?

What you said is right, the above equation is for normalizing the data with-in the range of [0,1]

Now, we can generalize using the below equation

To normalize in $[-1,1]$ you can use:

$$
x'' = 2\frac{x - \min{x}}{\max{x} - \min{x}} - 1
$$

In general, you can always get a new variable $x'''$ in $[a,b]$:

$$
x''' = (b-a)\frac{x - \min{x}}{\max{x} - \min{x}} + a
$$


Why should we use (or not) dropout on the input layer?

Why not, because the risks outweigh the benefits.

It might work in images, where loss of pixels / voxels could be somewhat "reconstructed" by other layers, also pixel/voxel loss is somewhat common in image processing. But if you use it on other problems like NLP or tabular data, dropping columns of data randomly won't improve performance and you will risk losing important information randomly. It's like running a lottery to throw away data and hope other layers can reconstruct the data.

In the case of NLP you might be throwing away important key words or in the case of tabular data, you might be throwing away data that cannot be replicated anyway else, like gens in a genome, numeric or factors in a table, etc.

I guess this could work if you are using an input-dropout-hidden layer model as you described as part of a larger ensemble though, so that the model focuses on other, less evident features of the data. However, in theory, this is already achieved by dropout after hidden layers.


Is multi-class classification the same with categorical classification?

Yeah, the answer is as simple as :

Binary Classification means only 2 targets to find.
Multilabel / Multiclass / Multinomial Classification means more than 2 targets to find.

In Supervised Learning, you will have :

Classification (class of 0, 1 etc..)
Regression (continuous values 84.2, 105.8 etc..)

Categorical and class means the same and are the opposite of Regression which are continuous variables

You can find example on https://scikit-learn.org/stable/supervised_learning.html#supervised-learning


Is there an algorithm that imputes missing values based on n nearest columns? (KNN hybrid)

You can extend scikit-learn's sklearn.impute.KNNImputer. The metric parameter can accept the custom function you described.


Optimizing co-occurrence matrix computation

From easiest to hardest:

Try running it in pypy or numba
Find a faster implementation. Unfortunately I can not recommend one.
Parallelize the loop over the documents. Not so hard since your vocabulary is precomputed. (Even if it weren't you could get away with it using the hashing trick.) Combine this with the first bullet.
Rewrite the inner loop in Cython.
Rewrite the whole thing in a faster language like C++ or Scala.


0.8 is the threshold value meaning you want to exclude columns(features) with at least 80% of values is 0 or 1: sort of unbalanced dummies. This is only applied well to dummies, but not necessarily general numerical values.
For numerical, you can test RFECV function in feature_selection package from sciki-learn, the syntax as follows:

feature_folds = ms.KFold(n_splits=10, shuffle = True) # cross-validation folds

selector = fs.RFECV(estimator = your_model, cv = feature_folds,
                      scoring = 'roc_auc')

selector = selector.fit(your_features, your_labels)

selector.support_


K-fold cross validation only helps you giving an estimation of the error you are going to make with a given model. K-fold can be used in order to tune hyperparameters: we have a model that depends on some hyperparameters, we compute the K-fold validation error using different values of the hyperparameters, and we take the model with the lowest cross-validation error. Finally, we train the model with the chosen hyperparameters with all the training data and we build a classifier based on that model.


Search the Number of occurrences of the particular words in data using Pandas.

Jurgy's answer should work. However, based on comments above, you can simply do like this,

df['yourcolumn_name_good_bad'].value_counts().plot(kind='bar')

Otherwise, if you want to use seaborn library and create a plot follow this.


Twitter uses Storm for real-time processing of data.  Problems can happen with real-time data.  Systems might go down.  Data might be inadvertently processed twice.  Network connections can be lost.  A lot can happen in a real-time system.

They use hadoop to reliably process historical data.  I don't know specifics, but for instance, getting solid information from aggregated logs is probably more reliable than attaching to the stream.

If they simply relied on Storm for everything - Storm would have problems due to the nature of providing real-time information at scale.  If they relied on hadoop for everything, there's a good deal of latency involved.  Combining the two with Summingbird is the next logical step.


How to treat a categorical variable which can have multiple values in regression?

In this kind of case, I would look into mean/median target encoding. Basically, you take the average/median target/response value of y (based on your training set only) per each category in your feature. This new statistic per group is now your encoding. For your validation/testing data, you simply join back the "learned" statistics from your training set as a lookup table.

It tends to work extremely well for features with high cardinality. Also, suppose you do use one-hot encoding. How sparse are some of the resulting columns? If you have multiple sparse columns (can be detected easily by looking for low variance features or by simply counting how many 0's to 1's there are) then likely, these categories can be grouped.

Finally, there are algorithms that have specifically been made to handle high cardinality situations internally; in fact, CatBoost (whose motivation in its creation was in part to create a gradient boosting framework that directly supported categorical variables) to my belief heavily popularized mean/median target encoding (what I explain above) and gives you many different options with respect to encoding categorical features.


What can we understand from max-activation generated images?

Those are the activation maps of the learned features.

In the case of this specific model, the filters learn that the "fishy" parts of the image:

Head-ish and dorsal fin-ish sections that define convex hulls
Scale-ish textures insides the convex hulls and surrounded by water-ish textures outside
Orange-ish and gray-ish colors insides the convex hulls and blue-ish colors outside
The head-ish section tend not to be in the corners of an image (since people frame pictures and people think faces are interesting)


Why adding combinations of features would increase performance of linear SVM?

Multiplication is not a linear operation. Your linear SVM constructs a (hyper-)plane
$$
w_0 = w_1 x_1 + w_2 x_2
$$
for some weights $w_0, w_1, w_2.$

By introducing the AND-feature, you add another dimension:
$$
w_0 = w_1 x_1 + w_2 x_2 + w_3 x_1 x_2.
$$

It might well be that your two-dimensional data set is not linearly separable, but the three-dimensional data set is.

A small addition: Would adding the OR-feature increase performance even further? No, because it is a linear combination of the other three features: $x \vee y = x + y - (x \wedge y)$ where $\vee$ is OR and $\wedge$ is AND.


How to setup my Ubuntu PC for these Deep Learning frameworks?

An easy way to run different versions of multiple frameworks alongside each other is to use Docker. With Docker, you can create a new "Container", kind of a light-weight virtual machine, for each project. Docker handles all hardware abstraction and separates the different Frameworks.

Note: To use the GPU within a Docker container, you need to install Nvidia-Docker, as a "normal" Docker installation can not share the GPU with a container.

In your example, you could have a Docker container for Digits, and a separate one for TensorFlow - then the two frameworks will be completely isolated from each other.

This approach comes with multiple advantages:

You can run the same container on a different machine without installing e.g. TensorFlow - all you need is (Nvidia-)Docker.
You can easily use different (TensorFlow) versions for different projects, as each project gets it's own container which is separated from the system.
Containers can use different CUDA versions, e.g. CUDA 8 for Digits and CUDA 9 for TensorFlow.
There are pre-built containers for almost all existing deep learning frameworks. Nvidia even provides some GPU-optimized containers with their free NGC (Nvidia GPU Cloud) service.


Numpy ndarray holding string with unknown data type

In Python 3, you shouldn't really specify the np.string_ dtype, as it is left there for backwards compatibility with Python 2. The S type you see using np.dtype is a map to the bytes_ type, a zero-terminated string buffer, which shouldn't be used.
The S just means string and the number gives the number of bytes.
In [1]: s = ""        # start with an empty string

In [2]: for i in range(5):    # make the string larger
            s += str(i)
            a = np.array([s], dtype=np.string_)
            print(f"{a}\t{a.dtype}")

    [b'0']  |S1
    [b'01'] |S2
    [b'012']    |S3
    [b'0123']   |S4
    [b'01234']  |S5
    [b'012345'] |S6

For python 3 you should instead use np.unicode:
In [1]: a = np.array(["hello", "world"], dtype=np.unicode)

In [2]: type(a)
Out[2]: numpy.ndarray

In [3]: b.dtype
Out[3]: dtype('<U5')

< means little-endian
U means a unicode string
5 relates to the number of bytes used to hold the string. If you had really long strings, that number would then increase.

Have a look at this documentation, which states:

Note on string types:
For backward compatibility with Python 2 the S and a typestrings remain zero-terminated bytes and np.string_ continues to map to np.bytes_. To use actual strings in Python 3 use U or np.unicode_.


Is it possible to use Generative Adversarial Networks (GANs) for text classification?

(without philosophically adding some layers on top of GANS) answer is NO.

You can expect to generate images, news etc (generative adverserial network) but not to actually perform classification


Why MA model order is from acf but not pacf

First, I feel like crossvalidated SE might be a better fit here. Regardless, the answer comes from some mathematical facts of both AR and MA models. I don't really understand what your link is trying to suggest in its explanation.

For AR(p) models, it is a fact that the partial autocorrelation function (PACF) cuts off (abruptly hits 0) after lag p. Thus, analyzing a PACF graph can be useful since if the process follows an AR one, we can possibly identify the order if we see that the pacf graph shows significant partial autocorrelation at the first few lags and then cuts off to insignificant (i.e. not enough evidence to suggest non zero lags) lags after.

In a similar way, for MA(q) models the autocorrelation function (ACF), NOT the pacf, cuts off (abruptly hits 0) after lag p. Thus, analyzing a graph of the ACF can be useful for identifying the order of an MA(q) model for reasons very similar to the AR(p) model and the pacf graph.

For the AR(p) model, we can show the pacf for lags k > p cut to zero by solving the Yule Walker system of equations (if my memory serves correct). For the MA(q) model, we can make use of the backshift operator and/or spectral analysis (this is how I learned it, anyway) to find the autocorrelation function for lags k > q (and again, see that it is zero for all lags after q).

This methodology is part of the "Box-Jenkins" methodology, which too be honest is a bit too subjective for my tastes in the sense that we are effectively eyeballing a graph. Rarely is the case (in my own studies, anyway) do we find a time series that perfectly exhibits these cutoffs and so it becomes tricky when picking orders. I prefer instead (and I believe that this is becoming much more popular) to select orders based off information criteria, namely, AIC/AICc/BIC instead due to less subjectivity.

One final note; there are other assumptions for these models that also make up the Box-Jenkins method. Namely, ensuring the series is stationary first (no trend or seasonality, which hopefully can be addressed using differencing/seasonal differencing though not always) is an important assumption for all AR/MA/ARMA (and with differencing, ARIMA) models.


Buyer classification is used to categorize users who purchase groups of items. Buyers are categorized in order to be targeted for advertising. When users buy similar items, they are more likely to buy similar items in the future. This is useful information when pricing items on a website.

A clear example, a website selling nutritional supplements may want to target different buyers by category. Men in their 20s are unlikely to purchase menopause supplements, and women in their 50s are also unlikely to buy creatine. So, by analyzing user purchase history and categorizing types of buyers, the site can send intelligent promotions - males in their 20s get ads for creatine while women in their 50s get ads for menopause supplements.

Also, if I want to run a sale to attract customers, I don't want to offer lower prices on items which are often purchased together. I'd rathe price one low to attract the buyer, and then hope they buy the complimentary item at full price. Categorizing buyers also helps with this problem.

You may want to read up on shopping cart analysis, which is not a new problem. Department stores have been analyzing shopping carts and classifying buyers long before online shopping was popularized. That's why you have to use membership cards to get the 'special' prices.

Fine tuning these details can increase revenue substantially.


You are right that the output of your encoder neural network is not a random variable, this is the mean $\mu$ and the standard deviation $\sigma$ of your latent random variable. For example if the output of your encoder is $\sigma = 1$ and $\mu = 0.5$, your latent random variable will be normal with mean 0.5 and standard deviation 1. Theb, you can calculate the KL divergence between your random variable and the prior for the latent random variable which is $\mathcal{N}(0, 1)$ in many tutorials.


When we do grouping of the data, Pig creates a new key named "group" and puts all the tuple matching that key into a bag and associates the bag with the key. So after the group operation schema of the grouped data will be something like

raw = load '$input' using PigStorage('\u0001') as (id1:int, name:chararray);
groupdata1 = group raw by (id1,name);
describe groupdata1;
{group: (id1: int,name: chararray),raw: {(id1: int,name: chararray)}}

The 'GROUP' in line 4 you are trying to access is one of the attribute of the schema from the last statement. These attribute name are case sensitive. It will produce the error saying it doesn't exist in the schema. So you need to use 'group' only to access it.


I know of a good few NLP regression tasks, but these are usually not about a predicting an extrinsic value from the semantic description, but rather predict some numerical property of the text itself:

MT quality estimation: predict the quality of a translated sentence, usually as a score. Data available at the WMT QE Shared Task series. There are probably other similar datasets about predicting the quality of a sentence in general.
Author profiling: predicting the age of the author of a text. Some datasets exist in particular through the PAN workshop series.
Predicting a rating based on a comment - there's data everywhere on social media, Amazon etc. You could even use StackExchange posts and predict votes :)

I'm not aware of any data similar to your example (btw it would be a very challenging task). I know that there are some works on detecting fraud from the text of an insurance claim, so maybe there are datasets that you could use in this area.


Which feature selection technique to pickup(Boruta vs RFE vs step selection)

There is a tradeoff between selecting features and precision. Fewer features probably have less precision (predictive power).

Select the features that make sense for your problem taking into consideration the trade-off of information vs performance.  The fewer features the model sees the predictive power that it has.


Out of curiosity, have you made any effort to quantify the accuracy of spaCy?  The way that it performs lemmatization leverages the syntactic similarity hypothesis, and as such 100% accuracy is an unrealistic expectation.

With that said, if there is a particular subset of words that are essential for you correctly classify, and you're willing to create a labeled training set, spaCy allows you to train custom models. I believe that you are able to train these models based on the existing language models, and so the amount of training data need not be prolific.


my question is what the stopping criteria of train CNN to extract features?

Generally speaking, the best features are the ones that, when classified, should have the best performance. So train your model, while monitoring a metric of your choice on a validation set. Stop the training phase when that metric stops improving.

  What the difference between extract features after train 50 epoch and 100?

Assuming no overfitting has occurred between epochs 50 and 100, the features of the latter will be more sophisticated and better equipped for classification on the given dataset.

If the model has started to overfit between these two epochs, then the latter model will have begun to memorize the training set. The features of this model will have begun to extract features tailored to the training set images.

  How to choose the best layer for extraction?

You should get the highest-level features available from the CNN. The most usual case is taking the previous layer of the first fully connected one. However, if that layer is a pooling one, you might want to take the previous one (which contains more information).

  Are it depend on the accuracy of model?

This comes back to the first question. If you think that accuracy is a good enough metric to represent your model's performance, then yes. There are many cases, however, where this is not the case (e.g. imbalanced dataset). Whichever metric you choose, it should be measured on the validation set.


First, let me give you an intuitive explanation. When you drop a pebble in water, you see the ripples being formed. Imagine that in reverse. All those ripples comes together at the point from which they started.
Node embeddings are like that. You take the information of neighbourhood nodes and combine it with the information in the original node. The art lies in how you combine the neighbourhood nodes and until which k-hop neighbours you do it. 1-hop means direct neighbours, 2-hop means neighbour of neighbours and so on.
In context of GraphSAGE, the combination of information is done using a simple feedforward neural network and the neighbourhood is incorporated iteratively. See link for further understanding.


LSTM for text with different sentences size, but same input-output sizes

The problem is, the LSTM results, after excluding the tags that correspond to the padding portion give outputs of different size than the input.

How do you get the "LSTM results"? Are the "LSTM results" extracted making a prediction for each word in the original sentence?

The LSTM is not a problem to do what you want to do. It might be an issue of your implementation. If you put a snippet of code maybe you could get some help.


The issue is due to your lamda function with the tokenizer key word argument.

>>> from sklearn.feature_extraction.text import TfidfVectorizer
>>> from joblib import dump
>>> t = TfidfVectorizer()
>>> dump(t, 'tfidf.pkl')
['tfidf.pkl']

No issues. Now let's pass a lambda function to tokenizer

>>> t = TfidfVectorizer(tokenizer=lambda x: x.split())
>>> dump(t, 'tfidf.pkl')

Which throws the following error:

  _pickle.PicklingError: Can't pickle  at 0x100e18b90>: it's not found as main.

To get around this, create a function to split the text:

>>> def text_splitter(text):
...     return text.split()

Try dumping again:

>>> t = TfidfVectorizer(tokenizer=text_splitter)
>>> dump(t, 'tfidf.pkl')
['tfidf.pkl']

Now you can you save the vectorizer.


Don't understand classification equation for hard margin SVM

Very good question. Its worth understanding following concepts about SVM to understand about your point

SVM get trained from the training dataset in such a way that for every instance Xi in dataset
W.Xi + b <= -1 if Xi belongs to -ve class
W.Xi + b >=1 if Xi belongs to +ve class
This also means that for edge instances of +ve and -ve classes the value will be 1 and -1 respectively. This is what it means by making the street as wide as possible.
Above condition is not necessarily true with new unseen instance 'U' because it hasn't been seen by SVM algorithm yet so it can also be inside the street separating the classes
Above equation also implies that the edge instances are separated by 2 units (whatever the unit is). To make the street gap 2 unit, W not give projection of Xi perpendicular to the street but it also scales it to make the gap 2 units. This means W is not necessarily a unit vector.
In other word we are not comparing the component of U in direction perpendicular to the street but scaled version of it so that the street is always 2 units wide

This is intuition based explanation of SVM but mathematically also above equation is the constrain of the optimization problem needed to solved for SVM


Relationships between groups of features against independent variables

One example would be to use ANCOVA (analysis of covariance) technique to capture association between continuous and categorical variables.
See here for more details.


Object Detection scratch implementation or high level API?

As you are looking to implement already known networks with a framework such as pytorch/tensorflow, I would definitely suggest reading into this post : https://towardsdatascience.com/object-detection-and-tracking-in-pytorch-b3cf1a696a98

Given that you wanted to implement this from scratch, its important you understand what the following:

Bounding boxes in images
Loss function - Intersection over union
Network architecture - You can use nettron to learn about the network architecture once you have written it.
Visualize the learning of your model after every 'x' epochs, to gain an intuituve understanding of what your network is trying to learn given the training dataset.


How many features do I select when doing feature selection for regression algorithms? Is R2 and RMSE good measures of success for overfitting?

You have overfitting when your model corresponds too closely to the training data and may therefore fail to fit additional data or predict future observations reliably. Basically, when the performance on training (or validation) set are much more better than on test set.
You have the opposite case: performance on test set much better than validation set.
This can happen if the two sets do not come from the same distribution (how did you divide train/validation/test set?). In this case, the data of the test set could be much easier to predict.
Another possibility is that the size of the test set is too small.
My suggestion is: shuffle your dataset; divide in 70% training and 30% test set. Do cross validation on training set. Compute the R2 on both sets.


Image data augmentation with this function

Data augmentation is a technique for increasing the dataset size by performing certain operations over images such as translation, rotation, hue shift etc and increase the training dataset size. It could be applied to any kind of image dataset.

The above mentioned function seems to be applying shift with images of shape 8x8 and as far as the shape of the image is of this dimension, it will definitely work.

Refer to this paper for further reference.


How to deal with item belonging to more than one category

There are multiple ways you can do it. The first two options you propose would be valid but I'm not sure if the log-loss allows you to use probabilities as target as opposed to just binary labels, but if that is the case they will work for what you want. I have another suggestion. Let's say there are at any point n possible moves available (In case of a 2x2x2 there are three dimensions and in every dimension 4 moves I think, so that would be 12 options) you could train 12 classifiers or in the case of a neural networks just twelve sigmoids in the output layer. Then you can represent every move as a probability, represent the training targets as a vector of 1s and 0s with 1 as being an optimal move and then during classification pick the one with the highest probability.

An extension of your first option would be to add all the optimal moves as state. Say move 2 and 4 are optimal you could just add the same state twice with different labels.

EDIT: That said, I think your second option is the best one if the math works out (I'm uncertain about this and don't have time to figure it out right now) but the others will work as well.

EDIT2: According to Neil Slater the second option will work if it's implemented properly and I think that is the cleanest solution so I would go for that if possible, otherwise I would go for either go for the multiple samples for multiple optimal moves or the 12-way binary classification.


Unsupervised text classification with R/Python

I can think of two approaches.

You could use term-frequency/inverse-document-frequency (tf-idf) to cluster the vectors. Personally, I would start by first clustering the full text of the original job vacancies and then use this to assign clusters to the vectors. I have the feeling that it will outperform clustering directly the vectors.
There are implementations both in R and in Python. I would use Python for clustering in a language other than English. You can use TfidfVectorizer from sklearn.
I've a post on medium with an example of clustering wikipedia articles in Python if you are interested.

You can start by a list of key/basic skills and then try to match your vectors to the items of this list. For defining a distance between words/phrases you can:
a. use Jaccard index (ex. see this)
b. use SpaCy in Python to find descriptions that are similar. I believe there is a language model for Polish. I haven't tried it but I would use something that is described in "The Beginner’s Guide to Similarity Matching Using spaCy". Since you are trying to find the similarity between phrases (not single words) you might also  want to consider this stackoverflow discussion.


Deciding the number of clusters in K-means clustering of descriptors

Measure, don't guess

It appears to be common to try k=1024, 2048, 4096, ... and use what works best for classification.

This is possivle because the clustering is actually a quantization task, and you do classification afterwards.


How to build a hybrid graph with CNN and RNN Architecture?

I write an answer here since I don't have enough reputation to write a comment.

What about treating your M matrices as sequence, feeding them to a TimeDistributed(Conv2D) layer(s) and then to an LSTM layer? In this way the Conv layer should extract feature maps that are treated by the LSTM in a sequence-like fashion, updating/accumulating the states up to the M matrix. Afterwards you should put a Dense layer with sigmoid/softmax activation and get class probabilities.


How to compute Hessian matrix for log-likelihood function for Logistic Regression

Beta is a vector of parameters, therefore:

$
\frac{\delta l(\beta)}{\delta\beta}=
[\frac{\delta l(\beta)}{\delta\beta_1}\quad\frac{\delta l(\beta)}{\delta\beta_2}\quad\frac{\delta l(\beta)}{\delta\beta_3}\quad...\quad\frac{\delta l(\beta)}{\delta\beta_n}]$ and so

$
\frac{\delta(\frac{\delta l(\beta)}{\delta\beta})}{\delta\beta^{T}}=
\begin{bmatrix}
\frac{\delta l^2(\beta)}{\delta\beta_1^2} & \frac{\delta l^2(\beta)}{\delta\beta_1\delta\beta_2} & ... & \frac{\delta l^2(\beta)}{\delta\beta_1\delta\beta_n} \\
\frac{\delta l^2(\beta)}{\delta\beta_2\delta\beta_1} & \frac{\delta l^2(\beta)}{\delta\beta_2^2} & ... & \frac{\delta l^2(\beta)}{\delta\beta_2\delta\beta_n} \\
\vdots & \vdots & \ddots & \vdots \\
\frac{\delta l^2(\beta)}{\delta\beta_n\delta\beta_1} & \frac{\delta l^2(\beta)}{\delta\beta_n\delta\beta_2} & ... & \frac{\delta l^2(\beta)}{\delta\beta_n^2}
\end{bmatrix}$, which is your Hessian.

The term on the right side of your equation is also a matrix, because there is a multiplication of vectors in it: $x_i \cdot x_i^T$, which gives a $n \times n$ matrix.


How to resolve too many indices for array Index Error

Your code is broken in two places.

The first is because you took the argmax of your class probabilities from y_pred. The line

y_pred = y_pred.argmax(axis=-1)

reshapes your prediction vector into (624,) to match your vector of classes. Thus, when you try to slice your array later with y_pred[:,i] it's going to bark since you no longer have a second dimension. This isn't really the behavior you want either, since the roc_curve function is interested in the exact class probabilities your model produces!

The second is for the same reason, attempting to index the second dimension of  a one dimensional numpy array, but for the Y_test vector.

So if you're interested in capturing TPR/FPR for both classes by treating each as the positive class, you need to drop these lines

#reshape the predicted labels and convert type
y_pred = y_pred.argmax(axis=-1)
y_pred = y_pred.astype('int32')

and you need to change the first line of your for loop to:

fpr[i], tpr[i], _ = roc_curve(Y_test, y_pred[:, i])

hope this helps


finding maximum depth of random forest given the number of features

The maximum depth of a forest is a parameter which you set yourself. If you're asking how do you find the optimal depth of a tree given a set of features then this is through cross-validation. For example, create 5 rf's with 5 different tree depths and see which one performs the best on the validation set.


How to improve a CNN without changing the architecture?

Are you in fact using the same architecture as they are? If not that could potentially be the problem.

Otherwise, are you using the same trainings protocol as they, i.e. optimizer, learning rate, learning rate schedule, batch size, preprocessing, weight initialization, number of training epochs? Depending on the size of your model and the amount of training data, 10 epochs might not be enough to judge about your models performance.

Can you link the paper?


Communication between Keras and Backend like Tensorflow, Theano

Keras is a high level API but the backend is readily available. You simply access it by doing:

from keras import backend as K

The K will then be the same as tf as if you imported Tensorflow like this:

import tensorflow as tf

So you can use K to perform lower level operations with the backend.

For more information you can read the Keras backend documentation.


The slide explains a limitation which applies to any linear model. It would equally apply to linear regression for example.

  What does he mean by hand generated features?

This means any features generated by analysis of the problem. For instance if you wanted to categorise a building you might have its height and width. A hand generated feature could be deciding to multiply height by width to get floor area, because it looked like a good match to the problem.

  I don't get the binary input example and why it is a table look-up type problem and why it won't generalize?

A table look-up solution is just the logical extreme of this approach. If you have a really complex classification, and your raw features don't relate directly (as a linear multiple of the target), you can craft very specific manipulations of them that give just the right answer for each input example. Essentially this is the same as marking each example in your training data with the correct answer, which has the same structure, conceptually, as a table of input: desired output with one entry per example.

In fact this might generalize, but only exactly as well as the crafted features do. In practice, when you have a complex problem and sample data that only partially explains your target variable (i.e. in most data science scenarios), then generating derived features until you find some that explain the data is strongly related to overfitting.

From your comment:

  In his video lecture, he says "Suppose for example we have binary input vectors. And we create a separate feature unit that gets activated by exactly one of those binary input vectors. We'll need exponentially many feature units. But now we can make any possible discrimination on binary input vectors. So for binary input vectors, there's no limitation if you're willing to make enough feature units." 1.What feature? 2.Why are we creating this feature? And why adding exponential such features we can discriminate these vectors?

Here is an example of the scheme that Geoffrey Hinton describes. Say you have 4 binary features, associated with one target value and see the following data:

data 0 1 1 0 -> class 1
data 1 1 1 0 -> class 2
data 0 1 0 1 -> class 1
data 1 1 1 0 -> class 2
data 0 1 1 1 -> class 2
data 0 1 0 0 -> class 1

It is possible to get a perceptron to predict the correct output values by crafting features as follows:

data 0 1 1 0 -> features 1 0 0 0 0 -> class 1
data 1 1 1 0 -> features 0 1 0 0 0 -> class 2
data 0 1 0 1 -> features 0 0 1 0 0 -> class 1
data 1 1 1 0 -> features 0 1 0 0 0 -> class 2
data 0 1 1 1 -> features 0 0 0 1 0 -> class 2
data 0 1 0 0 -> features 0 0 0 0 1 -> class 1

Each unique set of original data gets a new one-hot-encoded category assigned. It is clear that ultimately if you had $n$ original features, you would need $2^n$ such derived categories - which is an exponential relationship to $n$.

Working like this, there is no generalisation possible, because any pattern you had not turned into a derived feature and learned the correct value for would not have any effect on the perceptron, it would just be encoded as all zeroes. However, it would learn to fit the training data very well, it could just associate each unique vector with a weight equal to the training output - this is effectively a table lookup.

The whole point of this description is to show that hand-crafted features to "fix" perceptrons are not a good strategy. Even though they can be made to work for training data, ultimately you would be fooling yourself.


Is reinforcement learning a subset of unsupervised learning?

Reinforcement learning differs from Unsupervised learning as it uses additional information regarding the expected behavior of the agent in the form of a reward function. However, it also differs from Supervised learning as it does not require any labelled data for training or testing.
So, it is neither of them. You may also refer to the wiki page of Reinforcement learning for further related details.


High / low resources language : what does it mean?

High resource languages are languages for which many data resources exist, making possible the development of machine-learning based systems for these languages. English is by far the most well resourced language. West-Europe languages are quite well covered, as well as Japanese and Chinese. Naturally low-resource languages are the opposite, that is languages with none or very few resources available. Unsurprisingly many languages from Africa and other poorest parts of the world are low-resources. But this is also the case for some extinct or near-extinct languages and many local dialects. There are actually many languages which are mostly oral, for which very few written resources exist (let alone resources in electronic format); for some there are written documents but not even something as basic as a dictionary.

There are many different types of resources which are needed in order to train good language-based systems:

a high amount of raw text from various genres (type of documents), e.g. books, scientific papers, emails, social media content, etc.
lexical, syntactic and semantic resources such as dictionaries, dependency tree corpora, semantic databases (e.g. WordNet), etc.
task-specific resources such as parallel corpora for machine translation, various kinds of annotated text (e.g. with part-of-speech tags, named entities, etc.)

Many types of language resources are costly to produce, this is why the economic inequalities between countries/languages are reflected in the amount (or absence) of language resources. The Universal Dependencies project is an interesting effort to fill this gap.


How to know if a model is overfitting or underfitting by looking at graph

Overfitting is a scenario where your model performs well on training data but performs poorly on data not seen during training.  This basically means that your model has memorized the training data instead of learning the relationships between features and labels.

If you are familiar with the bias/variance tradeoff, then you can think of overfitting as a situation where your model has high variance, memorizing the random noise in the training set.

Overfitting is easy to diagnose with the accuracy visualizations you have available.  If "Accuracy" (measured against the training set) is very good and "Validation Accuracy" (measured against a validation set) is not as good, then your model is overfitting.

Underfitting is the opposite counterpart of overfitting wherein your model exhibits high bias.  This situation can occur when your model is not sufficiently complex to capture the relationship between features and labels (or if your model is too strictly regularized).

Underfitting is a bit harder to diagnose.  If Accuracy and Validation Accuracy are similar but are both poor, then you may be underfitting.

Edit 1:

Strategies to avoid overfitting/underfitting

Recall that overfitting is caused by the model memorizing the training data instead of learning the more-general mapping from features to labels.
This commonly occurs when training a model with so many parameters that it can fit nearly any dataset.  As von Neumann so eloquently put it, "With four parameters I can fit an elephant, and with five I can make him wiggle his trunk."

You can combat overfitting by reducing the complexity of your model (i.e. reducing the number of trainable parameters).  The specifics of how this is accomplished vary depending on the learning algorithm and the domain.

For neural networks, you can use fewer layers (shallower networks), fewer neurons per layer, sparser connections between the layers (as in convolutional nets), or regularization techniques like dropout.

In the same vein, you can combat underfitting by increasing the complexity of your model.  This has been the driving force behind the push for ever-deeper neural networks in recent years.  With more layers, the network can learn more sophisticated relationships and perhaps perform well on difficult real-world tasks.

Of course, you can't recklessly add layers to a network and expect great performance.  Training deep neural networks is hard, for a number of statistical and technical reasons (one of which is avoiding overfitting).

So to answer your question directly:  If your network is overfitting, adding more layers will almost certainly make the problem worse, since you're increasing model complexity.  If your network is underfitting, adding more layers can help, but it's rarely so straightforward.  You need to think carefully about how you expect the network to operate and what strategies you can employ to ensure that it doesn't begin to overfit.

Edit 2:

P.S. If you're new to the field of machine learning, then it may be helpful to experiment with more intuitive models than neural nets while learning about over/underfitting and the bias/variance tradeoff.

I would recommend decision trees (or model trees for regression problems).  Tree-based models are easy to interpret, and playing around with parameters like the max depth and minimum impurity decrease of a decision tree might help you gain some intuition about the relationship of model complexity to overfitting.


Handling huge dataset imbalance (2 class values) and appropriate ML algorithm

I'm not allowed to comment, but I have more a suggestion: you could try to implement some "Over-sampling Techniques" like SMOTE:
http://scholar.google.com/scholar?q=oversampling+minority+classes


Scaling often assumes you know the min/max or mean/standard deviation, so directly scaling features where these information is not really known, can be a bad idea.

For example, clipped signals may hide this info, so scaling them can have a negative result because you may distort its true values.

Below is an image of 1) a signal that can be scaled, and 2) a clipped signal that scaling should not be done.


Getting slightly different results is natural and should not be a problem. How to minimize the instabilities due to several contributing factors is discussed at length in the linked post below, for Keras using different backends including TensorFlow:

https://machinelearningmastery.com/reproducible-results-neural-networks-keras/


Setting sparse=True in Scikit Learn OneHotEncoder does not reduce memory usage

Based on @BenReiniger's comment, I removed the numeric portion from the ColumnTransformer and ran the following code:

    from sklearn.compose import ColumnTransformer
    from sklearn.preprocessing import OneHotEncoder
    from sklearn.impute import SimpleImputer
    from sklearn.pipeline import Pipeline

    categorical_transformer = Pipeline(steps=[
        ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
        ('onehot', OneHotEncoder(handle_unknown='ignore', sparse=True))])

    preprocessor = ColumnTransformer(transformers=[
        ('cat', categorical_transformer, selector(dtype_include="object"))
    ])

    X = pd.DataFrame(preprocessor.fit_transform(X))

    print_memory_usage_of_data_frame(X)

The result was Memory usage is 0.106 MB,
Running the same code above but with sparse option set to False:
    OneHotEncoder(handle_unknown='ignore', sparse=False)

resulted in Memory usage is 20.688 MB.
So it is clear that changing the sparse parameter in OneHotEncoder does indeed reduce memory usage.


Merge columns of coordinates and construct sequence

it probably depends on the use of your new generated column. Here I try to reproduce your example:

x1 <- c(0, 0)
x2 <- c(58.33546, 58.33604)
x3 <- c(15.14554, 15.14571)
df <- data.frame(x1, x2, x3)
df$x4 <- paste("(", as.character(df$x2), ", ", as.character(df$x3), ")")

And df$x4 will look like:

[1] "( 58.33546 ,  15.14554 )" "( 58.33604 ,  15.14571 )"

If you want to show the coordinates, the above format should be fine. If you want to do further analysis with that column, some other transformations are required.


The upper range of a collected dataset is most likely accurate, but the rest may suffer biased omissions: How to call this phenomenon?

The case described can be cast as imbalanced dataset problem, or rare events problem.
Even more generally as highly non-uniform underlying distribution problem (which is an umbrella term for both cases).
References:

Machine Learning Tips: Handling Imbalanced Datasets
Handling imbalanced datasets in machine learning


Weighted sum of word vectors for document similarity

Yes, your method is valid and it has been studied before it is known as Mean of Word Embeddings (MOWE) or Sum of Word Embeddings (SOWE), although your method is more a weighted mean of vectors. I think that a good starting point for knowing more about the academics of the method is this paper: How Well Sentence Embeddings Capture Meaning. It discusses your method and give some pointers to other papers that also discuss the validity of the method.


What's a good machine learning algorithm for low frequency trading?

Random forests, GBM or even the newer and fancier xgboost are not the best candidates for binary classification (predicting ups and down) of stocks predictions or forex trading or at least not as the main algorithm. The reason is that, for this particular problem, they require a huge amount of trees (and tree depth in case of GBM or xgboost) to obtain reasonable accuracy (Breiman suggested using at least 5000 trees and to "not be stingy" and in fact his main ML paper on RF he used 50,000 trees per run).

However, some quants use random forests as feature selectors while others use it to generate new features. It all depends on the characteristics of the data.

I would suggest you read this question and answers on quant.stackexchange where people discuss what methods are the best and when to use them, among them ISOMAP, Laplacian eigenmaps, ANNs, swarm optimization.

Check out the machine-learning tag on the same site, there you might find information related to your particular dataset.


ANN applied to Boston Housing dataset returns negative value

"This is to be expected. sklearn has changed their API to invert their cost functions. Nothing to be concerned about."

I got this answer from Dr. Jason Brownlee.


This is where Word Embedding play an amazing role. If you want to solve the problem you have at hand with Natural Language Processing approach, what you can do is learn vector representations for these documents and store them with the labels i.e tags for that text.

You can then use the similarity of vector representations between documents, to predict the tags.

This way, your computation is not intensive at all and you can keep retraining as and when you get more data.


Reversing Naive Bayes to find extreme points of data sets

What you are looking for is called "decision boundary", which is the set of (extreme) points laying on the boundary between classes.

Decision boundary of NB is a set of points $\boldsymbol{x}$ that satisfy at least one of these  $K(K-1)/2$ conditions $$i\neq j \in [1, K]:{\Bbb P}(\boldsymbol{x}, C_i)={\Bbb P}(\boldsymbol{x}, C_j) \overset{\forall k}{\geq} {\Bbb P}(\boldsymbol{x}, C_k)$$
Generally, Naive Bayes does not learn an explicit decision boundary (specially when categorical features are involved), however, for example here is the derivation of NB boundary for continuous variables when ${\Bbb P}(\boldsymbol{x}|C_i)$ is chosen from exponential family (e.g. a Gaussian). Nonetheless, you could still select a set of random points, then assign a class to each, and finally plot them using something like t-SNE to get a sense of decision boundaries. For example, a tool like this interactive t-SNE plot would be helpful:

Or a library like mlxtend.plotting that carries out a similar procedure:

On the contrary, models like multi-class logistic regression and SVM both have explicit decision boundaries, which are equations of learned parameters $\boldsymbol{w}$ such as $$w_0+w_1\text{fat}+w_2\text{salt}+w_3\text{sugar}=0$$ that you can access via common libraries.


Multi-label classification using output quantization

TL;DR

Bad idea, don't do that.

Explanation

1)
You are introducing an ordering in the classes that doesn't exist. Basically, with your example, you are saying that: "dog is close to cat", "cat is close to rabbit", but "dog is far from rabbit", which is an extra learning feature your network needs to learn.

For example, if the network outputs 0.35, then it is class cat but it is also close to dog and far from rabbit.

2)
Another problem, that some results are not possible to be represented.

For example, what if you changed the range to:

  0 <= output < =1

  cat [0.0 - 0.33]

  rabbit [0.33 - 0.66]

  dog [0.66 - 1]

How will you represent that the same example from before ("it is class cat but it is also close to dog and far from rabbit.")? It become impossible.

With the previous representation you could have:

output = [0.5, 0.0, 0.5] # cat, rabbit, dog

how would you map that to the new range where cat = [0.0-0.33] and dog = [0.66-1.0]? Can we pick the middle (0.5)? No, because that is class rabbit.

Conclusion

By doing that, you reduce the number of weights in the final layers (good), however, you introduce dependencies which needs to be learned and some results are not representable or interpretable.


Improving the performace of the Naive Bayes classifier by decorrelating the data

As you pointed out, a null covariance does not guarantee that variables are independent. You can have strongly dependent variables which show a covariance equal to 0. See a famous plot taken from Wikipedia page on Pearson correlation coefficient

A standard way to uncorrelate variables is to perform principal component analysis on the dataset, and use all components which are, by definition, uncorrelated. But please note that the Naive Bayes assumption is conditional independence, so independence for each class. In the general case, PCA projection would not be the same depending on the selected class.

Regarding the performance of Naive Bayes, I see no reason that a simple transformation should improve the model. This is supported by some reasonably well-received answers on similar questions, for instance this one on Stack Overflow, or this other one on Cross Validated.

However, I have read several reports of people getting increased accuracy when performing PCA prior to Naive Bayes. See here for instance, for a few exchanges on the subject, within a community of which trustworthiness is unknown to me; or search "PCA naive bayes" through your web search engine.

So in the end, I have no mathematical evidence to support this, but my feeling would be that the model performance variation before/after transformation could depend on the problem, especially the directions of principal components for each class.

Perhaps, if you perform some tests, you could share the result here. Otherwise, I think you could get answers with more mathematical background on Cross Validated.


How to determine layers structures appropriate for a specific application?

From the structure you have outlined it looks like you're going for a dense fully-connected neural network. A general rule of thumb for practitioners on this topic is that increasing the size of a single hidden layer is more likely to yield a more performant network than adding successive layers. This is obviously not true in all cases - but as you are experimenting with architectures it is more useful to start by increasing the size of your one hidden layer before you jump to two of even three. See this for more details.
I would recommend removing one of those hidden layers and increasing the number of nodes on the remaining.
Something else to note: if you are looking at image data you might consider using convolutional layers. These are much more appropriate for problems with data that are spatially related (images etc..). If you do decide to go with this approach the previous advice does not apply as it relates only to simple dense multilayer perceptrons. Instead review this.


What are the equations involved in calculation of the parameters of embedding layer?

Like all hidden layers in a neural network, an embedding layer can be thought of as a feature extractor with parameters that are automatically learned during training.
So, like any other layer, the parameters are adjusted during training by the backpropagation algorithm.

The specific equations used to compute error gradients and update weights are likely to depend on the optimizer used to train the network.  The classic optimizer is stochastic gradient descent, but more sophisticated optimizers like Adagrad, RMSProp, and ADAM are commonly used nowadays.


Techniques to manage high dynamic range in neural network regression?

You may try cycle GANs.
In normal Generative Adversarial Networks (GANs), a generative network -the generator- is trained to generate images with an auxiliary network, the discriminator. The discriminator learns to tell apart data generated by the generator (i.e. fake data) from real data, while the generator learns to generate data that fools the discriminator. GANs are known to generate good quality data where MSE and the likes generate "blurry" data that lack detail.
Cycle GANs are two coupled GANs, where the generators learn to "convert" the input data from domain A to domain B, and vice versa, plus an extra loss term to ensure cyclic consistency (i.e. generator1(generator2(a)) = a).
The results are very impressive. Here you can see cycle GANs trained to convert images, changing horses into zebras and vice versa, winter landscape into summer, etc:

If your data is continuous and your architecture is fully convolutional, then cycleGANs may actually work in your case.


When I use SHAP for classification problem, it shows an output that is not 0 or 1. How can I overcome this?

The default link function is the identity, so you are seeing log-odds rather than probabilities. To see the probability, try adding link='logit' to your call of force_plot like so:

shap.force_plot(explainer.expected_value,
                shap_values[0,:].reshape(1, 8),
                X_train[0,:].reshape(1, 8),
                link='logit')

You can read more at the SHAP documentation site.


High Cross Validation Score on Training Set, High Score on Test Set, But Low Score on Kaggle?

This "train split" you named train_set and test_set are not guarantee to be clean or even balanced.
When your test set has better performance than your training set that might mean that you have data leakage (some examples in the test set are equal to the training set) or just mean your test set is slightly easier than the training set.


How to chose a Machine Learning algorithm?

I would totally start here. You're going to want to tweak things when they don't perform as well as you want, but it gives you a great starting point.


In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document and word representations?

It is possible to produce a corpus from the learned LDA parameters ($\theta$ and $\phi$) according to the generative model of LDA but it is not realistic to expect that you would recreate the original documents (in bag-of-words form). To be more specific, it is possible - but highly improbable - that you would generate the bag-of-words documents corresponding to the input corpus.


NN with 1 hidden layer: Number of half planes visualised

A nice experiment!  I think you've just forgotten to include the final neuron's bias term.
As to the original approach, I think you're right that the hidden layer's output will be piecewise linear with 7 regions.  But after the activation at the final neuron you can cut each of those regions into two based on sign.  Plotting more of a heatmap for the actual outputs might be worthwhile.


Getting different chi square values than sklearn function

I had a similar problem using Scipy's chi2_contingency. It turns out that the Scipy function uses by default correction=True, and the statistic you showed doesn't use the correction. Perhaps sklearn's chi2 uses the correction as well, and that's the reason you don't get the same results. I suggest trying Scipy's with both correction set to true and false to see if it matches the two results.


Bag of words and word2vec clarifications

BoW is a text representation like word2vec or doc 2vec. If you already have the word2vec vectors then you can take the mean for each document and calculate the similarity but it's not the direct approach. The direct approach is to calculate doc2vec which models the documents directly and with more detailed precision than word2vec.


Maybe I'm being a bit too simplistic, but I would build a set of training data that looks like this (Good=1 means patient showed up for appt and is good by your definition, 0 = bad)

Recid, PatId, SurgeryId, DrId, DateAppt, TimeAppt, Gender, Age, Good
1, 1, 100, 10, 01jan16, 10:30, M, 31, 1
2, 1, 100, 12, 05jan16, 15:20, M, 31, 1
3, 1, 100, 10, 06mar16, 11:45, M, 31, 0
4, 2, 101, 15, 02Feb16, 12:35, F, 75, 1
....

I would then use one of the machine learning tools in R - there are a variety of them, to train a model of your data.

Then with another set of data, I would test the model you have just built to see how correct it is. If you don't have a second set of data, then randomly partition your original training set and only train with half of the data.

Some further suggestions to make your model more powerful, is create additional variables with the following information

a Flag to indicate if a person has missed a previous appointment with any doctor.
a Flag to indicate if a person has missed a previous appointment with the particular doctor they are going to be visiting - I know from person experience there are some doctors in my surgery that I have a preference to see.
The number of days since the last appointment
Day of week of appointment

Sounds like a nice dataset to be working with!


How to predict custom value after using linear regression?

As an alternative to pd.dummies() you could use sklearn.preprocessing.OneHotEncoder().The advantage here would be that after creating an encoder object you can reuse it to transform data points you have to the one hot form. You could use OneHotEncoder(categories) to manually assign categories for the encoder object or you could use OneHotEncoder.fit() to extract categories from a particular feature. You can then use OneHotEncoder.transform(feature) to transform it to one hot form. pd.dummies() does not provide the re usability part, therefore, sklearn.OneHotEncoder() makes things simpler.


can accuracy rise while precision and recall drop?

The explanation is simple, assume you have the following values:

True Positives (TP) = 1
True Negatives (TN) = 998
False Positives (FP) = 1
False Negatives (FN) = 1

Accuracy = (TP + TN) / (TP + TN + FP + FN) = 999/1001 = 0.998
Precision = TP / (TP + FP) = 1/2 = 0.5
Recall = TP / (TP + FN) = 1/2 = 0.5

In summary you have an unbalanced dataset i.e. the number of samples of class is much larger than the number of the other classes. So predicting that every sample belongs to class results in a high accuracy value.


Product Recommendation based on purchase history

Even if you don't have ratings or reviews, you can use the customer purchases to help creating your model and selecting the most appropriate one. If the customer has bought the product, you can suppose he has liked it. If he hasn't, we can suppose he has disliked it. This is an approach which is mainly used by e-shops.

You can find more on recommendation system selection and validation below. It is discussed how to choose the best recommender with both offline and online approaches.

https://medium.com/recombee-blog/evaluating-recommender-systems-choosing-the-best-one-for-your-business-c688ab781a35


How to implement word to word Co-occurence matrix in python

from nltk.tokenize import word_tokenize
from itertools import combinations
from collections import Counter

sentences = ['i go to london', 'you do not go to london','but london goes to you']
vocab = set(word_tokenize(' '.join(sentences)))
print('Vocabulary:\n',vocab,'\n')
token_sent_list = [word_tokenize(sen) for sen in sentences]
print('Each sentence in token form:\n',token_sent_list,'\n')

co_occ = {ii:Counter({jj:0 for jj in vocab if jj!=ii}) for ii in vocab}
k=2

for sen in token_sent_list:
    for ii in range(len(sen)):
        if ii < k:
            c = Counter(sen[0:ii+k+1])
            del c[sen[ii]]
            co_occ[sen[ii]] = co_occ[sen[ii]] + c
        elif ii > len(sen)-(k+1):
            c = Counter(sen[ii-k::])
            del c[sen[ii]]
            co_occ[sen[ii]] = co_occ[sen[ii]] + c
        else:
            c = Counter(sen[ii-k:ii+k+1])
            del c[sen[ii]]
            co_occ[sen[ii]] = co_occ[sen[ii]] + c

# Having final matrix in dict form lets you convert it to different python data structures
co_occ = {ii:dict(co_occ[ii]) for ii in vocab}
display(co_occ)

Output:

Vocabulary:
 {'london', 'but', 'goes', 'i', 'do', 'you', 'go', 'not', 'to'}

Each sentence in token form:
 [['i', 'go', 'to', 'london'], ['you', 'do', 'not', 'go', 'to', 'london'], ['but', 'london', 'goes', 'to', 'you']]

{'london': {'go': 2, 'to': 3, 'but': 1, 'goes': 1},
 'but': {'london': 1, 'goes': 1},
 'goes': {'london': 1, 'but': 1, 'you': 1, 'to': 1},
 'i': {'go': 1, 'to': 1},
 'do': {'you': 1, 'go': 1, 'not': 1},
 'you': {'do': 1, 'not': 1, 'goes': 1, 'to': 1},
 'go': {'london': 2, 'i': 1, 'to': 2, 'do': 1, 'not': 1},
 'not': {'do': 1, 'you': 1, 'go': 1, 'to': 1},
 'to': {'london': 3, 'i': 1, 'go': 2, 'not': 1, 'goes': 1, 'you': 1}}

PS

Do text preprocessing yourself (remove punctuations, lemmatization, stemming, blahblah)
Continue the code for any conversion you want. You have the dict, you can convert it to sparse matrix or pandas datframe


What is the intuition of using clustering for performing feature engineering in machine learning tasks?

Overall the paper is not very clear so there are a few uncertainties, but the general approach is this:

Their main idea is to create new features which represent the "outlyingness" of the instance. They use several different methods in order to detect outliers, however they do not explain how exactly are the new features created.
One of the methods they use to detect outliers is based on hierarchical clustering: the result of such clustering is a binary tree in which the most distant clusters/instances are connected last, i.e. close to the root of the tree. Their assumption is that an outlier tends not to be close to any other instance or cluster, therefore they are connected last. While the method makes sense, it's not clear in the paper whether they only retain the very last instance as outlier or the last few instances (or any other variant).

So the clustering and the feature creation are only indirectly related:

The clustering is used to detect one or several outliers in the instances, and several other methods are used for the same purpose.
Based on the detection of these outliers, one or several new features are created which contain a value describing the "outlier status" of the instance. The simplest option would be to create a single boolean feature which is true if the instance was detected as outlier by any of the methods, but one can imagine more advanced options. For example, based on the hierarchical clustering one can obtain the order in which the instances are connected, and the rank of every instance can be used as a feature.


Number of filters in Convolutional Neural Networks

Number of filters depends on the problem you are solving and the complexity of objects, which have to be recognized by your model. During a training step model receives images, implements transformation to them using current states of convolutional filter's weights to extract some significant information and then compute the error between output and groundtruth labels. Then using back propagation weights are recomputed to optimize the model's performance. You can find here more information, how it works https://towardsdatascience.com/how-does-back-propagation-in-artificial-neural-networks-work-c7cad873ea7 First layers usually extract some simple features of objects on images (for example, it could be horizontal or vertical lines). Going deeper the complexity of features are growing. There is link to the original unet paper with description of its architecture, which might clarify usage of filters https://arxiv.org/pdf/1505.04597.pdf


How to get K most different rows in csv?

To select the most different rows, you would need to define first what you consider different. For ages and scores, subtracting values would work, for example:
Row1

Age is 38
Score is 0.2

Row2

Age is 87
Score is 1.0

Difference by numeric feature:

Age Diff is 49
Score Diff is 0.8

Those values could be normalized or weighted to account for different importances between features.
For titles, you’d probably need to use a text similarity metric, such as tf-idf with cosine similarity or embedding vector distance. The most different ones are the ones with higher score/age difference and lower title similarity.


How many RNN units are needed for tasks involving sequences?

You don't need to assign an RNN cell to each entry in a sequence, you just need one.

A single cell can process sequences because of its recursive nature(its output feeds back into itself):

$$ h_t = f(h_{t-1}, W,  x_t) $$

where $h_t$ is the hidden state at step $t$, $W$ is the weight used for the cell, and $x_t$ is the current input at step $t$ .

Hopefully, you can see how it is only necessary for us one weight matrix, $W$, instead of multiple as your understanding would imply.

Where you got the notion of multiple RNN cells is probably when we feed the hidden state as the input, $x$, into another RNN cell. More layers of RNN cells, mean you can learn a more complex sequence.

Conclusively, your RNN cell cell can't work without new inputs, therefore it end when your sequence ends.
Also, it is recommended to use LSTM or GRU over vanilla RNN units.


inner workings of Mobile-net resolution multiplier - what does it do?

I contacted the authors of the mobile net architecture and papers and they provided me with the following answer:

The multiplier accomplishes what is being articulated in C): it acts as a re-scaling of the original input image to fit the value specified by the parameter. The default image size is 224x244, which corresponds to a resolution multiplier of 1.0. So 112x112 input image resolution corresponds to a 0.5 resolution multiplier which is achieved by simply downscaling the image to 112x112. Technically we scale the smaller side to target and crop the other dimensions.

I hope this proves as helpful to others as it was to me.


How many Hidden Layers and Neurons should I use in an RNN?

Number of layers is a hyperparameter. It should be optimized based on train-test split. You can also start with the number of layers from a popular network. Look at kaggle.com and see how many layers do they use in competitions.


Why is my model accuracy decreasing after the second epoch?

Decreasing accuracy as training progresses means that the learning rate for your model is too high. Your model weights are changing a lot due to the high learning rate and therefore moving away from the local minimum where your accuracy would be at its highest. Try decreasing the learning rate and see what happens.


How to treat time based ticket prices for train/test split

Indeed airfare data is different than many other price data. In To Buy or Not to Buy: Mining Airfare Data to Minimize Ticket Purchase Price the authors put it as following:

  Computational finance is concerned with predicting prices
  and making buying decisions in markets for stock, options,
  and commodities. Prices in such markets are not determined
  by a hidden algorithm, as in the product pricing case, but
  rather by supply and demand as determined by the actions
  of a large number of buyers and sellers. Thus, for example,
  stock prices tend to move in small incremental steps rather
  than in the large, tiered jumps observed in the airline data.

(tho the reality is not that black and white as airfares also depend on seat availability and stock prices are also indirectly driven by algorithms but generally they have a point)

These algorithm-driven price changes do not so much rely on historical price development which is why I would give your approach a try. In line with that there are multiple papers in which airfare prices have not been treated as timeseries (besides the one already linked also see Predicting Airfare Prices - tho in this paper they did consider an attribute related to the most recent previous price of the ticket!).

Nevertheless, there are many date-related aspects your might want to consider. "Days to trip" is one which is in your data already. Others, easily available from your data, could be

"weekday of departure",
"departure during holiday season or not" and
"weekday of price request".

And then of course there are tons of other variables which might be important, e.g. "number of stopovers", "overnight flight or not", "number of free baggage" (and obviously the cabin class or maybe even booking class) etc. (also see Airfare Prices Prediction Using Machine Learning Techniques)


CNN model for more than 3 channels input

A CNN does not care about the starting number of channels. It can be one, three, 20, or anything really. Here's a simple example in the python library Keras for how you might start out a CNN with 20 channels, assuming your images are 100x100. Obviously these numbers can be changed to whatever your problem needs.

from keras.layers import Input, Conv2D

# Assumes images are 100x100 with 20 channels
inputs = Input(shape=(100, 100, 20))

# First convolution will take in the 20 channels and output 32 channels
x = Conv2D(32, strides=(2, 2), activation='relu')(inputs)

# Second convolution will take in the 32 channels and output 64 channels
x = Conv2D(64, strides=(2, 2), activation='relu')(x)

...


Trouble with accuracy of multiclass perceptron

Firstly, in all the linear separator algorithms such as linear regression, logistic regression and the perceptron, adding the bias is as simple as adding a feature column consisting of all 1's. Then the third weight that will be trained will act as the bias $b$.

I have some working code for a multi-class perceptron

First let's generate some artificial data with 2 features. Each distribution of points is going to be Gaussian with a given mean and variance described as a list of lists for each dimension under the variable params.

def gen_data(params, n):
    dims = len(params[0])
    num_classes = len(params)

    x = np.zeros((n*num_classes, dims))
    y = np.zeros((n*num_classes,))
    for ix, i in enumerate(range(num_classes)):
        inst = np.random.randn(n, dims)

        for dim in range(dims): x[ix*n:(ix+1)*n,dim] = np.random.normal(params[ix][dim][0],
                                                                        params[ix][dim][1], n)
        y[ix*n:(ix+1)*n] = ix
    return x, y

params = [[[ 5,1],  [ 5,1]],
          [[ 0,1],  [ 0,1]],
          [[2, 1],  [ 2,1]],
          [[-2, 1],  [ 2,1]]]

n = 300
x, y = gen_data(params, 300)

plt.scatter(x[:,0], x[:,1])
plt.show()

Alright so now we have 4 distributions with different labels. Let's split the data for sanctity's sake.

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33)

Let's train the weights using the training data

def get_weights(x, y, n_epochs, verbose = 0):
    # Append a ones column to the feature for the bias
    data = np.ones((x.shape[0], x.shape[1]+1))
    data[:, 0:x.shape[1]] = x
    # Set the targets as integers for comparison
    targets = y.astype(int)

    # Initialize the weights as a matrix
    # number of classes by number of features
    weights = np.ones((len(set(y)), x.shape[1]+1))
    for epoch in range(n_epochs):
        for i, target in zip(data, targets):
            temp = np.dot(i, weights.T)
            pred = np.argmax(temp)
            # If wrongly predicted update prediction
            if pred != target:
                weights[target, :] = weights[target, :] + i
                weights[pred, :] = weights[pred, :] - i

        if verbose == 1:
            print('Iteration: ', epoch)
            print(weights)
            print('---------------------------------------------')
    return weights

weights = get_weights(x_train, y_train, n_epochs = 30, verbose = 1)

This converges to approximately this

  [[  23.62752045   16.03867499 -111.        ]
  [  -3.96545848  -8.66924406   47.        ] [  -0.94290763   -1.84413793   33.        ]  [ -14.71915434   -1.52529301   35.        ]]

We get an accuracy calculated using the score

def predict(x, weights):
    data = np.ones((  x.shape[0], x.shape[1]+1  ))
    data[:, 0:x.shape[1]] = x
    predictions = np.argmax(np.dot(data, weights.T), axis = 1)
    return predictions

def score(x, y, weights):
    pred = predict(x, weights)
    return sum(pred == y_test)/len(pred)

score(x_test, y_test, weights)

  0.8686868686868687

We can check our results using a confusion matrix. For the training set

from sklearn.metrics import confusion_matrix

predictions = predict(x_train, weights)
plt.imshow(confusion_matrix(y_train, predictions))
plt.show()

And the testing set

predictions = predict(x_test, weights)
plt.imshow(confusion_matrix(y_test, predictions))
plt.show()

So we see that in fact our algorithm is performing quite well.

We can then plot our points to see how it is classifying them. I will plot the training points as small circles, and the testing points as larger ones. The dark points are those which are misclassified

colors = ['y', 'r', 'b', 'g', 'k']

# Predict training set
predictions = predict(x_train, weights)
for i, t, p in zip(x_train, y_train, predictions):
    if t == p: plt.scatter(i[0], i[1], c=colors[int(t)], alpha = 0.2, s=20)
    else: plt.scatter(i[0], i[1], c=colors[int(t)], alpha = 1)

# Predict test set
predictions = predict(x_test, weights)
for i, t, p in zip(x_test, y_test, predictions):
    if t == p: plt.scatter(i[0], i[1], c=colors[int(t)], alpha = 0.2)
    else: plt.scatter(i[0], i[1], c=colors[int(t)], alpha = 1)

# Plot the linear separators
x1 = np.linspace(np.min(x[:,0]),np.max(x[:,1]),2)
x2 = np.zeros((weights.shape[0], 2))
for ix_w, weight in enumerate(weights):
    x2 = 1 * ( - weight[2] - weight[0]*x1) / weight[1]
    plt.plot(x1, x2, c = colors[ix_w])

plt.xlabel('Feature 1')
plt.ylabel('Feature 2')

plt.xlim([np.min(x[:,0]), np.max(x[:,0])])
plt.ylim([np.min(x[:,1]), np.max(x[:,1])])

plt.show()

This code generalizes to the binary classification task as well

params = [[[ 5,1],  [ 5,1]],
          [[ 0,1],  [ 0,1]]]

Stop training on convergence

If you want to stop the algorithm based on convergence you can use a stop criteria. For example you can stop training once every weight in your matrix changes by less than a very small number. The very small number we usually choose is machine epsilon 2.220446049250313e-16, which is essentially zero. Sometimes this requirement is too stringent so it can be replaced by any number of significant values.

Change the get_weights code to include the break criteria as

from copy import deepcopy

def get_weights(x, y, n_epochs, verbose = 0):
    # Append a ones column to the feature for the bias
    data = np.ones((x.shape[0], x.shape[1]+1))
    data[:, 0:x.shape[1]] = x
    # Set the targets as integers for comparison
    targets = y.astype(int)

    # Initialize the weights as a matrix
    # number of classes by number of features
    weights = np.zeros((len(set(y)), x.shape[1]+1))
    past_weights = np.zeros((len(set(y)), x.shape[1]+1))
    for epoch in range(n_epochs):
        for i, target in zip(data, targets):
            temp = np.dot(i, weights.T)
            pred = np.argmax(temp)
            # If wrongly predicted update prediction
            if pred != target:
                weights[target, :] = weights[target, :] + i
                weights[pred, :] = weights[pred, :] - i

        if np.abs(weights - past_weights).all() < np.finfo(float).eps:
            break
        past_weights = deepcopy(weights)

        if verbose == 1:
            print('Iteration: ', epoch)
            print(weights)
            print('---------------------------------------------')
    return weights


RandomForestClassifier : binary classification scores

Referring to http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html, you are using teh predict method to derive the binary response.  If you call predict_proba, you will get an array of probabilities corresponding to the binary response.


Choose CNN architecture first, then optimize parameters - validation vs test performance to pick architecture?

You shouldn't choose the best model based on the performance on test set. You should run cross-validation and pick the best model from there and then assess performance on a test set. Otherwise you would just getting the best model and overfitting your test set.

Regarding imbalanced data sets, you should be aware of metrics used to assess your model in this case, such as recall, precision, etc., and not only accuracy.
In python, you can do StratifiedKFold, where you will split your data getting the same proportion for train and test set regarding target variable.


How can access to modify feature_importances of Random Forest Classifier model?

I'm not sure it's possible to "transfer" the feature importances from model to model in a Random Forest Classifier. Although I think there are two work-arounds you may be able to use.
The first option you have is to retrain the second model on the exact same training data. This will have the effect of both models having close to the same feature importances, though I think doesn't guarantee they will be exactly the same.
Another way would be to try knowledge distillation (https://arxiv.org/abs/1503.02531). Here you're basically making a target, or student, model train to not only match the hard labels, but also a set of soft labels, generated from a teacher network. In this case, the model that you want to transfer the knowledge to would be the student network. However, I believe this will involve writing a custom loss function, which I don't think is possible in sklearn. The paper I linked is in the context of neural networks so I also don't know if there would be any gaps in the performance between model types, as presumably, you would doing this with a random forest.
Just to confirm, I ran this short script that trains two models on the same data and it seems to be doing what you want it to:
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1000, n_features=4,
                           n_informative=2, n_redundant=0,
                            random_state=0, shuffle=False)

clf = RandomForestClassifier(max_depth=2, random_state=0)
clf.fit(X, y)
print('model 1 feature importances: ', clf.feature_importances_)

clf2 = RandomForestClassifier(max_depth=2, random_state=0)
clf2.fit(X, y)
print('model 2 feature importances: ', clf2.feature_importances_)

The results I got were:
model 1 feature importances:  [0.14205973 0.76664038 0.0282433  0.06305659]
model 2 feature importances:  [0.14205973 0.76664038 0.0282433  0.06305659]

Here you can see that the results were the exact same. I would imagine that the integrity of the transfer may change as the data changes. Make sure both models are initialized with the same random state if you want the feature importance values to match exactly.


Tokenization of data in dataframe in python

You can try with this:

import pandas as pd
import nltk

df = pd.DataFrame({'frases': ['Do not let the day end without having grown a little,', 'without having been happy, without having increased your dreams', 'Do not let yourself be overcomed by discouragement.','We are passion-full beings.']})

df['tokenized'] = df.apply(lambda row: nltk.word_tokenize(row['frases']), axis=1)


Pickle is not a format but a protocol to serialize a python object to binary format.
In the documentation of gensim, you can read they actually use pickle for serializing to binary format.
You can of course use pickle library instead of the .save method in gensim, but it will be exactly the same, since gensim already use pickle. I would recommend you to use .save method however, because gensim likely has better way to handle the hierarchical structure and dependencies of your model.


How to compare the output of Self-organizing maps?

From Wikipedia:

  A self-organizing map (SOM) or self-organizing feature map (SOFM) is a type of artificial neural network (ANN) that is trained using unsupervised learning to produce a low-dimensional (typically two-dimensional), discretized representation of the input space of the training samples, called a map, and is therefore a method to do dimensionality reduction.

So you only have the original data itself; no additional data (like labels in a supervised setting). If you are also say the result has to have two dimensions, you basically look at functions

$$f: X \rightarrow \mathbb{R}^2$$

where $X \subsetneq \mathbb{R}^n$ in most cases. You already mentioned quantization error.

Up to my knowledge there is nothing better measure which does not include getting more knowledge about the data itself by human inspection / using other datasets.

With human inspection you can, of course, tell for a given dataset and a given human if one mapping seems to make more sense.

You might also consider other dimensionality reduction techniques:

t-SNE
Auto-Encoders


Does a CNN think things inside the filter are collocated aka dependent on each other?

Yes, the filter will learn as if they are spatially co-located.
The main purpose of convolutions is to detect local features, where the notion of locality comes from the positions over which the filter is applied.
Some neural network building blocks that you could use are:

Position-wise dense layers: this applies a linear transformation to each of the elements. The result would be an element for each of the elements in the input. The dimensionality depends on the matrix used for the multiplication. This can be achieved also by applying a 1D convolution with a filter of length 1.

Max/avg. pooling over time: taking the maximum/average value over all elements in the input, separately for each input channel. This gives you a single element, as it collapses all the elements in one. A variation is to take the $k$ maximum elements.

Of course, you may want to exploit the relations between the elements in your input, but not in their original meaningless ordering: you could sort your inputs (from lowest to highest or vice versa). Then you could apply convolutional layers.


Fast checking of set membership on large data

Somebody already tagged this with "bloom-filter".  I agree with them.

A Bloom Filter is a hash-based technique that never results in false negatives, but trades off probability of false positives for saved space.  It meets your specific performance requirements - "fewer than 10 bits per element are required for a 1% false positive probability, independent of the size or number of elements in the set (Bonomi et al. (2006))." (from the linked article)

The main reason you wouldn't use a Bloom filter for this is that removing spammers is computationally expensive, requiring you to rebuild the filter or track your list of removed spammers separately.


XG Boost Regression to model Log of Dependent Variable

It's actually good practice to perform some type of scaling of your continuous variables. The box-cox transformation (or positive variable) or the yeo-johnson transformation (which adapts the box-cox transformation) are in the class of power transorms implemented in scikit-learn.
I you look at the implementation, you see that the log transformation that you applied is a particular case of box-cox.
Gradient boosting includes an optimisation step that will rely on computing gradients of your loss function. A recurrent theme in ML is how gradient descent have a harder time finding optimal solution when the data has a high order of magnitude. Essentially the information is there at high order of magnitude but the optimisation space is "stretched" - scaling your data brings everything closer if you will (that's a really gross explanation of what happens).
So essentially you've done a great job at making the most out of your data by transforming it to get a better model.
And this is actually an excellent case because, you often hear that when using decision tree you should not scale the data, and by that what is meant are the features. When using a decision tree model for regression you should absolutely consider scaling the target.


How can I select data from a Tensorflow Dataset data collection?

You can use the ds.map() function to create dataset conatains only images or labels:

ds_images = ds.map(lambda d:d['images')

the original purpose of the map function is manipulating the data without converting to numpy, for example:

ds_images = ds.map(lambda d:d['images']/255)

hope I helped you.