Calculation and visualization of islands of influence

As you've explained it, you want to detect communities, which is a common problem in social network analysis. If your network had been directed and temporal, I would have suggested you look into influence analysis; e.g., as in Detecting Communities of Authority and Analyzing Their Influence in Dynamic Social Networks.


Filter phrases based on correspondent POS tags

Hey,

Here's how i would solve this.
The problem with regex is that you don't have any index so I find it better to convert your inputs into lists of single words/tags by splitting on spaces. Then it is easier to find the pattern in the tag list and get the matching words.

words = [word for key in key_list for word in key.split()]
tags = [tag for value in value_list for tag in value.split()]

pattern = ['JJ', 'JJ', 'NNS']
matches = []

for i in range(len(tags)):
    if tags[i:i+len(pattern)] == pattern:
        matches.append(words[i:i+len(pattern)])

matches = [' '.join(match) for match in matches]
print(matches)

# result : ['partial differential equations', 'partial differential equations']

Also, if I'm not mistaken your pos-tagger marked "differential" as a noun (NN) in "ordinary differential equations". Nltk tagger does the job and it is really fast to implement but it is not perfect.

You can try using another tagger, I personally use the Stanford one for nlp processing.


SHAP value analysis gives different feature importance on train and test set

Since SHAP gives you an estimation of an individual sample (they are local explainers), your explanations are local(for a certain instance)
You are just comparing two different instances and getting different results. This is normal and can happen in train and test set. This doesn't mean also that your train and test set have bad split, they could be good split.
In the end SHAP is done to help you understand how the model behaves in a particular instance. It should be done where you are interested in understanding. I guess that you can also try to find what is the difference between train and test with shap values, but they are local explainers so you might not find much success.
I wouldn't say anything about the quality of predictions given the feature importance.


Which recommender system approach allows for inclusion of user profile?

Try matrix/tensor factorization methods and data fusion.


Are there more layer types like convolution layers and fully connected layers?

Jan van der Vegt did a good job of mentioning some of the main irregular layer types, but I'll toss in a couple of others for the sake of completeness. I know from experience that layers in Teuvo Kohonen's Self-Organizing Maps can be fully connected, without requiring a square or rectangular matrix. I've gotten excellent results by creating topologies that match the underlying problems, which may feature jagged and possibly multidimensional shapes, then operating on them with sets instead of matrices. That's one reason I prefer using set-based languages like SQL to matrix math when coding neural nets. Another interesting neural net type worth exploring is Neural Gas, which expands across the space of data points to fit the natural topology of a problem, whatever that may be. It is easier to express in terms of manifolds than ordinary space. In a sense, it can be viewed as a generalization and abstraction of Kohonen nets.

Resources on Kohonen nets are easy to find, but the literature on Neural Gas is still scarce. Here are some worthwhile links I've read to Neural Gas and assorted variants:

• Martinetz and Schultern's original article on Neural Gas.

• The Wikipedia article on Neural Gas.

• A Master's Thesis on Growing Neural Gas, which dynamically creates the topology

• A Growing Neural Gas Network Learns Topologies, a research paper by Bernd Fritzke


How to calculate Temperature variable in softmax(boltzmann) exploration

the reason you get error is that when temperature annealed, the softmax would obtain a big value which the exponential cannot calculate. The softmax would approximate the value into nan or 0 or 1 which causes the error. I got the same problem and still fixing it. In my case, I just make sure the reward is not too big which can guarantee the softmax a not big q value.


R - time series decomposition without detection of seasonality

Without seeing your data it is hard to tell whether there is seasonality or not. The decompose() function will try to find seasonality using a different approach than tbats() as discussed in this post and the user comments of this blog post by the author of tbats.

I would recommend to post your data and discuss what you are trying to achieve to get more guidance.


Classification of phone numbers belonging to same client

So, finally i managed to come up with a plausible solution.

First of all, i built a dataset by combining each phone number pair, and calculated some mutual features, like proportion of same location per day, wether it is the same device type or not, the group of base station, and etc.

Having done that, i realised that i only have a positive class, hence, can't employ binary classification straight away. To solve this problem i used what's called a p-u (positive-unlabeled) classification, in which i trained 1000 decision trees with my labeled data points as positive class, and random sample from unlabeled data as negative class. Then, i averaged predictions for each entry.
This approach has resulted in decent F-score.

Here is a resource where you can learn more about pu classification: pu-classification


A model underfits when it is too simple with regards to the data it is trying to model.

One way to detect such situation is to use the bias–variance approach, which can represented like this:

Your model is underfitted when you have a high bias.

To know whether you have a too high bias or a too high variance, you view the phenomenon in terms of training and test errors:

High bias: This learning curve shows high error on both the training and test sets, so the algorithm is suffering from high bias:

High variance: This learning curve shows a large gap between training and test set errors, so the algorithm is suffering from high variance.

If an algorithm is suffering from high variance:

more data will probably help
otherwise reduce the model complexity

If an algorithm is suffering from high bias:

increase the model complexity

I would advise to watch Coursera' Machine Learning course, section "10: Advice for applying Machine Learning", from which I took the above graphs.


How can we handle cases were the quality of the prediction undermines the quality of the data?

Because of this, more and more patients who would have been classified
  as high risk are actually not catching the disease and therefore
  decreasing the accuracy of your model.

I'd rather challenge the assumption and say the algorithm didn't miss in this case: it correctly identified a high risk case, which was its original purpose. You're not building a model to identify people who will catch the disease, but to prevent the disease of actually happening.

Always think hard about which metrics make sense and what they represent. If you build a confusion matrix:

Predicted \ Actual | Caught the disease | Did not catch the disease
High risk          | A                  | B
Low risk           | C                  | D

Now, when you use accuracy, you're saying A and D are good, B and C are bad, all with the same weights. But is B really bad? Isn't C way worse than anything else? Or depending on the cost to prevent the disease, maybe B is worse? I don't know, it requires domain knowledge. But point being, using accuracy mindlessly is not a good way of dealing with this situation: adjust your metrics to your problem.


Model trained on CIFAR 10 performing badly when fed with internet downloaded images

Here are some possibilities that come to mind:

The ordering of the dimensions. Depending on the network you are training with and the original training data, you may need to transpose some of the dimensions and/or reshape them differently. The original cifar 10 data is a little strange in that the color-dimension precedes the height/width dimensions. It may help to visualize both an image from the training data and the internet data side by side with the same function and ensure they are displaying the same way as a test of this.
Was any pre-processing performed on the training images, for example rescaling of values? If so, the same pre-processing should be performed on these images. Even if pre-processing was not explicitly performed image data can be stored in a variety of different ways. I would check the range of values on your training/validation images and compare against the images you are getting from the net.
Something funky is possibly happening when these images are resized. I would suggest visualizing the above examples after they have been resized and confirming that they are reasonable.
Another possibility is that something unexpected is happening when loading the model - it may seem redundant given the validation results but I would nonetheless take a set of 20 or so images from the original validation set and check the predictions after retrieving the model with the above load_model function.


A good reference for the back propagation algorithm?

Perhaps we can forget the graph idea to start and just go through the mathematics of it and then we will map our computations to a type of graph just so that it makes it easier to scale.

One neuron network is called the perceptron

The first neural networks only had a single neuron which took in some inputs $x$ and then provide an output. A common function used is the sigmoid function

$\sigma(z) = \frac{1}{1+exp(z)}$

$\sigma(w^Tx) = \frac{1}{1+exp(w^Tx + b)}$

where $w$ is the associated weight for each input $x$ and we have a bias $b$.

How do we train these weights

We will use gradient descent to train the weights based on the output of the sigmoid function and we will use some cost function $C$ and train on batches of data of size $N$.

$C = \frac{1}{2N} \sum_i^N (\hat{y} - y)^2$

$\hat{y}$ is the predicted class obtained from the sigmoid function and $y$ is the ground truth label. We will use gradient descent to minimize the cost function with respect to the weights $w$. To make life easier we will split the derivative as follows

$\frac{\partial C}{\partial w} = \frac{\partial C}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial w}$.

$\frac{\partial C}{\partial \hat{y}} = \hat{y} - y$

and we have that $\hat{y} = \sigma(w^Tx)$ and the derivative of the sigmoid function is $\frac{\partial \sigma(z)}{\partial z} = \sigma(z)(1-\sigma(z))$ thus we have,

$\frac{\partial \hat{y}}{\partial w} = \frac{1}{1+exp(w^Tx + b)} (1 - \frac{1}{1+exp(w^Tx + b)})$.

So we can then update the weights through gradient descent as

$w^{new} = w^{old} - \eta \frac{\partial C}{\partial w}$

where $\eta$ is the learning rate.

What if we have 2 layers?

Take a look at the following answer: Compute backpropagation

Why model neural networks as a graph

In graph theory nodes are connected by vertices. Each neuron takes in some inputs and produces an output. The inputs to this node are not the same as the outputs of the previous layer. They are affected by some weight. Thus we can consider these weights to be the vertices of the graph that link nodes together.


Does encoding missing data with fixed values help in classification?

You can treat the mere presence of any value as signal - hence the 0 or 1.

What help this would be to your project depends on your dataset.


Unbalanced training set from balanced data

With a straightforward approach, you can just oversample one of the classes to get the imbalance. You can achieve it with using SMOTE for example. Usually this technique is used to get a balanced set from imbalanced, but it can also work vice-versa, just oversample only once class.
Some links to check:
https://imbalanced-learn.org/stable/over_sampling.html
https://machinelearningmastery.com/random-oversampling-and-undersampling-for-imbalanced-classification/
https://towardsdatascience.com/how-to-effortlessly-handle-class-imbalance-with-python-and-smote-9b715ca8e5a7


This problem is called record linkage, there are various techniques which can be used, usually involving some distance measure between record and/or approximate string matching between string fields.

Fyi it's a quite complex problem, especially if quality deduplication is expected and the volume of data is high.


Bayesian optimization with Keras tuner for time series

I finally came out with a way to implement bayesian hyperparameter optimization for a time series neural network model using walk-forward validation. You can find a gist with the code here

I used some helper functions for the walk-forward validation from this Jason Brownlee book

It is basically as follows:

original time series data:

define a function to convert your time series dataset into a supervised dataset format, another one for the train-test split (ordered in time as done in sklearn.model_selection.TimeSeriesSplit)

define your evaluation metric, root mean squared error in this case
we also add a differencing function, in case we want to make our time series stationary (by differencing, we can reduce trend and/or stationality)
define the walk-forward validation functions (walk_forward_validation and repeat_evaluate)
define the keras tuner bayesian optimizer, based on a build_model function wich contains the LSTM network in this case with the hidden layers units and the learning rate as optimizable hyperparameters
define the model_fit function which will be used in the walk-forward training and evaluation step

lastly, find the evaluation metric value and std

To keep in mind: this is a proof-of-concept example about how to use keras tuner for time series using walk-forward validation. Further analysis could be done to improve forecasts.


word/sentence alignment for English document

Use kaldi's align-text which align two sentences using Levenshtein distance.

Code: https://github.com/kaldi-asr/kaldi/blob/master/src/bin/align-text.cc


can you suggest best method for background / ambient noise removal from cough audio and keep only cough signal

First let's see some data: typical cough data for someone with pneumonia:

image source. This source paper also contains more information on processing cough data.

The simplest way I can think of would be to analyse and extract the average/base background noise of an many samples as you can, then simply subtract the background noise from the entire sound wave. This background noise is essentially the area outside of the start and end lines indicated on the image above.

Outline of steps will need to:

define the start and end of the cough itself (as per the image)
measure the average magnitude of the background: e.g. np.mean(np.abs(background))
subtract the average background noise from the sound clip

If you can plot the data again, you should notice that the lines outside of start and end are almost flat (but not exactly flat, we only subtracted the average background noise)

Links:

Have a look at this answer, which goes through similar steps in more detail
Here is a great list of libraries you can use to work with sound data in Python
Audacity (software package) has a lot of tools and is scriptable, using Python


Weighting MSE is a way to give more importance to some prediction errors than to others in the overall score. This is useful if you are using MSE as a performance metric for your model, especially during the model training (loss function) or validation (hyper-parameter setting).

In the case you cite as example, more importance is given to cases with more clicks. If you use this WMSE as performance metric for validation, your model will tend to be better on cases with high number of clicks, than if you had used MSE.

Just a note on your example: the used squared error ($(predictedClicks - observedClicks)^2$) is an absolute squared error (I would have expected a relative error), and therefore already increases with the number of clicks. So in this case, weighting that way reinforces the performance on cases with high number of clicks.


I solved this problem by using a keras function:
torch.tensor(tf.keras.preprocessing.sequence.pad_sequences(doc1,maxlen=longest_sent,padding='post')

torch.tensor(tf.keras.preprocessing.sequence.pad_sequences(doc2,maxlen=longest_sent,padding='post')


Using Linear Regression to Learn Polynomial Regression

It is quite simple to understand (and to implement using matrices).
Consider a specific example (to generalise later). You have a polynomial function of a single feature $x$):
$$ f(x) = \omega_0 x^0 + \omega_1 x^1 + \ldots \omega_n x^n $$
You can organise coefficients and features in vectors and get $f$ by a scalar product:
$$ \mathbf{\omega} = \begin{pmatrix} \omega_0, \\ \vdots \\ \omega_n \end{pmatrix}, \qquad \mathbf{x} =  \begin{pmatrix} 1, \\ x \\ x^2 \\ \vdots \\ x^n \end{pmatrix}$$
Hence $$ f(x) = \omega^T\mathbf{x}$$.
This is nothing else than a multi-feature linear regression where the $i$-th feature is now the $i$-th power of $x$.
In numpy, imagine you have an array of data x.
To create the vector $\mathbf{x}$ above, you can do (for $n=3$, for instance)
X = np.ones((len(x), 4))
X[:,1] = x
X[:,2] = np.power(x,2)
X[:,3] = np.power(x,3)

And then using sklearn LinearRegression,
model = LinearRegression()
model.fit(X, y)

UPDATE: In sklearn has been recently introduced PolynomialFeatures that precisely performs the transformation I described in numpy (you asked in numpy, but this might be useful as well).


Is deep learning a must in a Data Science MSc programme?

No, it's not problematic. Most data scientists do not need or use deep learning. Deep learning is very popular right now, but that does not mean it's widely used. Deep learning can lead to substantial overfitting on small to medium datasets (I'm arbitrarily going to say that means less than 2 GB), which are the sizes that most people have. Deep learning is primarily used for object recognition in images, or text/speech models. If you're not doing either of these two things, you probably don't need to use DL.


Problem regarding calculating correlation approach?

Most probably, you are using Pearson's correlation method. This method is used for two Continuous features.
Here, both the price_drop and the OHE features are Binary Categorical features.
So, you can use these methods -
Phi - Phi is a measure of the degree of association between two binary variables      (two categorical variables, each of which can have only one of two values)
Crammer's V - Cramer’s V is an extension of phi for tables larger than 2×2.
Both are extensions of the Chi-square test of Independence.
Since both the Features have 2 values, both of the above methods will output the same result.
# dataset is your DataFrame
s1 = dataset['Status']
s2 = dataset[product_type_OHE_01]

import pandas as pd
from scipy.stats import chi2_contingency
n = len(s1)
r,c = s1.nunique(), s2.nunique()
matrix = pd.crosstab(s1,s2).values
chi_sq = chi2_contingency(matrix)

phi = np.sqrt(chi_sq[0]/n)
cramm_V = np.sqrt(chi_sq[0]/(n*min(r-1,c-1)))

print(phi, cramm_V)


You can connect the output of the sentiment analysis to Test&Score widget together with SVM learner. Test&Score widget can perform different validations (e.g. cross-validation).

If you want to perform testing on different data please use Predictions widget. For more information please check widgets help available with pressing f1 shortcut in Orange.


Python Matplotlib Radar not correctly centered

Seems to me that fig.add_axes() is miscalculating the dimensions here, essentially doubling the "left" and "bottom" values, not sure why. So you should use 0.8 for "right" and "top" values, as in:

axes = [fig.add_axes([0.1, 0.1, .8, .8], polar=True,
            label="axes{}".format(i))
            for i in range(len(variables))]


Every ML algorithm with high complexity can overfit. However, the OP is asking whether an RF will not overfit when increasing the number of trees in the forest.

In general, ensemble methods reduces the prediction variance to almost nothing, improving the accuracy of the ensemble. If we define the variance of the expected generalization error of an individual randomized model as:

From here, the variance of the expected generalization error of an ensemble corresponds to:

where p(x) is the Pearson’s correlation coefficient between the predictions of two randomized models trained on the same data from two independent seeds. If we increase the number of DT's in the RF, larger M, the variance of the ensemble decreases when ρ(x)<1. Therefore, the variance of an ensemble is strictly smaller than the variance of an individual model.

In a nutshell, increasing the number of individual randomized models in an ensemble will never increase the generalization error.


What is exactly meant by neural network that can take different types of input?

In short, can I create neural network that takes different types of
  input individually or do I use different neural networks for each of
  them and then group their outputs?

Yes, you can. Check the Functional API of Keras, on how to define multi input/output networks. Then you can create different models for the processing of each input and fuse them together into a single multi-input model using the keras.models.Model() class.

In the following example, you can see that the main_input is processed differently than the aux_input and both are thereafter merged together to be propagated through the rest of the layers of the network.


Training Keras model with multiple CSV files

I would use tensorflow 2.0 with tf.data
import tensorflow as tf

filenames = ["filename1", "filename2", ...]

dataset = tf.data.Dataset.list_files(filenames, seed=42, shuffle=True)

# this reads 5 text files at a time, skips the first row of each file
dataset.interleave(lambda filename: tf.data.TextLineDataset(filename).skip(1), cycle_length=5, num_parallel_calls=tf.data.experimental.AUTOTUNE)

for line in dataset.take(5):
    print(line)

You can then pass in the dataset objects directly into tensorflow.keras for training.
There are some good examples here but please beware that this is NOT for tensorflow 2.0, and somethings may have changed


Why is there a difference of "ML" vs "MLLIB" in Apache Spark's documentation?

An RDD is a Read-only partition collection of records. RDD is or was the fundamental data structure of Spark. It allows a programmer to perform in-memory computations on large clusters in a fault-tolerant manner. Thus, speed up the task.

A dataframe on the other hand organizes data into named columns. It is an immutable distributed collection of data. DataFrame in Spark allows developers to impose a structure onto a distributed collection of data, allowing higher-level abstraction.

The two are very different and more details on the two can be found on this link https://data-flair.training/blogs/apache-spark-rdd-vs-dataframe-vs-dataset/

mllib was in the initial releases of spark as at that time spark was only working with RDDs. Now a lot of Spark coding is done around dataframes, which ml supports. Spark has also put mllib under maintenance.

  As of Spark 2.0, the RDD-based APIs in the spark.mllib package have
  entered maintenance mode. The primary Machine Learning API for Spark
  is now the DataFrame-based API in the spark.ml package.

  Q. What are the implications?

  MLlib will still support the RDD-based API in spark.mllib with bug
  fixes. MLlib will not add new features to the RDD-based API. In the
  Spark 2.x releases, MLlib will add features to the DataFrames-based
  API to reach feature parity with the RDD-based API. After reaching
  feature parity (roughly estimated for Spark 2.3), the RDD-based API
  will be deprecated. The RDD-based API is expected to be removed in
  Spark 3.0.

  Q. Why is MLlib switching to the DataFrame-based API?

  DataFrames provide a more user-friendly API than RDDs. The many
  benefits of DataFrames include Spark Datasources, SQL/DataFrame
  queries, Tungsten and Catalyst optimizations, and uniform APIs across
  languages. The DataFrame-based API for MLlib provides a uniform API
  across ML algorithms and across multiple languages. DataFrames
  facilitate practical ML Pipelines, particularly feature
  transformations. See the Pipelines guide for details.

  Q. What is “Spark ML”?

  “Spark ML” is not an official name but occasionally used to refer to
  the MLlib DataFrame-based API. This is majorly due to the
  org.apache.spark.ml Scala package name used by the DataFrame-based
  API, and the “Spark ML Pipelines” term we used initially to emphasize
  the pipeline concept.

  Q. Is MLlib deprecated?

  No. MLlib includes both the RDD-based API and the DataFrame-based API.
  The RDD-based API is now in maintenance mode. But neither API is
  deprecated, nor MLlib as a whole.

You will find more details on Sparks official release https://spark.apache.org/docs/latest/ml-guide.html#announcement-dataframe-based-api-is-primary-api


Default parameters for decision trees give better results than parameters optimised using GridsearchCV

My Suggestion:

The intrinsic separation of classes needs more complex model to be captured. I say this, because the difference between default model and your grid search is in max_depth parameter which is one of complexity indicators in Decision Trees. The default is None so it uses the maximum complexity it can get from max_depth but your parameter values are at most 10. To check this, you may try increasing the max_depth in your grid search (or leave it None) and see the result of grid search. If it improves then this is the point.


How to encode ordinal data before applying linear regression in STATA?

I think the best way is to dummy-encode teacher qualification. So each level of qualification enters the regression with a separate intercept term. Note that dummy-encoding always works against a contrast level. So when "Master degree" is the base-level, you will see the effect of "Bachelor" compared to "Master" etc.
You can dummy-encode in Stata by using the i. prefix, e.g. summarize i.size. In a regression you would use reg y i.x.
See the Stata docs for details.


Methods of disaggregating data to smaller units?

From a statistical point of view this is impossible if one doesn't have any data at the fine-grained level. Any statistical inference must be based on a sample from which specific patterns can be observed.
If there is no data at the fine-grained level, any calculation is based on assumptions. For example one may assume that a variable is proportional to population (linear relation). But why not assume that the variable is a polynomial function of the temperature? Or that it is related to the prevalence of a particular gene? The main issue is that without any data there's no way to test any of these assumptions, so no there can be not reliable conclusion.


I'm not sure I understand your problem very well but let's see.
First let me try to formalize the task as a ML problem:

Identifying the SMS of interest is a binary classification task. Your "completeness" score seems to correspond to the standard recall measure. It is usually a good idea to also look at precision, i.e. out of the SMS identified as relevant how many are truly relevant.
You don't really explain the second part of extracting information out of SMS of interest. In general this part might be some kind of sequence labelling problem (similar to Named Entity Recognition).

It's important to distinguish these two parts if you're going to use ML.
Now what you are currently doing is a rule-based system: you manually identify the patterns one by one, then the system just applies the list of patterns. rule-based systems are considered the ancestor of ML systems: as you noticed, it might be possible to automatize the part of identifying the patterns. Given that you already have labelled examples (training data), you could train a model to recognize the general patterns by itself. The main difficulty is to provide the model with the right features so that it can do the job as correctly as possible.
Note that in general it doesn't really work as a list of templates in the same way as what you do manually.
The question of identifying cases without looking at the whole corpus might be related to ''semi-supervised learning'', where a system iteratively learns to annotate unlabelled data from an small initial set of labelled data.


do the results change if you remove duplicate rows and you sum their weights?

With weighted linear regression, it is exactly the same, as the expression for the loss function is a sum of weights multiplied by errors in the prediction. This works, of course, for other methods with loss functions, such as logistic regression and neural networks. This is due to the fact that the loss function is linear with respect to the weights. As you save memory, it is totally recommendable.

With other methods, you should check if the criteria for choosing parameters or method is linear or not with respect to the weights. If not, you should not do it (to me it doesn't make sense that methods are not linear with respect to weights, but there might be a case where this happens).


Why replay memory store old states and action rather than Q-value (Deep Q-learning)

But since we already compute the Q-value of action a in state old_state in order to choose a as the best action, why don't we simply store directly the Q-value ?

That is because calculating a relevant TD Target e.g. $R + \gamma \text{max}_{a'}Q(S', a')$ requires the current target policy estimates for action value $Q$. The action value at the time of making the step can be out of date for two reasons:

The estimates have changed due to other updates, since the experience was stored
The target policy has changed due to other updates, since the experience was stored

Storing the $Q$ value at the time the experience was made should still work to some degree, provided you don't keep the experience for so long that the values are radically different. However, it will usually be a lot less efficient, as updates will be biased towards older less accurate values.

There is a similar, but less damaging, effect from the experience replay table even with $Q$ recalculations. That is because the distribution of experience may not match what the current policy generates - something that most function approximators (e.g. neural networks used in DQN) are sensitive to. However, there are other factors in play here too, and it can be beneficial to train on a deliberately different distribution of experience - for instance prioritising experiences with larger update steps can speed learning and keeping older experiences available can reduce instances of catastrophic forgetting.

Note that if you were using an off-policy Monte Carlo method, you could store the Monte Carlo return in the experience replay table, since it does not bootstrap by using current value estimates. However, the early parts of older less relevant trajectories would stop contributing to updates in that case once the target policy had changed significantly during learning.


Is a neural network suitable for this application

You may or may not apply neural network to your problem. Neural network will give predictions for sure but it can't be said how much efficient will it be for this problem. You have to code and check it for yourself. Also, if you have all the above data in tabular form and you know the labels(which I guess is valid/inavlid), I will say try it with xgboost. Neural networks are exceptional for unsupervised learning but in the case of supervised learning, there may be a model which may outperform a neural network.


Generate dataframe series from current series which is a list of objects

One way of doing it is by creating an auxiliary function to extract the category from your tag and return it when found or 'unknown' otherwise. Then using .apply() with that function will do the trick:

json_string = '''{"submissionTime":"2019-02-25T09:26:00","b_data":{"bName":"Masato","b_Acc":[{"id":0,"transactions":[{"date":"2019-12-19","text":"PERIODICAL PAYMENT","amount":3397,"type":"","tags":[{"institution":"University of MC"},{"lenderType":"private"},{"category":"birdy"},{"creditDebit":"credit"}]},{"date":"2019-12-03","text":"LINE FEE","amount":-460.21,"type":"Overdrawn Fees","tags":[{"category":"Overdrawn"},{"creditDebit":"debit"}]},{"date":"2019-12-31","text":"INTEREST","amount":-871.62,"type":"Interest Charge","tags":[{"category":"Fees"},{"creditDebit":"debit"}]},{"date":"2019-12-31","text":"LOAN SERVICE FEE","amount":-120,"type":"Loan Related Fees","tags":[{"category":"Fees"},{"creditDebit":"debit"}]},{"date":"2019-12-18","text":"PERIODICAL PAYMENT","amount":3397,"type":"","tags":[{"institution":"University of MC"},{"lenderType":"private"},{"category":"birdy"},{"creditDebit":"credit"}]},{"date":"2019-12-02","text":"LINE FEE","amount":-498.34,"type":"Overdrawn Fees","tags":[{"category":"Overdrawn"},{"creditDebit":"debit"}]},{"date":"2019-11-29","text":"INTEREST","amount":-794.4,"type":"Interest Charge","tags":[{"category":"Fees"},{"creditDebit":"debit"}]},{"date":"2019-11-19","text":"PERIODICAL PAYMENT","amount":3397,"type":"","tags":[{"institution":"University of MC"},{"lenderType":"private"},{"category":"birdy"},{"creditDebit":"credit"}]},{"date":"2019-11-01","text":"LINE FEE","amount":-484.87,"type":"Overdrawn Fees","tags":[{"category":"Overdrawn"},{"creditDebit":"debit"}]},{"date":"2019-10-31","text":"INTEREST","amount":-882.04,"type":"Interest Charge","tags":[{"category":"Fees"},{"creditDebit":"debit"}]},{"date":"2019-10-21","text":"PERIODICAL PAYMENT","amount":3397,"type":"","tags":[{"institution":"University of MC"},{"lenderType":"private"},{"category":"birdy"},{"creditDebit":"credit"}]},{"date":"2019-10-01","text":"LINE FEE","amount":-503.59,"type":"Overdrawn Fees","tags":[{"category":"Overdrawn"},{"creditDebit":"debit"}]},{"date":"2019-09-30","text":"INTEREST","amount":-916.98,"type":"Interest Charge","tags":[{"category":"Fees"},{"creditDebit":"debit"}]},{"date":"2019-09-30","text":"LOAN SERVICE FEE","amount":-120,"type":"Loan Related Fees","tags":[{"category":"Fees"},{"creditDebit":"debit"}]},{"date":"2019-09-19","text":"PERIODICAL PAYMENT","amount":3397,"type":"","tags":[{"institution":"University of MC"},{"lenderType":"private"},{"category":"birdy"},{"creditDebit":"credit"}]},{"date":"2019-09-02","text":"LINE FEE","amount":-489.65,"type":"Overdrawn Fees","tags":[{"category":"Overdrawn"},{"creditDebit":"debit"}]},{"date":"2019-08-30","text":"INTEREST","amount":-892.13,"type":"Interest Charge","tags":[{"category":"Fees"},{"creditDebit":"debit"}]}]}]}}'''
js = json.loads(json_string)
df = pd.DataFrame(js['b_data']['b_Acc'][0]['transactions'])

def extract_category(tag):
    dall = {}
    # we create a new unique dict with all the items in the tag
    for d in tag:
        dall.update(d)
    # if category is in our new dict, return it else return unknown
    if 'category' in dall.keys():
        return dall['category']
    else:
        return 'unknown'

df['category'] = df.tags.apply(lambda x: extract_category(x))


I have personally only heard the letters being spelled out:

  "Here we see the R-O-C curve on this pronunciation model is far from optimal"


What does n means in neural network neuron output?

n is the dimension of the vector x and also y, as you can see wT is a transpose of w with dimension (n,n), is the image z is y and a is x. and dont bother about l it indicates the index of layer.


Quality check for preprocessing of Text data

Evaluating any task consists in defining the task formally so that there is a way to define what is a correct output as objectively as possible. For example a good Machine Translation system produces a good translation if it has the same meaning as the input sentence and is grammatically correct in the target language.
Assuming that this task of preprocessing is formally defined, then the evaluation should measure how "correctly preprocessed" is the output:

are the stem and lemma always correct?
are the stop words and only the stop words removed?
Etc.

Usually one would build a test set, manually add the correct output and then compare the system output against this gold standard.
However "preprocessing" is generally not considered a task by itself, because by definition it's a step for another task. Importantly, the steps of preprocessing depend on the other task, they are not always the same. For example stop words removal makes sense only for tasks based on distributional semantics, i.e. related to the topic. Preprocessing may also include steps which depend on the volume of data.


Pivoting a two-column feature table in Pandas

If there is no value column - introduce it yourself!

df["value"]=1
pd.pivot_table(df, values="value", index=["city"], columns="cuisine", fill_value=0)

For your example I got (after fixing the misprint in 'Japanse' to 'Japanese')

cuisine  Chinese  French  German  Japanese
city
NY             1       0       0         1
SF             0       1       1         1


This is a common result of imprecise matching, such as with whitespace problems.

For example, if you imagine that the grey shading used in codeblocks on StackExchange represents the whitespaces in your Excel cells, then
  AK
is technically the same as

AK

You can trim and/or use wildcard chars to allow matching to skip spaces. For example:

=VLOOKUP(CONCATENATE("*",TRIM(A2),"*"), $E$2:$F$5, 2,FALSE)


Dealing with extreme values in softmax cross entropy?

Where exactly in the computations are these underflows manifesting? See here for a brief explanation around the extremes of the softmax.

Quick fixes could be to either increase the precision of your model (using 64-bit floats instead of, presumably, 32 bit floats), or just introduce a function that caps your values, so anything below zero or exactly zero is just made to be close enough to zero that the computer doesn't freak out. For example, use X = np.log(np.max(x, 1e-9)) before going into the softmax.

In any case, the softmax shouldn't have problems with your input, as the final activations are exponentiated:

$$
\sigma (\mathbf {z} )_{j}={\frac {e^{z_{j}}}{\sum _{k=1}^{K}e^{z_{k}}}}
$$

This means all values will now be in the range [0, 1].

The cross-entropy equation should also be able to deal with the output of this.

If none of this helps your specific issue, could you share a specific example of your problem?


Dividing data set into (almost) equal batches based on characteristics

If I got this right you want to split the dataset into train and test sets in a way that preserves the same proportions of examples in each class as observed in the original dataset?
This is called a stratified train-test split. See the stratify argument here sklearn split


Q learning and Neural Network for Tic Tac Toe

It is my understanding that both methods should be achieving the optimal policy, can you confirm or deny my understanding?

Yes, I would expect a neural network for Q Learning to find the optimal policy, provided it remains stable*. The value estimates might be slightly inaccurate, but the resulting policy should be completely optimal. That is because in tic tac toe, all the value estimates should be -1, 0 or +1, and the data is cleanly separated.

You should be able to get a neural network to learn the optimal Q table from the first experiment using supervised learning. In fact that would be a good test of whether your NN has capacity to learn that table.

* Neural networks added naively to Q-learning agents are often not stable. In fact that is so common a problem in scaling up RL agents that it has a name: "the deadly triad". This is generally not solved by elegant mathematical changes to the agent, but by some engineering tricks:

Experience replay. Save observations (S, A, R, S') and sample from this memory table later to train in mini-batches.
Alternating networks. Use an old frozen copy of the neural network to calculate $\text{max}_{a'} Q(S',a')$ for the TD target $R + \gamma\text{max}_{a'} Q(S',a')$


What are the factors to consider when setting the depth of a decision tree?

Yes, but it also means you're likely to overfit to the training data, so you need to find the value that strikes a balance between accuracy and properly fitting the data. Deciding on the proper setting of the max_depth parameter is the task of the tuning process, via either Grid Search or Randomised Search with cross-validation.

This page from the scikit-learn documentation explains the process well: https://scikit-learn.org/stable/modules/grid_search.html


How can I implement deconvolution on CNN (TensorFlow)?

Deconvolution have very simple structure: unpooling → deconv
like this:

# Unpooling
Ps = (tf.gradients(pooled, h))[0]
unpooled = tf.multiply(Ps, P)

# Deconv
batch_size = tf.shape(self.input_x)[0]
ds = [batch_size]
ds.append(self.embedded_chars_expanded.get_shape()[1])
ds.append(self.embedded_chars_expanded.get_shape()[2])
ds.append(self.embedded_chars_expanded.get_shape()[3])
deconv_shape = tf.stack(ds)
deconv = tf.nn.conv2d_transpose(
    unpooled,
    W,
    deconv_shape,
    strides=[1, 1, 1, 1],
    padding='VALID',
    name="Deconv"
    )


Predict method of the perceptron algorithm

Here is how the code appears to break down:

method parameters:

The train(...) method takes an input vector called pattern along with the target output called label and a weight update coefficient (learning rate) called step.

method execution:

train(...) first passes the input vector through our neuron using the predict(...) method.
The result of predict(...) is stored in a variable named out_label.
The out_label variable is either 1 or -1.
Depending on the pattern that we provide we want the target output (label) to be either 1 or -1.
If out_label is equal to label (1 == 1 or -1 == -1), then we return True
Otherwise, the neuron outputted a value we do not want (-1 != 1 or 1 != -1), so we update the weights accordingly, then return False

example where the neuron is correct:

Let's say we have the following parameters:

input vector (pattern) = [-5,2,8]

target output (label) = 1

The train(...) method executes as follows:

we call the predict(...) which does the following:

passes our vector [-5,2,8] through the neuron using the function np.dot(self.w, pattern) + self.b
stores the result of the above function in a variable called activation (let's say activation = 0.23)
returns 1 because activation > 0

we store the result of predict(...) in the variable out_label, which is 1 in this example
because our target output (label = 1), is equal to the actual output (out_label = 1) of the neuron, we return True and consider the neuron output correct.

example where the neuron is incorrect:

Let's say we have the following parameters:

input vector (pattern) = [-1,-5,3]

target output (label) = 1

The train(...) method executes as follows:

we call the predict(...) which does the following:

passes our vector [-1,-5,3] through the neuron using the function np.dot(self.w, pattern) + self.b
stores the result of the above function in a variable called activation (let's say activation = -0.65)
returns -1 because activation <= 0

we store the result of predict(...) in the variable out_label, which is -1 in this example
because our target output (label = 1), is not equal to the actual output (out_label = -1) of the neuron, we update the weights using:
self.w += step * (label - out_label) * pattern
self.b += step * (label - out_label)
we then return False because the neuron was incorrect


Understanding why in deep reinforcement learning correlations in the data reduce the effectiveness

One reason can be the high correlation between training data can cause bias the deep learner. In this way, if the learner gets a new data which is not correlated with the training data, it will have large error.


How to form and minimise custom features for classification in supervised learning

https://en.wikipedia.org/wiki/Support-vector_machine

polynomial kernel of degree 2, with C coefficient >.1 but you would have to figure it out based on your data

I can provide more info on request


Clustering Tweet Data using DBSCAN Algorithm

Two things: eps and quantitative representation of text.
You see that there is only for eps=0.15 a lot of clusters. But for others a lot less. This is hyper parameter that needs to be optimised (and min_samples)
And the other thing thats more important is what you use quantitative representation of text. You said Bag of Words, TFIDF, Spacy Vectors and also, Word2Vec, but did you tune them? DId you tree embeddings etc etc. There is a lot of improvement here, and when its good dbscan will function a lot better.


How can I create a classifier using the feature map of a CNN?

You can use restricted boltman machine to obtain feature map, and then build a CNN with this architecture : conv layer -> pooling->conv layer pooling-> ....->fully connected layer.   Add conv layer and pooling layer until the matrix size reduced into $1\times 1$, then treat them as input for your fully connected layer, then train the fully connected layer


How does multicollinearity affect neural networks?

Multi colinearity affects the learning of Artificial Neural network.  Since the information in the dependent variable is very less compared to the other variables, the neural network will take more time to converge.

In packages like sklearn, the dependent variables are identified and omitted from the calculation.  I have used the lm function in R and it marks the coefficient of the dependent variable with NA.  one can remove the variable from the calculation and still the coefficients are going to be same.  In these cases, the rank of the x matrix will be less than the number of columns.

Even though there are no inverse exists for xTx, most of the packages will not calculate the inverse directly, but they will calculate the pseudo inverse.


If a categorical feature only occurs a few times in a data set, should I drop it?

Indeed it's often a good idea to remove boolean features which are very rare, but the problem is that choosing a threshold by intuition is not necessarily optimal. Whenever possible the optimal value should be determined experimentally, and typically that should be possible for efficient methods such as log regression or SVM. The idea is simply to consider a range of values as threshold and run a grid search using a subset of the training set. The threshold is exactly like an hyper-parameter for the learning method.


When using Data augmentation is it ok to validate only with the original images?

You should validate only on the original images. The augmentation is there so that it can help your model generalize better, but to evaluate your model you need actual images, not transformed ones.

To do this in keras you need to define two instances of the ImageDataGenerator, one for training and one for validating. To train the model you need to set both generators to the fit_generator function.

train_gen = ImageDataGenerator(aug_params).flow_from_directory(train_dir)
valid_gen = ImageDataGenerator().flow_from_directory(valid_dir)

model.fit_generator(train_gen, validation_data=valid_gen)

It is possible to achieve a higher validation accuracy than a train accuracy if you heavily augment the training data.


There are plenty of different approaches you can use, and none is the universal best solution. However, in general, preprocessing in twitter data, especially for Doc2Vec, follow:

tokeninzing (nltk tokenizer, custom regex tokenizer) to identify words. Depending on the application, you can work on special cases such as english contractions, negations (sentiment analysis especially) and others
normalizing urls and/or mentions, transforming each of them to the same version to reduce voacbulary size (http://someurl, @mention) or totally removing them.
removing of numbers, punctuation or others. Note that this is highly domain dependent, for example "!" has been shown to express a lot in sentiment analysis.

In fact, doc2vec doesn't expect anything, it uses the vocabulary you give. So you decide what you want to keep. A good practice is to experiment different tokenizers and training phases if you have time to see what works best. Also use what has already been done in your field of application.

Don't forget that the input of Doc2Vec is an iterator over a list of TaggedDocument. See this tutorial for more.

Good luck !


Why does the non autoregresive transfomer model in fairseq require the prev_output_tokens input?

It is there to maintain consistency with the signature of the forward method of the base class TransformerModel and therefore allow to use it in place of any other autoregressive transformer, but it is actually not used. The same happens in the model decoder.


(ii) means after splitting the data you end up with a single instance in the node, so there is nothing left to do

(iii) means after splitting the data all the samples in the node are identical ("duplicates"), and again there is nothing left to do

  Assuming all instances are distinct, each instance is isolated to an
  external node when an iTree is fully grown, in which case the number
  of external nodes is n and the number of internal nodes is n − 1; the
  total number of nodes of an iTrees is 2n − 1

Isolating an instance means splitting the data until that instance ends up in a leaf by itself. If you have $n$ instances, you will need $n$ external ("end") nodes. One for each instance. The internal nodes are all the nodes before the end nodes, including the root that contains all the instances. One can show that their number is $n-1$. For example, if you have $n=4$ you can split it in (4)-(2,2)-((1,1)(1,1)). So you have $1+1+1+1=4=n$ end nodes, and $1+2=3=n-1$ inner nodes.

Of course, the splits don't have to be equal. It could also be something like (4)-(3,1) Where you then split the (3) into (2,1) and the (2) into (1,1) But the counts will be the same. You'll have 3 inner nodes (4),(3) and (2) and 4 external nodes. (Best to draw it out)

The total is therefore the number of end nodes plus the number of inner nodes $n +(n-1) = 2n-1$.


What you are experiencing is called overfitting and it happens because of your very small dataset.

All the model cares about is performance on the training dataset, so given the opportunity, it will simply attempt to memorize it. This is what happens in you case, you feed a model which contains over 130 Million parameters less than 319 images.

So regarding your questions:

The loss function shows a clear case of overfitting.
On general, it is okay to use a trained model, especially when you only have a small dataset, but in your case, the dataset is too small for any deep-learning model. When I say small dataset, I mean 10k images, not several hundreds.
You should not train for longer time, once the validation loss stops improving, it is a clear sign to stop. There is even a training technique named "early stopping" which is designed to stop training once the validation loss stops to drop.

You have to understand that currently, your dataset of 300 images, is irrelevant to the world of deep-learning. So if you still want to use it for object detection, you need to revert to more classic computer-vision techniques like using HOG or SIFT features, or even manually engineering the features for your special case.


What is the best way to visualize data that is over 100% capacity

If you want to see small numbers and large numbers on the same chart, you can use a log() scale for the y axis. This will work for bar charts, or other types of charts. Log scaling will show more details than linear over a larger range. Just need to point this out to the user as they may be used to seeing linear scales. For something that changes over time, it can be helpful to show a time based plot, with any data over the limit in red and the data under in green or blue. Also you can add a horizontal line showing the cutoff value.
Example:
Linear vs Log

Flag with red points including a horizontal line to show the cutoff


Omitting the technical details, boosting is a statistical technique where we train many additional weak models to attempt to correct the errors of the previous models we have learned so far. In each iteration of gradient boosting, a new model is trained (usually a decision tree) which tries to fit to the residual of the prediction made by the existing set of models.

The init_score value represents the prediction that you are trying to correct with the first boosting iteration. By default this probably predicts the majority class for all examples or a randomly selected class, but you can input the prediction that was outputted by any other model here if you like.

The first tree that is learned by LightGBM will try to correct the errors of this initial prediction. In the example that you linked, they train a LightGBM model for one iteration, and use the score to continue training another LightGBM model. So in practice, what they have done in this example is kind of pointless because they could have just trained a single LightGBM from the start, but I suppose it is just to illustrate the concept and how it works.


Should I Plot my Learning Curve Depending on Various Parameters to Diagnose Overfitting/Underfitting or is One Parameter Sufficient?

Overfitting is a result of the entire model. It is best to visualize every part of the model to understand how the combination of elements is overfitting.

Ideally, you should visualize all elements on the same interactive figure so how the combination is overfitting. If that is not possible or the interpretation is difficult, then many single element visualizations would be an option. However, that increases the possibility of missing interactions causing overfitting.


train half model in one time and other in another time

You can save a keras model during the training and then load it up again. To save a model you can do so using a json format, and then to save the weights you can do so using a callback as follows:

Let's make some random model,

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

Then we can save the model that we created itself

# Save the model
model_json = model.to_json()
with open("weights/model.json", "w") as json_file:
    json_file.write(model_json)

And we can use a callback to save the weights at some given checkpoint. In this example they are saved each time we get better model performance.

# Save the weights using a checkpoint.
filepath="weights/weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]

epochs = 4
batch_size = 128
# Fit the model weights.
model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          callbacks=callbacks_list,
          validation_data=(x_test_reshaped, y_test_binary))

Then we can load the model and weights in the future using

# load model saved as a json and create model
json_file = open('weights/model.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
loaded_model = model_from_json(loaded_model_json)

# load weights into new model
loaded_model.load_weights("weights/weights-improvement-04-0.99.hdf5")
print("Loaded model from disk")

# evaluate loaded model on test data
#loaded_model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
predictions = loaded_model.predict_classes(x_test_reshaped[0:10], verbose=0)
show_imgs(x_test[0:10])
print('Predicted classes: ', predictions)

Once you have the loaded weights, then you can resume training by once again calling the fit function

epochs = 4
batch_size = 128
# Fit the model weights.
loaded_model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          callbacks=callbacks_list,
          validation_data=(x_test_reshaped, y_test_binary))

This will perform 4 more epochs of training.


What's the difference between finding the average Euclidean distance and using inertia_ in KMeans in sklearn?

K-means does not minimize Euclidean distances, but squared Euclidean distances. This is not the same.

The nearest center is the same for both, but the mean only optimizes the squares. You can find the counterexample on my earlier answers here.

The proper value to use for the basic elbow approach is the "inertia", which is the sum of squares. But some evaluation measures may use other distances. For example the Silhouette index can be used with other distances.

Don't forget that these approaches are just heuristics for trying some k. Always also consider other values.


Multivariate time series forecasting with LSTM

1)First and most important, do not give Yi(t) history as feature. You will just end in a model that replicates the previous input to minimize the error, a cheating model. For more detail, you can have a look at my explanations at two different questions:

https://stackoverflow.com/questions/52778922/stock-prediction-gru-model-predicting-same-given-values-instead-of-future-stoc/52786399?noredirect=1#comment93125020_52786399

https://stackoverflow.com/questions/52252442/how-to-handle-shift-in-forecasted-value/53141558#53141558

2)Create your labels by sliding your Y(t) one step forward so that your each sample will have a label of Y(t+1). That means you will delete sample #1 as a result.

3)Use a time-window for each sample for your features. Do not provide just your features as x(t) for the label y(t+1). For example, with a window size w = 10, provide x(t-9),x(t-8),.....,x(t-1),x(t+1) as a single sample for label y(t+1). Then you will be boosting the sequential nature of LSTM, possibly acquiring greater performance.

4)You can use Keras for your LSTM regression task, have a look at simple code piece from my old works, I modified it for your task:

nn = Sequential()
nn.add(LSTM(80, batch_input_shape=(64,11,20), return_sequences=True, recurrent_dropout = 0.1))
nn.add(LSTM(60, recurrent_dropout = 0.2))
nn.add(Dropout(0.2))
nn.add(Dense(40, activation='relu'))
nn.add(Dropout(0.2))
nn.add(Dense(10, activation='relu'))
nn.add(Dense(1))
nn.compile(loss= 'mse, optimizer= 'adam')
nn.fit(x_train, y_train, epochs=15, batch_size=64, shuffle=True, validation_data=(x_dev, y_dev))
y_pred = nn.predict(x_dev, batch_size=64)

where batch_input_shape=(64,10,25) explains that your batch size is 64 (trains 64 samples per gradient descent), your window size is 10, and you have 25 different features. You can delete anything about dropout in the code if you get confused; they are just for preventing overfitting.

Note: Do not forget to normalize your numeric input data at the start!

Hope I could help. Good Luck!


Data preparation and machine learning algorithm for click prediction

Some suggestions

Remove items appearing too infrequently in the data. That will reduce the dimensionality by several orders of magnitude. If a feature occurs less than say 10 times, it's likely that it's not adding any predictive value, and it may lead to overfitting due to low frequency
Try a Linear SVM instead. They handle large dimensional data very well in terms of not overfitting. They also often have the option to assign relative weights to different classes, which may help address your unbalanced problem above. The sklearn svm (which simply wraps some other packages such as libsvm) has this option.
Don't use the ID column. Producing a model per user will most probably lead to overfitting. Instead, feed in attributes that describe the user that allows the model to generalize over similar users. You could try fitting a separate model per user, but you need a lot of data per user to do this well.
It sounds like you really need to try some feature selection here, to reduce the dimensionality of the problem. But try 1 and 2 first, as they may give you good results sooner (although the end solution may still work better with some good feature selection). Sklearn again has a number of options for feature selection.


sklearn ColumnTransformer creates new columns in output when there are overlapping columns between steps

The description says-

".................the features generated by each transformer will be concatenated to form a single feature space"

Based on this I would not expect it to "reduce" the number of columns.

On top of my mind, another pipeline which computes on dates column and feeds its output to numeric column transformation in columnTransformation.


Intuition Behind Restricted Boltzmann Machine (RBM)

RBM's are an interesting beast. To answer your question, and to jog my memory on them, I'll derive RBMs and talk through the derivation. You mentioned that you're confused on the likelihood, so my derivation will be from the perspective of trying to maximize the likelihood. So let's begin.

RBMs contain two different sets of neurons, visible and hidden, I'll denote them $v$ and $h$ respectively. Given a specific configuration of $v$ and $h$, we map it the probability space.

$$p(v,h) = \frac{e^{-E(v,h)}}{Z}$$

There are a couple things more to define. The surrogate function we use to map from a specific configuration to the probability space is called the energy function $E(v,h)$. The $Z$ constant is a normalization factor to ensure that we actually map to the probability space. Now let's get to what we're really looking for; the probability of a set of visible neurons, in other words, the probability of our data.
$$Z = \sum_{v \in V}\sum_{h \in H}e^{-E(v,h)}$$
$$p(v)=\sum_{h \in H}p(v,h)=\frac{\sum_{h \in H}e^{-E(v,h)}}{\sum_{v \in V}\sum_{h \in H}e^{-E(v,h)}}$$

Although there are a lot of terms in this equation, it simply comes down to writing the correct probability equations. Hopefully, so far, this has helped you realize why we need energy function to calculate the probability, or what is done more usually the unnormalized probability $p(v)*Z$. The unnormalized probability is used because the partition function $Z$ is very expensive to compute.

Now let's get to the actual learning phase of RBMs. To maximize likelihood, for every data point, we have to take a gradient step to make $p(v)=1$. To get the gradient expressions it takes some mathematical acrobatics. The first thing we do is take the log of $p(v)$. We will be operating in the log probability space from now on in order to make the math feasible.

$$\log(p(v))=\log[\sum_{h \in H}e^{-E(v,h)}]-\log[\sum_{v \in V}\sum_{h \in H}e^{-E(v,h)}]$$
Let's take the gradient with respect to the paremeters in $p(v)$

\begin{align}
\frac{\partial \log(p(v))}{\partial \theta}=&
-\frac{1}{\sum_{h' \in H}e^{-E(v,h')}}\sum_{h' \in H}e^{-E(v,h')}\frac{\partial E(v,h')}{\partial \theta}\\ & + \frac{1}{\sum_{v' \in V}\sum_{h' \in H}e^{-E(v',h')}}\sum_{v' \in V}\sum_{h' \in H}e^{-E(v',h')}\frac{\partial E(v,h)}{\partial \theta}
\end{align}

Now I did this on paper and wrote the semi-final equation down as to not waste a lot of space on this site. I recommend you derive these equations yourself. Now I'll write some equations down that will help out in continuing our derivation. Note that: $Zp(v,h)=e^{-E(v,h')}$, $p(v)=\sum_{h \in H}p(v,h)$ and that $p(h|v) = \frac{p(v,h)}{p(h)}$

\begin{align}
\frac{\partial log(p(v))}{\partial \theta}&=
-\frac{1}{p(v)}\sum_{h' \in H}p(v,h')\frac{\partial E(v,h')}{\partial \theta}+\sum_{v' \in V}\sum_{h' \in H}p(v',h')\frac{\partial E(v',h')}{\partial \theta}\\
\frac{\partial log(p(v))}{\partial \theta}&=
-\sum_{h' \in H}p(h'|v)\frac{\partial E(v,h')}{\partial \theta}+\sum_{v' \in V}\sum_{h' \in H}p(v',h')\frac{\partial E(v',h')}{\partial \theta}
\end{align}

And there we go, we derived maximum likelihood estimation for RBM's, if you want you can write the last two terms via expectation of their respective terms (conditional, and joint probability).

Notes on energy function and stochasticity of neurons.

As you can see above in my derivation, I left the definition of the energy function rather vague. And the reason for doing that is that many different versions of RBM implement various energy functions. The one that Hinton describes in the lecture linked above, and shown by @Laurens-Meeus is:
$$E(v,h)=−a^Tv−b^Th−v^TWh.$$

It might be easier to reason about the gradient terms above via the expectation form.

$$\frac{\partial \log(p(v))}{\partial \theta}=
-\mathop{\mathbb{E}}_{p(h'|v)}\frac{\partial E(v,h')}{\partial \theta}+\mathop{\mathbb{E}}_{p(v',h')}\frac{\partial E(v',h')}{\partial \theta}$$

The expectation of the first term is actually really easy to calculate, and that was the genius behind RBMs. By restricting the connection the conditional expectation simply becomes a forward propagation of the RBM with the visible units clamped. This is the so called wake phase in Boltzmann machines. Now calculating the second term is much harder and usually Monte Carlo methods are utilized to do so. Writing the gradient via average of Monte Carlo runs:

$$\frac{\partial \log(p(v))}{\partial \theta}\approx
-\langle \frac{\partial E(v,h')}{\partial \theta}\rangle_{p(h'|v)}+\langle\frac{\partial E(v',h')}{\partial \theta}\rangle_{p(v',h')}$$

Calculating the first term is not hard, as stated above, therefore Monte-Carlo is done over the second term. Monte Carlo methods use random successive sampling of the distribution, to calculate the expectation (sum or integral). Now this random sampling in classical RBM's is defined as setting a unit to be either 0 or 1 based on its probability stochasticly, in other words, get a random uniform number, if it is less than the neurons probability set it to 1, if it is greater than set it to 0.


As interesting as these kind of questions are; it will always be subjective and task oriented. There are a multitude of comparisons between these languages: here, here, here and many viewpoints here.

I won't list the technical differences, but just offer my perspective on each language.

Julia

If you want to learn something new and upcoming, perhaps even influence the development in the long term, Julia might be a nice candidate, because (after ca. 10 years of development) it recently made its first 1.0 release and may gain maintstream use over the coming years. Starting now would potentially make you one of relatively few experts in a few years. It is geared towards scientific computing, combining the strengths of more established languages and I agree, some features do sound too good to be true!

R

If you want to have a very well-founded language with more focus on the intracacies of statistics, proven robustness over decades and a strong community, maybe R is for you. There are packages for almost anything related to modelling, forecasting, Bayesian methods, ... the list goes on. It can be fairly painless to learn and provides a short path to results, as libraries oftentimes have a single function that does lots of work for you. R also provides hands-down the nicest plotting experience, whose importance is not to be underestimated!

Python

If you want a "Swiss Army Knife", able to analyse data, build a deep learning model and then perhaps write a webserver to stream API data, from a robot, finally pushing live results to a dynamic website (and so on). There are libraries and tutorials available on seemingly endless topics. Syntax is generally quite friendly, so the path to being productive is just as short as the others. Python isn't as well established (historically) within statistics and machine learning compared to R, but I rarley find myself lacking anything. Python is growing in popularity too, with the majority of academic research I see within Data Science appearing e.g. on GitHub in Python. So it is a mature language and is also expanding.

I think other people could offer their perspectives to give a fuller comparison, but that's my two cents. I personally switched from R to Python for almost all tasks - and haven't yet found time to take Julia for a full test-drive (but want to!).

Coming back to my starting point: all the languages are Turing complete, meaning you can achieve anything with any of the languages... but why open a can of beans with a spoon if there is also can-opener available?

For most practical cases: if you are more comfortable with one language - use that one. If a task requires a certain package from one language, or your company requires the usage of one language - your decision is made for you.

Having motivation to learn a language and get some actual work done should, in my opinion, be the main factor. If you try all three, it is likely one will feel more natural to your way of thinking.


XGBoost: what to do when Kfold is not enough?

Using normal KFold cross-validation for time-series data will yield a highly optimistic error estimate since you are using data from the future to predict the past. The model just has to learn to interpolate, not to predict.
Therefore you have to use time-wise CV:

Furthermore, if your goal is to predict the future for a time-series you do not know the past of, you have to use leave-one-group-out, time-wise-CV to get a realistic performance estimate: Train on all time-series except for a test-set of time-series, but only using data up to a certain point in time. Validate on the test-time-series but only using data after the timepoint.

I doubt you will have success with this approach since this is often a very difficult problem.

For choosing the right cross-validation method you have to be clear about the goal of your method.

Do you want to interpolate missing data for a couple of time-series?
Use K-Fold CV
Do you want to predict the future of some time-series, for which you
know the past? Use time-wise CV
Do you want to predict the future of some time-series, but you only
know the past of some other time series? Use Leave-one-Group-out,
time-wise CV

I assume you are after the second point since this is the most common problem.


You are essentially asking how Google calculates the score of the domains. The correspondence of the score with ranking is different for every search. Sometimes it is closer to x(i) = alpha*x(i-1) (exponential weights). Sometimes it could be linear x(i) = (n-i)*2/((n+1)*n). I've seen some papers that just use the number of the ranking for regression (linear), but this was not about internet domains, but some other topic.

I will try to answer the question about your weights and you weighting system:

You are using 1/1, 1/2, 1/3 ... 1/100 ... 1/1000

Instead you could use 1/1, 1/1.01, 1/1.02, 1/1.03 ... 1/2 ... 1/10

So your weights are $$weights = \frac{1}{1+ranking*0.01*\frac{9}{9.99}}$$
where ranking starts with 0 and ends with 999


sklearn predict: IndexingError: ('Too many indexers', 'occurred at index ')

Removing remainder='passthrough' resolved the problem. I just tacked the prediction onto 'X' and got what I needed since sklearn v20 can deal with pandas dataframes.


Why perceptron does not converge on data not linearly separable

The behavior appears to actually depend on the learning rate $\eta$; a smaller $\eta$ affects which points are misclassified in the next iteration, which affects the weight update more than just by the simple scaling you alluded to.
With appropriately small learning rates though, it seems you are guaranteed convergence to some local minimum, if you avoid certain degenerate situations that would give e.g. a constant loss surface.  Exactly what you describe happens at these minima: the loss to the misclassified points of either class equal each other.
I put together a short demonstration in this colab notebook (github link).  Below are some animations of the evolution of the decision line during the gradient descent, starting at the top with a large learning rate and decreasing it from there.  In this situation, there should be two local minima, the two "tangents" to the "blobs" (possibly pulled in a little, misclassifying a couple of extra points if it can decrease the loss on the central points a little more than the cost for the newly misclassified points).


Should I prevent augmented data to leak to the test/cross validation sets

Yes!

The accuracy of your algorithm should be tested on images that you expect to receive. This would not include data that has been rotated, shifted, blurred, etc. However, augmenting your data in such ways can and usually will lead to better results.

Split your data first. Then augment only the training data while staying away from the validation data. Only use your testing data when you are confident that your solution leads to good results on your validation set without overfitting it.


How does ML Clustering put to a practical real-world use?

You want to understand the data.

So you run a clustering, then study how the points in a cluster differ from the points not in a cluster. Then based on these observations, you form a hypothesis. For example, you may notice that a cluster contains countries who eat a lot of fast-food and who are overweight. Then you can formulate the hypothesis that fast-food causes overweightedness, and then test that hypothesis.

This is a form of explorative data analysis. There is not a mathematical function to maximize, but it is a tool for humans to understand their data and then be able to formulate new hypotheses that would not have sprung to your mind otherwise.


What is whole word masking in the recent BERT model?

phil ##am #mon is a subword encoding of the single word “philammon” into 3 tokens. The comment just means that they mask words as opposed to tokens by taking into account subword encoding.

For more on subword encodings take a look at the slides from cs224, especially Byte Pair Encoding, from the Feb 14 subwords lecture at http://web.stanford.edu/class/cs224n/index.html#schedule.


As far as I understand the topic embeddings are not portable between graphs since every graph has their own characteristics which you need to learn. At the field of word embeddings, it is different since domain stays the same (words).

In the field of graphs, you usually train the model on the same graph that will be embedded later. It is also important to train on the same graph because the labels of nodes are usually nodes IDs which are graph specific. In the field of words embeddings, the method recognizes words by their name (sequence of letters) while in graphs nodes are recognized by their IDs.

Hope that it explains a bit. If you want to understand graph embeddings, in general, I suggest reading this story.


How do you search for content, not words?

The general principle about finding text based on meaning similarity is called distributional semantics. The main idea is that words with a similar meaning statistically tend to co-occur with the same context words. This is the basis for many standard tasks like topic modeling, word sense disambiguation/induction, and generally for everything related to semantics.
The problem you're describing is essentially information retrieval: given a query expressed as one or several words, find the "documents" in a corpus which are semantically similar to this query.
[edit] Depending what the data looks like, you might be interested in the more specific task of semantic textual similarity.


Normalizing / standardizing training and validation data

Yes, the test data might contain something important that you didn't learn from the test set. No, you're not allowed to use the test set for learning.

The purpose of the test set is to understand how the system would behave on new data not seen during training. Fitting your scaler in the test set would defeat that purpose.

  1) Generate the mean and standard deviation stats on the 80% of training data and then apply the same mean/standard deviation to standardize the validation data.

That's what you should do.

  Or 2) Standardize the training data, and then standardize the validation data, i.e mean and SD is derived from the 80% for the training data, and then mean/SD is derived separately on the 20% of validation data?

This is valid, if your production system makes it like that. So if you get data in batches and normalize the batches it might be fine as well.


Is this a good classified model based confusion matrix and classification report?

The question 'is this a good model?' is essentially a business question.
There is always a precision/recall tradeoff and you decide based on your business goal what model to choose.

Data science can help you with the question on how to compare classifiers.

Let us start, that each (or nearly each) classifier can predict not only the outcome (Class2 or Class3), but also the probability of this outcome.

Your confusion matrix and all your metrics are suitable for making business decision, but, from data science point of view, have two problems

1) they do not take into account the agreement by chance

2) they are based on the threshold probability 0.5, which is not necessarily the optimal probability for deciding about the class.

To deal with (1), have a look at Cohen's kappa

To deal with (2), use metrics based on probability. The simpler one is Area Under the ROC Curve, see also here. Even finer, but not so straightforward to interpret is Logarithmic Loss

  Is recall depend on training data? How do I improve recall for class 2 along with accuracy?

Recall depends on the quality of your model (and thus also on the quality of your training data) and on the chosen probability threshold. By decreasing probability threshold, you improve recall but worsen precision. Which tradeoff to choose is again a business decision.


Keras supports lazy execution. The create_model and model.compile code are not executed until it is absolutely required which is right before the first training epoch. That increased time for the first epoch includes building the TensorFlow computational graph based on the plan in your create_model function. All remaining epochs re-use the same computational graph which is why they are significantly faster.


Machine Learning library in Python, list or numpy or pandas

Pandas does normally a decent job allowing dataframes to behave as numpy arrays.

My recommendation is to use numpy types, the reason is that, for consistency with pretty much what the industry is doing, you are much safer with numpy.

I love pandas, and I love the dataframes, but they provide extra functionality that the model does NOT need, the same way in general programming you will not use a String to represent a boolean (even though tou could do it with a String), simply because you should use whatever data types provides you the functionality you need... and nothing else.

So, numpy is the way to go. As for python lists, you do not get the mathematical operations that you get with numpy, so do not consider them.


Should I go for a 'balanced' dataset or a 'representative' dataset?

I would say the answer depends on your use case. Based on my experience:

If you're trying to build a representative model -- one that describes the data rather than necessarily predicts -- then I would suggest using a representative sample of your data.
If you want to build a predictive model, particularly one that performs well by measure of AUC or rank-order and plan to use a basic ML framework (i.e. Decision Tree, SVM, Naive Bayes, etc), then I would suggest you feed the framework a balanced dataset. Much of the literature on class imbalance finds that random undersampling (down sampling the majority class to the size of the minority class) can drive performance gains.
If you're building a predictive model, but are using a more advanced framework (i.e. something that determines sampling parameters via wrapper or a modification of a bagging framework that samples to class equivalence), then I would suggest again feeding the representative sample and letting the algorithm take care of balancing the data for training.


Is Gini coefficient a good metric for measuring predictive model performance on highly imbalanced data

The Gini Coefficient can also be expressed in terms of the area under the ROC curve (AUC): G = 2*AUC -1 link. The ROC curve, on the other hand, is influenced by class imbalance through the false positive rate FP/(FP+TN). If the number of negatives is a lot larger, this could be a potential issue.

In short, the Gini Coefficient has similar pros and cons as the AUC ROC metric.


Learning Algorithm that decide which model gives better results for each testing instance

Applying any non-linear model in a model stacking approach should do what you want. In brief the approach is to take predictions from other models as new features, plus the original data and labels, then use them to train a meta-model. Read the link, it offers practical advice on how to do this within a k-fold validation framework, which will give it a much better chance of doing well.

Non-linear models that combine simpler units - e.g. neural networks and models that use multiple decision trees (e.g. xgboost) - already perform this kind of internal split during training. So if you are already using those, you might not gain such a big improvement over simpler ensemble techniques, such as taking a mean or weighted mean over models.


How to perform a running (moving) standardization for feature scaling of a growing dataset?

I think you want
$$S_{n}=S_{n−1}+ (x_{n}−μ_{n−1})(x_{n}−μ_{n})$$
where S, x and μ are respectively the variance, value and mean.
See https://fanf2.user.srcf.net/hermes/doc/antiforgery/stats.pdf for explanation and derivation.


JRip implements a propositional rule learner, “Repeated Incremental Pruning to Produce Error Reduction” (RIPPER), as proposed by Cohen (1995) and OneR builds a simple 1-R classifier, proposed by Holte (1993).

Its hard to say which algorithm works better. Best approach is to compare different classification algorithms performance in terms of precision, recall, accuracy, f1 score, AUC, specificity and sensitivity on your train/test data set and pick the one that gives best results or use ensemble of top performing algorithms to build your final model.

There is a good white paper doing similar exercise of comparing different classification algorithms including OneR and JRIP here.

Hope this helps.


Why Scikit and statsmodel provide different Coefficient of determination?

You need to add an intercept to statsmodels manually, while it is added automatically in sklearn.
import altair as alt
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
import statsmodels.api as sm

np.random.seed(0)
data = pd.DataFrame({
'Date': pd.date_range('1990-01-01', freq='D', periods=50),
'NDVI': np.random.uniform(low=-1, high=1, size=(50)),
'RVI': np.random.uniform(low=0, high=1.4, size=(50))
})

X = data[['NDVI']].values
X2 = data[['NDVI']].columns
Y = data['RVI'].values

# Sklearn (note syntax order X,Y in fit)
regressor = LinearRegression()
model = regressor.fit(X, Y)
print("Coef:", model.coef_)
print("Constant:", model.intercept_)
print("R2:", model.score(X,Y))

# Statsmodels (note syntax order Y,X in fit)
X = sm.add_constant(X) # manually add a constant here
model = sm.OLS(Y, X).fit()
print(model.summary())

Results:
Sklearn:
Coef: [-0.06561888]
Constant: 0.5756540424787774
R2: 0.0077907160447101545

Statsmodels:
                            OLS Regression Results
==============================================================================
Dep. Variable:                      y   R-squared:                       0.008
Model:                            OLS   Adj. R-squared:                 -0.013
Method:                 Least Squares   F-statistic:                    0.3769
Date:                Tue, 19 May 2020   Prob (F-statistic):              0.542
Time:                        11:18:42   Log-Likelihood:                -25.536
No. Observations:                  50   AIC:                             55.07
Df Residuals:                      48   BIC:                             58.90
Df Model:                           1
Covariance Type:            nonrobust
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const          0.5757      0.059      9.796      0.000       0.457       0.694
x1            -0.0656      0.107     -0.614      0.542      -0.281       0.149
==============================================================================
Omnibus:                        5.497   Durbin-Watson:                   2.448
Prob(Omnibus):                  0.064   Jarque-Bera (JB):                3.625
Skew:                           0.492   Prob(JB):                        0.163
Kurtosis:                       2.122   Cond. No.                         1.85
==============================================================================


Creating a pie chart with historical motion in Rstudio

Shiny App, will be the place to display the graphics, however the animation you are looking for is not something that plotly would create, to the best of my knowledge.

d3 however is a different beast. It can create aesthetically pleasing graphics. For your specific case, here is the example

https://bl.ocks.org/mbostock/1346410

Or you can look at the entire gallery


How distribution of data effects model performance?

Good question. Your interpretation is adequate. Using a logarithmic function reduces the skewness of the target variable. Why does that matter?

Transforming your target via a logarithmic function linearizes your target. Which is useful for many models which expect linear targets. Scikit-Learn has a page describing this phenomenon: https://scikit-learn.org/stable/auto_examples/compose/plot_transformed_target.html

Important to note

If you modify your targets before training, you should apply the inverse transform at the end of your model to compute your "final" prediction. That way, your performance metrics can be comparable.

Intuitively, imagine that you have a very naive model which returns the average target regardless of the input. If your targets are skewed, it means that you will under-/over-shoot for a majority of the predictions. Because of this, the range of your error will be greater, which worsens scores such as the Mean Absolute or Relative Error (MAE/MSE). By normalizing your targets, you reduce the range of your error, which ultimately should improve your model directly.


Stacking and Ensembling methods in Data Science

It sometimes is called "ensemble learning" where several "weak learners" make a prediction. These predictions are "combined" by some meta-model. A simplistic approach would be that you just use the majority vote. You can also use logistic regression. You can (and should!) of course check the performance of the stacked model(s) by predicting on test data. Related ensemble techniques are boosting and bagging.
Simplified Example: Assume you want to predict a binary outcome. Say you have two different models, which do not perform extremely well, but perform better than random guessing. Also, assume the models are independent. Suppose each of your models makes a correct prediction with $0.8\%$ probability. Now when you have two models, the possible outcomes are:
Correct, Correct:   $0.8*0.8=0.64$
Wrong, Wrong: $0.2*0.2=0.04$
Correct, Wrong: $0.8*0.2=0.16$
Wrong, Correct: $0.2*0.8=0.16$
You see that the probability that both models are wrong is "only" $0.04$. So if you manage to identify the remaining cases as "correct predictions" by some intermediate/meta (stacking) model, you end up with quite a good result. Essentially you would "boost" the expected accuracy of the model(s) from $0.8$ (single model) to $0.64+0.16+0.16=0.96$ (ensemble) here.


How is the Gaussian noise given to this BLSTM based GAN?

As stated in 3.2 Model setup

The generator G is made up of embedding layer, one bidirectional  long  short-term  memory  (BLSTM)  [21]  layer,  one fully connected (FC) layer.

And

Gaussian noise is 10-dim vector concatenated with the output of BLSTM

So the noise is concatenated to the embeddings computed by the BLSTM for each time step. I think they concatenate the same 10-dim vector to each embedding output of the BLSTM but this is not clear.
The obtained vector (embedding concatenated with noise) is fed to the Fully-connected layer with softmax activation to compute the probability that the word should be translated or not.
Concerning page 2 footnote about "ignoring the noise"
I think they simply ignore it in their explanations of the model, but it is effectively used in the model by simply appending a gaussian noise vector to the embeddings output of the BLSTM.
As you see in the following figure they do not show the noise, but it is actually concatenated to the output embeddings of the BLSTM in the Generator.


Source of Arthur Samuel's definition of machine learning

The exact quote exists in neither the 1959 paper nor the 1967 paper (second version).

These are the closest quotes from the 1959 paper:

  A computer can be programmed so that it will learn to play a better
  game of checkers than can be played by the person who wrote the
  program.

And

  Programming computers to learn from experience should eventually
  eliminate the need for much of this detailed programming effort.

Also, Wiki page of Arthur Samuel states that:

  He coined the term "machine learning" in 1959

and references the 1959 paper.

Either the quote is created as a gist of Arthur Samuel's 1959 paper, or it is said but not written by him. In my opinion, the former is more probable, since it is not even remotely mentioned in the 1967 paper.


There is a good tutorial here:

http://www.unal.edu.co/diracad/einternacional/Weka.pdf

Basically you want to use a StringToWordVector filter to convert your string into word vectors (!) and then train a Bayes classifier on that data.


Approaches to pre-processing the huge but organised text data, with & without the generators

Could you reduce data types, for example, int32 to int16, but you should be careful, you have to be sure that you don't lose important a piece of information by reducing of memory.
Iteratively read CSV and dump lines into the SQLite table. Working on database is faster than on CSV file.
Use the library for parallel computing in Python like Dask or Pandarallel.


The 'dplyr' package in R is ideal for these types of data manipulation tasks. The arrange function for example can group a dataframe by a certain column, and then sort by another column. For example:

arrange(df, desc(mileage), group_by = year)

See arrange for documentation on the arrange function, and dplyr for the dplyr package description.


Poisson distribution, Standard deviation, fitting line

This sounds like simple linear regression. Your x-value is the vector of means and the y-value is the vector of corresponding standard deviations:
$$Y = X\beta$$
$\beta$ a vector containing the slope of your regression line and the offset term.
In R for example you can do fit <- lm(Y~X).


Sub-sampling so that sample statistics match population statistics

The term you are looking for is stratified sampling : https://en.wikipedia.org/wiki/Stratified_sampling. It's a way to sample from population that can be partitioned into sub-populations. More specifically for your problem, look at what they do for clinical trials, when they parition patients in sub-groups.


Clustering Observations by String Sequences (Python/Pandas df)

As URLs are usually unique strings, I am not sure if NLP methods are proper things here. That is interesting that TFIDF+Kmeans gave you an intuitive results however as the problem is unsupervised, it´s pretty difficult to say what is intuitive that way.

In my opinion your problem is a typical Community Detection in Bipartite Graphs e.g. this paper. This approach is certainly worth to try. Of course the result can be combined with other algorithms e.g. in a voting schema.

The last but not the least is to take a clustering method which does not limit to resolution e.g. in Community Detection, any Modularity-based method is only able to capture clusters larger than a resolution limit. One approach to this is a hierarchical approach to partitioning the data in which you again apply clustering on the significantly large cluster you find.

Hope it helps!


how to prevent machine crash while searching for hyper parameters of XGBoost with GridSearchCV

I assume its crashing because not enough RAM. So I also assume your data is quite big. Your search grid is quite big. So this will definitely take some time.
In order to speed up the training and overloading the RAM. You can fit the model in a subsample of data. Theoretically if your data is big enough and you sample it, when you use the whole model the optimal hyperparameters should be quite similar.
Once you do, see what is the actual improvement of changing hyperparameters, hyperparameters in xgboost shouldn't be sooooo important. They will make a change but not a huge change.
So my suggestion will be to sample your data by 1% or something similar to search for hyperparameters and then fitting the model in the whole training set.
And not search in such a huge hyperparameter space.


tf.keras.preprocessing.image_dataset_from_directory
Generates a tf.data.Dataset from image files in a directory.

ImageDataGenerator.flow_from_directory
Takes the path to a directory & generates batches of augmented data.

While their return type also differs but the key difference is that flow_from_directory is a method of ImageDataGenerator while image_dataset_from_directory is a preprocessing function to read image form directory.
image_dataset_from_directory will not facilitate you with augmented image generation capability on-the-fly.

which one should I use?

It's quite common to generate augmented images when working with CNN, so better to use flow_from_directory.
If you do not need augmented image, you may control the same by the parameters of ImageDataGenerator
Reference - Keras docs


Can we use k fold Cross Validation without any extra (excluded) Test Set?

The key question is

  Is K-fold cross validation is used to select the final model (or algorithm)?

If yes, as you said, then the final model should be tested on an extra set that has no overlap with the data used in K-fold CV (i.e. a test set).
If no, the average score reported from K-fold CV is the final test score, no extra set is required.

Therefore, if authors only picked the best model from all K models, it should have been tested on an extra set. The score of model on the validation fold (from K-fold CV) is not acceptable.

Also, here is a related post on this site (my answer) that goes into more detail about test and validation sets (scores).

EDIT:

I have found a similar question on stats.stackexchange.com. Also, this comment by amoeba suggestions "nested CV" instead of "CV + test set", which I think it is worth fleshing out here.

K-fold CV evaluation

1. For k = [1,..,K]
  1. tr = (K-1)/K of data, ts = 1/K of data
  2. m[k] = model trained using tr (can be further split into tr2 + v)
  3. score[k] = score of m[k] on ts
3. Test score = average of score[1],...,score[K]

K-fold CV selection and evaluation

1. tr = 80% of data, ts = 20% of data (or some other ratio)
2. For k = [1,..,K]
  1. tr2 = (K-1)/K of tr, v = 1/K of tr
  2. m[k] = model trained using tr2 (can be further split into tr3 + v2)
  3. score[k] = score of m[k] on v
3. M = best of m[1],...,m[K]
4. Test score = score of M on ts

Nested k-fold CV selection and evaluation

1. For k = [1,..,K]
  1. tr = (K-1)/K of data, ts = 1/K of data
  2. For k2 = [1,..,K2]
     1. tr2 = (K2-1)/K2 of tr, v = 1/K2 of tr
     2. m[k2] = model trained using tr2 (can be further split into tr3 + v2)
     3. score[k2] = score of m[k2] on v
  3. M = best of m[1],...,m[K2]
  4. score[k] = score of M on ts
2. Test score = average of score[1],...,score[K]

Note that in these algorithms, model trained includes parameter learning, hyper-parameter tuning, and model selection (of course, except for the outermost model selection). For example, for selecting between hyper-parameters 8 and 12, we need a deeper loop inside model trained.


In CycleGAN are there two different generators and two different discriminators?

There are two generators and two discriminators.

Check out the code here


Basically, you need to create a whole system which contains multiple ML algorithms. We can go feature-wise and see how the system could work.

Description: This will consist of text which could be easily vectorized using Doc2Vec. This vector will act as a feature for the final model.
Product images: You will need to create a model which make some sort of classification. Once, the model is trained to remove the last layer ( mostly the Softmax layer ). This will fetch us the encoded representation of the image. You may use an AutoEncoder for this.
Manufacturer and other properties: I think you may need to omit this feature as they will be product-specific and create complexities while training.

Once, we have all the above models trained, we can feed the features ( generated from these models ) to a neural network, which will produce the final result.


Choosing between two object detection model checkpoints

Choosing the model with the lowest loss on the validation set would be more appropriate. There are cases where the lowest loss model also exhibits the highest mAP, but that definitely is not the norm.

Regarding your question, if you are to choose the model based on accuracy-like metrics, consider what would be less harmful in your specific example. For instance, consider the case of cancer classification, predicting that a healthy patient has cancer (False Positive) might result in some frustration but it is not that big of a deal. On the other hand, predicting a cancer patient as of being healthy might cost his life.