Take a look at professor Andrew Ng's course about deep learning. Its homework is written in Keras and TensorFlow. It is a very good point to begin.

I guess it is not fair to mention non-free lectures because the owners of other non-free lectures may get disappointed but take a look at complete guide to TensorFlow for deep learning with python, which is very useful.

I have also found these lectures from Stanford university very useful. Furthermore, you can take a look at the official paper of google research team about TensorFlow where you can read TensorFlow is an interface for expressing machine learning
algorithms, and an implementation for executing such algorithms.
A computation expressed using TensorFlow can be
executed with little or no change on a wide variety of heterogeneous
systems, ranging from mobile devices such as phones
and tablets up to large-scale distributed systems of hundreds
of machines and thousands of computational devices such as
GPU cards. The system is flexible and can be used to express
a wide variety of algorithms, including training and inference
algorithms for deep neural network models, and it has been
used for conducting research and for deploying machine learning
systems into production across more than a dozen areas of
computer science and other fields, including speech recognition,
computer vision, robotics, information retrieval, natural
language processing, geographic information extraction, and
computational drug discovery. This paper describes the TensorFlow
interface and an implementation of that interface that
we have built at Google. The TensorFlow API and a reference
implementation were released as an open-source package under
the Apache 2.0 license in November, 2015 and are available at
www.tensorflow.org..


Constant Learning Rate for Gradient Decent

Intuitively, if $\alpha$ is too large you may "shoot over" your target and end up bouncing around the search space without converging. If $\alpha$ is too small your convergence will be slow and you could end up stuck on a plateau or a local minimum.

That's why most learning rate schemes start with somewhat larger learning rates for quick gains and then reduce the learning rate gradually.


Policy Gradients - gradient Log probabilities favor less likely actions?

Yes, you are right that the term $\frac{1}{\pi_\theta(a_t|s_t)}$ itself is favoring the less likely actions. But don't forget how $\nabla \pi_\theta(a_t|s_t)$ is calculated, and you will see that $\nabla \pi_\theta(a_t|s_t)$ is over-emphasizing on the first action and $\frac{1}{\pi_\theta(a_t|s_t)}$ is correcting it.

Recall that the derivative for y =softmax(x) is $\frac{\partial y_i}{\partial x_j} = y_i(\delta_{ij}-y_i)$. Notice that $y_i$ is multiplied in the front, i.e. over-emphasizing the more likely $y_i$. By multiplying $\frac{1}{y_i}$ in the front, the over-emphasizing effect is being corrected.

In your example, when ${\pi_\theta(a_t|s_t)}=[0.5,  0.1,  0.1,  0.1,  0.1,  0.1]$, we can write down that
$$\nabla \pi_\theta(a_t|s_t)=\left(\begin{array}{cccccc}
0.5\times0.5 & -0.5\times0.1 & -0.5\times0.1 & -0.5\times0.1 & -0.5\times0.1 &  -0.5\times0.1\\
 -0.5\times0.1 & 0.1\times0.9 &  -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 \\
 -0.5\times0.1 & -0.1\times0.1 & 0.1\times0.9  & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 \\
-0.5\times0.1 &  -0.1\times0.1 & -0.1\times0.1 & 0.1\times0.9 &-0.1\times0.1 & -0.1\times0.1 \\
 -0.5\times0.1  &  -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 & 0.1\times0.9 & -0.1\times0.1 \\
 -0.5\times0.1  &  -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 & -0.1\times0.1 & 0.1\times0.9\\
\end{array}\right)$$
And by multiplying it with $\frac{1}{\pi_\theta(a_t|s_t)}=[\frac{1}{0.5},\frac{1}{0.1},\frac{1}{0.1},\frac{1}{0.1},\frac{1}{0.1},\frac{1}{0.1}]$, you effectively get
$$\nabla \log\pi_\theta(a_t|s_t)=\frac{1}{\pi_\theta(a_t|s_t)}\nabla \pi_\theta(a_t|s_t)=
\left(\begin{array}{cccccc}
0.5 & -0.5, &-0.5, &-0.5, &-0.5, &-0.5, \\
-0.1 & 0.9 & -0.1 & -0.1 & -0.1 & -0.1 &  \\
-0.1 & -0.1 & 0.9 & -0.1 & -0.1 & -0.1 &  \\
-0.1 & -0.1 & -0.1 & 0.9 & -0.1 & -0.1 &  \\
-0.1 & -0.1 & -0.1 & -0.1 & 0.9 & -0.1 &  \\
-0.1 & -0.1 & -0.1 & -0.1 & -0.1 & 0.9 &  \\
\end{array}\right)$$

In case you are still thinking that because $0.9>0.5$ in the above result, we are slightly favoring the less likely action (although not 5 times more favorable), I would say that's the desired behavior. Because when we already have a relatively high score for action 1 (pre-activation 0.5) and now increase the score a little bit, we don't want the output probability to increase that much because otherwise the probability would more easily explode out of the range of $[0,1]$.

This kind of saturating gradient effect is very common for anything using the log derivative trick, not only in RL. You can also think of the example in the original REINFORCE algorithm
paper (on page 6) where a Bernoulli random variable
$$g_i(y_i,p_i)=\left\{
\begin{array}{c l}
     1-p_i & \text{if } y_i=0\\
     p_i & \text{if } y_i=1
\end{array}\right.$$

has a log derivative as

$$\frac{\partial \log g_i}{\partial p_i}=\left\{
\begin{array}{c l}
     -\frac{1}{1-p_i} & \text{if } y_i=0\\
     \frac{1}{p_i} & \text{if } y_i=1
\end{array}\right.$$
Here when $p_i$ is already large and $y_i=1$, the derivative would be relatively small.

Finally, regarding your question what if we only got such a large large reward because we took the first action? Don't worry, here you are estimating the expected reward (and its gradient) of your current policy $\pi$, not exploiting this policy, so you just want to have a fair (unbiased) estimate, not a best possible reward.


Is it possible to decode which neuron represent which feature and why does it represent it?

Short answer: NO.

Think about it, you could have 5 or 5000 neurons, and each neuron would represent a different thing in different models with different inputs.

One of the problems of neural networks is that it is quite difficult to understand what is going inside them, they are a bit of a black box. Other techniques such as decision trees are much more easy to understand or explain to non-technical people.

The closest thing you could do is visualize the hidden layers and that might give you some sort of insight.


Imagine you have a blue line drawn from the point (0,0) to the point (0, 0.51). Do you see a common shape now?

That happens because you have few or no predicted probabilities above 0.9 or something close to it, so the point with 0 false positives is never touched.

The ROC curve you are watching is perfectly normal.


How do you know if your Autoencoder network is fully connected?

Is Autoencoder with more than one hidden layers a type of deep neural
  network (DNN)?

Yes.

  Is DNN always fully connected?

No. For example Echo State Networks are sparsely connected DNNs.

  how do you know if the Autoencoder network that you build is fully
  connected?

I am not sure about R because I don't use it but for Keras/Python you can read the layer descriptions. The most common predefined layers (Dense, Convolutional, RNN, LSTM..) are fully connected. If you want to have a sparsely connected layer, then you can probably create a custom one using a Lambda Layer.


It would help if you clarified what "relevance" and "important" mean, but you should take a look at Term Frequency-Inverse Document Frequency.

tf-idf weights words by the frequency of appearance within a document but penalizes words that appear across many documents. The concept is that words appearing many times in a single document are likely noteworthy, but they could also just be common words that appear frequently in all documents.

tf-idf has implementations in most popular langauges including Python and R.
The Python Implementation is in Sci-kit learn and is called TfidfVectorizer. It takes a list of filenames or a single document as input. You can also specifiy the stop words to apply to the documents and allows for tokenization. You probably also would want to apply a stemmer to your documents which can be done using Sci-kit learn.
If you prefer to work in R, you would want to use the tm package. This package has numerous functions for text mining.


Neural Network - distinguishing between several normalized values is impossible?

The math you are talking about is in equation (7):

  Let x′ be a new data point obtained by re-scaling x by δ. Then we have,

  It is easy to see re-scaling individual data points does not change
  the model’s prediction under layer normalization. Similar to the
  re-centering of the weight matrix in layer normalization, we can also
  show that batch normalization is invariant to re-centering of the
  dataset.

It proves that the prediction over x' would be the same as in x. So what you say is right, they are indistinguishable.

And I think this is what we want because improves generalization in predictions!

For example in image classification, you are trying to detect orangutangs, if an image  I represent an orangutang, ( I x 5)+2 still is an orangutang!


What is the minimum size of the test set?

This is a great question since it is illuminating to examination the best practices of both about traditional statistics and machine learning as they are brought together.

Two Separate Rules/Best Practices -

First, the two items that you mentioned should be examined separately and not conflated i.e. they should be carefully combined as you suggest in your question.

Significance: You have estimated that you want greater than 1000 cases to have statistical significance and would further benefit from greater than 3000 test cases.

Cross validation: Cross validation is performed by splitting the data (often 80% train-20% test) so that tests for bias (~underfitting) and variance (~overfitting) can be assessed.

Combining significance with cross validation: Now we know that we want to have significance in our tests so we want greater than 3000 records.  We also want to perform cross validation, so in order for both the testing and training data to return significant results we want both to have a minimum of 3000 records.  The best scenario for this would be to have 15,000 total records. This way, the data can be split 80/20 and the testing set is still significant.

Lets assume you only have 4000 records in your data set. In this case, I would opt to make my training data significant while allowing the testing set to drop to lower significance.

More rigor: Everything above has been quite hand wavey and lacks statistical rigor. For this, I refer you to a couple of papers referenced in another Stack Exchange question -

Dietterich, "Approximate Statistical Tests for Comparing
Supervised Classication Learning Algorithms"

Salzberg, Data Mining and Knowledge Discovery, 1, 317–327 (1997), "On Comparing Classifiers: Pitfalls to Avoid and a
Recommended Approach".

Hope this helps!


Pytorch DataLoader is a generator, it will generate new batches when iterated through. Hence, as per the best of my knowledge, you can only change the data on the fly. For eg, if you want to replace the first element of your dataset with second element of the datase, you can do something like -
T = torch.tensor(([101,123,414,463],[234,903,313,341]))
train = TensorDataset(T)
train_dataloader = DataLoader(train)

for idx,i in enumerate(train_dataloader):
  if idx == 0:
    i = i[0] # Get the first and the only instance of the batch
    i = train[1][0]

Otherwise, you will need to change your dataset (train variable) and then again create a dataloader for this new dataset.


Is finetuning from a pretrained model always better than training from scratch?

One limitation of using pretrained model is that you are forced to use the same architecture and weights. There could many scenarios where the pretrained architecture is limiting. If you train from scratch, you can define a custom architecture for the specific problem.


Random Forest Classifier gives very high accuracy on test set - overfitting?

rand_forest.fit(X,y)

Why are you using the whole data set for training? You are using the test set for training then evaluate the performance on it again?

In your code, I didn't see you actually used the training set you created.


How to apply dataset balancing techniques whilst using Pipeline in Sklearn?

Should it be done before scaling or after?

According to this post, you should scale the data first :

  My thought would be to standardize (normalizing is typically using the min and max values not mean and standard deviation) the data first and then over-sample if that is what you are thinking in terms of balancing. I say this because you will want to use that same mean/std dev of the original set when you standardize new data so that it mirrors the training set that was used.

Should it be done train/test split or after?

According to this post :

  Sampling should always be done on train dataset. If you are using python, scikit-learn has some really cool packages to help you with this. Random sampling is a very bad option for splitting. Try stratified sampling. This splits your class proportionally between training and test set.

You should also set the class_weight parameter of sklearn.svm.SVC :

  Set the parameter C of class i to class_weight[i]*C for SVC. If not given, all classes are supposed to have weight one. The “balanced” mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y)).

Hope this helps.


Data-generating probability distribution, probability distribution of a dataset, in ML

It's more of a theoretical distribution, that a concrete one.
The main idea is this:
we consider all data to have an underlying distribution which generates the data. Through the procedure of creating a dataset we effectively sampled some instances from it. Now I like to think of this distribution as a theoretical notion of all possible data of this type that could ever exist.
Let me give you an example:
suppose we have the cats vs dogs dataset. This dataset contains $25000$ images of cats and dogs. Now we can consider the cat images as samples from a larger population. But what would this population include? All cat images on the web? All cat images in existence? or all cat images that could ever conceivably exist? Let's refer to this as this population as $C$. This population follows a certain distribution (not every image is an image of a cat); this distribution essentially tells us what makes a cat, a cat. Now if I were to take a picture of a cat tomorrow (let's call this $c$), I would have effectively taken a sample out of this dataset (i.e. $c \sim C$).
Where does this come into play in Machine Learning?
Well, generative models essentially try to learn this distribution, and they try to do this through its samples (i.e. our dataset). They look at its samples and try to generalize to identify what distribution spawned them. Essentially they try to answer the question what makes $c$ a sample of $C$?
Furthermore, even discriminative models make several assumptions about the data (e.g. that the samples are independent and identically distributed, that the training and test set follow the same underlying distribution)
More formally

The training and test data are generated by a probability distribution over datasets called the data-generating process. We typically make a set of assumptions known collectively as the i.i.d. assumptions. These assumptions are that the examples in each dataset are independent from each other, and that the training set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption enables us to describe the data-generating process with a probability distribution over a single example. The same distribution is then used to generate every train example and every test example. We call that shared underlying distribution the data-generating distribution, denoted $p_{data}$. This probabilistic framework and the i.i.d. assumptions enables us to mathematically study the relationship between training error and test error.
- I. Goodfellow et al. "The Deep Learning Book" sec 5.2

I'd suggest reading chapter 5 from this book because the authors explain a lot of well known ML concepts (bias, variance, overfitting, underfitting etc.) through the scope of this data-generating distribution.

Edit after comment suggestion:
The question is essentially,

how does the data-generating distribution fit into the training process of a neural network?

The answer isn't so obvious, mainly because neural network classifiers are discriminatve models (i.e. they don't try to identify the data-generating distribution; rather they try to find out what features separate the classes among one another). Also I'd like to add that, as stated previously, the data-generating distribution is a theoretical concept, not a concrete one employed during training.
There is a way, through, we can tie this into the whole training procedure. Initially, consider that NNs try to minimize the cross-entropy loss between its predictions $\hat y$ and the actual labels $y$:
$$
Loss(y, \hat y) = - \sum_i y_i \, log \, \hat y_i
$$
Now let's think of $y$ and $\hat y$ not as tensors but as probability distributions. The first represents the probability that a sample would belong to class $y$, while the second represents the probability with which the network thinks a sample belongs to that class.
We can take this one step further and compute the KL divergence between $y$ and $\hat y$. This metric essentially tells us the difference between two distributions (higher values mean distributions are more different).
$$
KL \left( y \|\| \hat y \right) = \sum_i y_i \, log \, \frac{y_i}{\hat y_i}
$$
Note that minimizing cross-entropy is equivalent to minimizing the KL divergence between these two distributions. If the two distributions are identical their KL divergence has a value on $0$; this value increases the more they differ.
Minimizing the KL divergence between two distributions is the same as minimizing the JS divergence between them. This is a metric derived from KL, which can be used as a distance function between distributions (i.e. *how close $y$ is to $\hat y$).
So if you think of it this way, Neural Networks are trained to minimize the distance between the actual data-generating distribution $y$ and their perception of the data-generating distribution $\hat y$.
In order for this to be achievable, some assumptions must be held:

The samples we have must be representative of the distribution (i.e. $y_i^{train} \sim y$).
The test samples we'll use to evaluate our network on must follow the same distribution (i.e. $y_i^{test} \sim y$).
The network must have sufficient capacity to learn this distribution.
The right optimization strategy needs to be followed to minimize the distance.
etc.


Why do probabilities sum to one and how can I set optimal threshold level?

Question 1. Why do probabilities sum to 1: Probability theory. Probabilities sum 1 because that’s how we define them. It just so happens that, by forcing them to sum 1, they have an intuitive interpretation and also calculations end up being easier. But this is mere convenience. Probabilities (or, more specifically, probability measures) could have been defined to sum 12 or 100 or whatever number you prefer. It doesn’t really matter.

Further sub-questions:

Its not the data thats exclusive its the classes.
Classifier is able to classify just with lower probability ( I would actually argue that 99999% is to much maybe he is overfitting)

Question 2 Can I set the threshold, like 90%, and conclude if the probability is less than 90% , let the users manually classify the data? Let the data tell you optimal cut-off level/threshold (from the ROC curve, read it up)

def optimal_cutoff(ground_truths: np.array,
                             predictions: np.array) -> float:
        """

        :param ground_truths: array of arrays (matrix) of all ground truths
        :param predictions: array of arrays (matrix) of all predictions
        :return: optimal cut-off level
        """
        optimal_thresholds = []
        for y_train, y_pred in zip(ground_truths, predictions):
            fpr, tpr, threshold = roc_curve(y_train, y_pred)
            optimal_idx = np.argmax(tpr - fpr)
            optimal_thresholds.append(threshold[optimal_idx])

        return sum(optimal_thresholds) / (len(optimal_thresholds))

So you would calculate cut-off on your available GROUND truths. Now if, when you specified optimal cut-off level your metric (lets say f1) falls behind certain level you can tell users to check it up.


Distribution of training dataset in binary classification problem

In general it is better to have a good mix of positive and negative labels for a model to learn from. That said, many classification problems suffer from significant label imbalance (e.g. fraud detection).
I don't see an obvious problem with a 70/30 mix of labels, but as you start modeling be sure to check the confusion matrix and look for where you model isn't performing well. If it appears that the imbalance is causing issues, you could try techniques like under-sampling or SMOTE to create a better balance (artificially). The linked article below has more details if you are interested:
https://www.jeremyjordan.me/imbalanced-data/


How to include validation set in the pipeline to tune parameters for an SVM?

Editing my answer -
def predict(self, train_file: str, test_file: str, lower_case: bool) -> pd.DataFrame:
    "Train model using sklearn pipeline"
    train_df = self.read_data(train_file, lower_case)
    dev_df = self.read_data(dev_file, lower_case)
    learner = self.pipeline.fit(train_df['text'], train_df['truth'])
    # Fit the learner to the test data
    test_df = self.read_data(test_file, lower_case)
    # assuming dev_df is validset
    dev_df['pred'] = learner.predict(dev_df['text'])
    test_df['pred'] = learner.predict(test_df['text'])
    return test_df, dev_df

use two variables when you are calling the predict function to store tes and dev results


Regression: How to deal with positive skewness in continuous target variable

Problem in this competition is VERY similiar. Instead of me copy-pasting some interesting ideas just check out winner write-ups in the discussions.

Some takeways:

Add count of values of features as new features.
This is information that LightGBM can’t see
Check unique value counts for all features of train and test set.
Model stacking almost always works better than single model (forget about it in production)
Check for feature independence. For this competition, features were all uncorrelated (naive bayes worked great)
etc...

And finally and independently from this competition general note: Its all about the right data representation and not about the model. If you can represent data in the right way than basic linear regression will score good. I would do some serios data analysis for residual values in range negative until 200 and 500 until infty and see what confuses your model, obviously there are some conflicting features that force your model to predict low when it should be high. Maybe this is not satisfying to you but to really solve this problem you have to find what confuses him so that he predicts high when it should be high, and then engineer feature/data representation so that he can discriminate around this.


TF Addons computes the F1 score and more generally the FBeta Score


In R many great improvements are typically made by creating new libraries rather than changing the actual functions in base R itself. datatable, read_csv in readr, tibbles, etc are very good examples.
read.csv vs fread as answered by Matt Dowle himself
read.csv coerces strings as factors by default which is not done by read_csv and fread. So if that is what you need, read.csv will work well. Now answering your question,there is no disadvantage using the fast readers to read small files apart from a small speedup that you might get.


Should I transform my feature into normal distrubition before Isolation Forest

Its better to use data as is for anomaly detection as the underlying data is not normal. Apart from what you asked, Isolation Forest has a problem that it has the chances of creating ghost clusters where anomaly scores would be incorrect. Hence the recent one is EIF works better.


How to train a model on top of a transformer to output a sequence?

I think you should treat this problem as a binary classification problem.  For each word in the changed sentence, you will have a binary label: correct or incorrect.  I would recommend relabeling so that "correct" words will have a label of 0 and "incorrect" words will have a label of 1.  In your example you would have:
correct_sentence = "we used to play together"
changed_sentence = "we use play to together"
labels = [0, 1, 1, 1, 0]

And instead of padding with some special value, pad with the "correct" label (which would be 0 if you use my suggestion above).
Conventionally, class labels always start at index 0, so this labeling scheme will match what PyTorch expects for binary classification problems.

Next, you will need to change the activation function for your final Linear layer.  Right now, your model ends with just a Linear layer, meaning the output is unbounded.  This doesn't really make sense for classification problems, because you know that the output should always be in the range [0, C-1], where C is the number of classes.
Instead, you should apply an activation function to make your outputs behave more like class labels.  For a binary classification problem, a good choice for the final activation is torch.nn.Sigmoid.  You would modify your model definition like this:
class Camembert(torch.nn.Module):
    """
    The definition of the custom model, last 15 layers of Camembert will be retrained
    and then a fcn to 512 (the size of every label).
    """
    def __init__(self, cam_model):
        super(Camembert, self).__init__()
        self.l1 = cam_model
        total_layers = 199
        for i, param in enumerate(cam_model.parameters()):
            if total_layers - i > hparams["retrain_layers"]:
                param.requires_grad = False
            else:
                pass
        self.l2 = torch.nn.Dropout(hparams["dropout_rate"])
        self.l3 = torch.nn.Linear(768, 512)
        self.activation = torch.nn.Sigmoid()

    def forward(self, ids, mask):
        _, output = self.l1(ids, attention_mask=mask)
        output = self.l2(output)
        output = self.l3(output)
        output = self.activation(output)
        return output

Your output will now have dimension (batch_size, 512, 1).  Each of the 512 outputs will be a number between 0 and 1.  You can treat this as the probability of each particular token being "incorrect".  If the output is greater than 0.5, then the label becomes "incorrect".  Otherwise, the label is "correct".
Finally, since you're treating the problem as a binary classification problem, you will want to use binary cross-entropy loss (torch.nn.BCELoss).  Note that you will have to unsqueeze the labels to make their dimension match the dimension of your output.
model = Camembert(cam_model)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

input = <tokenized, padded input sequence>
labels = torch.tensor([0, 1, 1, 1, 0, . . .  , 0])
output = model(input)
loss = criterion(output, labels.unsqueeze(1))

optimizer.zero_grad()
loss.backward()
optimizer.step()


Modules on Python which are useful for missing Word/Letter prediction in text paragraphs from a coprpus

This problem seems to be exactly (or at least very similar to) that in the Billion Word Imputation challenge at Kaggle. Some of the approaches discussed in the forums might help you out. My initial thoughts for useful tools/methods would be word2vec, the stanford NLP tools and sklearn.


How fit_transform, transform and TfidfVectorizer works

I'm not really sure what you're asking, but in general, you need to fit an Estimator to data so it can learn what it has to do, then you transform data with it. fit_transform just does fit and then transform. Here you fit the transformer to Name_clean, and then apply it to both in turn. That's pretty normal.


Keras - Autoencoder different from Encoder + Decoder

When you do autoencoder.fit, your model is learning something so your weights are changing for the encoder model and decoder model.
After that while creating your encoder/Decoder like "encoder = Model(inputs=x, outputs=Encoder()(x), name='encoder')", you are creating another encoder model with differnt weights, you have to use the same weight vectors.
Check "autoencoder.layers[1].get_weights()" is equal to "encoder.get_weights()" or not, then you will get to know what is happening in the model.( similar thing for decoder as well). You can check that using below code,

for i in range(len(autoencoder.layers[1].weights)):
    print((autoencoder.layers[1].get_weights()[i]==encoder.get_weights()[i]).all())


OpenCV has a module to detect faces, eyes etc. With pretrained models you may also be able to detect cars.
https://www.pyimagesearch.com/2018/11/12/yolo-object-detection-with-opencv/

There are many good tutorials online. Just have a look.

So if you are happy with this, you don‘t need to know much about data science. However, if you want to go a step further and train your own models, you will need to look into e.g. Keras (with pretrained models).
https://keras.rstudio.com/articles/applications.html

My view is that OpenCV is a cool tool and you can do a lot with it. However, off-the-shelf applications have their limitations (of course).


If you have historic data of earlier purchase by customers. Try to build any classification algo ( Decision tree/ RF) to build rule associated with customer and product.
   Now you can suggest product to a new customer associating his properties to rules which are already created. Just goole classification algos in R/Python

Clustering ( K means/Hiearchical etc)  would be good for customer segmentation, not directly useful for prediction.

https://machinelearningstories.blogspot.com/2017/09/hierarchical-clustering-bottom-up.html

You can also use market Basket analysis for recommending similar products to existing customer.

https://machinelearningstories.blogspot.com/2016/11/recommendation-engine-market-basket.html

http://courseprojects.souravsengupta.com/cds2016/product-recommendations-a-multi-classification-problem/


sklearn SimpleImputer too slow for categorical data represented as string values

Searching the source code of Sklearn for SimpleImputer (with strategy= "most_frequent"), the most frequent value is calculated within a loop in python, therefore that is the part of code that is so slow. In the source code of SimpleImputer there is also the comment that explains why they do not use the scipy.stats.mstats.mode, which is mush faster:

  scipy.stats.mstats.mode cannot be used because it will no work properly if the first element is masked and if its frequency is equal to the frequency of the most frequent valid element. See https://github.com/scipy/scipy/issues/2636

So if you want to use the SimpleImputer with this strategy, a faster way would be to use the  "constant" strategy and pass the most frequent value by yourself (ser.mode()[0]) then the time is almost the same:

t0 = time.time()
imp = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
imp.fit_transform(arr)
print('Simple Imputer (Most Frequent) Time Elapsed:', time.time()-t0)

t0 = time.time()
imp = SimpleImputer(missing_values=np.nan, strategy='constant', fill_value=ser.mode()[0])
imp.fit_transform(arr)
print('Simple Imputer (Constant) Time Elapsed:', time.time()-t0)

t0 = time.time()
ser.fillna(value=ser.mode()[0])
print('Pandas Time Elapsed:', time.time()-t0)

And the time elapsed for each strategy:

Simple Imputer (Most Frequent) Time Elapsed: 14.320188045501709
Simple Imputer (Constant) Time Elapsed: 0.052472829818725586
Pandas Time Elapsed: 0.04726815223693848


How can I replace outliers with maximum non-outlier value?

My answer to the first question is use numpy's percentile function. And then, with y being the target vector and Tr the percentile level chose,  try something like

import numpy as np
value = np.percentile(y, Tr)
for i in range(len(y)):
   if y[i] > value:
       y[i]= value

For the second question, I guess I would remove them or replace them with the mean if the outliers are an obvious mistake. But your approach seems reasonable otherwise.


The explanations are as follows:

$m_H:(0,1)^2 \rightarrow \mathbb N$ is a similar notation to $f:R^n\rightarrow \mathbb N$ which means it takes a n-dimensional input consisting of real numbers only. In the case of PAC learning the input is 2 dimensional consisting of numbers between $0$ and $1$ only which stand for values of $\epsilon, \delta$ respectively.
The integer $2$ as explained above is the dimension of the input vector.
$\rightarrow \mathbb N$ means mapped to natural numbers. In case of PAC learning that for each value of $(\epsilon, \delta)$ a function $m_H$ maps it to natural numbers, simply put $m_H(\epsilon, \delta) = $


Chi-square test - how can I say if attributes are correlated?

Chi-Square is used to determine which of the attributes are most informative. Its used in feature Selection.

So, if you have an attribute A ,B and C and output Y, we are trying to know Y depends on A or B or C ? A or B or C might be independent also i.e. has no affect on output Y.

So Chi-Sqaure is a statistical test to find out which attribute is independent and can be removed.

A contingency tables is created for each attribute value and its frequencies/occurrence is recorded and p_values above/below threshold determines if its relevant or not.

More about it here-
https://machinelearningmastery.com/chi-squared-test-for-machine-learning/

  Online, I read about significance level, but i didn't quite understand
  it. In particular, is it at 0.5 or 0.05? Which values are "ok"?

Please read about p-value


There might be many reasons for NAN's-

https://stats.stackexchange.com/questions/325451/cost-function-turning-into-nan-after-a-certain-number-of-iterations

https://stackoverflow.com/questions/33962226/common-causes-of-nans-during-training

basically LSTM is not able to train. try changing parameters esp activation function.

I haven't worked on python but what is the use of metrics=['accuracy']) in ur code, when u have already given cross entropy as loss function.


Orange3: Is it possible to get the total number of rows for a given column from inside 'Feature Constructor'?

Python Script widget looks like it might fit your needs, where, according to the docs, a code like this should give you the number of rows in a data set:

n_rows = len(in_data)


Is consecutive multiple transformation of Y variable a valid approach statistically?

Combining analytical transformations (say $f$ and $g$), as you suggest, is nothing else than applying a single $u = f\circ g$ transformation, so it is as valid as applying any simple transformation.

You just need to make sure that the transformation works (in this case, $y>0$ in the whole dataset and in the predictions, so that $\sqrt{log(y+1)}$ is real), and of course that it remains invertible, i.e. the initial variable can be retrieved from the transformed value, without ambiguity (this requires the transformation to be injective).

The transformation you suggest seems valid. You could also try a second log-transformation instead of the square root.


Linear Regression in python with multiple outputs

Both Lasso and SVM are available in sklearn library. Lasso: sklearn.linear_model.Lasso. SVM: sklearn.svm.SVT

An example from Lasso page:

>>> from sklearn import linear_model
>>> clf = linear_model.Lasso(alpha=0.1)
>>> clf.fit([[0,0], [1, 1], [2, 2]], [0, 1, 2])
Lasso(alpha=0.1, copy_X=True, fit_intercept=True, max_iter=1000, normalize=False, positive=False, precompute=False, random_state=None, selection='cyclic', tol=0.0001, warm_start=False)
>>> print(clf.coef_)
[0.85 0. ]
>>> print(clf.intercept_)
0.15...

In your case clf.fit looks like this:

clf.fit(X, Y)

X should be the size (nn,n)
Y should be the size nn
Where nn is the number of observations (points) and n is the number of variables. So rows in X are observations and columns are different variables.

If you have more variables than observations then you should read this post about the problems you can have with it and how to solve them.


What is done first, cross validation or grid search?

Well, grid search involves finding best hyperparameters. Best according to what data set? a held out validation set. If that's what you mean by cross validation, then they necessarily happen simultaneously.

It doesn't really make sense to do something called cross validation before testing hyperparams - indeed, what would you be evaluating?

CV as in k-fold cross validation can also happen within each model fitting process in the search, to produce a better estimate of the loss (and its variance, which is useful in more sophisticated tuning procedures). I think this is less usual but valid.

It's possible to use CV when fitting the final model after hyperparameter search. It might give you a better estimate of the loss, or confusion matrix, as you compute many of them. But each model you fit isn't using all available data. I think it's probably more conventional to take the best model's parameters and loss / confusion matrix, from the fitting process, as an estimate of generalization, and then refit the final model on all data. This means no CV at that stage.


Linear Discriminant Analysis + bayesian theorem = LDA classifier??

I found the answer in: https://machinelearningmastery.com/linear-discriminant-analysis-for-machine-learning/

  Making Predictions with LDA

  LDA makes predictions by estimating the probability that a new set of
  inputs belongs to each class. The class that gets the highest
  probability is the output class and a prediction is made.

  The model uses Bayes Theorem to estimate the probabilities. Briefly
  Bayes’ Theorem can be used to estimate the probability of the output
  class (k) given the input (x) using the probability of each class and
  the probability of the data belonging to each class:

  P(Y=x|X=x) = (PIk * fk(x)) / sum(PIl * fl(x))

  Where PIk refers to the base probability of each class (k) observed in
  your training data (e.g. 0.5 for a 50-50 split in a two class
  problem). In Bayes’ Theorem this is called the prior probability.

  PIk = nk/n

  The f(x) above is the estimated probability of x belonging to the
  class. A Gaussian distribution function is used for f(x). Plugging the
  Gaussian into the above equation and simplifying we end up with the
  equation below. This is called a discriminate function and the class
  is calculated as having the largest value will be the output
  classification (y):

  Dk(x) = x * (muk/siga^2) – (muk^2/(2*sigma^2)) + ln(PIk)

  Dk(x) is the discriminate function for class k given input x, the muk,
  sigma^2 and PIk are all estimated from your data.

So yes, it uses Bayes Theorem to estimate the probabilities.


Application of ideas from graph theory in machine learning

Graphs are a very flexible form of data representation, and therefore have been applied to machine learning in many different ways in the past. You can take a look to the papers that are submitted to specialized conferences like S+SSPR (The joint IAPR International Workshops on Structural and Syntactic Pattern Recognition and Statistical Techniques in Pattern Recognition) and GBR (Workshop on Graph-based Representations in Pattern Recognition) to start getting a good idea of potential applications. Some examples:

Within the Computer Vision field, graphs have been used to extract structure information that can later on be used on several applications, like for instance object recognition and detection, image segmentation and so on.
Spectral clustering is an example of clustering method based on graph theory. It makes use of the eigenvalues of the similarity matrix to combine clustering and dimensionality reduction.
Random walks may be used to predict and recommend links in social networks or to rank webpages by relevance.


select hyperparameters using Latin hypercube sampling (LHS) from a large matrix/grid of parameter combinations

dials from Tidymodels has a grid_latin_hypercube function you can use for this https://dials.tidymodels.org/reference/grid_max_entropy.html


Can the dependency between variables be deduced from data? And if so, how?

Yes, We can do that. We can measure the linear relation between two variables by correlation metric to know more see this. If you want to identify the linear relationship between more than two variables we can do it by VIF scores. If a VIF score of a variable is >5 we can say this variable can be expressed as linear combination of other variables to know more see this. If you want to implement in python consider this.


Large scale Autoencoder for friendship recommendation

Autoencoder will not predict potential friendship. Autoencoders learn a lossy compression of the data. A lossy compression of existing friendship relationships will not predict new relationships.


Why is the variance of my model predictions much smaller than the training data?

The first thing you can do here is to scale your output data before training the model. For instance
$$
\dfrac{x_i - \bar{x}_{train}}{\sigma_{x_{train}}}
$$
If that does not work, then you can have a look at the model and / or training parameters.


F1 will never be zero, but very near to zero for a bad classifier. If TP or TN is zero then there isn't any need to check F1.


how bidirectional neural networks can be applied on time series while we do not know the future data?

You are correct in your general understanding of bidirectional recurrent neural networks, they do utilize information from both the past and the future.

However, they are not usually used for predicting the future. Instead they are mostly used for tasks like: Speech Recognition, Translation and Handwritten-Recognition.

For these uses, the "prediction", or more generally the output of the model, is based on a global-scale (a big chunk of text like a full sentence or a paragraph), while the bidirectional behavior works on a local-scale (single words).

In simple terms, when we want our model to predict the meaning of a full sentence. We need it to understand the meaning of the specific words composing it. But in order to do that, we use the words that come before (past) and after (future) each specific word.


Creating data model out of .csv file using Python

In my opinion this is probably achieved fastest with a rule set. You can easily load and manipulate data via pandas. Then you just have to determine what are rules. But without any details its difficult to go more into details. Regular expressions might be a powerful tool to use but that is speculation given the sparse info you provide.


What to choose: an overfit model with higher evaluation score or a non-overfit model with lower one?

What library are you using?
Dropout is used during training to prevent overfitting.

Make sure dropout is not applied for the validation (which is standard for Keras). This could artificially decrease your validation accuracy.

Also, accuracy is a bad metric to evaluate your performance. See this answer to find out why. Try ROC-AUC to evaluate your model performance.


Use following WHERE predicate

-- data (Jan-Dec) of the year 2015
year = 2015  OR
-- data for the year 2014 from May-Dec
year = 2014 and month in ('May','Jun','Jul','Aug','Sep','Oct','Nov','Dec')

Adjust the column names and/or litaral values as required.
If the month is stored as number you may use month between 5 and 12


How can I compare the grammatical complexity between two texts using their sentences dependency length?

If your goal is only to determine whether the level of complexity is the same between these two sources based on these particular features, yes you can simply compare the distributions with a Student's t-test if the distributions are normal or a Wilcoxon test if they are not.
Spoiler alert: it's very likely that they are different. A statistical significance test doesn't give you much information, it's usually much more useful to try to quantify the level of complexity, but it's also much harder. Based on your previous linked question, you might be interested to read about text complexity/readability, quite a lot of research has been done on this topic (e.g. here, here, or there, among many others). There are also a few general tutorials apparently (e.g. this or that). I also know that there have been a number of readability/complexity metrics proposed in the literature, but I don't have references.
Note that looking at the features from the text is unlikely to be very useful on its own. Probably you will need either some kind of corpus annotated by complexity, or to use a metric which has been proved to correlate with sentence complexity.


Value Error: Operands could not be broadcast together with shapes - LSTM

The answer that OP provided is correct, yet I would like to elaborate a little more on it, in an attempt to shed more light.
First of all, you have to understand what is performed by the call
reframed = series_to_supervised(scaled, 1, 1)

in line 50.
Shortly, let us assume that the input has n samples of m features (variables). This means that the shape of the scaled dataframe is (n) x (m). The particular call to series_to_supervised() in line 50 implies that the scaled dataframe will be reshaped into a (n-1) x (2m) dataframe, to include the current and the 1-step look-back values, after dropping the row of NaN values due to the shift operation inside the body of series_to_supervised(). Most likely, your error is caused by line 52:
reframed.drop(reframed.columns[[9,10,11,12,13,14,15]], axis=1, inplace=True)

because you are probably dropping an arbitrary number of columns. The code is structured so that it forecasts one time series based on the past values of multiple time series (including the signal of interest). Therefore, you have to drop (m-1) columns, so that in the end all m past values persist, along with the current value of the signal of interest that will be used as ground truth. This means that the shape of reframed should be (n-1) x (m+1) after line 52. If not, you probably get that error with the dimension mismatch when reverting the scaling.
At this point I would like to emphasize on the fact that you should be very careful with logical errors when dropping columns, because you want to drop all but the signal of interest in line 52. If you keep any other column the code will still run, but you will certainly get logical artifacts.
Moreover, if you drop any column other than the first one (at index 0), you have to modify the concatenation in lines 85 & 90 accordingly, otherwise you will also get logical artifacts.


What is the best way to pick the optimized configuration from this dataset?

I have not executed the below code but I would do something like this,
df.loc[(df['Lighting Levela (Max)'] == max(df['Lighting Levela (Max)']) &
       (df['Energy Consumption (Min)'] == min(df['Energy Consumption (Max)']) &
       (df['Visual Comfort (Max)'] == max(df['Visual Comfort (Max)']) &
       (df['Thermal Comfort (Max)'] == max(df['Thermal Comfort (Max)']) &
       (df['CO2 levels (Min)'] == min(df['CO2 levels (Max)']), :)


How to compare two dataframes and put the counted unique values in a first dataframe's column?

What you want can be achieved using themerge function in pandas. Here is the code:

df1 = pd.DataFrame({'FID_1':[1,2,3]})
df2 = pd.DataFrame({'NEAR_FID':[1,1,1,1,2,2,3,3]})

df_freq = df2.NEAR_FID.value_counts().reset_index()
df = pd.merge(df_freq, df1, how='inner', left_on=['index'], right_on=['FID_1'])
df
       index  NEAR_FID  FID_1
0      1         4      1
1      3         2      3
2      2         2      2

You can remove any of the column index or FID_1 to get your desired output.

For more on merge function look at the official documentation
 and for a very good explanation of its working look here.

EDIT 1:
Merge function only accepts Dataframes and not series. So, if your data is in series format, convert it to dataframe first.
You can check the type of your data by:

print(type(df2))

And that should output pandas.core.series.Series

Use this to change it to a dataframe:

df = df.to_frame().reset_index()


Unbalanced classes -- How to minimize false negatives?

Class imbalance is a very common problem.  You can either oversample the positive class (or undersample the negative) or add class weights.

Another thing to remember in this case is that accuracy is not a very useful metric here.  You might consider AUC or F1 score.

Changing your decision threshold may seem appealing, but will obviously lead to (in this case likely drastically) increased false positives (though perhaps FPs aren't as bad as FNs in the case of medical diagnosis, if tests aren't prohibitively expensive).

A more in-depth analysis of options in the case of class imbalance is provided here.


How to adapt output with multiple columns for better presentation?

You can use kable() (see manual to create tables that are directly in markdown format.

Your code would become:

```{r,results='asis'}
 knitr::kable(opneg1)
```

See also this great manual on rmarkdown.


How is Bayes theorem being applied in expanding the formula of Binary Independence Model?

It can be read as the probability of R=1, given (x and y) as your second formula.
Anything after | stand for the given part.

Since:

$P(x | r=1 \wedge q) = \frac{P(x \wedge R=1 \wedge q)}{P(R=1 \wedge q)}$

$P(R=1|q) = \frac{P(R=1 \wedge q)}{P(q)}$

$P(x|q) = \frac{P(x \wedge q)}{p(q)}$

Therefore:

$$P(R=1|x,q) = \frac{P(x | R=1 \wedge q)*P(R=1|q)}{P(x|q)} =$$

$$\frac{P(x \wedge R=1 \wedge q)}{P(R=1 \wedge q)} * \frac{P(R=1 \wedge q)}{P(q)}* \frac{P(q)}{P(x \wedge q)} =$$
$$\frac{P(x \wedge R=1 \wedge q)}{P(x \wedge q)}$$


Why TREC set two task: document ranking and passage ranking

The idea behind the two tasks is to explore how document length affects the effectiveness of the different retrieval models.

Ellen Voorhees
TREC project manager
NIST


Data Exploration Led Conversion to Ordinal Variable

You are describing highly correlation features. The most common way to measure the correlation between two variables measured on at least an ordinal scale is the Spearman rank-order correlation coefficient.
Generally if two features are near perfectly correlated, one feature can be dropped from analysis.


Predicting correct match of French to English food descriptions

As you suspected, the best approach would be to take a massive multilingual pretrained language model and make use of the information about French and English that it has already learned. You can read about some good options here.

The basic idea is to train a new, lightweight network to make predictions based on the output from the pretrained model; its usual to just have a single layer feed forward network for this “fine-tuning”. Some implementations will already have this conveniently coded up for you, so check the documentation for whatever you decide to use!

Your problem is specifically a sentence pair classification problem, and there is a tutorial for that here. Pay close attention to the data processing phase of the tutorial. Overall, the differences you need to apply to what the tutorial describes are

You need to use multilingual BERT
You need to prepare your data exactly as the tutorial says about how to set up your data, but use your own snippets in place of the sentence pairs


EM parameter estimation for conditional Gaussians

When $X_1$ is unobserved, at iteration $k=1$ of EM, the posterior mean value (when $X_2=3$) is $5.18$ by using an inference algorithm, i.e. Junction tree/kalman filter.
Then the sufficient statistics for $X_1$ is:

$s_1=\Sigma_{i=1}^nx_{i1}=9+4+5.18$ and $s_{11}=\Sigma_{i=1}^nx_{i1}^2=9^2+4^2+(5.18^2+\sigma_{11.2})$

where $\sigma_{11.2}$ is the posterior conditional variance of $X_1$ given $X_2$, when $X_1$ is unobserved.

$s_2=\Sigma_{i=1}^nx_{i2}$ and $s_{22}=\Sigma_{i=1}^nx_{i2}^2$. When $X_2$ is unobserved a term $\sigma_{22.1}$ is also added to $x_{i2}^2$.

So parameters $\mu_{X_1}$ and $\sigma_{X_1}$ are obtained. Similarly, posterior mean of unobserved $X_2$ is also obtained by using i.e. Junction tree inference.

The sufficient statistics for $s_{12}=\Sigma_{i=1}^nx_{i1}x_{i2}$, and $s_{21}=\Sigma_{i=1}^nx_{i2}x_{i1}$.

Mean vector is then: $\mu^{(k)}=(s_1/n, s_2/n)$.

Covariance matrix is: $\Sigma^{(k)}=$$\left( \begin{matrix}
   \Sigma _{11} & \Sigma _{12}  \\
   \Sigma _{21} & \Sigma _{22}  \\
\end{matrix} \right)$$=$$\left( \begin{matrix}
   s_{11}/n-(s_1/n)^2 & s_{21}/n-(s_2/n)(s_1/n)  \\
   s_{12}/n-(s_1/n)(s_2/n) & s_{22}/n-(s_2/n)^2  \\
\end{matrix} \right)$$
 $

Other parameters in the question are obtained. Conditional mean and conditional variance of $X_2|X_1$ is calculated by the formula in the reference of the question.


Which loss function is the best loss function when using XGB regression with highly skewed dataset?

You could try weighting the loss for each data point, such that some datapoints do not dominate the loss.


NEAT XOR Example gives different results at each run

Provided each network implements the XOR function approximately, but within reasonable error bounds, then this is NEAT behaving as designed.

Neural networks allow for multiple equivalent solutions. Even using a fixed architecture and stochastic gradient descent to learn a simple function, there is a good chance of ending up with very different weights each time. With NEAT, it also explores alternative architectures, and the search includes many calls to random number generators in order to make decisions.

The main deterministic step in NEAT is the selection process - comparing population members with each other to rank them in terms of fitness. However, as multiple designs of neural network can all solve the XOR problem roughly equally well, it is unlikely this will find the same one each time.

If you need to exactly reproduce an experiment in NEAT, you can achieve that by setting the RNG seed. I think this could just be a call to random.seed() for the library you are using, but some libraries may have other RNG instances that need setting up too.


Intensity image to RGB for transfer learning

If you mean images classification networks like AlexNet (and I suppose images segmentation ones too), then at first layer they detect color-agnostic features as well as color-dependent ones:  (https://arstechnica.com/science/2018/12/how-computers-got-shockingly-good-at-recognizing-images/3/). Here blue color mean that blue component is necessary for the filter to fire stronger, but (I believe) that G and B and not important for it. For magenta (R+B) red and blue are necessary anb green is not important. If the filter is white on the left, black on the right, it will fire when intensity raises from right to left.

Transfer learning has to work if your image has the same nature as the train photos (at least partially). So you want filter detecting legs to detect something similar to legs on your images. And it will detect if lower-level filters detect parts of leg (say rectangle - increase of intensity when background is darker than the leg and then decrease back – background-leg-background). If your leg is detected as a set of petals connected with dogs’ noses instead, there will be no sense in this, even if source net detect this as flowerbed with confidence. This will be a noise.

So you of course could pack your color intensities into RGB triples. Say for simplicity in the form of C letter on the R+G plain. You will get 256*3 values. But this would break the correspondence with the nature of your image. Since the net will see the packed values. Even if everything is good when you move from RG (0, 0) -> (255, 0) -> (255, 255), instead of intensity raising, it will decrease on the way -> (0, 255). This will likely rotate inside out filters that should detect this exact raising of intensity. Sometimes maybe it’s not important, there are pairs of opposite filters or somehow else… But I think far from always so still…

I would simply map your 0-N to gray (0, 0, 0)-(255, 255, 255), loosing of precision is so small and typical problem in real-life tasks... If you really need this resolution, maybe take windows of maximally appropriate size (e.g. v=0 for input 0-1000, linearly to 255 for input 1000-2000, 255 for bigger input values). Everything is MHO and as a first idea. More information about your task could help.


Estimating effort of building machine learning model

Data Loading & Pre-Processing : depending on your data-size, this is a highly variable measure. we can agree that this will be the second longest task.

Edit : This phase ,for me, was about collecting the relevant columns and structuring a csv file to be read and loaded in a python environment.

Model Development : again, this will highly depend on your data ( and i'm referring to complexity and structure of your data rather than the size) , if you have a lot of features at hand, and your problem is complex ( demands a lot of effort in feature engineering and feature selection ) , this will be the longest task at hand and will probably take up most of your time
I would say deployment would take something in weeks like you mentionned depending on the complexity ( yes again ) of your production environnement.

Please take my statement lightly as i don't have much experience in this field. I've had the opportunity to deploy some models as webservices, and it took me less than your projections. However, our problems are quite different.. so my numbers could be out of match.


I received this error message: Found input variables with inconsistent numbers of samples: [15573, 15600]. How do I resolve it please?

You likely get problems because you remove rows containing nulls in X_train and y_train independent of each other. y_train probably has few, or no nulls and X_train probably has some. So when you remove a row in X_train and the same row is not removed in y_train it will cause your data to be unsynced and have different lenghts. Instead you should remove nulls before you separate X and y.

Before this:

X = df['text']
y = df['label']

Do this:

df.dropna(inplace=True)

And remove this:

X_train.dropna(inplace=True)
X_test.dropna(inplace=True)
y_train.dropna(inplace=True)
y_test.dropna(inplace=True)


Dealing with low-information centroids using Nearest Centroid Classifier and bag of words method

This is a standard problem with distance/similarity measures between texts of different length. I'm not aware of any standard way to solve it, but in your case I would simply try to remove any email shorter than a certain length from the training set (you can experiment with different thresholds). This would hopefully force the centroids to be more specific, the goal being that none of them can easily attract all the instances.


how to update the pre-trained word2vec model with new train data using genism

You cannot use keyedvectors for resuming training as per Gensim documentation. You will need to save the full model in previous training run to allow resuming training. Keyedvectors remove training related information and other metadata to reduce memory footprint.


Should I standardize first or generate polynomials first?

Depends on what are you using the standardization for and how the features relate to your problem, but I guess for most situations you’re better off standardizing them afterwards.

If you are looking for some standardization in order to make optimization of some algorithm faster (e.g. neural networks rather than logistic regression), it might be a better idea to standardize afterwards, as otherwise it defies the purpose.

If you are doing it in order to introduce some regularization, then standardizing them to 0-1 first will make the polynomials smaller overall and thus more likely to be given weights compared to the original features, which is probably not what you want.


If you're interested more generally in speech understanding or speech-to-text, some approaches to natural language parsing and speech-to-text use recurrent neural networks or Hidden Markov processes for learning, as well as a number of signal-processing algorithms to extract more data from the input stream that just raw audio. Keep in mind people have spent their whole careers on this work, so it's not a good problem to just pick up and run with unless you're a MS/PhD candidate looking for a capstone/dissertation project. Here's the iconic paper from Bell Labs that inspired a lot of the DFA/HMM solutions. I've yet to find a paper that does a good job of explaining how to actually implement the RNN style solutions, but here's one in case you're interested.

It's likely that Alexa uses some combination of these methods, but I doubt you're going to get any good answer out of anyone here. After all, it's an important Amazon project and it's not like their engineers are going to come on Stack Overflow and start giving away trade secrets.


I take a look at their github.

Here are the relevant parts:

local stepSize = lr/biasCorrection1 --math.sqrt(biasCorrection2)/biasCorrection1

stepSize=math.pow(2,torch.round(math.log(stepSize)/(math.log(2))))

and also

state.v:copy(torch.pow(2,torch.round(torch.log(state.v):div(math.log(2)))))
state.v:add(epsilon)
tmp:addcdiv(1, state.m, state.v)

It seems that for $\alpha \oslash (1-\beta_1^t)$, what is being done is $$2^{\lfloor \log_2 \frac{\alpha}{1-\beta_1^t} \rfloor}$$

which is a left shift if $\lfloor \log_2 \frac{\alpha}{1-\beta_1^t} \rfloor > 0$ and a right shift  if $\lfloor \log_2 \frac{\alpha}{1-\beta_1^t} \rfloor < 0$.

On the other hand, for $m_t \oslash v_t^{-1}$, what is being done is

$$\frac{m_t}{2^{\lfloor \log_2 v_t\rfloor}}$$

which is a right shift if $\lfloor \log_2 v_t\rfloor> 0$ and a left shift if $\lfloor \log_2 v_t\rfloor < 0$.


How do you determine the architecture of a model(keras)?

There is no theoretical understanding, that would take a problem and specify the optimal network architecture for you. So no, there is no formula, I'm afraid. What does exist, are strategies to arrive at good solutions.

Good strategies are to start simple and try to make iterative improvements. Define a single KPI (error metric) that you're trying to optimize, build the simplest model you can imagine that will hopefully work a little bit, then try to improve from there. You get improvements by trying to understand why your model does not perform optimally. You need to be able to distinguish between underfitting and overfitting to make the right modifications. Study the errors that your model makes on specific images and try to make improvements to address those mistakes with changes to the model.

I think you may find this course on Coursera interesting: Structuring Machine Learning Projects.


Using a pipeline and transforming data with imputing and OneHotEncoding performs worse than get_dummies

"(Making) dummy variables" and "one-hot encoding" are the same thing, yes.  (The phrasings originate from statistics and computer science, respectively.)

pandas.get_dummies and sklearn's OneHotEncoder both implement this process, but have some differences in their (current) implementations.

Skip to the end if you're not interested in some details that might not be relevant for your underlying question.

I think there are three basic questions to ask:

What do we do if the train set has levels the test set doesn't?
What do we do if the test set has levels the train set doesn't?
Do we want one dummy variable for every level?  [Otherwise, you get perfect multicollinearity, bad for some models.]

Does "missing" count as a level?

(And to be more general, production sets count here as test sets, and validation sets can count as either training or test sets depending on your usage.)

In both (1) and (2), we'll have problems applying a model fitted on the train set to the test set without rectifying the differences.  In (1), we just need to provide the extra columns to the test set (which will be all-zeros, but that's what we expect).  In (2) we have more serious troubles: the model has no idea what to do with those new levels.  The only thing to do is to find a reasonable way to convert those new levels to the "closest" training levels (which requires domain knowledge and/or a separate analysis), or failing that, just encode those levels as all zeros (essentially converting those levels to the "reference level" of the training set, i.e. none of the training levels).

pandas is more about data manipulation and aggregation, not about modelling, so doesn't really need to care about what to do with these issues.  However, align gives us a way to work it out: it will conform the two sets' column names, filling in gaps with np.nan.  Using the left-join as in your code takes care of (1).  (Using outer-join would take care of the programmatic problem in (2), but the train set will still have a column of all nans, so the model won't actually know what to do with those levels.)  Finally, get_dummies helpfully has the options drop_first and dummy_na to take care of (3).
(pandas version 0.25)

sklearn of course is all about modeling, and cares a great deal about train/test splits.  OneHotEncoder then has the fit method that bakes in the fitting set's categories (later available with the attribute categories_), and the transform method uses only those categories for later use (with e.g. test sets).  It has a useful option handle_unknown, that when set to 'ignore' takes care of (1).  There has been discussion/work toward giving options to handle (2) [namely, providing an "other" level], as well as dealing with missing values [currently it produces an error].  It can take care of (3) with the option drop.  Things are a little more difficult here too because sklearn uses numpy arrays internally instead of dataframes, so the columns don't actually get names.  (It might be worth mentioning that earlier versions of sklearn was much more restrictive on the inputs to OneHotEncoder.)
(sklearn version 0.21)

Alright, finally to the base question.  It may be that the slightly different options above produce different enough encoded datasets as to cause you problems in modeling.  (Something else to check: the methods are dummifying the same columns, right?)  It is more likely, I think, that the imputation method you've employed has hurt performance.  I'm not familiar enough with the Housing Prices dataset to know, but if the missing values aren't missing at random, then imputation might misrepresent the truth.  And, since XGBoost can handle missing values without imputing, you might have more similar results by skipping this step.


Improving SVM binary classification model on new dataset

Online learning for SVMs is supported only for certain SVM options (a linear SVM with SGD can easily update for example).

There are implementations out there to address the issue of online learning for all SVMs.

https://stats.stackexchange.com/questions/26041/can-svm-do-stream-learning-one-example-at-a-time

If your model isn't terribly large it also probably isn't too bad to train a new one on all data periodically.


CNNs like U-Net extract lower level features like edges on lower layers (i.e. the first convolutional layers) and higher level features on higher layers (i.e. convolutional layers closer to the final linear layers). This principle is losely inspired by how visual perception is implemented in the Visual Cortex among humans (and other animals).

In a CNN the feature maps could for example look like this:

As you can see the lower level feature maps detect simple structures like edges while higher level feature maps recognize more complex structures like eyes or faces.

Colors, however, are processed differently by CNNs than spatial features since color images are usually fed to CNNs using three input channels (one for each color in RGB format). So colors are not detected in the same way as spatial features but instead the first convolutional layer receives a 3-dimensional input image with one dimension for each color-component (one for red, one for green, one for blue).

The article What exactly does CNN see? and the paper Visualizing and Understanding Convolutional Networks (which are also the sources of above images) provide a more detailed explanation.


Which text clustering algorithms should I use to group free text fields in Python?

That is called "entity resolution" or "record linkage". It is a very hard problem.

If it all possible, I would try to add "type ahead" to the form. That would encourage users to select already used categories.

fuzzywuzzy is a Python package for approximate string matching.

dedupe is "a Python library that uses machine learning to perform fuzzy matching, deduplication, and entity resolution quickly on structured data."


Green circles and blue squares are samples of two different classes. How does it matter that "X" belongs to which one of either of these "Green circle" or "Blue Square" class?

Likelihood should be done based upon the number of classes and the K value, instead of dataset samples.

  For me likelihood weight will be:

  Likelihood of being blue square ~ 0.66 i.e. 2/3, whereas likelihood of
  being green circle is 1/3.

Prepared another example:
Where I have 2 Green circle and 16 blue square, whereas K=5

If I follow your approach, likelihood of
Green Square is 1/2, Blue square is 4/16 = 1/4, whereas from the image it's clear that "X" belongs to Blue Square class only.

[Update]
Thanks @Ben Reiniger, for correcting me.

I spent more time and what I understood is, it's not always correct to put equal weights(likelihood) across different classes in dataset. My earlier observation was based upon the assumption that all classes have equal weights and I was wrong, even though data is bit skewed.

Consider an another example:

We have a huge dataset of patient reports, where features are created based upon the several tests done so far. Our task is to identify the possibility if a particular person suffering from cancer or not.

In this case, even if a single feature point towards cancer disease, we cann't neglect it and should predict towards rare class, such that patient can go for further analysis.

In such cases weights are not equivalent across different classes.
According to me, wights should be defined based upon the use case.

A very well explained its calculation in How does Weighted KNN works?


Can theano work on mapreduce or on spark?

I don't think theano have spark support yet, however there are at least 2 deep learning libraries that are good with that. The first one is mxnet, they have support for spark, R, python and C++. Another option is deeplearning4j, made in Java with direct access to Spark.


Machine Learning and Natural Language Processing : Project Initiation

Starting with a small chunk of documents (~50) as a training set with a set of rules implements would be a good starting point. Train using a few algorithms and see if your training accuracy is coming out as acceptable or better than the human accuracy.

Then slowly increase your training batch size and see the impact on the results.

Once you believe that the training set is a partially good representation of the dataset, go ahead do a test run. Then keep growing from there.


Why is not AUC or other metrics used for splitting nodes in decision trees?

On accuracy:
Why we use information gain over accuracy as splitting criterion in decision tree?

AUC has been explored; it seems to work well, but is slower:
https://www.semanticscholar.org/paper/Learning-Decision-Trees-Using-the-Area-Under-the-Ferri-Flach/46e40f487e555277033f188778d6c5c05df8daa4
http://proceedings.mlr.press/v7/doetsch09.html


Supervised learning approach - creating my own labels

The approach you're describing might be good but the main question is how the automatic labeling works. You say you can "create a function to label the data based on behavior": what is the behavior? Do you manually decide the label or is entirely automatic? If it's completely automatic and general enough so that it can work for any instance, then you don't need any ML since you can directly apply your function. On the other hand if it's specific to a subset of data (or requires some amount of manual decision) then it makes sense, and the challenge will be whether the features are informative enough to predict the label for fresh instances from a different subset. ML assumes that your test instances come from the same distribution as the training set.

  Create a new feature column called "label" that classifies accounts based on a user defined function that IDs spikes in volume. Is this feature engineering ?

It's not feature engineering if it's the class that you will try to predict later with new instances. Feature engineering would be selecting particular features, for example instead of volume amount you could have minimum, maximum and mean volume, or discretizing the time, etc.


Computer vision: Identifying different items in screenshot

This is a good case for computer vision.  You know exactly what the icon looks like (basically pixel-perfect), and you want to find whether it exists anywhere in the screenshot.  This is an image matching problem.

Normalized cross-correlation (NCC) is a standard way to check whether small image $I$ (your icon)  appears anywhere in the large image $I'$: you compute the NCC and then look for a location where the value of the NCC is large; that is your candidate match.  You can now apply this to the set of candidate icons: for each candidate icon, search whether it appears anywhere in the screenshot, using NCC.  OpenCV and other computer vision libraries should include normalized implementations of the NCC computation.

For speed, you can also try using image pyramids, but try using NCC directly on the full screenshot first.


How regression algorithm works on categorical features

TL;DR: It is a convention to convert categorical features to numeric features before you can use them in regression, or any other machine learning algorithm for that matter. Technically, there is nothing that is stopping ML algorithms from working with categorical features but their software implementation would be prohibitively expensive, hence this convention in ML practice.

Turning to your problem, if you are trying to convert categorical label values into numerical values, there are various methods but it appears the best one for your data would be One-Hot encoder. Both Scikit-Learn and PySpark, and most other libraries provide handy functions for it, as it is very common operation. For example:

from sklearn.preprocessing import OneHotEncoder
one_hot = OneHotEncoder()
one_hot.fit(...your_columns...)

Once you have all data in numeric form, you can use pretty much any algorithm.

As for the last figure, it is scatter plot between x and y, and it is not obvious what x and y are. Weirdly enough, it is not one-on-one relationship because a given value of x seems to produce multiple value of y?!!! And I am not sure what you are plotting in earlier figures that is giving you straight lines!


What is the distribution of the goal feature in this dataset

It looks like it's probably pareto-distributed- meaning that most goals are small, some are medium, and a few are REALLY big. It's fat-tailed and not normally distributed at all.
A nice way to see it is to do a histogram plot with log=True and bump up the number of bins.


How to grow a list of related words based on initial keywords?

The word2vec algorithm may be a good way to retrieve more elements for a list of similar words. It is an unsupervised "deep learning" algorithm that has previously been demonstrated with Wikipedia-based training data (helper scripts are provided on the Google code page).

There are currently C and Python implementations. This tutorial by Radim Řehůřek, the author of the Gensim topic modelling library, is an excellent place to start.

The "single topic" demonstration on the tutorial is a good example of retreiving similar words to a single term (try searching on 'red' or 'yellow'). It should be possible to extend this technique to find the words that have the greatest overall similarity to a set of input words.


How to add more features in addition to a 100D word vector

You can add a second Input layer to your architecture.

Look at this link Keras Guide Multiple Inputs for a more detailed explanation.

The output of the embedding layer can be combined with the secondary input and passed to the next layer.

auxiliary_input = Input(shape=(5,), name='aux_input')
x = keras.layers.concatenate([lstm_out, auxiliary_input])


tensorflow categorical data with vocabulary list - Expected binary or Unicode string, got [0,1,2,...]

I'm not completely familiar with TF API, but here's what I think is happening.

The library tells you that it can handle a binary column or a string. But you have all the ingredients listed in a single column. So the integer conversion of ingredient label is not helping.

You can instead create one column per possible list of ingredient and setting it to 1 if that ingredient is present or absent. For example, Italian cuisine will have column for tomatoes or garlic set to 1 for many records.

You can read more about get_dummies function in pandas library. If the original ingredient list comes in form of text, you can read up more about text feature extraction / bag of words APIs in scikit-learn libary.


Master of Statistics or Master of Data Science

As everyone here knows, data science as a field is broad, and similarly the experience required for a particular job is broad.

Depending on where you want to end up, either degree could suit you better. If you would like more of a traditional statistician's job which has been rebranded to data scientist in order to gain more attention, then perhaps the MSc Stats would be more applicable. However, if your field requires more complex computer programming or database management, you may want to look into the masters in data science or masters in computer science, depending on your background and interest. Going with a "hot" degree like data science means that you will likely learn a lot of common analyses in superficial detail in order to be employable quickly, but with a more traditionally academic MSc Stats or MSc Com sci, you will learn the theoretical basis with the potential loss of applicability. My personal bias is towards to academic side (MSc/PhD Stats or applied math), but I think this is a decision you should think hard about. Regardless of your decision, in general data science / analysis / programming / whatever this field is requires quite a lot of self directed learning, so whatever program you choose must provide you with the skills and tools to be able to teach yourself whatever you need to know.

I would personally take a hard look at what specific kinds of jobs appeal to you, and which programs may prepare you better. Talk to the admissions officers and look up the career paths of past students. Where did they end up? Is that where you want to be? Keep in mind the kind of life you want to live and the kind of work you want to do. I personally find anything financially or profit driven absolutely soul-sucking, but enjoying doing almost the same work in the medical field. Different skill sets, different programming languages, and different vocabulary are required, and the more you know before you enter the field the better you can prepare yourself for the the journey to come.

Regardless, have fun, and don't worry too much about making mistakes. No one knows how things will work out in the long run. You can prepare yourself with as much knowledge as possible, but you'll never know which was the truly right choice, just which one was your choice.


Where to store dataset when creating neural network models and using github?

If the dataset isn't too large, you may add datasets on GitHub itself. Generally project datasets aren't large, but again, I can't be sure as you haven't mentioned any approximate size. But if it is large, you may refer to data source in your ReadMe or within your .py file as an external hyperlink.

For example, save dataset on Google Drive & turn on Viewable option by generating shareable URL. Now that generated link can be refered to in your ReadMe file as something like: "Kindly refer this external link to access dataset". Alternatively, you may also read data directly within your .py file from that generated link, and any project reviewer would notice that external link being used to read dataset.

Hope that answers your query!


You can use ImageHash

The difference between the hashed images will give you a similarity score

from PIL import Image
import imagehash

hash = imagehash.average_hash(Image.open('test.png'))
otherhash = imagehash.average_hash(Image.open('other.bmp'))

print(hash - otherhash)

https://pypi.org/project/ImageHash/


Why are the ANN training and validation accuracy graphs not smooth?

Considering that your validation accuracy has bigger steps than your training accuracy, this may simply be an issue of data size. Accuracy counts correct/not correct, so if the model switches its opinion on a sample the accuracy increases suddenly. Because your validation data is likely smaller than the training data the steps are bigger there.
There is nothing to worry about, this looks normal. On the other hand your validation loss is increasing, so you are overfitting.


Using machine learning technique to predict commodity prices

Based on your question there are couple of things which I would assume to answer your question:

As you need to predict the commodity price the data which is collected is time series data.
Since you want to use other commodity to predict, it means that you don't have any past data of the product which you want to predict.

The answer could be derived by performing some exploratory analysis on the existing data. i.e., based on your business understanding you need to decide which product is similar to the new product. This kind of techniques is used to understand the sale of the new product/how is it going to perform after the launch.

Techniques which can be used over here are Time Series Analysis like ARIMA, if seasonality is present then SARIMA, if no trend then Exponential Smoothing(too many spikes),  there are other models like Auto Regression, Moving Average, Croston if there are 0's etc.

This is one way of looking at your problem.


Confusion for considering accuracy or standard deviation in selecting the best parameters

You are perfectly right to pay attention to the std dev across CV folds, especially with a small dataset. As you observed, different models show different values for the performance but also for the std dev, so you have to arbitrate a tradeoff between performance and stability:

The safe option is to choose the model with lower accuracy and low variance. It might not always perform optimally but at least it won't perform too bad.
The risky option is the high accuracy, high variance model: in average it will perform best, but you have a higher risk that it actually performs poorly.

This choice depends on the context, i.e. what the model is intended for.


Which dissimilarity/similarity measure use after a dimension reduction ( PCA / AutoEncoder / ... )?

This question will invite a lot of opinions and it will be very specific to the problem that you are looking at. But there's no harm in laying out the popular options -

1. Start with the data

a. Images
 - If you are working with images and plan to use a classifier like ConvNet, PCA probably isnt a very good idea. You should just resize the image using popular libs like opencv

b. Unstructured Text
 - I have personally looked at a lot of free text problems, and my goto method is using Cosine Distance after doing a TruncatedSVD (dimensionality reduction) on the bag-of-words data (or any way of structuring for that matter)

c. Structured
 - Usually PCA is the go-to solution, but only if you are ok with losing the interpretability in your downstream predictive models. Scree Plots are a good way to evaluate how good a PCA exercise was / how many Principal Components you need.

2. Methods

a. Auto-encoders
 - These are neural-network based dimensionality reduction technique and I haven't looked at a single production instance where in a predictive modelling workflow Auto-encoders outperformed PCA. I guess we just have to wait for the right use-case

b. Distance metrics - Euclidean is no-doubt the most popular one, but Cosine catches up well to free text problems. You also have Hamming distance, Mahalanobis distance and countless others but its really upto you to try that. I haven't come across a case where these were performing better.

While we are talking about distance metrics, also look at DBScan for clustering.

Also, Silhouette Distance is another effective way of measuring how good your clusters did.


What is a good approach for embedding both textual and spatial features for document classification?

When text is available as scanned image:

Divide your image into small grids.
Assign  each grid a row/column number like (i,j)
Now to your word vector append 2 more cells which are the row and column number of cell to which the word belongs.

When text is available as
Have the document in the form of html. Then have the embeddings for entire html DOM tree of that document, which would include both  tags and the actual text. This way html tags would be giving the spatial/positional information.


What do we learn from training a dataset for logistic regression

I hope this is what you are looking for:

In Linear regression, the parameters are learned (e.g. theta1,theta2, etc.) which are then used to help us in predicting the real value ( as it is the regression problem)
But In case of logistics regression (as it is used for classification purposes) the same parameters are learned but different activation function is used like sigmoid which results in the range of 0 to 1( which actually shows the probability of an outcome). So, based on your requirement, if the result of the sigmoid or any other activation function crosses a certain threshold value which can be any between 0 to 1, then it is classified as one class otherwise the other class.

For example, If after multiplying all the parameters and input values and then applying activation function on the multiplication result, the output value is 0.7 which let say is great than our threshold value(e.g. 0.5) then it is classified as YES otherwise NO.


Looking for an RL solution for sequence generation

To formulate your problem as an RL problem, we first need to index the functions from $1$ to $n$. This way, there would be $n+1$ actions in total, i.e.
$$\begin{align*}
a^i=\left\{\begin{matrix}
\text{stop} && i=0\\
\text{add } f_i && 1\le i\le n
\end{matrix}\right.
\end{align*}$$
Accordingly, state $s_t$ would be represented by a sequence of $t$ actions, where a final state ends with action $a^0$. Also, the reward at step $t$ (after doing action $a_t$) would be as follows
$$r_t = \left\{\begin{matrix}
-0.5 + 1_{F(s_ta_t)=True} + 3_{T(s_ta_t)=True} & a_t \neq a^0\\
0 & a_t = a^0
\end{matrix}\right.$$

To prepare such a representation for a neural network, we need to represent the actions with one-hot encoding. Here is an example,

s = [
      [0, 0, 0, 1, 0]  #a^1
      [0, 0, 1, 0, 0]  #a^2
      [0, 0, 0, 0, 1]  #a^0
    ]

which represents a final state with three actions (and four functions).

The tricky part of your problem is the ever-changing representation of the state. As an opposite example, for those video games that are studied in DQN paper, each state is a constant number of fixed-sized 2D images.

To solve this design challenge, we can opt for one of these two approaches (among possibly many others):

Limit the final length of states to $T$ and add a "not taken" action for those steps that are not taken (yet). For example,

 s = [
    [0, 0, 0, 0, 1, 0]  #a^1
    [0, 0, 0, 1, 0, 0]  #a^2
    [0, 0, 0, 1, 0, 0]  #a^2
    [1, 0, 0, 0, 0, 0]  #not taken
    [1, 0, 0, 0, 0, 0]  #not taken
  ]

for four functions and $T=5$. This way, we end up with a total of $n+2$ symbols for representing each state.
Using a recurrent neural network (RNN, LSTM, etc.) to encode the variable-length states to a fixed length representation; this is similar to encoding variable-length sentences to a fixed length representation in NLP tasks.

The problem is now completely formulated as an RL problem along with taking some implementation challenges into consideration.

From this point onward, we can do RL  by, for example, designing a 1D Dense network on top of RNN representations, or a 2D Dense or Convolutional network on top of fixed-length 2D representations (with $m \times (n+2)$-sized kernels that only move along actions, not along $n+2$ dimensions of one-hot encoding) that receives the representation of state $s$, denoted as $\phi(s)$, and outputs $Q(s, a)$ for each of $n+1$ actions, the same as DQN paper (see Algorithm 1: deep Q-learning with experience replay).

Extra notes

You may take a look at this post for feeding variable-length multi-dimensional sequences to LSTM using Keras.


Is it possible to use word2Vec to derive hyperonymy (hyponymy or ISA relation)?

A paper entitled "Dependency-Based Word Embeddings" found that word embeddings that defined context using dependency-based parsing generated cohyponyms. For example, "florida" was embedding near  "texas", "louisiana", "georgia", "california", and "carolina".


Predicting probability for each tag given already chosen tags

welcome to ML and data science. This is a classic situation where a RNN would be usefull. You could either train one from Keras yourself which would help you learn and get a better model but might take more time. Or you could use a more pre-made soloution if this is your first project maybe something like TextGenRNN (will still recquire a bit of modification).

RNN theory

Examples in python

More examples

TextgenRNN Library


LSTM Sequential Model question re: ValueError: non-broadcastable output operand with shape doesn't match broadcast shape

The final dense layer's units should be equal to the number of features in your y_train.
Suppose your y_train has shape (11784,5) then dense layer's units should be 5 or if y_train has shape (11784,1), then units should be 1. Model expects final dense layer's units equal to the number of output features.
You have to identify which features you need in input and output.
'open', 'low', 'high' and 'close' and 'volume' - these are your features.
What do you want to predict? Is this a classification or regression problem? What is your problem statement?
Based on these, identify your input and output features.
Put input features in x_train and output features in y_train.
Now, in the final dense layer, you should use units equal to the number of features in y_train.


How to combine and separate test and train data for data cleaning?

There are several methods to choose from. If you insist on concatenating the two dataframes, then first add a new column to each DataFrame called source. Make the value for test.csv 'test' and likewise for the training set.
When you have finished cleaning the combined df, then use the source column to split the data again.
An alternative method is to record all the operations you perform on the training set and simply repeat for the test set. This won't work it you normalise values based on the population.


What is the best way to predict time series data?

You can find the seasonality in your data (if there exist) and by removing the seasonality, find the trend of tomato price and predict the future price using the seasonality data and trend data. If you are using python, you can find the seasonal library useful to apply this method to your data.

The advantage of this method to your method is you don't need to think over finding dependent features and any feature engineering.

Moreover, as you have mentioned in tags of your question, you can using RNN to predict the future value by as your data is time series and sequential intrinsically. You can find this article useful.


The PCA projections do not look not orthogonal because your figure axes are not equal.

Set all axis equal with something like this:

ax.axis('equal')

or

ax.xlim(-5, 5)
ax.ylim(-5, 5)
ax.zlim(-5, 5)
ax.gca().set_aspect('equal')


How to use SimpleImputer Class to replace missing values with mean values using Python?

Your error is due to using Simple Imputer's fit and fit_transform on a numpy array. Here's how i used it on a Dataframe

imr = Imputer(missing_values='NaN', strategy='median', axis=0)
imr = imr.fit(data[['age']])
data['age'] = imr.transform(data[['age']]).ravel()

  X.fit = impute.fit_transform().. this is wrong. you can't assign a value to a X.fit() just simply because .fit() is an imputer function, you can't use the method fit() on a numpy array, hence your error!

Use x[:, 1:3] = imputer.fit_transform(x[:, 1:3]) instead

Hope this helps!


"Deep Noether's Theorem": Building in Symmetry Constraints

From Emre's comment above, Section 4.4 of Group theoretical methods in machine learning by Risi Kondor has detailed information and proofs about creating kernel methods that inherently have symmetries. I will summarize it in a hopefully intuitive way (I am a physicist not a mathematician!).

Most ML algorithms have a matrix multiplication like,
\begin{align}
s_i &= \sum_j W_{ij}~x_j \\
  &= \sum_j W_{ij}~(\vec{e}_j \cdot \vec{x})
\end{align}
with  $ \vec{x} $ being the input and $ W_{ij} $ being the weights we wish to train.

Kernel Method

Enter the realm of kernel methods and let the algorithm handle input via,
\begin{align}
s_i &= \sum_j W_{ij}~k(e_j,~x)
\end{align}
where now we generalize to $ x, e_j \in \mathcal{X} $.

Consider a group $ G $ that acts on $ \mathcal{X} $ via $ x \rightarrow T_g(x) $ for $ g \in G $. A simple way to make our algorithm invariant under this group is to make a kernel,
\begin{align}
k^G(x, y) &= \frac{1}{|G|} \sum_{g \in G} k(x, T_g(y))
\end{align}
with $ k(x, y) = k(T_g(x), T_g(y)) $.

So,
\begin{align}
k^G(x, T_h(y)) &= \frac{1}{|G|} \sum_{g \in G} k(x, T_{gh}(y)) \\
  &= \frac{1}{|G|} \sum_{g \in G} k(x, T_{g}(y)) \\
  &= \frac{1}{|G|} \sum_{g \in G} k(T_{g}(x), y)
\end{align}

For $ k(x, y) = x \cdot y $ which works for all unitary representations,

\begin{align}
k^G(x, T_h(y)) &= \left[ \frac{1}{|G|} \sum_{g \in G} T_{g}(x) \right] \cdot y
\end{align}

Which offers a transformation matrix that can symmeterize the input into the algorithm.

SO(2) Example

Actually just the group that maps to $ \frac{\pi}{2} $ rotations for simplicity.

Let us run linear regression on data $ (\vec{x}_i, y_i) \in \mathbb{R}^2 \times \mathbb{R} $ where we expect a rotational symmetry.

Our optimization problem becomes,
\begin{align}
\min_{W_{j}} &\sum_i \frac{1}{2} (y_i - \tilde{y}_i)^2 \\
\tilde{y}_i &= \sum_j W_{j} k_G(e_j, x_i) + b_i
\end{align}

The kernel $ k(x, y) = \| x - y \|^2 $ satisfies $ k(x, y) = k(T_g(x), T_g(y)) $. You could also use $ k(x, y) = x \cdot y $ and a variety of kernels.

Thus,
\begin{align}
k_G(e_j, x_i) &= \frac{1}{4} \sum_{n=1}^4 \| R(n\pi/2)~\vec{e}_j - \vec{x}_i \|^2 \\
  &= \frac{1}{4} \sum_{n=1}^4 ( \cos(n\pi/2) - \vec{x}_{i1} )^2 + ( \sin(n\pi/2) - \vec{x}_{i2} )^2 \\
  &= \frac{1}{4} \left[ 2 \vec{x}_{i1}^2 + 2 \vec{x}_{i2}^2 + (1 - \vec{x}_{i1} )^2 + (1 - \vec{x}_{i2} )^2 + (1 + \vec{x}_{i1} )^2 + (1 + \vec{x}_{i2} )^2 \right] \\
  &= \vec{x}_{i1}^2 + \vec{x}_{i2}^2 + 1
\end{align}

Note that we needn't sum over $ j $ because it is the same for both. So our problem becomes,
\begin{align}
\min_{W} &\sum_i \frac{1}{2} (y_i - \tilde{y}_i)^2 \\
\tilde{y}_i &= W \left[ \vec{x}_{i1}^2 + \vec{x}_{i2}^2 + 1 \right] + b_i
\end{align}

Which yields the expected spherical symmetry!

Tic-Tac-Toe

Example code can be seen here. It shows how we can create a matrix that encodes the symmetry and use it. Note that this is really bad when I actually run it! Working with other kernels at the moment.


That is indeed quite odd. I can't really find many other sources on SQ-RBF, except here, where the definition differs from wikipedia, but when I plot this, it does not recover the image they show. However, if you define

y =
    1 - x^2/2        for |x| <= 1
    0.5*(|x|-2)^2    for 1 < |x| <= 2
    0                for |x| > 2

you can recover a sensible shape.