PCA formulation - Deep Learning book by Ian Goodfellow

He means that $D$ must work well on all data points, not just a single one. I.e., $$ z_i = Dx_i \;\;\text{ so that }\;\; || (I - D^TD)x_i ||_2^2 \;\;\text{ is small } $$ for all $i$. If we could choose a different matrix, say $M_i$ for each $x_i$, then we could minimize the error $E_i = || x_i - M_i^TM_i x_i ||_2^2$ separately for each $x_i$ in isolation. But since we want a single matrix that works for all points, so we stack them all together and instead optimize
$$ D^* = \arg\min_D \mathcal{E}= \arg\min_D|| X - D^TDX||_F^2 $$
i.e., minimizing across the whole dataset simultaneously.

See e.g. here or here for more.


How to Normalise features for small datasets?

You can use a MinMaxScaler on your train set, which will normalize your features inside [0, 1]. The same scaler can transform the test set and if there are values greater than the ones found in the train set, the scaler handles that by returning values greater than 1. Essentially, your test set will be normalized.


Categorical Clustering of Users Reading Habits

Sounds like you are set to do collaborative filtering - you have readers (users) and documents (items) creating a unary-response matrix with 1 cells indicating the reader (row) read the given document (col). There are various types of recommender systems (e.g., user-based and item-based collaborative filters) and several of them can account for the meta-data associated with your documents (e.g., content-based systems). You should look into R package {recommenderlab}


It completely depends on the context of the data that is being considered. For example, $2\sigma$ from the mean ($\mu$), depends on the distribution of the data. What is the value of $\mathbb{P}(-2\sigma < X - \mu <2\sigma)$.

Also, there are many methods for outlier detection, and all of them depend on the context. Hence, you cannot say which method should be used by taking it outside the context. You should do some experiments by these methods over the data, and then base on the real outliers sample, decide which method is proper for the current data.


ML/NN as Function Evaluator for further Optimization (maximization) - Practical Example

This post seems similar to yours and may help. It seems that what you are looking for is a derivative-free optimization method. The Wikipedia page for the concept lists such methods.
Intuitively, these techniques will sample the function (the network in your case) with various inputs (pressure, temperature, speed) and will figure out which inputs optimize it. Where they differ is in their sampling strategy, as it may be impractical or expensive to sample.
You can use scipy.optimize.minimize to do that. Pass your network as func and use an initial guess, which can be the last values of your variables. Scipy expects a function with the following signature: fun(x, *args) -> float where x is a one-dimensional numpy array. This might mean you will need to wrap your network in something like this:
def wrapper(x, *args) -> float:

    network_input = _numpy_to_valid_network_input(x)

    network_output = network.predict(network_input, *args)

    scipy_output = _network_output_to_float(network_output)

    return scipy_output

And then, you can pass wrapper as your func. Negating the output of scipy_output will turn the minimization problem into a maximization problem. If your input variables are bounded, i.e. [0, 100], you can do two things:

Use algorithms that allow you to define these bounds explicitly (i.e. L-BFGS-B using the bounds parameter)
Implicitly bound your inputs by using a sigmoid function for instance. By doing something like I did below, you can create a function bounded_value that always returns values within the range of your choice, even if the optimization algorithm might try any float.

import numpy as np

def sigmoid(x):
  return 1 / (1 + np.exp(-x))

def bounded_value(x, min_value=0, max_value=100):
   return min_value + (sigmoid(x) * (max_value - min_value))

If you want to bound your output, that's even easier. If your goal is to minimize, return a very large value if your network outputs any value outside of your range. Obviously, ensuring that your network always returns sensible values could be achieved by tweaking the loss used during training (which is difficult with AutoKeras).
If you want to implement your own method, I would recommend coordinate descent as you don't have many input dimensions (3) and it is quite simple to implement from scratch. Obviously, a brute-force approach where you randomly sample the space and choose the inputs that yielded the best function value is even easier to implement.


Convert a list of lists into a Pandas Dataframe

import pandas as pd

data = [['New York Yankees', 'Acevedo Juan', 900000, 'Pitcher'],
        ['New York Yankees', 'Anderson Jason', 300000, 'Pitcher'],
        ['New York Yankees', 'Clemens Roger', 10100000, 'Pitcher'],
        ['New York Yankees', 'Contreras Jose', 5500000, 'Pitcher']]

df = pd.DataFrame.from_records(data)


Resources on Data Science for Football / Soccer?

There's this website called Sportsdatamart which allows you to
download CSV and XLS file formats for a number of Football series
like the England Premier League, the Italian Serie A, the German
Bundesliga and the Spanish La Liga.
On Football Data UK, you can find EXCEL and CSV data files to
use for quantitative testing of betting systems in spreadsheet
applications. League tables, head2head statistics and information on
goalscrores, first scorers and top scorers could be accessed through
the Livescore service.
Soccer Wiki is a collaborative database and anyone can create
and edit data. This community driven database contains information
on players, clubs, stadiums, managers, referees, leagues and other
data related to the world of soccer.
If you are looking for data sets centered on European Football,
check this out.


How to gauge overfit with MLPClassifier and cross_val_score?

If I understand your question correctly, you would like the algorithm to detect overfitting by itself, without you manual inspection.

For this you need another split of data. So first you split your entire dataset into training and test, and then you split again your training into two subsets: the actual training and validation. You use these two subsets internally during the learning process to estimate accuracy and detect overfitting (and you may use cross_val_score() for this purpose if you want, just do not touch the holdout test part).

On a side note, looking at your two loops (for activation and neuron), you may want to use Grid Search for this.


Help with DDP Mining algorithm for Effective Classification of data sets from 2 groups

I didn't read the paper in detail but I can see that the class label is used in Algorithm 2 "branch and bound" p. 3: it's not very clear because it's only used through Information Gain (IG). I assume that the IG is calculated between the feature and the class, in order to find the "most discriminative feature", i.e. the one which gives the most information about the class.


Using federated learning the model training does not require the whole data to be present at a centralized server instead the model training is decentralized such that the model gets trained collectively on individual nodes and then summarizes the changes as a small focused update. Only this update to the model is sent to the cloud, using encrypted communication, where it is immediately averaged with other user updates to improve the shared model. All the training data remains on your device, and no individual updates are stored in the cloud.

References-
https://ai.googleblog.com/2017/04/federated-learning-collaborative.html

PySyft is a library hooked into Pytorch that provides federated learning capabilities.

PySyft github repository link-
https://github.com/OpenMined/PySyft

Demo code for federated learning using PySyft library-
https://colab.research.google.com/drive/1F3ALlA3ogfeeVXuwQwVoX4PimzTDJhPy#scrollTo=PTCvX6H9JDCt


Understanding of number of cells in layers of sequential models

RNN cell versus neuron
You are hesitating between RNN cells and neurons. I understand you refer to RNN cells in your question. And so, traditionally any layer of a sequence model will always have the same number of cells as the length of your sequence or embedding size.
See difference between cell and neuron here: Difference between cell state and hidden state
See input size for sequence models here: How can I picture an unfolded RNN as a normal Feed Forward Network?
Re; code snippet
In SimpleRNN, the first argument is the number of units in each cell i.e. number of neurons in each cell. The number of cells is not an argument in sequential layers, and always is equal to the length of your input sequences or size of embedding layer.
model = Sequential()
model.add(Embedding(maxWordsCount, 256, input_length = inp_words))
# Below, 128 is the neurons of each cell, and relates to the cell memory capacity.
model.add(SimpleRNN(128, activation='tanh'))


There are a number of methods to automate the optimisation of your hyper-parameters, such as GridSearch and RandomSearch which the article you linked discusses briefly.

The main reason to choose one over the other is if you want the best possible parameters, and don't care how long it takes to get them: go for GridSearch. On the other hand, if you don't want the optimisation to take a long time, but still want some good parameters then go for RandomSearch.

These two implementations in ScikitLearn aren't exactly "advanced packages" but they'll get the job done for any model in Scikit (Random Forest, MLPClassifiers, etc). Emre's comment also has some pretty cool advanced packages which are Scikit compatible or Tensorflow compatible.


Why is Keras fit_generator different from fit?

The Keras fit_generator takes in a python generator as an input to train the model over an array of training data. Usually, these generators are formed using built-in Keras functions such as ImageDataGenerator that takes in the image array as inputs and performs user-defined operations (such as augmentations, normalization, transformations, etc.) on the same in real time. This considerably reduces the memory consumption.

For more, refer Keras image preprocessing docs


Integer encoding and weighing when one feature consists of more names

That's a good question. This task would be referred to as multi-label encoding. Bascially if a movie belongs to several genres, you one hot-encode each genre and add the vectors.
If there were only 6 genres (horror, romance, action, adventure, comedy, fantasy)
For instance a movie that is horror, action and comedy (The Dead don't Die?):

horror = [1, 0, 0, 0, 0, 0]
action = [0, 0, 1, 0, 0, 0]
comedy = [0, 0, 0, 0, 1, 0]

So a movie that belongs to all three would be encoded as: [1, 0, 1, 0, 1, 0]. A movie that only has a category then its encoding is a one-hot encoded label.
You can perform this task in scikit-learn with a multi label binarizer


How can I create a pixel labelled image for Semantic Segmentation?

The answer can be found here.

Go to the APPS tab, and search for Image Labeler, or Training Image Labeler

Click on Add Images to add your training images.

Click on Add ROI Labels to add class names for the regions of interest.

Proceed to select the regions of interest manually from the uploaded images.


how to work with a large dataset in python?

Before exploring more sophisticated tools like Spark or Dask, one option would be to read the data in chunks instead of loading the whole file. For example, if you are using pandas the read_csv method accepts chunksize argument.

The main idea is that often what you need to do is reduce each chunk down to something much smaller with only the parts you need (some average/count/sum etc.. of some classes) and store a series/array/dict etc accordingly.


Why in deep learning or another AI technique python is used rather than other languages?

Short answers

Tooling. Python has fantastic math, statistics, and linear algebra libraries.
Less Code, Same Result. Python provides quick and simple ways of achieving programming solutions compared to C#, Java, C++, etc. This means you'll write less code and achieve the same result.


Here are a few comments from my perspective:

First, methodologically this kind of approach must be done carefully, but people who work this way are rarely careful. training/evaluating many different types of models and/or parameters is equivalent to doing parameter tuning, i.e. it's essentially training a model. As a consequence the performance should be evaluated on a validation set, and after that the best model is selected and only this model should be evaluated on the (unseen) test set. A common mistake is to evaluate all the models on the same test set and assume that this gives the real performance. This is wrong, the best performance could have been obtained by chance and therefore the true performance is likely lower.
While it is normal to try to maximize performance and to some extent this can involve a fair amount of experimental evaluation, trying many different models/parameters blindly is in my opinion very poor practice and it rarely leads to the best results (but it can lead to an illusion of good result, see point above). Most of the time there is much more to gain by doing a fine-grained analysis of the data, including studying expert knowledge about the context, than blindly relying on black box algorithms. A simple example is feature engineering: sometimes some simple redesigning of the features can drastically improve performance, but there's no ML technique which can replace this (DL methods may come close to it, but usually at a much higher computational cost).
Final point: the quest for maximum performance is often misguided. Sometimes performance is used as a lazy excuse for not doing the job properly: what really matters in production? Is it really 0.3% more F1-score or an interpretable result? Was the evaluation measure chosen to really reflect the quality of the prediction or just "because it's standard"? And the training/test sets, do we know if the production data will look exactly the same? If not, is the ML method robust enough to handle these variations? When optimizing performance one often relies on the model catching the most subtle patterns, which means a higher risk of overfitting and mediocre performance in production. More generally, there are more questions (at least in research) about explainability, the role of human actors in a ML process, and of course the computational complexity and the environmental cost of the ML method. All of these points are not reflected by a performance measure, but they matter.

Overall I would consider this kind of approach as mediocre: it's inelegant, it's prone to mistakes (which often won't be detected) and it's short-sighted because it ignores other dimensions of the quality of a ML system.
By the way this approach can be automated. Personally I would not be very proud (and probably even worried) if my job could be done entirely by a program.

Have people found that this is fairly common?

In research my guess is that this would be common enough in low-level publications, but it's unlikely to pass at any reputable journal/conference (because expert reviewers can usually detect it).

How can you work on a machine learning problem in a non-random-try-everything kind of way?

That's the thing, there's no recipe... But that's the beauty of it, isn't it? :)
In my opinion it's a bit like medicine: a good doctor has a good intuition because they have a solid theoretical background and a lot of practice. They don't just follow some rigid manual, they see the patient as a unique individual and they do their best to understand every aspect of the problem.


What's the best way to train a model when having a very big dataset split in batches?

The first solution might give you a lot of headache as some convergence problem may arise from changing the dataset every n iterations. The second one seems the way to go, but before considering a more complex approach with generators, you might want to have some considerations at your memory usage.
Basically memory usage is : number of rows * number of columns * average memory usage. As it happens there are simple solutions to reduce each of those terms independently that often makes wonder as their reduction is multiplicative.
Average memory usage: this one has to be done on each datasets. Basically, working on your datatypes you can reduce the memory usage. This need to be done careffuly so as not to loose too much information. But basically reducing your float types from float64 to float32 already divide your memory usage by a factor of 2. This factor can go up to 8 as you could use small int8 or categories for some variables. (Plus it speeds up calculations, which might end up also giving you statistical performance boost). In the case of multiple csv files it is best to have a dictionnay of types that ensure general coherence.
Number of columns: with Machine Learning there is always the temptation to dump the whole database in the SotA algo and let it sort things out. This is not really a good approach. Usually it is best to start with a simple model (linear / logistic regression), including a smaller set of features. Selecting a subset of pertinent variables means that you will have to do some explanatory analysis. It will help you to handle your problem : having a better understanding of the relationship between features and target, observing problems like outliers and missing values, and being able to explain it. You can even use things like glmnet that do a bit of feature selection for you. With a dozen variables we are talking a factor 10 in memory usage reduction.
Number of rows : when building a model it is often practical to downsample it significantly, like at a 10% rate. This allows you to check you model building step by step without having to actually do all the calculations. Then you can reduce the downsampling. depending on your problem learning on 25% or 50% of your dataset might be more than enough, while giving a memory reduction of a factor 2-4.
Depending on your problem - if there is a variable that mainly drive the underlying behavior - you might want to build multiple models, one for each category of said variable (or buckets of a continuous variable). It's not always intuitive as most Data Scientist will prefer having one model, but sometimes it is more practical to use / handle / explain. And in your case it would allow for handling smaller databases.
All in all, those simple considerations would be enough to reduce your memory usage by a factor of 40-80.


How to add a new category to a deep learning model?

If this is just a one-time case, you can simply re-train the neural network. If you frequently have to add new classes, then this is a bad idea. What you want to do in such cases is called content-based image retrieval (CBIR), or simply image retrieval or visual search. I will explain both cases in my answer below.

One-time case

If this just happens once - you forgot the 11th class, or your customer changed his/her mind - but it won't happen again, then then you can simply an 11th output node to the last layer. Initialize the weights to this node randomly, but use the weights you already have for the other outputs. Then, just train it as usual. It might be helpful to fix some weights, i.e. don't train these.

An extreme case would be to only train the new weights, and leave all others fixed. But I am not sure whether this will work that well - might be worth a try.

Content-based image retrieval

Consider the following example: you are working for a CD store, who wants their customers to be able to take a picture of an album cover, and the application shows them the CD they scanned in their online store.
In that case, you would have to re-train the network for every new CD they have in the store. That might be 5 new CDs each day, so re-training the network that way is not suitable.

The solution is to train a network, which maps the image into a feature space. Each image will be represented by a descriptor, which is e.g. a 256-dimensional vector. You can "classify" an image by calculating this descriptor, and comparing it to your database of descriptors (i.e. the descriptors of all CDs you have in your store). The closest descriptor in the database wins.

How do you train a neural network to learn such a descriptor vector? That is an active field of research. You can find recent work by searching for keywords like "image retrieval" or "metric learning".

Right now, people usually take a pre-trained network, e.g. VGG-16, cut off the FC layers, and use the final convolutional as your descriptor vector. You can further train this network e.g. by using a siamese network with triplet loss.


What's the difference between Error, Risk and Loss?

Error

In this context, error is the difference between the actual / true value ($\theta$) and the predicted / estimated value ($\hat\theta$)

$$Error = \theta - \hat\theta$$

Loss

Loss and Risk are both measurements of how well a model fits the 'data'. The difference is what 'data' means.

Loss ($L$) is a measurement of how well our model perform against your training data. You calculate loss using a loss function, many of which uses error in its equation; for example, simple loss functions may include:

Mean Absolute Error (MAE) - average of the errors

$$L = \frac{1}{n}\sum_{i=0}^n|\theta - \hat\theta|$$
Mean Squared Error (MSE) - average of the squares of the errors

$$L = \frac{1}{n}\sum_{i=0}^n(\theta - \hat\theta)^2$$
Lp - MAE and MSE apply a power of 1 and 2 to the errors, respectively, and averages them. Because of this, they are also called L1 and L2 loss functions. However, you can apply higher (or negative) powers, or $p$, to the errors.

$$L = \frac{1}{n}\sum_{i=0}^n|\theta - \hat\theta|^p$$

Whilst error is simply difference between $\theta$ and $\hat\theta$, loss can differ depending on which loss function you pick. The loss function you pick depends on your data and your objectives. For instance, MAE, or L1, are sensitive to outliers; so if you want your model to also be sensitive to outliers, then you can use MAE.

Your loss depends on the loss function used, which depends on your use case; on the other hand, the error is always the same.

Risk

To re-iterate, loss measures how well your model fits against your training data. However, our end goal is not to fit our model to our training data, which can lead to overfitting. Instead, it is to fit against our validation and test data, or simply any new unseen data.

This is where (true) risk comes in. Risk is the average measure of loss, or expected loss, across your whole data distribution.

$$R(\theta,\hat\theta) = \mathbb{E}[L(\theta,\hat\theta)]$$

To illustrate, let's imagine that you have an overfitted model. Both the errors and loss of your model would be very low (because they are measured against your training data). But because it is overfitted, you'd expect the loss for any new data to be high. Thus, you have a model which has low error and loss, but very high risk.

Empirical Risk

To bring this back full circle - when we train our model, we do not have the full distribution of the data. This may be because some of our data is used for validation and testing, or that new data points are produced in real-time. The best we can do is to pick our training data in a random way and assume that our training data is representative of the real data.

Therefore, because we don't have all the data, the best we can do is to minimize the empirical risk, from data that we do have (our training data), and use regularization techniques to generalize (i.e. avoid overfitting). This is why minimizing loss and minimizing empirical risk are roughly the same thing.

Summary

When we are training our model, our focus should not be on minimizing errors or loss, but to minimize true risk. But most of the time, we can't, so we minimize the empirical risk and regularize.


The name of the model is ARMAX, which is a model dependent of exogenous and endogenous (lagged) variable.

https://stats.stackexchange.com/questions/351146/fit-an-armax-model-in-r


After choosing top models in classification? Can I apply it on the rest of my dataset

This is an interesting question.

In general the split of data is about the underlying distribution. That means you split a dataset into train-test sets in a way that random split of train-test does not dramatically affect the distribution. But Splitting based on topics is not random!

Specially in your case you are talking about text in which the distribution is super sensitive to the domain i.e. if you collect the commentary of 1000 football games and the narrations of 1000 documentary movies about wild life, you will see that they are literally two different things. The conceptual difference between products most likely affects the distribution of words/terms/phrases therefore the model trained on reviews of mp3s MUST NOT be validated on reviews of football shoes!

In your case, I would say the train-test split (CV folds) should be done on whole data together so that you maintain the original topology of word distribution (topology here is not a Math term but I simply mean the shape of distribution).

In this case if you do Topic Modeling on the whole training data you simply see 5 different product topics. Or if you use word2vec or doc2vec you hopefully see 5 different clusters. Then you can run your models in this setting for evaluation.

NOTE:

If the size of classes are very different you need to come up with some solution for small classes. If it was the case, just drop me a line in comments and we can discuss solutions.

Good Luck :)


How propagate the error delta in backpropagation in convolutional neural networks (CNN)?

So you are correct that the principle of backpropagation is to do the reverse of the operations. The same is true about the convolutional layer.
The forward pass of the convolutional layer can be expressed by
$x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$.
Where $m$ and $n$ is the shape of the convolutional kernel that you will pass over your input image and $w$ is the associated weight for that kernel. $o$ is the input features and $x$ is the resulting value represented by their respective layers $l-1$ and $l$.
For backpropagation we will want to compute $\frac{\partial x}{\partial w}$.
$\frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}} = \frac{\partial}{\partial w^l_{m', n'}} (\sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l)$.
By expanding the summation we end up observing that the derivative will only be non-zero when $m=m'$ and $n=n'$. We then get
$\frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}} = o^{l-1}_{i+m', j+n'}$.
We can then put this result into the overall error term we have calculated.


This technique of gradually increasing the learning rate linearly during early phase of training is called learning rate warmup.
This is typically needed in order to prevent overfitting during the initial phase of training where a certain subset of observations may significantly skew your model towards some features and push you towards a bad local optimum.
The rectified adam paper suggests that this warmup heuristic serves as a variance reduction technique.

Due to the lack of samples in the early stage, the adaptive learning rate has an undesirably large variance, which leads to suspicious/bad local optima

They perform a few experiments that validates their hypothesis.


Reinforcement Learning with static state

The framework you describe is the Bandits framework. Bandits are algorithms that solve stateless MDPs. So the problem is not sequential in the sense that actions affect the transition to the next state. Take a look at a nice Contextual Bandits tutorial which is accompanied with Python Jupyter notebook. Please note that Contextual Bandits are a special category of Bandits but at least you will be exposed to the general Bandit idea and formalization.

You can try Q learning but with the Rescorla-Wagner rule $r(a) - Q(a)$ as the reward correction $\gamma\max_a' Q(s',a')$ does not make any sense as there is no transition to a subsequent state $s'$.


Conditional Multivariate Gaussian Distribution - Section 2.3, equation 2.74

\begin{align}&-\frac12(x - \mu)^{T}\Sigma^{-1}(x - \mu) \\&= -\frac12(x_a - \mu_a)^{T}\Lambda_{aa}(x_a - \mu_a) \tag{1}
\\&-\frac12(x_a - \mu_a)^{T}\Lambda_{ab}(x_b - \mu_b) \tag{2}
\\&-\frac12(x_b - \mu_b)^{T}\Lambda_{ba}(x_a - \mu_a) \tag{3}
\\&-\frac12(x_b - \mu_b)^{T}\Lambda_{aa}(x_b - \mu_b) \tag{4}\end{align}

Now let's focus on $(1)$,  the linear part in $x_a$ are

$$-\frac12 (-x_a^T\Lambda_{aa}\mu_a - \mu_a^T\Lambda_{aa}x_a)=x_a^T\Lambda_{aa}\mu_a \tag{5}$$

Now focus on $(2)$, the linear part in $x_a$ is

$$-\frac12x_a^T\Lambda_{ab}(x_b-\mu_b) \tag{6}$$

Now focus on $(3)$,  the linear part in $x_a$ is

$$-\frac12(x_b-\mu_b)^T\Lambda_{ba}x_a=-\frac12x_a^T\Lambda_{ab}(x_b-\mu_b) \tag{7}$$

There is no linear term of $x_a$ in $(4)$,

Adding $(5)$ to $(7)$, we have

\begin{align}x_a^T\Lambda_{aa}\mu_a-\frac12x_a^T\Lambda_{ab}(x_b - \mu_b)-\frac12x_a^T\Lambda_{ab}(x_b - \mu_b)&=x_a^T\Lambda_{aa}\mu_a-x_a^T\Lambda_{ab}(x_b - \mu_b)\\
&=x_a^T(\Lambda_{aa}\mu_a -\Lambda_{ab}(x_b-\mu_b))\end{align}


Best way to fix the size of a sentence [Sentiment Analysis]

The easiest way is to average the word- embeddings. This works quite well.
Another thing you can try is to represent each document as a bag of words - i.e. - to have a vector in the size of your vocabulary, where each element in the vector represents the number of times a certain word had been mentioned in your document (for example, the first element in the vector will represent how many times the word a was mentioned, and so on).
Afterwords, to reduce the size of the vector you can use techniques like LDA, SVD, or autoencoders.


model to predict annual outcome based on previous years data

I guess you need to predict following week's performance based on previous month's performance. Well, I think its more or less like predicting tomorrow's weather based on previous data.

I guess you need to use Time series forecasting model (ARIMA or LSTM ) for the same by setting the lag for one month. You can build a simple Deep Learning model for the same by adding layers with lag as a parameter


In sequence models, is it possible to have training batches with different timesteps each to reduce the required padding per input sequence?

The answer to your needs is called "bucketing". It consists of creating batches of sequences with similar length, to minimize the needed padding.
In tensorflow, you can do it with tf.data.experimental.bucket_by_sequence_length. Take into account that previously it was in a different python package (tf.contrib.data.bucket_by_sequence_length), so the examples online may containt the outdated name.
To see some usage examples, you can check this jupyter notebook, or other answers in stackoverflow, or this tutorial.


Sentiment analysis using sources other than the IMDB data

One of the most fundamental assumptions in machine learning is that the training data is 'similar' to the test data. Training makes no sense otherwise.

So the question is: How similar are reviews of, say, movies and stocks? Maybe somewhat, but not too much.
Your movie-trained algorithm would certainly be able to deal with statements like 'This company is awesome' or 'Warning, do not buy this stock'.

But what about 'The stock price will explode/implode' or 'Concerning this stock, I am bearish/bullish'? The words explode, implode, bearish and bullish have probably never been expressions of sentiment in movie reviews.


Why is my test data accuracy higher than my training data?

Your test-set is unlucky.

Use Cross-Validation

Also make sure to handle proper/random split for test and train data. So that it's not biased


Cropping and/or resizing is very trivial using OpenCV. You could write a 10 line script to iterate over all your images and apply the transformation you require.


How to convert a non gaussian distribution into a gaussian destribution?

I recommend order normalization from R package bestNormalize, good info at:
https://cran.r-project.org/web/packages/bestNormalize/vignettes/bestNormalize.html

Below a short comparison of a few methods:

Best Normalizing transformation with 6283 Observations
Estimated Normality Statistics (Pearson P / df, lower => more normal):
 - No transform: 21.8352
 - Box-Cox: 4.0105
 - Log_b(x+a): 4.5358
 - sqrt(x+a): 7.1273
 - arcsinh(x): 4.5268
 - Yeo-Johnson: 4.0185
 - orderNorm: 0.9744

It is not written anywhere, but logic says, that you should rbind train and test datasets before this normalization if you want to use all of its advantages.


How should I sample my validation set if I randomly sample training data?

Actually you should never use any sampling techniques on your testing/evaluation data because this could lead to wrong classification results.
If your dataset is imbalanced you could perform upsampling or downsampling techniques (like SMOTE) on your training data only.
If you want to benchmark your multi-class classification you need to rely on e.g. the confusion matrix, recall, precision and F1 measure. Please keep in mind that the accuracy measure cannot be interpreted if you have too imbalanced data.


Role and need of Mathematics in Data Science

Firstly, I think this question is both broad and opinion-based, however most people don't think so. I'll explain why:

Data Science is a domain which various very much with the industry it is applied in, so very much application-based.

For example: Someone like me in the growth and customer support segment, I use markov models and reinforcement learning, as customers moves can most likely be modeled as a (stochastic) graph. So, when I look out for hiring someone into my team, I look for graph theory skills, linear algebra (obviously) and statistics.

However, for someone working in the segment where NLP is used, the math needed is different, and so does with every other application of data science.

However, some really very important courses which help someone in the path of a data scientist according to me are:

Linear Algebra
Statistics
Probability theory
Discrete Math
Analytical methodology (Numerical Analysis would be a close to precise name for that)

Some would argue that Information Theory is also very important, however I haven't really used it heavily in my day job, so I don't think it's absolutely necessary.

Helpful Post


Significance of comparing Receiver Operating Characteristic (ROC) curves

Although should one be using ROC curves when comparing different types of classifiers (e.g. Random Forest vs Neural Net vs Logistic Regression)?

Yes, because you can clearly see which model is performing best overall. The more you get close to the top left corner of the graph, the better your model is. (the blue line in your example, Random Forest)

If you compute the Area under the ROC curve in the example, you'll get:

AUC(RF) > AUC(MLP) > AUC(SVM)

So AUC is a good index to show which curve is "higher".

Now, sometimes you could be interested in a model that has a "steeper" ROC curve in the bottom left corner (higher chance of improving TPR, with low cost in term of FPR), and maybe that model isn't the best in term of AUC. (Not the case in your example)

In that case you would not choose AUC as a term of model choice, because it doesn't select the "best" model for you.

Overall the ROC curve between different models helps you a lot, showing not only the performance over different thresholds, but also comparing different solutions simultaneously.

Edit:

Each model gives a number for each observation that is the probability to belong in one class (a number p between 0 and 1).
Say you get p = 0.3 for one observation, then with a threshold of 0.5 you will label that observation to one of the two classes (assuming there are only two labels).

Your rule is:

If p < 0.5 (threshold) then obs is labeled to class "one"
If p >= 0.5 (threshold) then obs is labeled to class "zero"

With ROC curve, you basically have a "high" number of thresholds, say going from 0.01 to 0.99. For each model.

So for each model, you will assign each observation to the respective class, using for t = 0.01, then t = 0.02, and so on till t = 0.99. (t = threshold)


How predictions of level 1 models become training set of a new model in stacked generalization.

You have created a model pipeline and must run all trained models ("lower level" ones first) in order to make a prediction on new data using the stack.

With test data set, it is slightly easier, since you can store the predictions from the "level 1" models when testing them, and only run the final model across this stored data.

In addition to your brief description, usually to avoid bias from re-using training data, you would use k-fold cross-validation or similar mechanism, and your training data for the final model should be the cv predictions from each model. You do not want to use the training predictions from those models, because they are likely to be overfit whilst "level 1" test and production predictions will not be, and this would introduce population differences between train and test data in your "level 2" model.

It is also quite a common variation to use the M new features from your "level 1" models alongside some or all of the original features. This gives the meta model more data to base its decision on when deciding the relative weights between the first stage models (assuming this top-level model is non-linear).


Looks like the gmaps package is what you're looking for.
You can do things like this with it:


What to do if my target variable is column of lists?

you can have multiple Y. Each item will be a binary variable which means that item exists or not.


Identify given patterns in unstructured data like text files

That is exactly what the Trifecta product does (in addition to other features). It uses the Wrangle language which is a DSL (domain specific language) designed for data manipulation. There is a much earlier research project called Wrangler from the same people. The Wrangler papers might give you ideas.


Model for balancing 2 sport teams with multiple players?

There is some lack of information on your post.

Does it take long time to try a match? (can I do 10.000 simulations or matches in minutes?)
What's your goal? Is to create teams that are as equal as possible?
Do the players really have better performance when they are together or you are assuming it?

But, I will try to give you an answer:
If it takes long what you can do is shuffle the players and measure the win/loss ratio, that resumes all you need, statistically speaking you will be conducting what is known as a experiment. If the players really "interact" you will have to measure the win/loss ratio by pairs.


Validation accuracy greater than training accuracy in cnn

As @Suren has eluded to, it could be the case that your validation accuracy is higher than your training accuracy due to the distribution of the classes over the training and validation sets. If the training set contains a higher proportion of a particular class and the validation class contains examples of that particular class as well, then of course you will see validation accuracy being high.

The advice would be to balance out the classes over the training and validation and sets. You can do this data augmentation, in that you artificially increase the number of examples based not he data you already have.
Have a look at this paper which evaluates the effect of different methods of class balancing using data augmentation on generalisation performance of CNNs:
https://arxiv.org/pdf/1710.05381.pdf


Yes there is. Actually using NNs on sequences is a big part of Deep Learning and one of the most powerful NNs come from this field of research. They are called Recurrent Neural Networks or RNNs. I would recommend you to read this article or watch this Stanford lecture, to learn more about them.

Especially a variant of RNNs, the so called LSTM that stands for Long Short Term Memory enables networks to memorize earlier inputs.

Based on your explanation I would suggest you have a many-to-one task where you input many frames and output one class. These types of tasks where you input images is most often solved using a combination of CNN and RNN. You could read this article to get more insights, to that technique.


What I need to write in the line level of torch.vison for 21 classes?

I solved my problem. Here are the steps I changed :

add a 'class_name' column to the "marking" DataFrame
replace 'source' with 'class_name' in data splitting (df_folds…)
modify load_image_and_boxes() function to also return labels modify load_cutmix_image_and_boxes() function to also return labels (most work to do here)
replace labels = torch.zeros((boxes.shape[0],), dtype=torch.int64) with torch.tensor(labels, dtype=torch.int64)
in get_net() function, set config.num_classes = number of classes in your dataset


How to run a saved TensorFlow Model? (Video Prediction Model)

The TensorFlow saver is used to save the weights of a specific model at some given point. When you want to use a trained model, you must first define the model's architecture (which should be similar to the one used for saving the weights), then you can use the same "saver" class to restore the weights:

with tf.Session() as sess:
    # Restore variables from disk.
    saver.restore(sess, "../my_saved_model.ckpt")

Regarding your initial question. I think that if you are just starting with deep learning and TensorFlow, this is the wrong place to start and you should first understand how TensorFlow works in general by applying it at easier tasks like image classification (start with MNIST).

From what I understand, you need to use the "construct_model" function and pass it your initial image sequence (video) and some action tensor, and it should output a predicted frames:

def construct_model(images,
                actions=None,
                states=None,
                iter_num=-1.0,
                k=-1,
                use_state=True,
                num_masks=10,
                stp=False,
                cdna=True,
                dna=False,
                context_frames=2):
"""Build convolutional lstm video predictor using STP, CDNA, or DNA.
  Args:
    images: tensor of ground truth image sequences
    actions: tensor of action sequences
    states: tensor of ground truth state sequences
    iter_num: tensor of the current training iteration (for sched. sampling)
    k: constant used for scheduled sampling. -1 to feed in own prediction.
    use_state: True to include state and action in prediction
    num_masks: the number of different pixel motion predictions (and
           the number of masks for each of those predictions)
    stp: True to use Spatial Transformer Predictor (STP)
    cdna: True to use Convoluational Dynamic Neural Advection (CDNA)
    dna: True to use Dynamic Neural Advection (DNA)
    context_frames: number of ground truth frames to pass in before
                feeding in own predictions
  Returns:
    gen_images: predicted future image frames
    gen_states: predicted future states


Multi-class classification with only one feature

Yes! You can use only one feature for multi-class classification. For example, you have one feature height and you want to classify short , medium or tall.


you can just use time library and give time counter like time save to variable and after classification take time now save to variable and subtract both and print the result you get the execution time


problem of choosing right statistical method for scheduler prediction

Do you know if the scheduler has a memory?

Let us assume for a moment that the scheduler has no memory.  This is a straightforward classification (supervised learning) problem: the inputs are X, the outputs are the schedules (N->M maps).  Actually, if every N gets scheduled and the only question is which M it gets, the outputs are lists which channel (or none) is scheduled to each block, and there is only a certain possible number of those, so you can model them as discrete outputs (classes) with their own probabilities.  Use whatever you like (AdaBoost, Naive Bayes, RBF SVM, Random Forest...) as a classifier.  I think you will quickly learn about the general behavior of the scheduler.

If the scheduler has a memory, then things get complicated.  I think you might approach that as a hidden Markov model: but the number of individual states may be quite large, and so it may be essentially impossible to build a complete map of transition probabilities.


Keras - Implementation of custom loss function with multiple outputs

As you are mixing y_true[0] with y_pred[0] and y_true[1] with y_pred[1], you could consider having different losses for each, and using loss={'Output_Dist': custom_loss, 'Output_Value': losses.MSE} when compiling. Internally, it will add the result of each one in a final loss.
def custom_loss(y_true, y_pred):
    return -K.dot(K.transpose(y_true), y_pred)


How to interpret my neural network with high accuracy but low probability in test results

First possibility:

How well is your dataset balanced? Are there mostly 0s? If I understand your graph well, it would seem that is the case.

If you in fact have 90% 0 and 10% 1 in your dataset, then your results would indicate the model is actually just performing random selection. This means it hasn't really learny anything about the data.

Second possibility:

If your training accuracy is high, but your test accuracy is low, this generally does point to overfitting, as you suspect.

Your model is essentially memorising the training data set, making it able to simply recall the correct result quite reliably, but without learning the underlying function that describes the dataset.

Here are a few things that might be behind this:

training for too many epochs
using a model that too big and complex
not using enough regularisation

I would recommend first checking the class balances of your dataset. If it highly imbalanced, you could look into performing stratified sampling to create your train/test split. This ensures that each batch has roughly the same number of 0 and 1 samples.

Secondly, you might consider using a slightly larger network. In the first instance, this could be as simple as increasing the number of neurons in each of your Dense layers. I would suggest something like (100, 50, 1) to begin with. That will immediately tell you if the size of the model is restricting learning.


How to train an image dataset in TensorFlow?

If you are interested in training tensorflow model for image recognition in general, it is a good idea to go through tensorflow tutorials on MNIST:

https://www.tensorflow.org/get_started/mnist/beginners
https://www.tensorflow.org/get_started/mnist/pros

If you want to collect the data yourself and train model on it, here is something which could be interesting: https://digits-draw-recognize.herokuapp.com/

This is my site with handwritten digit recognition. I have collected dataset by myself - I drew ~1k digits and processed the data. After this I trained two models, one of them is CNN in Tensorflow. And these models train on the new images drawn by people. You can find some description of functionality on this site.


Understanding ELBO Learning Dynamics for VAE?

There is a nice paper that discusses this problem of named as 'Modelling Bias' in VAE's. ELBO objective is:

$$L_{ELBO}(x) = E_{q_φ(z|x)}[\log p_θ(x|z)] − D_{KL}(q_{φ}(z|x) || p(z))$$

The first term forces the VAE to put all the mass of $q_φ(z|x)$ the mode of $p_θ(x|z)$ while the second term forces it to be as diffuse as possible. But generally  for images, the dimensionality of X is often orders of magnitude larger than the dimensionality of Z.

Because the same per dimensional modeling error incurs a much larger loss in X space than Z space, when the two objectives are conflicting
(e.g., because of limited modeling capacity), the model will tend to sacrifice divergences on Z and focus on minimizing divergences on X. Hence it may lead to over-fitting rather than generalization.


CNN has been used many times in genomics, what you described is not new. A 5s quick search on Google gave me this paper:

  https://www.biorxiv.org/content/biorxiv/early/2017/12/31/241414.full.pdf

  ...predict phenotypes from genotypes by using a deep convolutional neural
  network (CNN)...

"Genotype" is a better terminology than "SNP", as short indels are also important. Your idea is certainly not new, and in fact insufficient as a research topic anymore.

Predicting phenotype from just genotype is old and quite limitied. No good journal will take a topic like that anymore. No matter how good your model is; no matter how many layers you have (e.g. 1000000 trillion layers), how many GPUs you have (e.g. every single GPU in the world), how many engineers you have (e.g. every single ML engineer in the world). CNN will never solve problem perfectly as it neglects the vital information from proteins. Whatever you see from a paper would be limited in applications (e.g. strong assumptions). Genotype data don't tell you how exactly your genes are transcribed.

Instead, the current research trend is on applying deep learning networks integrating mutliple data sources (proteins, transcripts, genes, genotypes etc), and form a single unifed model. This is why we have AI for precision medicine.


Popular classification algorithms over time

Gradient Boosting performs well for problems that have a medium amount of tabular data. Many problems on Kaggle are in that style.

Currently, Deep Learning holds many of the state-of-the-art benchmarks. Deep Learning generally performs better as the amount of data grows. Deep Learning performs better on other types of data (images and text).

No single machine learning algorithm is best in all cases, often called the "no free lunch theorem".


Different types of analysis which can be done with Webdata of users

Recommendation Engine is one such application of ML that can be used to extract the useful information from the user available data and offer them the product that is the most relevant to him. You can map this to classic classification problem where  expected output of ML algorithm is to come up with the product that customer will buy.


Perhaps you can see if Graduated_high_school is correlated in any way to GPA_college? If there is no correlation, you can try to fit a model by dropping the Graduated_high_school column.
Else, you can try to drop rows belonging to under-represented high schools. However, one problem I foresee is that future predictions might have Graduated_high_school that are unseen in the training dataset, leading to problems (e.g. schools that weren't mentioned in the dataset, or if someone  decides to use your model, on a dataset from another country). So, if the Graduated_high_school is not important, I would consider dropping it altogether.
Or, maybe you can change Graduated_high_school to something else that is related, such as number of teachers in high school, teacher-student ratio etc.


Support is the number of occurrences of each label in the ground truth. For example in the results with test size=0.1, class P has only 3 samples. Based on this, if the support values are not close to each other, it only means that your data is unbalanced. This documentation might be helpful.


Naive Bayes Predict type = 'raw' returning NA

I am assuming that you are referring to this Stackoverflow post that mentions to add noise to the data since the error seems to be coming when there is one (or small) instance of a class in the dataset. Is that the case with the training data? If what you're trying to predict is a rare-event, then a suggestion might be to balance the training data by oversampling the rare class (hence adding noise).

Provided the above is not working, another suggestion is to remove infrequent terms in your term-document-matrix using the function removeSparseTerms.

Going beyond, given the amount of training data you have, it would be good to evaluate if the term document matrix with the words it contains or frequency of specific words is sufficient to differentiate the classes. If not, you should consider adding new features to describe the dataset.

Few suggestions:

count of positive/negative words or a sentiment index that ranges from -1 to 1, if relevant for your data
types of words in dataset (index or count of adjectives or nouns or verbs), again depending on your problem & data
rather than using term-document-matrix, try noun-phrases

Finally, I'm assuming that your test data contains records for both classes. If not, it is difficult to evaluate the model.

Hope that helps. If you could formulate your question more clearly with the data problem and provide some examples of the data, that would help.


Slightly different from VMware or Virtualbox, but Polynote in Docker is supported by the developers and they have created a number of ready-made Docker images - more info here, although the instructions are targeted at a Linux host. Below are the instructions modified for a Windows host:

Once you have Docker installed, make a folder somewhere on your host system, e.g. C:\polynote:

cd \
mkdir polynote

Then create a text file config.yml in that folder, with the contents:

listen:
  host: 0.0.0.0

Then, you can get the container running with a single command - although make sure you customise the paths to your config file and working folder:

docker run --rm -it -e PYSPARK_ALLOW_INSECURE_GATEWAY=1
        -p 127.0.0.1:8192:8192
        -p 127.0.0.1:4040-4050:4040-4050
        -v C:/polynote:/opt/config
        -v C:/path/to/your/working/folder:/opt/polynote/notebooks
        polynote/polynote:latest
        --config /opt/config/config.yml

This exposes some ports to the host machine, mounts the folder with your config file into the container, mounts your working folder in the container at /opt/polynote/notebooks, pulls the latest Polynote Docker image and tells Docker to use your custom config file.

Once the container is spun up, you can use your web browser to access Polynote at http://localhost:8192/.


What do Python's pandas/matplotlib/seaborn bring to the table that Tableau does not?

Don't worry - your hard-earned Python skills are still important ;)

Tableau is not a replacement - it is essentially a means of sharing your insights/findings. It is a wrapper around your normal toolkit (Pandas, Scikit-Learn, Keras, etc.). It can do some basic analysis (just using basic models from sklearn), but the powerful thing is it can deploy your models to allow people to run inference on stored data/new data, and then play around with it in an interactive dashboard.

Watch this video for a good overview of everything it can do, and how it connects to Python (and R/MatLab). There is just a bit of boiler plate code around your normal Python code.

Tableau also offer TabPy to set up a server, allowing nice deployments of your work, but in the end you need their desktop application to view the results (i.e. your customers need it to look at the results). This is not free: https://www.tableau.com/pricing/individual

In summary, I'd say Tableau is more of a business intelligence tool, allowing e.g. your non-data-scientist boss or other stakeholders to interactively explore the data and the results of your modelling. Similar to Microsoft's PowerBI.


Why TensorFlow can't fit simple linear model if I am minimizing absolute mean error instead of the mean squared error?

I tried this and got same result.

It is because the gradient of .abs is harder for a simple optimiser to follow to the minima, unlike squared difference where gradient approaches zero slowly, the gradient of the absolute difference has a fixed magnitude which abruptly reverses, which tends to make the optimiser oscillate around the minimum point. Basic gradient descent is very sensitive to magnitude of the gradient, and to the learning rate, which is essentially just a multiplier of the gradient for step sizes.

The simplest fix is to reduce the learning rate e.g. change line

optimizer = tf.train.GradientDescentOptimizer(0.5)

to

optimizer = tf.train.GradientDescentOptimizer(0.05)

Also, have a play with different optimisers. Some will be able to cope with .abs-based loss better.


A CNN automatically extracts features, so hand-crafting features has become unnecessary for most applications.  It learns what features to extract via backpropagation.

You can introduce handcrafted features in a few ways to boost the performance of a CNN:

inject hand-crafted features into the fully connected layer
feed the CNN output into another classifier alongside the hand-crafted features

This paper explores such concepts.


Can't help if you're fixed on KNIME, but if you can use Orange Data Mining, I've had success clustering (took a minute or so) and previewing a scatter plot of a dataset with 32k rows.


How do you choose an appropriate $k$ to achieve $k$-anonymity for data?

In most cases $k$ emerges from the volume and nature of data, plus trhe anonymity method used. Rarely does one have explicit control over $k$, except implicitly through these options.
Think of $k$ as a score instead of as a parameter.
It is possible, for example, some records will have higher $k$-anonymity than others. Then the average $k$ counts, or even the minimum.
If anonymity is a requirement, then the highest possible value of $k$ is what is needed. Since for each record there are only $k-1$ similar records, so methods can be used to exhaustively find the anonymised info, thus the highest possible $k$ is needed in order to slow down this process and make it practically impossible.
Of course the maximum $k$ is achieved when all data columns are anonymised, but this creates useless data, so the tradeoff between useful data and maximum anonymity results in a range of $k$ values to achieve (and this depends on the actual nature and volume of data).


Why some ML models can't take advantage of text ordering information?

This tutorial assumes the text will be represented as a fixed length feature vector. There are several ways to do that. The good old one is using n-grams, indexing and several tricks around that.

First, N-grams can be seen as kind of words (or rather, phrases). The extreme version of it is unigram, or 1-gram, which is just a single word. Bi-grams, tri-grams and so on encode certain word ordering, of course, but the ordering of the n-grams in the text is usually ignored. Let's assume you use unigrams and defined the vocabulary:

Index assigned for every token:
{'the': 0, 'mouse': 1, 'ran': 2, 'up': 3, 'clock': 4 }

By using the count encoding, we come up with the following vector representing the text:

'The mouse ran up the clock' = [2, 1, 1, 1, 1]

But if you simply shuffle the words, you will end up with the same vector.

2-grams, 3-grams and so on capture the word ordering better and it will be more difficult to get similar vectors for different texts. So, this is not completely true that they cannot leverage any information about text ordering. However, just by swapping sentences in a text, you can get very close representations. Especially, if some rare 3-grams are dropped from vocabulary (which is widely used to keep the vocabulary of reasonable size).

Recurrent neural networks and word embeddings are more modern approaches in NLP. In case of RNNs, words (in form of embeddings) are fed to the model sequentially (not independently like in case of n-grams), so the model is much more sensitive to the word ordering.


Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

Yes you can get the embedding of words. In sentence-transformers you need to see where and how the embedding of words are extracted. Usually the sentence embedding in BERT is simple a max or mean over all word embeddings, so BERT does have word embeddings.
The question is if it helps you at all as, like you mentioned, at the end of the day you have a bunch of words not a language. The point is that if you have enough data you can fine-tune BERT (see sentence-transformers documentation) and probably capture the difference-similarity between each sample.
Another way is to get your embedding trough LSTM which is more suitable for such sequences and you can train it using for example Keras.
I suggest trying SGT as well and compare its embedding capability with other methods so you can choose the best.


Multidimensional Dynamic Time Warping Implementation in Python - confirm?

Could the difference be caused by the step pattern you are using? Dtw in R defaults to the symmetric2 step pattern. Explanation of the step pattern from Toni Giorgino's paper:

  symmetric2 is normalizable, symmetric, with no local slope constraints. Since one diagonal step costs as much as the two equivalent steps along the sides, it can be normalized dividing by $N+M$ (query+reference lengths).

I thought this might be the problem after reading Comparing Dynamic Time Warping in R and Python.

If your problem is the same try adding step=symmetric1 to your R.dtw arguements.


What is the use of using width/height shift in data augmentation?

Since CNNs are translation invariant, aren't the translational shifts
from keras useless as those shifts will not result in new images per
say?

Invariance to translation means that if we translate the inputs the CNN will still be able to detect the class to which the input belongs. Translational Invariance is a result of the pooling operation. In pooling operation, we replace the output of the convnet at a certain location with a summary statistic of the nearby outputs such a maximum in case of Max Pooling. As in the case of max-pooling, we replace the output with the max, so even though we adjust the input slightly, it will not impact the values of most pooled outputs.
The major issue with max pooling is that the network fails to learn the spatial relation between different features, and thus will give a false positive if all features are present in the wrong position with respect to one another. This happens for Cases like Image Segmentation where we require position.
So, Data Augmentation comes in handy to handle heavy distortion in images and make the model more robust to these distortion when we have less images.
Reference:
Translation Invariance


LSTM future steps prediction with shifted y_train relatively to X_train

Unfortunately it is more likely that this approach itself is bad. It's not the fault of your LSTM or neural netowrk.

You may be able to find a lot of online tutorials using RNN/LSTM to predict stock price/crude oil price/bitcoin price or whatever price, based on its price history, and their results are almost surely meaningless.

Note that I am not saying analyzing a market price is meaningless, but predicting future prices based only on historical prices only is usually meaningless, given that the market is well established and most relevant information influencing its price is readily available; or in other words, the market is efficient.

Outside of the domain of data science, the assumptions for the efficient-market hypothesis are not entirely true for most markets, but its consequence is close to the truth for large markets: the asset price follows a random walk. In terms of prediction into the future, the price behaves as a martingale, i.e. the best prediction for tomorrow's price is today's price.


Tool/dataset for matching first names and nicknames

Maybe this helps. I could not check the content, but it says that it provides a csv mapping names and nicknames.

https://github.com/carltonnorthern/nickname-and-diminutive-names-lookup


How to combine features with different temporal scale in machine learning

Machine Learning does not have an specific method from mixing data with different sampling rates.

An extract from the paper:

We approach the problem of dealing with multiple sampling rates from an aggregation perspective. We propose Accordion, a new embedded method that constructs and selects aggregate features iteratively, in a memory-conscious fashion.

Our algorithms work on both classification and regression problems. We describe three experiments on real-world time series datasets, with satisfying results.

Mining Multivariate Time Series with Mixed Sampling Rates


Light GBM Regressor, L1 & L2 Regularization and Feature Importances

With regularization, LightGBM "shrinks" features which are not "helpful". So it is in fact normal, that feature importance is quite different with/without regularization. You don't need to exclude any features since the purpose of shrinking is to use features according to their importance (this happens automatically).

In your case the top two features seem to have good explanatory power, so that they are used as "most important" features. Other features are less important and are therefore "shrunken" by the model.

You may also find that different features pop up as top of the list (the list may look different in general) when you run the model multiple times. This is because (if you don't fix a seed), the model will take different pathes to obtain a best fit (so the whole thing is not deterministic).

Overall you should get a better fit with regularization (otherwise there is little need for it).

I wonder if it makes sense to use both (l1 and l2)!? L1 (aka reg_alpha) can shrink features to zero while l2 (aka reg_lambda) does not. I usually only use one of the parameters. Unfortunately, the documentation does not provide too much details here.


You may try Metropolis–Hastings sampling if multi-dimensional, or else Adaptive rejection, unless you've already tried.


Somehow you have to come up with some sort of numerical classification system for your movie genres.
I would start by creating a relationship tree between genres.  For example action movies and then action movies with comedy and then action movies with comedy with animation etc.

You could develop a whole Forest of trees that relate movie genres to one another.  You can then test the genres paths of individuals to compare.


Should weight distribution change more when fine-tuning transformers-based classifier?

Its difficult to judge by just looking at the weight distribution. You will be in a better position to identify the situation if you check the following things also:

1) Check if the loss is decreasing during training. It is very unlikely that a randomly initialised last layer will not need training to succeed in a downstream task. This is because even if the initialise layers are producing meaningful values, the randomly initialised final layer will output garbage values in the beginning. If the loss is decreasing then the model training is fine. Alternatively, check if the final if the final metric is improving e.g., accuracy.

2) Check the gradient received by your custom layers. One of the reason that the weights distribution is not changing could be due to zeroed or negligible gradient. I don't think gradients are zero because your plots show the weights to be changing, but still if If the gradient is zero then check your implementation. Something must be off like not using the actual output of the model while computing the loss for backpropagation. If the grads are very very small, try increasing the learning rate of the optimzer.

Conducting the above 2 analysis will strengthen your understanding of the current situation for the model training.


Why I get a very low accuracy with LSTM and pretrained word2vec?

Its huge discrepancy, I suspect a lie.

While +33% can be achieved, you said that you tried very different architectures and you did not get even close. Dont expect that one tweak, one layer, one xyz can give you all of a suddenly such a huge increase. If you did not get closer using suggestions from the paper there is also a possibility (not saying they did, but it has been done before in ML papers- lying about accuracy achieved without providing code) that they lied.


Eigenfaces VS Deep neural network for classification

As it stands, neural networks outperform the Eigenfaces approach. With this I mean that neural networks can solve a large range of problems with more accuracy.

Eigenfaces are nice because they can work already with a small amount of training samples, specially compared to neural networks that are known to be data intensive. So, in a small amount of data setting, you could start with Eigenfaces for feature extraction and pair it up with an SVM for classification. However, now a days, there are pre-trained neural network models (e.g. Facenet), which offer pre-trained embeddings for you to use for feature extraction. Then, all you need to do is train "a last layer" for classification. Bottom line is, you don't need to start from scratch, making it obsolete to use Eigenfaces for feature extraction.


What is a suitable loss function and evaluation metric for a classification model with large number of unbalanced target classes?

You could look at sensitivity and specificity. They can be combined effectively to either provide a basis for a correct classification, or the basis for an exclusionary classification within each class.  That said, if you are looking for a measure for the model as a whole, you'll probably want to look at some sort of weighted average of the sensitivity scores.

sensitivity is the same calculation as precision, but specificity is not the same as recall.

That said, you may need to have several models that specialize in different areas or, perhaps as worst case scenario, you may need to produce a one-vs-the-rest ensemble of models to get the best results. This might help if, as anymous.asker suggested, perhaps you can introduce some cost around positive or negative outcomes.

HTH


Class weights for imbalanced data in multilabel problems

I think what you are looking for in this case is "cost-sensitive classification" - you can look it up on google scholar to find some papers. You'll likely have to define "costs" for each type of misclassification.

Alternatively, if you are doing one-vs-rest classification, you can reweight/upsample/downsample differently for each class.

If your dataset is small, you could also try to set weights as the objective of some constrained optimization objective that would make them more balanced.


how do I predict the next's alarms ? (time series)

The first step is to pick a target variable. What are you trying to predict "problem name", "severity", "sub category", …?
Based on the target, it becomes either a conventional regression, ordinal regression, binary classification, multi-class classification, or multi-label classification problem.
Another issue might be how the date is encoded. "t1", "t2", … is not a conventional datestamp format.


Convolutional network for classification, extremely sensitive to lighting

It's called overfitting, your model learned to predict labels based on features which are not actually responsible for particular predictions, so when they are not presented it fails to predict right. Although there are various ways to deal with overfitting (e.g. Dropout) what you seem to need is image augmentation. It is a simple yet very powerful way to train a robust neural network. For your case - just simulate different lighting conditions by e.g. increasing/decreasing pixel values randomly for your inputs for same labels. It's also common practice to add random noise, random rotations, flip etc.

For more info check this stackoverflow answer.

Also, I recently published project where I used various of Augmentation functions which you may find useful. See: https://github.com/Naurislv/P12.1-Semantic-Segmentation/blob/master/augmentation.py


Have you changed the max depth parameter? This parameter limits the depth of each tree. If you have set it up to max_depth=1 you wont be able to grow your tree further.

From the documentation:

  max_depthint, default=None
  The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.

Also, it could be that you have your target on the train and your tree does not need to develop further.

Or that you are training with a small number of rows or columns


built-in cov in pandas DataFrame results ValueError array is too big

Christopher is right about the size of the array. To be simplistic about it, if this translates to 1.6B floats, at 16 bytes per float (32-bit version; 64-bit is bigger), then you're trying to create an array of about 26 GB. Even if you have the RAM for that, I'd imagine that it's probably going to overload something else along the way.

(Maybe not, but generally speaking, any operations that are that computationally intensive should first raise the question of whether you are doing the right calculation in the first place. And if you do need to do something of that magnitude, you should then try to break it down into more manageable chunks that can be run in parallel or distributed across machines.)

But given that you are describing a very, very wide dataset (~40k columns x 180 rows), I wonder whether you really want to take the covariance matrix of the transposed dataset (so 180x180 = 32,400 covariances)? That would be a far more tractable problem, and it's easier to see how it might be useful.

In any case, you're probably far better off calculating each pairwise covariance (or at least, the vector of cov(x_i,x_k) for all x_k != x_i) at the point where you'll actually use it, rather than calculating a giant matrix initially then referring back to it later. Memory issues aside, it'll make your life much easier if you start running things in parallel, and will help ensure you don't waste resources on unnecessary calculations.


Classifying objects based of a varying number of the same type of feature vector for each object

If I did not get you right please comment me. I would not go for doc2vec as you do not want to discriminate docs but persons. So better to concatenate speeches of each person to a single document and then feed it to word2vec/doc2vec (I assume you would like to use ANNs otherwise there are other options e.g. TF-IDF, etc.) In this manner each person will have a 300-d feature vector including a everything. (I would still try TF_IDF and CountVectorizer as well!)

If you insist on your current option, you may discard person info (you want to get a speech and say what kind of political-party the speaker is coming from, right?!) and set up your data as a 300-d feature vector for each speech whose tag is a political view (instead of taking mean of all speeches of 1 person). Then your feature vectors are all the same size (in this approach, each person might have several entry in the data which may bias the data so you may add a column including the speaker e.g. "speaker a", "speaker b", etc. Here you end up with a 301-d data).

Hope it helped. Good luck!


Smith-Waterman-Gotoh Algorithm - how to determine an overall similarity percentage

I don't know how to write math algebra like on the Wikipedia page for Smith-Waterman, so i'll use pseudo code. I found the logic at SimMetrics in the SmithWatermanGotoh java code.

str1 = PELICAN
str2 = COELACANTH
matchValue = 1 #in the comparisons below, when characters are equal, assign this value
mismatchValue = -2 #in the comparisons below, when characters are not equal, assign this value
gapValue = -0.5 #the gap penalty used in smith-waterman

# get the maxDistance which is the smallest number of characters between str1 and str2 multiplied by
# the largest of matchValue and gapValue
maxDistance = min(length(str1), length(str2)) x max(matchValue, gapValue);

# function to compare character at index aIndex of string a with character at index bIndex of string b
function compareCharacters(a, aIndex, b, bIndex, matchValue, mismatchValue) {
  if a[aIndex] === b[bIndex]
    return matchValue
  else
    return mismatchValue
}

v0 = an array
v1 = an array

lengthOfStr1 = number of characters in str1
lengthOfStr2 = number of characters in str2

# do the smith waterman similarity measure (currentMax)
currentMax = v0[0] = max(0, gapValue, compareCharacters(str1, 0, str2, 0, matchValue, mismatchValue))

for (j = 1; j < lengthOfStr2; j++) {
  v0[j] = max(0, v0[j - 1] + gapValue,
            compareCharacters(str1, 0, str2, j, matchValue, mismatchValue))

  currentMax = max(currentMax, v0[j])
}

for (i = 1; i < lengthOfStr1; i++) {
  v1[0] = max(0, v0[0] + gapValue, compareCharacters(str1, i, str2, 0, matchValue, mismatchValue))

  currentMax = max(currentMax, v1[0])

  for (j = 1; j < lengthOfStr2; j++) {
    v1[j] = max(0, v0[j] + gapValue, v1[j - 1] + gapValue,
            v0[j - 1] + compareCharacters(str1, i, str2, j, matchValue, mismatchValue))

    currentMax = max(currentMax, v1[j])
  }

  for (j = 0; j < lengthOfStr2; j++) {
    v0[j] = v1[j]
  }
}

# calculate the overallSimilarity between the strings
overallSimilarity = currentMax / maxDistance #<- 0.4767 for COELACANTH vs PELICAN


Multivariate Time Series Binary Classification

You can add all features as input to RNN/LSTM (Day #, F1, F2, ... F5) and binary class as output.
This article has an example of such network.


How do I know if my training data set is sufficient?

We can not judge whether our training data set is sufficient or not directly from the shape description for your data.

At the most of the time, we sample some instances, hopefully they are distributed as real-world data, so that the final model constructed through these data can reveal the truth of the real-world. Theoretically, the size of training data is not the problem, but we shall collect training samples as many as possible to ensure our model will have good performance on predicted instances.

Anyway, we can get the answer by testing whether the distribution of our predicted value is consistent with that of truth.


What is the difference between TDNN and CNN?

They are the same, as far as i can see. The name convolution usually applies to spatial not time dimensions, but that is only convention. Wikipedia also links the two.


TL;DR Ist just a bayesian varianat that generalises LDA.
See here
Basically in the bayesian Setting you dont Need the params, but you can infer them from the data. Using the same dirichlet generative process logic you can cluster grouped data.


Storing engineered features in a database

Depending on the technical level of your users, the frequency of the update, the complexity of the transformation, the need to share these features among the users, etc. would a custom VIEW for each user be a feasible solution?

Alternately, would you consider some ETL tools where you can create/ modify calculated columns, customizing the data pipeline as needed?


Multiclass Regression for density prediction

If you want to predict the raw number of users, then this is a classical regression problem. Set an output layer with a node for each antenna, and no activation function.

If instead you need to predict a probability distribution / frequency on all antennas, use a softmax activation, so that each output vector would sum up to 1.


SVM with polynomial kernel: strange results with high degree?

Setting the polynomial kernel degree to 50 is likely causing the SVM to severely overfit to the data, which would explain the 9% you are seeing.  Increasing the degree helps the SVM make an appropriate generalization, but when you start to see the validation/test accuracy decrease, then the SVM is starting to overfit.


Suggestions on what patterns/analysis to derive from Airlines Big Data

There really is no wrong answer here, but I recommend predicting flight cancellations (#22) and/or delays (25-29), since this is how I often see this data set being used. It could also have practical significance to you if you should ever find yourself flying to or departing from one of the worst offending airports/airlines.

I'm not sure if you have a choice (perhaps your employer requires it), but don't use Map Reduce -- it's incredibly difficult to learn/maintain, it's slow, and on top of that it has become obsolete. Use something like Spark's ML lib (http://spark.apache.org/docs/latest/mllib-guide.html). It's much easier to use and is much more current.


Predict_proba on a binary classification problem

sklearn provides us with two methods to calibrate a probabilistic classifier via their CalibratedClassifierCV class; one using Platt's scaling (sigmoid) and one using isotonic regression.

Another method is by using a Venn-Abers predictor, which is not implemented in sklearn, however you can find a custom implementation here.

As for imbalanced datasets, make sure to exploit the class_weights argument of an sklearn classifier, to increase the weight of under-represented classes.


Why can decision trees have a high amount of variance

It is relatively simple if you understand what variance refers to in this context. A model has high variance if it is very sensitive to (small) changes in the training data.

A decision tree has high variance because, if you imagine a very large tree, it can basically adjust its predictions to every single input.

Consider you wanted to predict the outcome of a soccer game. A decision tree could make decisions like:

  IF

  player X is on the field AND
  team A has a home game AND
  the weather is sunny AND
  the number of attending fans >= 26000 AND
  it is past 3pm

  THEN team A wins.

If the tree is very deep, it will get very specific and you may only have one such game in your training data. It probably would not be appropriate to base your predictions on just one example.

Now, if you make a small change e.g. set the number of attending fans to 25999, a decision tree might give you a completely different answer (because the game now doesn't meet the 4th condition).

Linear regression, for example, would not be so sensitive to a small change because it is limited ("biased" -> see bias-variance tradeoff) to linear relationships and cannot represent sudden changes from 25999 to 26000 fans.

That's why it is important to not make decision trees arbitrary large/deep. This limits its variance.

(See e.g. here for more on how random forests can help with this further.)


Optimizing regression weights for NN outputs with PyTorch

My solution is to update the variable manually and creating a new graph for autograd. Works perfectly well now!
my_nn.train()
for epoch in range(300):
    # zero the parameter gradients
    optimizer.zero_grad()
    my_nn.zero_grad()

    Y = my_nn(X)
    loss_output = criterion(Y, y)
    w = torch.zeros_like(target)
    w.requires_grad = True
    for nest_epoch in range(50):
        y_w = torch.matmul(X, w)
        loss_w = criterion(y_w, Y)
        grad = torch.autograd.grad(loss_w, w, create_graph=True)
        w = w - 0.1 * grad[0]
    loss_regression = criterion(w, target)
    loss = loss_output + loss_regression
    loss.backward()
    optimizer.step()


How to use survival analysis for predictive maintenance for time series data?

Whenever your task includes something like "...when XY will fail...", i'd say go for survival analysis first, it is easy and fast and it will give you overview of your data.

With your data you can either turn them into intervals to be able to plot survival curves, or proceed directly to Cox regression, which can work with continuous data and will yield the hazard ratio.

You can start with Kaplan-Meier curve (as bonus there are Confidence intervals):

km <- survfit(Surv(datetime, Failed) ~ 1,conf.int=0.90, conf.type="log-log", data=Dataset)
summary(km)
plot(km, xlab="month", ylab="estimated S(t)", main="Kaplan-Meier with log-log, C.I.=90%")

The curve will look somewhat like this:

Further you can split the curve to see if any of your parameters have different influence. You can do that by simply replace the ~1 with something like ~AttributeX

So you should get this kind of plot:

Of course R will give you various tests and p-values as well, such as Log-Rank to verify whether the influence is significant (survdiff).

Then you can proceed to Cox Regression, which will tell you what is the hazard ratio (=whether the attribute influences the hazard positively or negatively and to what extent). It looks like this in R:

cox<-coxph(Surv(datetime, Failed)~AttributeX, data=pbc)
summary(cox)

It is good practice to verify the assumptions - proportional hazard and functional form (again R will give you p-values or you can plot the residuals - Martingale or Schoenfeld).

If you are interested to know WHEN the event will occur, search for Accelerated Failure Time models, which will give you the parametric survival time distribution, where you can simply put the time and obtain the probability.

In R:

wei<-survreg(Surv(datetime, Failed)~ AttributeA + AttributeB + AttributeC,data=Dataset)

There are more possible distribution, you can check which one fits the best your data. I have never done the prediction, but there is a function predict which is described in documentation or there are already similar questions with answers on Crossvalidated, such as this.


Is Mac OS the most suitable OS for a beginner Data Scientist?

You will not find the most suitable OS for data science. It will come down to preference and what tools you will use. Data science is broad, what tools people work with will vary, some will have support in one OS and less in another.

All OSs are suitable and used by data scientists. Often you don't get to pick yourself. But very few things are exclusive to one OS, so it all works out. I would say you should pick the one you are most comfortable with. You want to spend more time learning data science and less time learning a new OS.


the models in deep-learning on a little corpus

With Transfer learning.
There are already some pre trained models that most likely will do fine with you text. You just have to retrain them with some simple API. Try fastai for example. here and here


I'm not aware about for which condition I have to give the rewards. Is my reward logic is correct or I have to change it?

The reward structure in reinforcement learning is a little bit flexible, provided it expresses the goal of your agent.

In your case, you want to avoid two bad scenarios in a continuous problem. There is also a good scenario - providing enough water - which overlaps with the negative consequences of not providing water on demand, but may be subtly different.

Your suggestion of granting a negative reward for spilled water, and a negative reward for unavailable water seems quite reasonable. You have a few choices within that that could make differences to the behaviour of an optimal agent. Given your problem description so far, I think these are a free choice for you:

You can decide on a fixed cost for an error e.g. $-1$ reward, or base the cost on the amount of water spilled or missing when required.
You can weight rewards for water spilled differently to water missing when required. This is the sort of thing that might be a business decision, based on different costs for the different kind of errors. If spillage costs are low, demand is high and impact of failed distribution is high, then the optimal policy may involve over-filling the tank and spilling water deliberately.
You can have a positive reward for the amount of water supplied. This is similar to having a negative reward for failed supply, but again might be weighted differently in a business scenario. If you are supplying water to paying clients, then you would care about profits here and have 3 types of reward covering the profits from selling water, the loss from wasted water and clean-up costs, and (the maybe more nebulous, but still estimated) loss from failing to meet customer demand when supply is not available.

You should bear in mind, that combined with the behaviour of the environment, these decisions may change what the optimal policy is. Until you define these values, there is no single optimal policy. As your problem is simple, there may be a lot of overlap between optimal policies, but choices you make in reward functions and how the environment behaves will result in differences perhaps in how willing the agent is to risk over-filling in some cases.

As the problem is continuous, the long-term reward that you want to optimise could become infinite. To avoid this, and make the problem tractable, you need to make one of the following choices:

Use a discount factor, $\gamma$, which can be from $0$ to $1$ (but not actually 1 for a continuous problem)
Formulate your goal as maximising average reward.

In your case I suggest using a highish discount factor, e.g. $\gamma = 0.99$, as most of the RL literature for value-based agents uses discount factors.

Finally, you may have an issue here:

  The distribution of removal of water is not fixed. we can remove any amount of water from bottle, i.e. any continuous value between [0, 5]

RL depends on the environment behaving as a Markov Decision Process. Which means the theory works best if there is a fixed distribution here. That could be a flat distribution in $[0,5]$, or it could be drawn from a complex real-world scenario, but the assumption from the theory is that the distribution of changes to state is a function of most recent state and action plus a stochastic element.

When you simulate your problem in order to train the agent, you will have to use an actual distribution. The agent will then optimise against the scenarios it experiences. Depending on what you simulate, it may do poorly in other situations. However, provided you don't take this to extremes, the agent should learn to optimise over a broad range of states, and many reasonable distributions of the removal of water.


How to display an Heat Map (using Tableau or other BI tool)?

Set the latitude and put it under rows, select longitude and put it under cols.
Select value and drop it to the center of the sheet.
And under the marks dropdown select filled maps.

Make sure you see filled maps in the dropdown and that the value column is selected for color.

Edit: Do this for gradual color in heatmap-
click on color tab.
Select edit Color-now choose stepped color. This should do the work of filled maps with gradual colors.

Hope it was helpful.


Neural Network Multiple | Averging predictions

There are multiple ways you could do. All of them are categorized under Ensemble methods in machine learning.

Voting classifiers: which is the simplest way. You just take votes based on the label from all models and uses the majority label. That means, you should first round up all labels to 0 or 1 and then use the majority.

Weighted voting classifiers: similar to previous one, but some models have higher weights in voting.

For more information look here.


What does the color coding and normalized values in confusion matrix actually specify?

Each element_ab shows the probability of predicting label a (horizontal axis) when the true label is b (vertical axis)
For example, when the true label is 0, it will be predicted as label 2 with the probability of 0.14

The color intensity indicates the probability of each element in a row


Difference - Batch Training or training multiple times?

There will not be any difference between :

Training all batches + Epoch in one go
Saving a checkpoint and resuming the training later

Saving a checkpoint is a very good practice. It is not uncommon for training to fail due to environmental issues (E.g.: Spot instance on AWS gets kicked out , PC crash).

With Libraries like Keras, it is very straight-forward to checkpoint models.


Gradient vanishing means drastic decrease of gradients when backpropagating through many layers. This problem is also known for recurrent neural networks as they are mathematically equivalent to very deep networks.
That is true that gradients decrease during training. However, that is not gradient vanishing. That is the sign that the network has been trained, i.e. that is what you normally expect in the end of network training.


In general, you perform batch normalization before the activation.  The entire point of the scaling/bias parameters ($\beta$ and $\gamma$) in the original paper is to scale the normalized value ($\hat{x}$ in the paper) so that it fully captures the dynamic range of the activation operator.

For example (and this is the example used in the paper), suppose the normalized output of the layer in question (convolution or dense) had unit variance (pointwise within the matrix). Then if you applied the sigmoid function pointwise as activation, you would be constrained to be within the linear region of sigmoid function, hence negating any non-linear properties that are supposedly crucial to making the network work.

However, I'm sure you can find an architecture where the order is switched. In the end, people will do what gets the best results empirically for their use-case.

As far as the updating the parameters go (your fourth question), he does update the parameters. I took a look at the source. Parameter updates occur in the "update" functions associated with the convolutional layer. I.e. he only packages batch norm with other layers, which is a reasonable optimization to make.

For questions 2&3: You are correct except for the following caveat: if you are only using dense layers in the head of a CNN, you don't need to suddenly switch strategies. You would use the MODE_PER_ACTIVATION when you expect each dimension of the feature map to have a different distribution. Think non-vision use cases where the input map might be different types of features. If you are using dense layers in a CNN, you would still still assume the normalization is the same across the "spatial" scale. After you flatten in the final layer, you essentially flatten into $C$ channels with 1x1 spatial dimension. So your "spatial" Batch Norm will have $C$ parameters anyways.

For question 5: No. Think of the BatchNorm as a learned re-scaling and a bias operation together. You wouldn't expect the output of your neural network to be consistent if you trained it with bias then removed the biases at inference time.


Approach to semantic similarity between documents

If I understand correctly, you're trying to map abstracts to their research papers.

Here is a simple starting point:

Compute a TF IDF model using the entire corpus (all abstracts + research papers). Use this model to transform your abstracts and research papers into a weighted vector representation. Under the TF IDF weighting scheme, these documents will be represented by vectors that point in the direction of the words that are most discriminating for them. Put another way, these vectors point towards the words that are the best at telling you what the document is about. So if two vectors are close to each other, then the two documents are likely to be about the same topic because they are using similar words.

This is where cosine similarity comes in. Take an abstract, and then iterate across all the research paper vectors, computing the cosine similarity between them. Then, map that abstract to whichever research paper has the highest cosine similarity. Repeat this for each abstract.

Once you have this simple baseline approach, you can start looking at more sophisticated models for capturing semantic similarity.


Observing a set of features while training is same as seeing it one time. How?

This really depends on the type of algorithm you are using. With iterative approaches like stochastic gradient descent this is exactly what does happen. With tree based methods it will not matter except that the interpretation of your hyperparameters surrounding regularization changes by artifically creating a bigger dataset.


Is it scientifically correct to derive conclusions unrelated to hypothesis from A/B test data

It looks analagous to drug testing, where reporting of side effects during drug trials is obviously very important - i.e. the increase in Y seems analagous to a side effect. And some famous drugs have begun their lives as research into a side effect. Viagra is probably the most famous case, being a spinoff from a drug developed as angina medication. So in your write-up on your experiment you should definitely report the apparent effect on Y.
However, if the effect on Y is commercially important, then you still need to go back and do an experiment around a hypothesis that references the increase in Y to validate the existence of the effect properly.


Assuming you have multiple cities in the dataframe. you can create some new features in the dataframe . For example  , I created a few features below to try and match you PACF and ACF graphs .
df['lag_1'] = df.groupby(['city'])['temperature'].transform(lambda x: x.shift(1))
d=1
df['d_1'] = df['temperature'] - df['lag_1']
p =  1:
df['p_1'] = df.groupby(['city'])['d_1'].transform(lambda x: x.shift(1))
q = 1:
df['ma_1'] = df.groupby(['city'])['d_1'].transform(lambda x: x.shift(1).rolling(1).mean())
P=2 (and other terms)
df['lag_t12'] = df.groupby(['city'])['temperature'].transform(lambda x: x.shift(12))
df['lag_t24'] = df.groupby(['city'])['temperature'].transform(lambda x: x.shift(24))
.
.
.
df['lag_t120'] = df.groupby(['city'])['temperature'].transform(lambda x: x.shift(120))
Q=10 :
df['Q_10'] = df[col for col in df if col.startswith('lag_t')].mean()
After this try using LightGBM , XGBoost or other regression packages to regress against  these newly created features  with temprature as your target variable.
Alternatively , you can forego the ACF/PCF approach altogether and instead create  bunch of commonly used features using :

shift
rolling mean
rolling standard deviations
max() , min() within groups

and regress against those and check which features  minimize RMSE/AIC/BIC in your Regression Hyperparameters.
Since Cross validation is different in cases of Time Series,consider using TimeSeriesSplit in scikit-learn  . Check-out this post in case you want to do grouped time series cross validation .