How to reduce dimensionality of 3.2B categorical features?

Based on this article, I have decided to run with an autoencoder (tanh, tanh, tanh) neural net as opposed to PCA.

  "[for] thousands of dimensions the autoencoder has a better chance of unpacking the structure and storing it in the hidden nodes by finding hidden features ... [Also,] in contrast to PCA, the autoencoder has all the information from the original data compressed in to the reduced layer. [Finally,] the autoencoder is better at reconstructing the original data set than PCA when k is small, however the error converges as k increases. For very large data sets this difference will be larger and means a smaller data set could be used for the same error as PCA."

  https://www.r-bloggers.com/pca-vs-autoencoders-for-dimensionality-reduction/

Autoencoders for dimensionality reduction are also explained in OReilly's TensorFlow book.

UPDATE:
It looks like an embeddings layer as the first hidden layer would also be a good start to reducing dimensionality.

UPDATE: keras pooling layers also seem relevant


How R performs the operations like Inf + Inf, Inf * Inf, Inf - Inf and Inf / Inf?

Inf is a reserved word in R, so it is handled at interpreter level. That is, similar to FALSE & TRUE $\rightarrow$ FALSE and FALSE | TRUE $\rightarrow$ TRUE, when interpreter is faced with Inf ? Inf, it decides what to do based on the operation ?. You can think of a set of if-then rules under the hood that resolve the expression to Inf for + and *, and to NaN for - and /, no complicated inference is involved.


How to measure uncertainty and prediction errors for a naive Bayesian Classifier?

You can check the probability returned by your model. Just be aware that probability is not uncertainty but if you calibrate your probabilities, you can have an idea of the belief of event realisation from a frequentist point of view.

For the error of prediction, with a calibrated classifier if you have a probability of 0.8 to belong to a certain class, you can expect that there is a 80% chance that your sample actually belongs to this class.


What are some good loss functions used to minimize extreme errors in regression and time series forecasting?

As you increase the harshness of big misses, you make the model less willing to miss big.

For instance, absolute loss considers missing by $2$ to be twice as bad as missing by $1$, but square loss considers missing by $2$ to be four times as bad as missing by $1$!

Imagine if you use cubic loss:

$$L_3(y,\hat{y}) = \sqrt[3]{\sum\big\vert y_i - \hat{y}_i \big\vert^3}$$

Imagine if you use quintic loss:

$$L_5(y,\hat{y}) = \sqrt[5]{\sum\big\vert y_i - \hat{y}_i \big\vert^5}$$

This gets at the general form of $L_p$ loss, which I'm guessing you can guess before reading what I type.

$$L_p(y,\hat{y}) = \sqrt[p]{\sum\big\vert y_i - \hat{y}_i \big\vert^p}$$

This relates to the idea of $L^p$ norms.

At the extreme of $L^p$ norms, you can set $p=\infty$, which is equivalent to just looking at the maximum. If you use $L_{\infty}$ loss, you minimize the greatest error.

I've wanted to play around with $L_{\infty}$ loss for a while and would be very curious what happens if you choose this loss function. I do, however, think this would be a very extreme loss function.

(Remember that $RMSE$, $MSE$, and $SSE$ all have the same $argmin$; by similar logic we are free to take the $p^{th}$ roots to get the cubic and qunitic loss functions I gave.)


Training Examples used in Stochastic Gradient Descent

In a given iteration of the stochastic gradient descent algorithm, all 45 predictors are updated using a randomly generated subset of your 10,000 observation sample.  This subset may consist of only 1 observation, but typically cross validation is used to determine the optimal subset size.  You could even try randomly generating different subset sizes each iteration.


What are features in the context of reinforcement learning?

What are "features" in the specific case of RL?

In RL, the supervised learning component is used to learn a function approximation for either a value function or a policy. This is usually a function of the state, and sometimes of the state and action. The features are therefore explanatory variables relating to the state (or state and action) and need to be sufficient to explain the value or optimal action of that state.

In RL literature, there is often no difference between terms "state representation" or "features". Although you may need to process the state representation using feature engineering, into suitable features for whatever supervised learning you are using. With neural networks, this may involve normalising values that might otherwise cause problems.

  What are examples of features in RL?

In OpenAI's gym, CartPole-v1 has the following features describing state:

Cart Position
Cart Velocity
Pole Angle
Pole Velocity At Tip

The positions and speeds are measured in arbitrary units in this case, but you might consider them to be in SI units - metres along the track, metres per second, radians, metres per second.

Typically in RL, you will want features that:

Describe the state (and possibly the action, depending on implementation)
Are feasible to collect by observation of the environment
Together with the action choice, can adequately explain expected rewards and next state. In RL terms, they should in aggregate possess the Markov property

  Which RL algorithms require the specification of features?

None of them*. However, without descriptive features of state, you are limited to enumerating all possible states, and using tabular methods. Working with tabular methods constrains you in practice by the time it takes to fully explore all the states, and the memory space required to represent all the individual estimates of value.

If you are looking into solving your problem using DQN, A3C or other "deep" RL method that uses neural networks, then you need to be thinking in terms of a state representation that is composed of features, and to treat those features as if they were inputs to the same neural networks used in supervised learning.

  Why do we need "features" in RL?

To work with complex environments with large state spaces that cannot be solved without some form of approximation and (hopefully) generalisation.

* This is slightly more complex with policy gradient methods, as they do require that you use a function approximator, such as a neural network. However, it is possible to make the "features" in that case a one-hot coded vector of the state enumeration, which is the same representation as tabular value-based methods. There is little reason to do that in practice, but it is possible in principle.


Signature verification - machine learning

I think a Siamese Network is the solution you are looking for. Arxiv has a paper which precisely addresses your problem.


Does RandomForest convergence imply I can solve a problem with a NN too?

To me those are separate things since both models have a different cost function to be optimized.
On the other hand you could combine those models by constructing embeddings based on random forest splits and then using those embeddings as inputs for a neural network.
Toy example shows that there is a non-trivial configuration of a neural net that can get as good results as those obtained a by a random forest:
from sklearn.datasets import load_iris
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import RandomTreesEmbedding

X, y = load_iris(return_X_y = True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 42, test_size = .2)

params = dict(n_estimators=100,
    max_depth=None,
    min_samples_split=2,
    min_samples_leaf=1,
    min_weight_fraction_leaf=0.0,
    max_leaf_nodes=None,
    min_impurity_decrease=0.0,
    min_impurity_split=None,
    random_state=42,
    verbose=0)

mlp = Pipeline([("embeddings", RandomTreesEmbedding(**params)),
                ("model", MLPClassifier(activation = "identity",hidden_layer_sizes=(1000,), max_iter = 10000, random_state = 42))]).fit(X_train, y_train)

rf = Pipeline([("model", RandomForestClassifier(**params))]).fit(X_train, y_train)

mlp.score(X_test, y_test)
rf.score(X_test, y_test)


In addition to the answers provided, you can:
1.) Train jointly a CNN (or Autoencoder) with clustering on your data. (DCN, kmeansNet,..)
2.) Pretrain a CNN using self-supervision on your data. (Have a look into the vast self-supervision literature, e.g. this work).
3.) Use an alternating scheme to train a CNN classifier on soft-labels provided by a clustering algorithm, e.g k-means (e.g. this work).


I know which forums specific user read - how to cluster the data?

There are different approaches from easy to complicated
Similarity-Based

Distance or similarity matrix is calculated for different topics (forums) and if two topics are more similar than a threshold, you suggest them to the users of the other one. For this, you need to gather all texts of different forums and vectorise them (from TF-IDF to Neural Embedding). Then define a similarity score which tells you how related topics are. Then suggest them to users.
Second way of doing the same is to calculate similarity of different topics based on the number of mutual readers. Then doing the same as above.

Community Detection (Clustering) in Bipartite Graphs
Familiar yourself with the concepts of Community Detection and Bipartite Graphs. Then apply this clustering on users based on the forums they read. Select top topics inside each cluster of users and suggest them to all members of that cluster.
Link Prediction in Bipartite Graphs
Familiarize yourself with concept of Link Prediction. Applying link prediction on your problem gives you most probable links between users and forums.


It's of course technically possible to calculate macro (or micro) average performance with only two classes, but there's no need for it. Normally one specifies which of the two classes is the positive one (usually the minority class), and then regular precision, recall and F-score can be used.
Commonly there is a majority and a minority class, and naturally the majority class is easier to predict for the classifier. That's why the minority class is usually chosen as the positive class: by choosing the most difficult class to predict, the performance value represents more precisely the real ability of the classifier. As a consequence, the macro-average performance is often better than the performance on the positive class, since the former includes the "easy" class.


How to choose the number of output channels in a convolutional layer?

I recommend you reading the  guide to convolution arithmetic for deep learning . There you can find very well written explanations about calculating the about size of your layers depending on kernel size, stride, dilatation, etc.

Further you can easily get your intermediate shapes in pytorch by adding a simple print(x.shape) statement in your forward pass and adapting your number of neurons in your fully connected layers.

Last but not least. When you cange your input size from 32x32 to 64x64 your output of your final convolutional layer will also have approximately doubled size (depends on kernel size and padding) in each dimension (height, width) and hence you quadruple (double x double) the number of neurons needed in your linear layer.


Insights betwwen two columns/variables in Dataframe

The first step would be to bin cvsc100 in the same bin you have for your input, that will surely help your comparisons.

The second step would be to build a frequency table with your input range in columns and the binned CVS100 in rows, and at the intersection the count of values. This would help to observe how CVSC100 is distributed compared to your input and understand the underlying process.

Assuming that the score you have can be written as score for period n and period n+1,  with some hypotheses on the stationarity of the underlying process and modulo some slight renormalisation you can get transfer probabilities and use this matrix for simulations to get the distribution of scores for period n+2,... and so on.

Without more data (explanatory variables or actual default values), you can't really get more usefull metrics.


When do we scale features and should it be done to label encoded features?

You should not use Label Encoding for Categorical data unless there is a known ranking and that also in the specified ratio between the level values.
In this case, the model will assume 10 as 2 times of 5.

One-hot will add a lot of dimensions as I can see in your data.

You must try other Categorical encoding techniques esp. Sum Coding Or Helmert.

You should also try Dimension reduction techniques post-One-Hot to see if it reduces the dimensions.

Please do read these references -
Beyond-one-hot
Library
Kaggle tutorial
Feature Engg


How can I find anomalies in each row of data?

I am assuming your goal is a time series anomaly detection, in which you want to detect the occurrence of outlier as time goes by. To achieve that, you need to set up a time series prediction model first, and then perform outlier detection base on that.

For unsupervised outlier detection, just check Andrew Ng's machine learning course on Coursera, there is a chapter about anomaly detection based on gaussian kernel. You can also read sklearn docments for OneClassSVM, LocalOutlierFactor, etc. Basically, the core idea of unsupervised anomaly detection is setup a metrics to describe the affinity between samples, and then use a user defined threshold to decide if a new sample is similar enough to other samples.


Losses of keras CNN model is not decreasing

you can try to remove some of the irrelevant features that may be the reason or you could try different solvers and activation methods.I previously answered similar question earlier in the following linklink


running model.evaluate many times results different accuracy and loss value tensorflow 2

If you train your model and then run model.evaluate N times (do not retrain the model) you should get the same answer each time PROVIDED your test data is the SAME each time. However if you train your model then run evaluate and do that combination N times results will vary due to the random weight initialization of the network.


Any idea about application of deep dream?

There's already at least one application out, if you interpret 'application' broadly enough: Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation by Hong, Noh and Han. They use it for image segmentation. Standard image recognition networks can only give you a bounding box for each object recognized on an image. If you want to know which pixels constitute that object, you have to do image segmentation.

Basically, after finding a dog on an image, Hong et al's architecture back-propagates the dog-ness through the neural network down to the pixel level, to find the pixels that were the most responsible for the dog appearing. (They then use this heatmap as input for a supervised segmentation network, there's no deep dreaming in that part.)

This is already kind of an existence proof that the Deep Dream idea can be useful outside image manipulation. But I wouldn't downplay image manipulation itself either. I mention two things that are not immediate applications of Deep Dreaming, and we don't have them currently, but I can kinda see a plausible road from the original Deep Dream algorithm towards these:

Beautifying pictures and human faces and bodies. (Automating what a Photoshop retouch artist does.)
CSI-style image upscaling with fake but believable interpolated detail.


Does k fold cross validation become less useful when number of observations is very large?

Yes, you are right that when number of observations are very large, k fold cross validation (CV) are less useful. Let's look at why this is so:

1) Very high number of observations imply high training time for model and validation. Already the number of observations is large for the model to be trained and validated and now we are demanding it to be done k times. This is a huge burden on resources that is why in the Deep Learning regime we don't generally follow k fold CV as the data needed for training good neural networks are very high compared to traditional ML algorithms.

2) The higher the number of observations, the higher is the number of data chosen for cross validation set. This inherently makes it less likely that the sampled data points do not represent the original distribution. As you would know, the more data we sample, the better we approximate the original distribution.

Because of these reasons, k fold CV is inefficient when the number of observations is very high so a hold out set for CV will do the job.


I would say cross validation is unnecessary here since the multiple partitioning of the data and variables is already implicit in Random Forests.   But it's still a good practice to hold out a testing set that is distinct from the training set.  This is mostly because you may introduce changes in your random forest to improve the performance on the test sets overall, thereby introducing the bias that the random forests are trying to overcome.   So if you withheld a portion of your data and judged the final performance of the RF on that withheld set only in the predict step, then it's fine.


What can I infer from similar accuracy using two different DNN architectures?

I believe the only way to make a science-backed inference, would be if you fine-tuned both of your proposed solutions, to ensure that both are performing at their maximum potential. Only then you can say that method A is better than method B, otherwise it might just be a matter of improper selection of parameters that leads to those results.

However, if both methods still yield similar results after you have fine-tuned them, then you can follow Occam's razor and select the simplest one as the best one. By simplest, you can consider the one with the least amount of parameters and least time needed for training/inference.


Dimensionality of the target for DQN agent training

I am of the opinion that this architecture is only one among others that can solve the same problem (for example one may have only 2 outputs one for the chosen action and one for the $Q$ value of that action, but I will not elaborate further on this).
What this architecture does is to output the whole function $Q(a)$, that is the $Q$ value as a function of action $a$. So each output node represents the $Q$ value for a certain action $a$ corresponding to that node (so node 1 corresponds to the $Q$ value for action $a_1$, node 2 to $Q$ value for action $a_2$ and so on..)
BUT this is NOT a vector output, in the usual sense of the term. This is a functional output that represents the whole function $Q(a)$ for each action $a$.
As far as the learning/decision rule is concerned things are as usual. Hope this is clear.


Binary encoding and its interpretation in Python

Your categorical variable has two levels, so there is no actual difference between dummy-coding vs. simply entering the variable into the analysis. That is, to dummy code you would create one new variable with two values but your original variable is already one variable with two values. Dummy-coding is important for variables with more than two possible values. So, in this case the computer won't consider Pave > Grvl.

But if you have more than two variables then you should use dummy variables.

For your data, you can use pandas.get_dummies() or sklearn's one hot encoder to achieve your result.


When to One-Hot encode categorical data when following Crisp-DM

To me it depends, because I would separate some types of Categorical Variables :

Categorical variables with few classes : OneHot as fast as you can
Categorical variable with some highly-represented classes and some low-represented classes : You can pre-process and regroup both low-represented classes in a huge "Other" class, and then OneHot and get a reasonable number of variables
Categorical variables with A LOT of low-represented class : If you OneHot directly, you'll create a lot of variables, so this feels impossible. You can, for example, browse those data so you calculate, for each class, the rate of "1" classes on your X_train. You then transform your class by this number, which is continuous, between 0 and 1, and so have information and is accepted by all models. This is called Target Encoding, and some packages built to be compatible with sklearn exist to do it automatically (like TargetEncoder, LeaveOneOut, WeightOfEvidence or JamesStein).

These are the kind of changes you can do, the choice of OHE directly, or pre-process before, it depends on the variable...
If your question is, for example, to know if you make feature selection before OHE or after, I'd suggest you mainly making it after : Remove useless variable (with no info), then OHE/preprocess remaining ones, and then make feature selection again.
Let's take an example : a variable called Age being classes like [0;10], [10;20], ... it's often significative if the value is >80 or <20, but doesn't care if it's 35 or 45, so the OHE will only select Age_[0;10], Age_[10;20], Age[80_90] and Age_90+


Can i expect good results having low correlation attributes?

The correlation does not effect your model using decision trees in a classification problem.
In the theory of decision tree models, you don`t need correlation or check of multicollinearity. Because the split in decision trees is made of entropy/information gain.
The correlation does only check linear dependencies. The same is, when the dataset is highly correlated. You will get very good results with decision trees, because there you don´t need to delete correlated features or do dimension reduction (if you don´t have to).
It can be, that you don´t get very good results, when you use linear structured models like multiclass neural network, or multiclass logistic regression. There you will see that dimension reduction etc. can have a high influence on the accuracy in these models.
I had a similar question but with highly correlated features:
decision -tree regression to avoid multicollinearity for regression model?
In your case I would say, if we use decision trees, it is not noticeable. However we should check this with the permutation importance of the features and check the polynomial dependencies. Of course you should ask the interviewer more question about his questions and the target of his question, to get more background information. This is very important in interviews.


Do we have to split our dataset into training & testing when using ARIMA model?

Splitting in Train and Test sets or not depends from the purpose of your analysis. You can follow a statistical approach or a machine learning approach.

In the classical, statistical approach, you fit a model on the whole batch of data. Your goal here is to check the sign of the variables' parameters, and whether they are significant or not. Scientifically speaking, each of those parameters represents the test of an hypothesis.

In the machine learning approach, you want a model that is good at predicting data it has never seen before. You don't care whether a given variable has a positive or negative association with you dependend variable, you don't care whether your parameters are 95% significant or not, you just care that the model predicts the output as precisely as possible.

So, the answer to your question is: it depends! What do you need your model for?


Public dataset for account to account payments

There is this dataset and if it doesnt match, maybe you could contact the authors
at Brussels University for more information.
https://data.world/raghu543/credit-card-fraud-data

The datasets contains transactions made by credit cards in September
2013 by european cardholders. This dataset presents transactions that
occurred in two days, where we have 492 frauds out of 284,807
transactions.

This one uses simulated data but sounds more like what you are looking for. There is also an academic paper that goes with it.
https://www.kaggle.com/ntnu-testimon/banksim1

We ran BankSim for 180 steps (approx. six months), several times and
calibrated the parameters in order to obtain a distribution that get
close enough to be reliable for testing. We collected several log
files and selected the most accurate. We injected thieves that aim to
steal an average of three cards per step and perform about two
fraudulent transactions per day. We produced 594643 records in total.
Where 587443 are normal payments and 7200 fraudulent transactions.
Since this is a randomised simulation the values are of course not
identical to original data.


Convolutional neural network with 1 channel images/1 input channel

The following will be the updated code for grayscale images:
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 1)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))

Why?
Because for an RGB image, there are 3 channels, 'R', 'G' and 'B'
So, the input shape will be (height, width, 3)
But since grayscale has only one channel, the input shape becomes (height, width, 1)
Note that if you are using Keras with Tensorflow backend, then the data_format is channels_last, which means that the input shape should be  (height, width, channels).
Otherwise, if you are using Theano as the backend, then the input shape should be (channels, height, width) since Theano uses the channels_first data format.
Hope this is of help.
Please upvote if helpful.


Why do decision trees have low accuracy?

It's not true in general. Decision trees tends to overfit in comparison to other algorithms, which provide too low accuracy. But if you use a decision tree in the right way i.e you prepare data in the proper format, use feature selection and perform k-fold cross-validation everything should be ok.

I am sure that you misread it. There is no reason why DT could be a much worse algorithm compared to others.


By default scikit-learn's KNNImputer uses Euclidean distance metric for searching neighbors and mean for imputing values.
If you have a combination of continuous and nominal variables, you should pass in a different distance metric.
If you want to use another imputation function than mean, you'll have to implement that yourself.


First of all, your question is not that clear. Next time try adding an example of your dataset.

As you probably know, Machine learning algorithms are described as learning a target function (f) that best maps input variables (X) to an output variable (Y): Y = f(X)  .

As @ankit-seth mentioned,
If you want to classify the data you need to have the labels, and then you can use ML algorithms to do that.
I'm guessing you need to cluster(group) your data - try reading about K-Means and go from there ...

good luck :)


Policy gradient: why does this converge with Adam and not SGD?

Stochastic gradient descent (SGD) can get stuck in saddle points which results in the model not converging. A saddle points gradient is where zero in many directions but not all directions.
Adam adaptively changes the learning rate based on past gradients. This additional information can allow the model to escape saddle points, thus the model is more likely to converge.


Improving Naive Bayes accuracy for text classification

Regarding your first question...

Do you anticipate the majority category to be similarly over-represented in real-world data as it is in your training data? If so, perhaps you could perform two-step classification:

Train a binary classifier (on all your training data) to predict membership (yes/no) in the majority class.
Train a multi-class classifier (on the rest of the training data) to predict membership in the remaining minority classes.


The validation loss < training loss and validation accuracy < training accuracy

How did you split your training and test sets? My guess is that the minority class is under-represented in your test set. Consider the case where the minority class has frequencies of 25% and 5% in the training and test sets, respectively. If my model predicts everything is the majority class, the recall for the minority class is 0% on both sets but we get 75% training accuracy and 95% validation accuracy. Now let's say we train the model a bit more and  achieve 50% recall while still correctly labeling all of the majority class samples: now we're up to 87% training and 97% accuracy. Because the training and test sets exhibit significantly different class frequencies, the test set will basically always have a higher accuracy than the training set (up to the point where you overfit, at which point the validation accuracy could be be lower than training accuracy but not necessarily).

The solution here is to either use stratified sampling to make sure your classes have equal representation between your different datasets, or to monitor other evaluation metrics like precision and recall. Plotting the taining and validation accuracies could also be helpful, since it should be visually pretty obvious if your model starts to overfit, regardless of class imbalance between your training and test sets.


How do I build a DQN which selects the correct objects in an environment based on the environment state?

The DQN agent does not need to care what the actions represent, in your case it only needs to make a discrete choice, and it is simple to enumerate the action space. Ignoring the meaning of them for a moment, you have 16 discrete actions. The simplest way to model that is to have a single index discrete action space of 16 actions for the agent which you then map to the selections that you need in order to assess the results. As long as you do this consistently (e.g. take the binary representation of the action index number), this is fine.

It is also possible that using a more direct representation would help, depending on the true nature of the value function. In which case you could use it, provided you model the neural network for $\hat{q}(s,a,\theta)$ with action vector concatenated to state vector in the input, and a single output of the estimated action value for that specific combination. To assess which action to take, you would create a minibatch of 16 inputs, all of which have the same state component, and cover the 16 possible input variations. Then you would pick the combination with the highest estimate and look at the action part of the input vector to discover which action was estimated to be best.

If you are not sure which approach would suit the problem best, you could try both.


I will restrict myself to the finite-dimensional situation.

First, if we are talking about normal eigenvalues and not generalized ones, we need a square matrix. If $M\in\mathbb{R}^{n\times n}$ (or $\mathbb{C}^{n\times n}$) is a diagonizable matrix, it has an eigendecomposition
\begin{equation}
M = V\Lambda V^{-1}
\textrm{,}
\end{equation}
with $V:=\begin{bmatrix}v_{1},v_{2},\ldots,v_{n}\end{bmatrix}$ the matrix with its $n$ eigenvectors $v_{i}$ as columns and $\Lambda :=\operatorname{diag}\left(\lambda_{1},\lambda_{2},\ldots,\lambda_{n}\right)$ the diagonal matrix with the corresponding eigenvalues on its main diagonal.

As you can see from this equation if you want to get matrices with the same eigenvectors, i.e. $V$ fixed, you can only change the eigenvalues $\lambda_{i}$. In general the relationship between the eigenvalues and the entries of the original matrix $M$ is non-trivial.

The easiest situation arises if the eigenvectors $v_{i}$ form the standard basis, i.e. only $1$ at the $i$th position, $0$ otherwise. Then $M=I\Lambda I^{-1}=\Lambda $ ($I$ the identity matrix) and you can change every entry on the main diagonal without changing any eigenvector.

Edit

I understood the question differently than victor. I thought the question asks what kind of different datasets can be described by the same eigenvectors.


Try to find a pattern in the sentences which are failing.
Use negspacy, find out words which are negative, replace them with opposite words. eg -
'not good' -> 'bad'
try using bert transformers pretrained sentiment classification models.


Your Pipeline is now a model that expects similar input as before. You can use a CV scheme to choose your K inside your Pipeline, after it has evaluated the options and chosen the correct K it will use that to predict. For actual predictions, enter a list of tweets and it will classify them, no need to manually specify which features to use. If you want to know which features your Pipeline has chosen you can look in the properties of your 'p' object.


Modelling for similarity between two descriptions

So the questions asks for how to compute similarity between the organisation description and project titles.
One initial thought would be to use a Doc2Vec model (concept, implementation), which will take the organisation descriptions and project titles as input and output a n-dimensional vector in semantic space for the given text.
From this, you can at least has a baseline which you can use with cosine similarity to see how similar the organisation description is to the project title.


Can bidirectional RNN use variable sequence length?

The short answer is no, a bidirectional architecture will still take in a variable sequence length. To understand why, you should understand how padding works.

For example, let's say you are implementing a bidirectional LSTM-RNN in tensorflow on variable length time series data for multiple subjects. The input is a 3D array with shape: [n_subjects, [n_features, [n_timesteps...] ...] ...] so to ensure that the array has consistent dimensions, you pad the other subject's features up to the length of the subject with features measured for the longest period of time.

Let's say subject 1 has one feature with values = [22,20,19,21,33,22,44,21,19,26,27] measured at times = [0,1,2,3,4,5,6,7,8,9,10]. subject 2 has one feature with values = [21,12,22,30,13,42,20] measured at times = [0,1,2,3,4,5,6]. You would pad features for Subject 2 by extending the array so that the padded_values = [21,12,22,30,13,42,20,0,0,0,0] at times = [0,1,2,3,4,5,6,7,8,9,10], then do the same thing for every subsequent subject.

This means the number of timesteps for each subject can be variable, and the merge you refer to occurs with the dimension for that particular subject.

Below is an example of a bidirectional LSTM-RNN architecture for a model that predicts sleep stages for different subjects using biometric features measured over variable lengths of time.


Does feature selections matter to Decision Tree algorithms?

For ensembles of decision trees, feature selection is generally not that important. During the induction of decision trees, the optimal feature is selected to split the data based on metrics like information gain, so if you have some non-informative features, they simply won't be selected.

Feature selection can still be important for small datasets, where spurious relationships between features and class labels are more commonly seen.

As a side note: it's common practice to remove some features from datasets for all learning algorithms, like ID fields. These features are especially bad for decision trees, as they usually end up being the feature with the highest information gain (and therefore selected to be split on) while actually containing no generalisable information. So in this way, feature selection can be useful when using decision trees, although it's arguable that the ID field should be considered a feature in the first place.


When imputing data, one is looking not to modify the true distribution of your data. So a way to test how good your imputation was is to make a test to contrast the true distribution of every feature that has been imputed vs the true (via KS test for example) distribution of the feature (prior imputing) if you can sate with a level. of confidence that your imputation preserved the distribution that would be a way.
Another way would be in case you have a supervised task, you can compare the performance of your model on each imputation technique. Like in the below's image from Scikit-learn documentation:


Why does logistic function use e rather than 2?

Since you are going to minimize later on the log likelihood, there is actually no big difference between $\log 2^x=x * \log2$ and $\log e^x=x$. You see the difference is simply a constant.
Nevertheless one could argue to use $2^x$ instead of $e^x$ und also use $\log_2$ instead of $\log$ when it comes to the optimizing step. In fact it is possible to use $2^x$ and also many other functions, which show some desired properties.
Which are:

$\lim\limits_{x \rightarrow \infty}{f(x)}=1$
$\lim\limits_{x \rightarrow -\infty}{f(x)}=0$
$f(x) = -f(-x) + 1$, (symmetric in $(0, 0.5)$

Here is an example of suitable functions from wikipedia.


R-square or adjusted R-square for one variable model?

They're going to be very similar (practically the same), for a model with only one independent variable. So I'd say it doesn't matter without understanding better your purpose in using R2 / Adj R.


Is it possible for a ML model to perform worse than random classification?

Yes, it's possible.

Just means the model is adjusting to noise, so it's valuing the "wrong" features.

As an analogy, if you were randomly guessing basketball game outcomes, you'd probably perform better than someone who thinks less points is better and is guessing based on previous games' scores


Tools for visualisation: making a video of a sliding windows moving on the time series

Using this, I've written a script for it.

%matplotlib notebook

import numpy as np
import matplotlib.pyplot as plt
import time

plt.ion()
size_x = 128
def pltts(ts1, ts2, ax, colors, lbl):
    x = np.linspace(0,2.5,size_x)
    if ax.lines:
        ax.lines[0].set_xdata(x)
        ax.lines[1].set_xdata(x)

        ax.lines[0].set_ydata(ts1)
        ax.lines[1].set_ydata(ts2)
    else:
        l1, l2 = ax.plot(x, ts1, colors[0], x, ts2, colors[1])
        fig.legend((l1, l2), ("BLUE", "RED"), loc='center', fontsize=fnt_siz-2)
        ax.set_title(lbl, fontsize=fnt_siz-2)
    fig.canvas.draw()

fnt_siz = 16

fig,ax = plt.subplots(2,1, figsize=(10,10))
fig.suptitle('Data', fontsize=fnt_siz)
ax[0].set_xlabel('Time', fontsize=fnt_siz-4)
ax[0].set_ylabel('Magnitude', fontsize=fnt_siz-4)
ax[1].set_xlabel('Time', fontsize=fnt_siz-4)
ax[1].set_ylabel('Magnitude', fontsize=fnt_siz-4)
plt.subplots_adjust(hspace=0.75)
# ax[0].grid(True)
# ax[1].grid(True)
for f in range(50):
    ts1 = np.random.normal(0, 4, size=(size_x,1))
    ts2 = np.random.normal(1, 1, size=(size_x,1))
    pltts(ts1, ts2, ax[0], ['b', 'r'], "ORGINAL")
    ts1 = 2*ts1
    ts2 = -3*ts2
    pltts(ts1, ts2, ax[1], ['b', 'r'], "TRANSFORMED")
    time.sleep(1)

This is a sample output:


How to see which transformation is the best

There are two main reasons people typically transform their data.  Either to help meet assumptions or to help boost predictive performance.

Help meet assumptions
In your case of a linear regression,  you have several assumptions that you have to meet including things like linearity and residuals have a constant variance and appear independent of each other.  Transforming your response can help make your relationship more linear and help with your residual assumptions, while a transformation on your predictors primarily helps with those residuals.  Typically if you are doing inference, you need normal residuals (or large sample size for CLT to kick in) which you can assess with a qq-plot or a test for normality, but if you are doing predictions you don't have to worry about this.  Trying a few popular transformations (sqrt, square, log) is a good idea, as is the Box-Cox transformation (on predictors) which basically finds a transformation that maximizes the normal likelihood function of the residuals.  The box-cox goes through several power transformations in a systematic manner, and it primarily used to make the residuals normal, though it can help the other issues too.
Improve predictive power
This is more of an open game, and I would be wary of over fitting.  See what works best and go with it (can be tough, might have to go with cross-validation if you're data isn't too big).
Your Case
Usually your x-axis for those plots is the fitted values (you can plot a glm object, not sure about lm, so run through most of your basic diagnostic plots). just with those plots its also tough to assess normality, which is typically done with a qq-plot or a hisotgram of the residuals.  It looks like your response variable new_users is a count; maybe a poisson model would be another option.  "best" transformation is the one that helps you meet your assumptions as noted in (1), its more of finding the transformation that makes you feel most confident that your assumptions are met.  If you are looking to increase performance, $R^2$ will help indicate a helpful transformation, but again be wary of overfitting.  The power transformations e.g.$(x^{1/3}), log(x+0.1), x^{2}$ are usually a good route to go.


Can the performance of a CNN be dependent on the train-test-val split random seed?

Yes, it can depend on it because it changes the data distribution with the network is trained.
You shouldn't consider random seed as a hyperparameter. Keep the same random seed and run comparison. Do this for at least 5 or 10 random seeds. You will definitely have a winner. If you don't go for more than 10 unless you get a winner. That will give you sufficient proof about the efficacy of the method. Otherwise since one is good in random seed or other one in other, just choose one of those random seeds, choose the method that gives best in it and continue your work.


How to get the tagset for hindi pos tagging?

I'm not familiar with NLTK but the tagset must come from the annotated corpus which was used to train the tagger. According to https://www.nltk.org/book/ch05.html (section 2.2), if you can find the name of the Hindi corpus you should be able to access the words with their tags with:

nltk.corpus.<corpus>.tagged_words()

Apparently this is the list of corpora available in NLTK: http://www.nltk.org/nltk_data/, it might help finding the one used to train the Hindi POS tagger.

Once you identify the corpus, it's likely that you can find explanations about the tagset and annotation process by searching for papers about its creation by the original authors.


Using Pandas, how do I strip part of a cell from an Excel file?

String operations on particular columns (or series) can be done by appending .str to the series. You can use the following command to get your output:

>>> df['Case Number'].str.strip('Case ID Number - 00')

You can refer to the documentation here Working with text.


choosing model based on last or best iteration on validation set

Too much noise in the curve may indicate a too-high learning rate. You could try reducing the learning rate with the number of epochs, when the curve becomes flat.

You can also use early stopping with a "window". That means, stop training if the loss is not getting better after N epochs. Where getting better means a % or a fixed number at your choice too.

I would use the best epoch. Using the last epoch sounds weird to me and could introduce some bias too if you wait to see some initial results before choosing a maximum number of epochs (and you are probably going to do that).

Also, you may want to spend some time trying to improve your training set if you have not done so already. Maybe you can find more data or augment it.

If you are unable to overfit, you may as well want to try with a more complex model. Regularization/dropout could help with your leaning curves too.


How to calculate similarity between 2 users based on the images they share?

Use a similarity vector with the dimension of all possible similarities, which initially are set to zero everywhere. Go over the shared images of one user and for each image set the vector to 1 at the positions of the similarities from the image.
Do the same for another user.
Use the scalar product as similarity metric between the two users.
You can add weights for similarities which appear more than 1 time on the shared images.


Evaluating a trained Reinforcement Learning Agent?

If your goal is optimal control, then you will want to measure the agent by how well it does at the task. You should use some aggregate measure of reward, such as total reward per episode (aka "return"), or mean reward per time step.

If you are working with a toy problem, or one designed so that it is easy to identify a maximum bound on the reward-based measure, then you can compare your agent against this known value. It is reasonable to expect a good agent to approach the maximum value.

In practice, many interesting problems do not have a known upper bound on reward totals or averages. For those problems, typically the best you can do is compare between agents. You can compare with:

A randomly acting agent. This would normally be just as a baseline, to show that the agent had learned something.
An automated agent using a simple action choice heuristic, which might be something natural or obvious in the given problem.
One or more humans on the same task
Other ML-trained agents including previous instances of the same agent

You will probably want to run multiple tests and average the results, if either the policy or environment are stochastic, in order to assess an agent with expected values as much as possible. It is also important, if you are using any off-policy technique such as DQN, to switch off any exploration during tests, to get a fair measure of how well the trained agent behaves (as opposed to how it behaves during training, which will be different).

If your agent is designed to continually learn and explore, and/or uses an on-policy approach, you can use results during training to assess it. For instance you can take a rolling average of total reward over the last N episodes or something similar. This is not a bad metric to monitor training, even for off-policy approaches, although for off-policy you will likely get an underestimate of performance compared to separate test runs.

There are other approaches and other metrics to assess an agent - e.g. how much experience, or how much computation the agent needs to learn to a certain level is often of interest.

If you want to "conclude that the agent is trained well or bad" for an optimal control task, then this assessment of total reward might be all you need. However, you can also look at loss metrics inside any neural networks - you would not do this in order to rank agents as "better" or "worse" but might do so in order to identify problems. These loss metrics are generally the same as supervised learning equivalents. So for instance in DQN, or for the "critic" part of PPO, you would be interested in whether the predicted value of any state matched the eventual value, and use MSE loss. For stochastic environments there is a caveat to this, that any collected data will be noisy so it is hard to tell the difference between a high loss due to variance in the data and high loss due to poor training or incorrect hyperparameters.


Keras cNN Transfer Model: Reduce Final Model Size?

There's actually some pretty interesting research on this topic. Key words would be model compression or CNN pruning (doing things like reducing the model size by removing filters with low activations).

Another interesting method uses a technique called teacher-student networks. Basically, this involves training the big network (the teacher) and using the output of the teacher as the labels when you train the small student network. This usually serves to improve the student over training with the ground truth.

For example, if you were training on MNIST, you'd have 10 classes and a softmax output of the teacher network that might look like [0, 0, 0, 0, 0.33, 0, 0, 0, 0, 0.67], in the case of a 9 that looks like a 4, when the original ground truth was [0, 0, 0, 0, 0, 0, 0, 0, 0, 1].

I'm not totally current on this research so give these a google first, there might be some better best practices out there. Hope this at least gets you going in the right direction.


Finding the worst affected industry due to COVID in terms of unemployment

Thank you clarifying the question and the current data available. This is a really interesting project idea and I look forward to hearing more developments on this.
So, with what I have seen so far, without the monthly breakdown of number of businesses by month, then you you can't look at the affect of COVID-19 on an industry-level, unless you compare 2019 with 2020 data.
However, with the data so far you can look at how COVID-19 is affecting unemployment rates and see if they affect certain groups of counties which have a certain industry composition/distribution.
If, for example, particular counties with similar distributions of industries (e.g. 90% Manufacturing, 5% Retail and 5% Information) show greater unemployment from those with a different composition.
Let me know if I need to clarify further. But again, very interesting project, please do keep me in the know with how this goes.


TensorFlow stopped working after installing opencv

The first and foremost thing we needs to aware is , from installation perspective is there any micro level relationship between tensorflow and openCV , so that it could help us out in troubleshooting the above issue.
But yes    this link will help you out in accomplishing your tensorflow/openCV setup installation


I think you can just apply a t-test or Wilcoxon test if you're don't believe in the normality of your observations.

In R:

A <- c(36.52 , 36.27 , 35.9 , 35.22 ,37.13 , 35.53 , 35.3 , 34.14 , 35)
B <- c(36.85 , 35.64 , 36.37 , 36.82 , 36.74,36.46 , 35.77 , 37.31 , 36.68)
wilcox.test(A,B,alternative="less",paired=T)
t.test(A,B,alternative="less",paired=T)

I've done paired tests under the assumption that you did the same folds for each architecture. If that is false, then do unpaired tests.

Edit: You may be interested in a question I posted a few months ago. Cross-validation for model comparison: use the same folds?


Eliminate low quality predictions in a classification task

An alternative to thresholding the classification probability would be to set a threshold on the ratio between the highest reported probability and the second highest reported probability. For example, a threshold of 2 would be interpretable as: "Only retain classifications where the likelihood of the class assignment is at least twice as likely as the next most likely class."


Dendrogram: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

Just in case someone else is searching for the same issue, by converting the labels to list, it will work.
samples= df_euro.iloc[:, 2:7].values[:42]
country_names= list(df_euro.iloc[:, 1].values[:42])

mergings = linkage(samples, method='single')

# Plot the dendrogram
fig, ax = plt.subplots(figsize=(15, 10))
fig  = dendrogram(mergings, labels=country_names)
plt.show()


Seeing your graphs it does not show that the model is not learning, as the training accuracy is high. If the model wasn't learning anything even this would have been low. Also, you are right that the model is overfitting because it is performing very well on the training set and poorly on the test set.

The overfitting scenario is also confirmed by the fact that you have a very small dataset. You might want to fine-tune all the layers of the resnet as shown here. Also, try early stopping and dropout to prevent the model from overfitting.


Support Vector Machines (SVM) vs Minimum distance between two convex hulls

You have to add a constraint of linear separability to make the comparison. The problem is that in reality most problem are not linearly separable. Either they need non-linear transformations (data processing or usage of kernel), or they are not separable at all (underlying randomness, noise). SVM can be adapted to these non-linearities quite well (allowing for missclasification, usage of kernels), while the second algo would need significant rework. Basically, as is, the second algo wouldn't work for most real life datasets.


Strange plot on a simple dataframe Rstudio

Update:
It was only a matter of transform string numbers in numerical numbers.
I did it by transforming each "non numeric" column in character and used the as.numeric().
The code looks like this now:
    #Nomes das colunas
names(df)[2] = 'Produçãox1'
names(df)[3] = 'Produçãox2'
names(df)[4] = 'Remuneraçãoy1'
names(df)[5] = 'Remuneraçãoy2'

df$Produçãox1 <- as.numeric(as.character(df$Produçãox1))
df$Produçãox2 <- as.numeric(as.character(df$Produçãox2))
df$Remuneraçãoy1 <- as.numeric(as.character(df$Remuneraçãoy1))
df$Remuneraçãoy2 <- as.numeric(as.character(df$Remuneraçãoy2))

#Regressão
reg1 <- lm(df$Remuneraçãoy1 ~ df$Produçãox1)
reg2 <- lm(df$Remuneraçãoy2 ~ df$Produçãox2)

#Plot
plot(df$Remuneraçãoy1 ~ df$Produçãox1,xlab='Produção do setor empresarial',
     ylab='Remuneração real')
abline(reg1, col='blue')

plot(df$Remuneraçãoy2 ~ df$Produçãox2,xlab='Produção do setor empresarial não agrícola',
     ylab='Remuneração real')
abline(reg2, col='red')


What you can do is discretize happiness index using cut() and assign labels to it.
For plotting then you can use groupby boxplot which will show effect of both income sources on a particular happiness index bin.
Refer GroupbyBoxplot
If you want distribution of happiness index then you can try:
Color Scales


IsolationForest Decision Function vs. Anomaly Prediction Question

It looks that your PCA is not mapping well the data. I recommend you to look at other dimensionality reductions techniques, such as, UMAP, TSNE, in order to see if you can achieve better representation. If you do not have too much data, you could use also MDS since it will preserve local distances (however, computationally very expensive):

https://scikit-learn.org/stable/modules/generated/sklearn.manifold.MDS.html

If only the visualization is what your are after, then you could just simply run Isolation Forest on 2 Principal components, then the boundaries should be more obvious in the first picture. Of course, the actual results may be misleading.


specific feature engineering for specific algorithm

In general, features are engineered so as to retain optimum relevant information present in the dataset with succinct representation and then features are adapted so that an algorithm can accept it as input.

Feature engineering generally involves methods like binning, PCA, etc. Adapting those features to pass to algorithm is another step which is a very small part of the feature engineering step. Adapting example: for an image, we may have to reshape the image like below so that x[0] points to the image

x = image.img_to_array(img)
normalise(x) // feature engineering
x = x.reshape((1,) + x.shape) // Adapting for algorithm

With this understanding, if features are generated once for a dataset, it may be used for any relevant algorithm with corresponding adaption.


How to create domain rules from raw unstructured text using NLP and deep learning?

I think your question can be solved using Case Based Reasoning .

Basic principle how it works is, you need to train the model using whole lot of different cases which you have. Based on the symptoms which you give the outcome is predicted(which disease).

Please refer to the links below for deep diving into the topic, appended couple of links with respect to health care industry:

Link -1
Link -2
Link -3
Link -4
Link -5
Link -6
Link -7

Do let me know if you need any additional information.


Metric to evaluate words generated by Neural Network

Maybe somebody will have a better idea but the default method would be to generate a set of names, then ask a few annotators to label them as good or bad (possibly scoring them from 1 very bad to 5 very good), and finally train a supervised model to recognize the good from the bad ones. This approach would also give you the opportunity to check the inter-annotator agreement, i.e. assess how subjective the choice good vs. bad is. In general a subjective task is difficult (or impossible) to do with an unsupervised metric.


How to determine the number of the training images in Keras after data augmentaion?

The images in the dataset are not used directly. Instead, only augmented images are provided to the model. So, your answer is 2000 images


Use distribution probability as a feature in ML model

As a general approach I would say you need to generate new features, that use your prior knowledge. For example, if you have a known size distribution, then for each specific size you can calculate its probability and use it as a new feature.
As I side-note, the geometric distribution of cow sizes seems very surprising to me, I would expect to see some gamma distribution or just normal (if size is measured in cm/inches).


ValueError: cannot reshape array of size 136415664 into shape (2734,132,126,1)

You need $2734 \times 132\times 126\times 1=45,471,888$ values in order to reshape into that tensor. Since you have $136,415,664$ values, the reshaping is impossible. If your fourth dimension is $4$, then the reshape will be possible.


How do I compare columns in different data frames?

If you want to check equals values on a certain column let's say Name you can merge both Dataframes to a new one:

mergedStuff = pd.merge(df1, df2, on=['Name'], how='inner')
mergedStuff.head()

I think this is more efficient and faster then whereif you have a big data set


Classification of data generated by radar using FFT

If you are dealing with a classification problem, you should treat the $0$ and $1$ as labels.

The goal of FFT is to convert time domain to frequency domain.

You do not want to include the label as part of your FFT procedure.


gzfname, h is doing tuple unpacking from the return of urlretrieve.  From the docs, urlretrieve returns:

  Return a tuple (filename, headers) where filename is the local file name under which the object can be found, and headers is whatever the info() method of the object returned by urlopen() returned (for a remote object). Exceptions are the same as for urlopen().

And the info() method of the urlopen object returns:

  info() — return the meta-information of the page, such as headers, in the form of an email.message_from_string() instance (see Quick Reference to HTTP Headers)

So, h is meta information about the retrieved page.


Did you try to start your training with a smaller data set and that worked?

How about using generators to gradually input your data? For me that worked quite well (in Keras : model.fit_generator(...))


Spacy Text classification (Binary Classification)

You're very close to having a working script. The textcat training example in the spacy repository shows how to save the model, reload it, and run it on a new text: https://github.com/explosion/spaCy/blob/master/examples/training/train_textcat.py

From around line 103:

    # test the trained model
    test_text = "This movie sucked"
    doc = nlp(test_text)
    print(test_text, doc.cats)

    if output_dir is not None:
        with nlp.use_params(optimizer.averages):
            nlp.to_disk(output_dir)
        print("Saved model to", output_dir)

        # test the saved model
        print("Loading from", output_dir)
        nlp2 = spacy.load(output_dir)
        doc2 = nlp2(test_text)
        print(test_text, doc2.cats)
```


How to get back stationary data from non stationary data after performing diff operations in var (time series)?

You simply have to add the resulting time series. The cumulative sums are comparable to your original input.


Issue with Custom object detection using tensorflow when Training on a single type of object

How many images are you using to train?  You might need more... let's say  at least 1000 for a single category - depending on how big your MobileNet is (there are different variants).

Are you using a pretrained MobileNet, or are you training it from scratch. If Mac'n Cheese was not a category in the original data used for a pre-trained model, then you might need to train from scratch, or only keep the first few layers of the model frozen, as they learn very general features from images that will liekly be useful for your target class (and many other). If you are unsure, search for transfer learning, to see what is possible.

The nominal value of the loss is quite hard to interpret, so values of 0.3 or 1.0 don't mean a lot without context. You can use them for comparison - so if you change something and loss values then start dropping to 0.005, you know that you are likely onto a good idea.

You can check out this tutorial notebook on MobileNet, which might give you more ideas about how to play with the network


Can there be in-active neuron in output layer

Yes, it is pretty much possible that you can have an in-active neuron in an output layer.
FYI:
Nodes at the input layer called passive nodes
Nodes at hidden layer called active nodes or unobservable outputs
Nodes at output layer called observable outputs


The question requires some preliminary clarification IMHO. The choice of activation and loss function both depend on your task, on the kind of problem you want to solve. Here are some examples:

If you are training a binary classifier you can solve the problem with sigmoid activation + binary crossentropy loss.
If you are training a multi-class classifier with multiple classes, then you need softmax activation + crossentropy loss.
If you are training a regressor you need a proper activation function with MSE or MAE loss, usually. With "proper" I mean linear, in case your output is unbounded, or ReLU in case your output takes only positive values. These are countless examples.

With activation function here I refer to the activation of the output layer. The activation that performs the final prediction scores doesn't have to be (and usually isn't) the same as the one used in hidden layers.
ELU and SELU are typically used for the hidden layers of a Neural Network, I personally never heard of an application of ELU or SELU for final outputs.
Both choices of final activation and loss function depend on the task, this is the only criterion to follow to implement a good Neural Network.


Metrics for Evaluating Performance of Logistic Regression

Since Logistic regression is not same as Linear regression , predicting just accuracy will mislead.
** Confusion Matrix** is one way to evaluate the performance of your model. Checking the values of True Positives, False Negatives ( Type II Error) are really important.

** ROC Curve** Receiver Operating Characteristic(ROC) summarizes the model’s performance by evaluating the trade offs between true positive rate (sensitivity) and false positive rate(1- specificity)

The below link will give you more information:

https://www.analyticsvidhya.com/blog/2015/11/beginners-guide-on-logistic-regression-in-r/


How to apply SelectKBest in azure machine learning

I have found there are other feature selection algorithms there

Please see the below screen shots


Is there an R tutorial of using LSTM for multivariate time series forecasting?

I like the Jena data example. In multivariate settings, you only need to generate lookbacks over all X.

https://blogs.rstudio.com/tensorflow/posts/2017-12-20-time-series-forecasting-with-recurrent-neural-networks/


What is the difference between Linear SVM and SVM with linear kernel?

As you can read in the documentation (http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html) it's conceptually similar, but implemented in another way to add flexibility:

  Similar to SVC with parameter kernel=’linear’, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples.

So if you only want to use linear support vectors use this one because it scales better and you get more freedom with your loss functions, but if you want to more easily try different kernels in a grid search use the more generic one.


ValueError: Error when checking input: expected dense_36_input to have shape (None, 12) but got array with shape (140, 2)

@Angel
Input dimetion means how many no of feature or columns your input is supposed to have.
The neural network will try to adjust to this input shape.
In your case you have specified input_dim = 12 but in your data (140,2) there are only 2 features or columns.
So there are two options

Set input_dim =2

Reshape input array to in multiples of specified input dim i..e 12.

In your case ONLY option 1 works because (140,2) cannot be reshaped in multiples of 12.
Note:: other valid input_dim for you can be 14 (reshape input to (20,14)), 10 (reshape input to (28,10))


Is there a deep learning method for 3D labels?

Since the probability are summing up to zero, so you can simply treat it as Multi-class problem and use a network with Softmax at the end.
Last layer and compile -

model.add(keras.layers.Dense( 3, activation="softmax"))
model.compile( optimizer='adam, loss="categorical_crossentropy", metrics='accuracy')

Metrics - Accuracy is not appropriate. Define a custom metrics based on the interpretation of 3 probabilities
The labels will be as per the probability-
e.g. This is for MNIST 10 digits -

Digit 1 - [0.05, 0.55, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05]

Prediction - [0.064, 0.356, 0.059, 0.069, 0.068, 0.050, 0.044, 0.122, 0.064, 0.101]

Code for MNIST - Colab link


Deep learning error: All layer names should be unique

The problem is, you are trying to instantiate two VGG16 models at the same time and its confusing for the kernel to figure out which graph it needs to use. Atleast that is the speculation I have, because of the problems faced when trying to load multiple models in keras with tensorflow backend, its not straight forward.

Try commenting the code from when you are instantiating the second model and this error should go away.


Strategies for handling unlabeled data which is slightly different from the labeled data

There is no straightforward way to deal with this scenario, but here are some ideas:

Augment the A/B/C label set with an extra label "U" for "unknown". Now all samples are labeled.
Predict A/B/C labels from the unlabeled samples from the other pieces of data in the sample. Now all samples are labeled.
Train a system for data with labels and another system for data without them. Use them according to the availability of the label. It would be great if your ML approach allows for sharing parts of the model for labeled data and the one for unlabeled data (e.g. neural networks).


Aggregate several attributes into one using PCA

I see. Yes PCA is a good tool that you can use on your choice of attributes. The tricky part is that PCA is based on variance. Hence, if you have 2 attributes which are more important and have low variance and then 5 more with a lot of variances then the latter will be predominant on the first PC (I guess some standardisation would help). Hence, maybe it makes more sense to incorporate some corresponding weights on each attribute, a weighted average for instance (dummy example, but may fit the cause).


making a picture of a neural network in tensorflow

This should do the trick:
tf.keras.utils.plot_model(
        model, to_file='model.png', show_shapes=False, show_layer_names=True,
        rankdir='TB', expand_nested=False, dpi=96
    )

This will generate an image like:

Example: https://www.tensorflow.org/guide/keras/functional


Why does a simpler model performs better than a complicated one?

High variance means that your model's performance varies from data to data, which is bad for the model.
For example:
You used a polynomial classifier with high degree, so it will overfit your training data resulting in good accuracy, but when tried on a different dataset it will yield less accuracy , resulting in variance.

The image on the right is way too complex which results in overfitting.

The middle image shows a simpler model than the image on right, but fits the data better ie. generalize the data better


Multiple XGBoost models or just 1 for a cetain type of category?

This is the main advantage of ML: if the variable has any predictive value (that is not included in another variable), it should be used by the model. So, generally speaking it doesn't really make sense to handpick your variables to make different versions of your model (however it make sense to handpick some you want to throw). That would just be equivalent to picking the first binary split in your first tree... you are not achieving much with that.
Edit: ok, it seems that the target is not really well defined as you aggregate things that are monthly payments and things that are the house value. In that case it make sense to have two models. (Honestly it would even make more sense not to aggregate those two distinct dataset in the first place).


Pyspark: Filter dataframe based on separate specific conditions

You can use the filter method on Spark's DataFrame API:
df_filtered = df.filter("df.col1 = F").collect()
which also supports regex
pattern = r"[a-zA-Z0-9]+"
df_filtered_regex = df.filter([df_filtered.c.rlike(pattern) for c in df.columns]).collect()`


First for RNN part, you should consider to read documentation about input_shape and batch_input_shape.

Then your Activation layer seems to be useless because you already apply relu activation in RNN layer.

Finally, your network returns output's shape of (20,) (RNN return (20,) and activation don't change shape) and you compare this with y_train with shape of (1,). Shapes dismatch so it return this error

EDIT: You can print model.summary() to monitor input/output 's shape


nltk.corpus for data science related words?

I have a way through which you can solve your problem. For it you will require a,

A pretrained embedding generator. It can be Word2Vec or GloVe. Any of them could work.

  Next, we have a corpus of words which have higher frequencies. Suppose
  we have a set of 100 such words where the 1st word has the highest
  frequency.

Now, we convert every word in this set to a vector using our pretrained word embedding.
Hence you will have a set of vectors for the words from the corpus. Let's call it $z_i$

We have the word "data science". Get a vector for this too. Let's call it $x$

Measure the euclidean distance between the vector $x$ and $z_i$.
Or, you can measure cosine similarity betwee $x$ and $z_i$.
Both of above methods will produce a set of values which would show proximity of $x$ with values of $z_i$.
From these 100 values, we get the least 10 values and convert them to words again.

These 10 words would have the highest similarity with the word "data science".


Dealing with NaN (missing) values for Logistic Regression- Best practices?

Use caution when removing features with missing values. Sometimes the fact that a feature has missing values is valuable data in and of itself.

What you are asking about is called imputation. A google search will give you a TON of literature about various imputation methods. Here are some of the most common:

-Mean imputation: replace all missing values with the mean of other values

-Mode imputation: same as above, with the mode

-Median imputation: get the idea? Replace all with the median

-Multiple Imputation by Chained Equations (MICE): basically do linear regression to predict the missing values based on other variables. This is a high-variance solution so some domain knowledge may be necessary. This can also take a long time. There are packages available to do this in R and Python.

-Replace missing values with 0 or 1 (looks like you have already tried this)

Play around with the different methods and see which ones give you the best result!


Is it bad practice to use multi-class over multi-label classification?

Perhaps I understood the question wrong, but multi-label and multi-class problems are fundamentally different, as the multi-class is exclusive (no instance can be classified into two classes a the same time), while the multi-label relaxes this condition. As Wikipedia on multi-label states:

  in the multi-label problem there is no constraint on how many of the classes the instance can be assigned to.

So, using multi-class or multi-label depends on the problem itself and not on how you solve it.


The best way to understand that part is by looking at figure 1 in the AlphaGo Zero paper.

The neural network (NN) minimizes the differences between its own policy $p_t$ and the MCTS policy $\pi_t$. The value of $\pi_t$ is produced by the MCTS self-play which in return uses the NN from the previous iteration.

The same goes for $v_t$ and $z$. In each iteration the weights of the NN are adjusted to minimize the distance between $v_t$ (output of the NN) and $z$ (output of the MCTS) as defined by the loss function. $z$ does not have a time index here as the full self-play produces just a single value for $z$ each time it is conducted.

TLDR for your first question: Both, $\pi$ and $v$, are being produced by the MCTS as input to the NN.

(The indexing in the paper is a bit confusing in my opinion so it is probably easiest to just look at it as stated above)

Now, with "input" I do not mean input on the input layer of the NN. As described in the appendix under "Neural network architecture" the input is a "19 x 19 x 17 image stack". which contains the following information:

The positions of player 1 for the latest 8 rounds (8 feature planes)
The positions of player 2 for the latest 8 rounds (8 feature planes)
A color feature indicating whose turn it is (1 feature planes)

And those 17 feature planes ($8+8+1$) combined with the $19\cdot19$ sized board is the $19\cdot19\cdot17$ input the NN receives thru its input layer. $\pi$ and $z$ are passed to the NN via the loss function only (i.e. they are the target values in this supervised learning problem!).

TLDR for your second question: $\pi$ and $z$ are not fed to the NN thru the input layer but just via the loss function as target values.


A data set to distinguish man from woman?

You can use deepdetect which provides image classification functionality over API.

There is a gender classification model available on the models page. I have used it without problems before.


How to use Statistical Learning theory in real analysis

the development of the basic calculus , necessary for both formulating problems and numerical techniques for finding the minimum of a function
the theoretical development of theory of learning, such as the VC theory, in the same way it is used in statistics to do things like prove the central limit theorem


Why is the F-measure preferred for classification tasks?

The Fı-score is preferred to simple classification accuracy in order to counter the problem of imbalanced datasets; if the thing you are looking for occurs only rarely anyway then a naive classifier can always say no and appear to be working very well! A variant on Fı is Fß, where

Fß = (1+ß²) × [ (P × R) ÷ ( (ß² × P) + R ) ]

Vary ß to balance precision and recall. As to the why F or G, I believe it to be empirical - you don't say if you are classifying or clustering in your own application?


How to structure my data into features and targets for PCA on Big Data?

First of all, if the "weight" etc. attributes should not be used for clustering, then don't include them. It makes you question hard to follow. On the other hand, if you don't include them  there is obviously no difference between the kids... So I'm not certain you have understood what you want, need to do, or ask for.

PCA is not a magic tool that makes you data clusterable! It is a very specific tool to decorrelate data that should only be used when you have strong, known, linear correlations in your data that you need to ignore. It will not work when you have non-linear correlations (e.g., income and number of yachts), or when your data is not continuous on a linear scale. In particular you shouldn't use it on your first attributes such as date, state, etc.

What you first will want to do - and that has nothing to do with PCA nor clustering, is to pivot your data to separte the kids. Consider transforming your messy table into "tidy data" form first, this will make such cleanup much more structured and easier to do.


Applying a keras model working with greyscale images to RGB images

Your model is not sufficiently complex to adequately classify the CIFAR 10 data set. CIFAR-10 is considerably more complex than the  Fashion-MNIST data set and therefore you need a more complex model.You can add more hidden layers to your model to achieve this. You should also add DROPOUT layers to prevent over fitting. Perhaps the easiest solution is to use transfer learning. I would recommend using the MobileNet CNN model if you want to try transfer learning. Documentation for that can be found here. Since CIFAR-10 has 50,000 sample images I do not think you will need data augmentation. First try a more complex model without augmentation and see what accuracy you achieve. If it is not adequate then use the keras ImageData Generator to provide data augmentation. Documentation for that is here.


That's an interesting problem, thanks for bring out here on stack.

I think this problem is similar to when we apply LSA(Latent Semantic Analysis) in sentiment analysis to find list of positive and negative words with polarity with respect to some predefined positive and negative words.

Good reads:

Learning Word Vectors for Sentiment Analysis
Unsupervised Learning of Semantic Orientation from a Hundred-Billion-Word Corpus

So, according to me LSA is your best approach to begin with in this situation as it learns the underlying relation between the words from the corpus and probably that's what you are looking for.


Approach fpr extracting/cropping features images using deeplearning and no annotations

So all you have is a set of images containing only cropped hats?
One idea is to leverage synthetic data to learn an object detector.
It would be better if you had crops of natural images with hats, not just hats alone (as it will lessen the domain shift).

Basically, the following: (1) take your hat images and generate "ground truth" images by randomly pasting a random number of hats (in random poses) into a random background image (from anywhere, ideally similar to the backgrounds of images that you plan to run on at test time). (2) Train an object detector model (e.g. Mask-RCNN, Faster R-CNN, or YOLO9000) using your synthetic dataset (since you know the "real" hat positions in the images, since you made them). (3) given a video, decompose it into a sequence of image frames, and run your trained detector frame-by-frame. (4) use the output of the detector to obtain the hat positions and crop them out.

A simple extension would be to generate synthetic videos for training, where your hats move around and you keep track of them; another one might be to try to make the synthetic images more realistic, by say attaching random people or at least heads underneath the hats.

The main challenge is the domain adaptation between synthetic and real images. Any of the many modern techniques for handling domain shift would likely be helpful. For instance, using GANs (also here).

It may interest you to know that this simple synthetic methodology has proven useful in practice. For instance, Tremblay et al at NVIDIA recently used it to also train an object detector via domain randomization. Other applications include robotics.

A much more trivial, but easier to implement solution is to train a hat classifier $C$, which takes in an image or image patch, and output whether the patch contains a hat or not. Using your cropped hat dataset as the true positives, and getting true negatives by randomly cropping patches from any random dataset of images, you can train $C$. Given a video, you then simply use a sliding window approach, where for every image patch at every frame, you run $C$ on it. You can then threshold the output of $C$ and/or crop out areas with high values from the classifier.


Neural network recommendations if only few features

There's all sorts of rules of thumb for the structure of a neural net (if n features use n+1 or 2n or 1.5n nodes, etc). I wouldn't take any of them as gospel.

The size and structure of your neural network chiefly depends on the complexity of the data you're trying to learn. If your data is already pretty well represented by, say, linear regression, then you don't really need much additional size to your neural network to represent it well. On the other hand, if your five inputs were, say, the move and shoot commands of a recording of a game of Doom, you're gonna need a large neural network to accommodate all the deep and meaningful interactions between the individual commands.

Each successive layer of nodes of a neural network deals with increasingly complicated shapes. The first hidden layer is all sigmoid or ReLU functions, the second layer is  combinations of the first layer, the third is combinations of the second layer, etc.

The general structure of the hidden layers is cone-shaped, tapering towards the right. There are lots of nodes in the first layer, and fewer in each successive layer. You can have constant numbers of nodes per layer, though in my playing around it doesn't seem to help much, and many of the layer nodes go relatively unused. The opposite direction (having few first layer nodes and many last layer nodes) is actively harmful, because the later nodes can only ever be combinations of the earlier nodes, and if the first nodes don't create enough shapes for the later nodes to combine, it won't be able to fit anything.

I've provided a couple of interactive examples through tensorflow's playground.

Normal structure (Decreasing nodes per layer)

Same no. of nodes per layer

Rising no. of nodes per layer

The first two should converge relatively easily, whereas the third shouldn't converge at all.


Which algorithms should I use for recommendation system using a graph database?

A biadjacency matrix of a bipartite graph admits matrix factorisation.

For $m$ items and $n$ users, the biadjacency matrix is an $m \times n$ matrix which can be factorised into two lower-rank factor matrices of sizes $m \times k$ and $k \times n$ respectively. This provides a lower-dimensional representation of how users' preferences vary over products.

This model also has the property that the inner product of the $i$th row and $j$th column of the two factor matrices provide an approximation of user $j$'s rating of item $i$. In other words, the product of the two matrices is an approximation of the original ratings matrix.


How to obtain unique count of categorical variable based on another categorical variable?

Try nunique(). That should do it. Here is a toy example:

import pandas as pd

df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
                          'foo', 'bar', 'foo', 'foo'],
                   'B' : ['one', 'one', 'two', 'three',
                          'two', 'two', 'one', 'two']})

print(df)

gives

     A      B
0  foo    one
1  bar    one
2  foo    two
3  bar  three
4  foo    two
5  bar    two
6  foo    one
7  foo    two

and

print(df.groupby('A')['B'].nunique())

then gives

A
bar    3
foo    2
Name: B, dtype: int64

For more details check this question on SO.


How to represent genre or artist name in a neural network

You can represent this kind of discrete information by means of embeddings.
An embedding is simply a table of vectors. It is defined by the number of vectors in the table and the length of each vector (i.e. dimensionality). You need to define a priori all the elements you want to support, e.g. for genres, you should list all the genres you will support, and list them. The index of each item is how you represent it.
You can have an embedding for each of the discrete features you need (genre, artist, etc).
In Keras, there is a layer for Embeddings.


Appropriate Machine Learning algorithm for modeling clustered time-varying binary outcome

At first, choosing the best model is probably not as important as getting the data-set setup properly. Most of data science and modelling is spent on building the best data-set to use. Random Forest Classification, SVM, XGBClassifier, even Logistic regression can perform adequately if given the properly prepared data-set to analyze. Also, the metric you are comparing you model to should be appropriate for imbalanced data. Just a FYI, accuracy is not one of them. Here are just a few suggestions to get you started:
Data preparation/cleansing:

you can try more methods to balance, such as down-sampling (think
this is usually better than up-sampling as the data you are using
was real data, not synthetic). SMOTE is okay also, but can be a bit
more complicated to use properly
you can try using a weighting function in your model to give more
importance to the minority value, if you think a mistake in
predicting that value is more important or more costly than a
mistake in the majority, or just want to balance your predictions
evenly between the classes
You can use PCA, SVD or LDA to remove some of the correlations
between your features
Normalize the data as some algorithms still don't handle large
differences in features very well (ex: feature one mean is .001 vs
feature 2 mean is 1000). This is a good practice in any modelling.
Look for fliers. Sometimes fliers can cause problems. Look for
values that don't make sense, or are too far out of normal range.

Metric to use:

I usually use ROC_AUC to find which model is best at predicting both
majority and minority classes.
Sensitivity and Specificity are also 2 good metrics to use to see
how good your model is at predicting each class


Knn distance plot for determining eps of DBSCAN

You

take the last column of that matrix
sort descending
plot index, distance
hope to see a knee (if the distance does not work well. there might be none)


Hindsight experience replay: strategy for sampling goals

I’ve been there for quite a while until i read the implementation of thr algorithm in OpenAI Baselines library, you can find it on Github.
Basically as of what i understood from the algorithm the K factor mainly adjusts the percentage of sampled data (being replayed for the update step in Q weights) that if we sample 1 transition without HER we sample another K with HER. This can be implemenated in the prescribed fashion if we sampled a batch of data and replayed K/1+k with HER by simply changing their goals by other states observed at any time in the future


Product Prediction to group of customers

This is an interesting problem statement. If we have products purchased by all the users, We can try the following

Perform Collaborative filtering for each metric e.g. ALL_FEES, ASSETS
While performing Collaborative Filtering, we use userGroups instead of users
For arriving at the top product recommendations for each user group, we choose products that have been recommended in most of the collaborative filtering metric model outputs


We are using Singular Value Decomposition in the much same manner as you, except rather than clustering similar items, we are using a reduced-rank matrix to power a recommendation engine based on a term-document matrix in Latent Semantic Indexing.

From your brief description, your approach seems sound enough. However, I highly recommend reading Berry, Dumais & O'Brien's Using Linear Algebra for Intelligent Information Retrieval.

Key to using SVD is selecting an acceptable rank-k approximation to the original sparse matrix. You should carry out some exploratory analysis to see how much variance can be explained, using the singular values in the diagonal matrix Sigma. This question was brought up in this question on CrossValidated.

A lot of the papers I've read suggest anywhere a rank k from 200 to 300 singular values. In a proof-of-concept implementation, we had original sparse matrix of about 10000 rows (unique terms) to about 1000 columns (unique documents), and we were capturing just under 85% of the variance with only 300 singular values.

However, that really hinges upon the nature of your data, so your mileage may vary.


Is it possible to implement a classifier according to quarters? What about missing data?

Would that be a good idea?

That is hard to tell from your description. It is not an immediately bad idea. If it results in a better classifier (according to cross-validation), then it has probably worked.

The main things that would concern me about splitting behaviour data by quarter and treating as independent are:

Your data samples will very likely be correlated when they share a person. You can work around this by careful splitting between training and cross-validation / test sets. Do not make a fully random split, but split by person - any individuals records should appear only in one of the training, cross-validation or test sets (assuming your goal is to take similar data in production from users who are not in your current database, and predict their class).
There could be seasonal variation in the records that reduce the effectiveness of the split. So a "type 1" person's records in APR-JUN might look like a "type 0" person from JAN-MAR.
How will you receive data in production - when you want to classify new users? If you only want to work on single-quarter data, then your new classifier is fine. If you have more data, you have to deal with your classifier maybe predicting different target variable for the same person depending on the quarter. You could combine these in some way - but if you do so, you should also do this in test to see what the impact of doing this is, which may be counter-productive (you end up with the same number of test examples as if you had not done the split). It might also be OK, perhaps it will add some regularisation.

  Do I have to be cautious of a particular hypothesis that could impact my results?

You have to be very cautious about testing your classifier, because you could get data leakage from the cross-validation and test sets to the training set, which would make you think the classifier is generalising well when in fact it is not. The fix for this described above - split by person when deciding train/cv/test split.

  I have some data available for some quarters of the year but sometimes it is missing (for instance let's imagine I don't have "Age" available for my last quarter) ; do I have to drop the feature ?

Handling missing data is a complicated topic in its own right, there are lots of options. You can start with:

If data is missing at random (i.e. there is no reason to suspect it is related to the target variable, or only impacts certain types of record), you can substitute the mean value of that feature from the training set, or impute it based on a statistical model from the other features.
If data is missing for reasons that might impact the target variable, then you should give that information to the classifier, because it might be an important feature in its own right. You can take the mean or more complex imputed value as before for the original feature, but also you should add a new boolean feature "feature X was missing".

Whether or not you should use the partial data or drop it is not possible to say in general. If you are not sure, then try both and pick the version with the best cross-validation result.


Variance as criteria for feature selection

You are correct that variance depends on the scale and typically it is not a good proxy for how informative a feature will be in terms of the response. The exception is zero variance features. A feature with zero variance has the exact same value for all observations and is therefore completely uninformative. Sklearns VarianceThreshold function defaults to removing only the features with exactly zero variance.

Another group of non-informative features is the near-zero-variance feature. However, typically, classifying a feature as a near-zero-variance feature does not actually rely on computing the feature's variance at all. See, for example, the NearZeroVar function documentation from R's caret package:

  "nearZeroVar diagnoses predictors that have one unique value (i.e. are zero variance predictors) or predictors that have both of the following characteristics: they have very few unique values relative to the number of samples and the ratio of the frequency of the most common value to the frequency of the second most common value is large."

You can see that the algorithm is not scale-dependent and does not rely on variance.


Without more information all I can say is that:

the say you're storing it is fine in general
you can further transform/store your data depending on your use case

To expand on #2, if I want to study Distance vs Energy across all subjects, then I would format my data like this:

> library(reshape2)
> dt2 <- dt[dt$measurement %in% c('Distance','Energy'),]
> dt_cast <- dcast(dt2, subject+Try~measurement+speed+condition, value.var='result')

The transformed data (dt_cast) would then look like:

  subject  Try Distance_speed1_Control Distance_speed1_Experm Distance_speed2_Control
1       1 Try1                       1                      3                       5
2       1 Try2                       2                      4                       6
  Distance_speed2_Experm Energy_speed1_Control Energy_speed1_Experm Energy_speed2_Control
1                      7                    21                   23                    25
2                      8                    22                   24                    26
  Energy_speed2_Experm
1                   27
2                   28

Allowing me to, for example, look at the relationship between the Distance_speed1_Control vs Energy_speed1_Control columns.

Basically subset/aggregate your data and then use the dcast to get the rows and columns the computer needs.


How would I load a database with multiple inconsistent headers?

$ cat one.csv
    site_no,datetime,00060_00003
    11481500,2019-10-05,7.54
$ cat two.csv
    site_no,datetime,00010_00001,00010_00002,00010_00003,00060_00003
   11523000,2019-10-05,15.0,14.1,14.6,1920

code:

df2 = pd.read_csv('two.csv')
df = pd.read_csv('one.csv')

pd.concat([df,df2], join='outer', join_axes=[df2.columns])

output:

   00010_00001  00010_00002  00010_00003  00060_00003  00060_00003        datetime   site_no
0          NaN          NaN          NaN          NaN             7.54  2019-10-05  11481500
0         15.0         14.1         14.6       1920.0              NaN  2019-10-05  11523000

the pd.join or pd.concat works on entire data frame, you don't need to loop through the variables.

using merge: (from the documentation)

t = pd.DataFrame({'key':['foo','foo'], 'lval':[1,2]})
>>> right = pd.DataFrame({'key':['foo', 'foo'], 'rval':[4,5]})
>>> left
   key  lval
0  foo     1
1  foo     2
>>> right
   key  rval
0  foo     4
1  foo     5
>>> pd.merge(left, right, on='key')
   key  lval  rval
0  foo     1     4
1  foo     1     5
2  foo     2     4
3  foo     2     5


Let me give you a few simple approaches in time series analysis.

The first approach consists in using previous values of your time series $Y_{t}$ as in $Y_{t} = \phi_{1}Y_{t-1} + ... + \phi_{n}Y_{t-n}$. In case you don't know, these models are called autoregressive (AR) models. This answers your first question. Of course it is useful to include the previous value of your time series. There is a whole set of models based on that idea.

The second approach is taking a window and extracting some features to describe the time series at each point in time. Then you use a conventional machine learning technique to predict future values as typically done. This is more common in a classification or regression setting but future values can be thought of as classifying future values. This technique has the advantage of dramatically reducing the number of features, although you usually lose characteristics associated with time. This addresses your second concern.

Another model that could be helpful in your case is the vector autoregressive model (VAR) (using Wikipedia's notation):

$$\left( \begin{array}{ccc}
y_{1,t} \\
y_{2,t}
\end{array}\right) = \left( \begin{array}{ccc}c_{1} \\ c_{2}\end{array}\right) + \left( \begin{array}{ccc}A_{1,1} & A_{1,2} \\ A_{2,1} & A_{2,2}\end{array}\right)\left( \begin{array}{ccc}
y_{1,t-1} \\
y_{2,t-1}
\end{array}\right) + \left( \begin{array}{ccc}
e_{1,t} \\
e_{2,t}
\end{array}\right)$$

Here you can see that $y_{1,t}$ has a contribution from its previous value $t_{1,t-1}$ but also includes the value of the other series $y_{2,t-1}$ in a linear combination. As usual, the purpose is to find the elements of $A_{i,j}$ that minimize some measure of error between observed values and estimated values.

A general suggestion: The first thing you need to do is to test the autocorrelation of your first series in order to confirm that an autoregressive approach is suitable and then test the cross correlation between both series to support the idea that using the second series to improve your predictions is appropriate.


Understanding action space in stable baselines

I guess I got answer to Q1. Sharing my understanding below:

PPO extends OnPolicyAlgorithm
PPO calls super(..,env,..), that is OnPolicyAlgorithm(..,env,..)
OnPolicyAlgorithm extends BaseAlgorithm
OnPolicyAlgorithm calls super(..,env,..), that is BaseAlgorithm(..,env,..)
BaseAlgorithm seems to obtain action_space from env

Still I am unable to get answer Q2.


In neural networks model, which number of hidden units to select?

Unfrtunately not, there is not theory to tell us what is the right number of units to choose. As there is not theory for the number of hidden layers to choose. On this respect, Deep Learning is still more an art than a science.

It's true that with one hidden layer we could theoretically solve any problem, but most of the problems are complicated enough to require unimaginable amounts computation.

I think in the end it all boils down to two main issues:

Your specific task, i.e. how large and deep a Network must be so that your model works as you need.
The compute power at your disposal.

When it comes to this, I strongly suggest to prioritize depth over width. Deeper Networks (the ones with many hidden layers) are much more efficient and powerful than large Networks (the ones with fewer, larger layers). It seems they are better at producing abstractions on the input data, to transform and process the signal in more sophisticated ways.