How can I read in a .csv file with special characters in it in pandas?

There is df.drop command that can be used as follows to remove certain rows (in this case, 15 & 16):

df.drop(df.index[[15,16]])

If the rows you don't need are regular (e.g. you never need row 15) then this is a quick and dirty solution.

If you only want to drop arbitrary rows containing some value, this should do the trick:

df = df.drop([df.column_name == ©1990-2016 AAR])


I will assume by C1, C2, etc, you mean convolutional layers, and by P1 ,P2 you mean pooling layers, and FC means fully connected layers.
We can calculate the memory required for a forward pass like this:
One image
If you're working with float32 values, then following the link provided above by @Alexandru Burlacu you have:
Input: 50x50x3 = 7,500 = 7.5K
C1: 50x50x32 = 80,000 = 80K
P1: 25x25x32 = 20,000 = 20K
C2: 25x25x64 = 40,000 = 40K
P2: 12x12x64 = 9,216 = 9.2K <- This is a problem (and my approximation is a very hand-wavy guess here). Instead of working with 50, 25, '12.5', it would make more sense to work with multiples of 32. I've heard working with multiples of 32 is also more efficient from a memory standpoint. The reason this is a bad idea is 2x2 pooling doesn't divide the space properly, as far as I can tell. Feel free to correct me if I'm wrong.
FC: 1x500 = 500 = 0.5K
Output: 1 x 10 = 10 = 0.01K (next to nothing)
Total memory: 7.5K + 80K + 20K + 40K + 0.5K = 157.2K * 4 bytes = 628.8 KB
That's for one image.
Minibatch
If you're working with a minibatch size of 64, then you're reading 64 of these into memory at once and performing the operations all together, scaling everything up like this:
Input: 64x50x50x3 = 480,000 = 480K = 0.48M
C1: 64x50x50x32 = 5,120,000 = 5.12M
P1: 64x25x25x32 = 1,280,000 = 1.28M
C2: 64x25x25x64 = 2,560,000 = 2.56M
P2: 64x12x12x64 = 589,824 = 590K = 0.59M
FC: 64x500 = 32,000 = 32K = 0.032M
Output: 1x10x64 = 640 = 0.64K = 0.00064M (we don't care, this is tiny)
Total memory: 10M x 4 bytes ~ 40MB (I'm saying approximate because the website also says an approximate value)
EDIT: I misread the website, sorry.
According to the website, a backward pass requires about triple this, because of the need to store:

the activations and associated gradients for each neuron - these are of equal size;

the gradients of the weights (parameters) which are the same size as the parameters;

the value of the momentum, if you're using it;

some kind of miscellaneous memory (I don't understand this part)


Finally solved the issue.
You should go to:

Data –> your_data_source –> Edit Aliases –> Measure Names

And see full mapping between variables/aggregations and aliases.


Plotting Stacked Histogram for Time-series data

This solution uses plotnine which is based on R ggplot and the grammar of graphics.

Generally you have to transform the timestamp to months first and then understand that:

X = Months
Y = itemcount
Group = item

This should help also for your matplotlib solution!

from datetime import datetime
import pandas as pd
from plotnine import *

df['month'] = pd.DatetimeIndex(df['timestamp']).month

p = (ggplot(pdf, aes(x = 'month', y = 'itemcount',fill='item')) + geom_bar( Star = 'identity',position='fill')

# fill here acts the same as group in matplotlib
```


Dueling DQN - Advantage Stream, why use average and not the tanh?

The goal of these models is to estimate the value of each action choice. They chose to use the average function to estimate the update value because the average function produces a single scalar influenced by each value.

Tanh function is not an appropriate nonlinear activation function. The tanh function only takes a scalar as an input, thus would not weight each value.

The authors tried an appropriate nonlinear activation function for many values - softmax:

  We also experimented with a softmax version of equation (8), but found it to deliver similar results to the simpler module of equation (9).


Why is performance worse when my time-series data is not shuffled prior to a train/test split vs. when it is shuffled prior to the split?

Without looking further into the data myself, I can surmise that something has changed recently with your data such that if you split without shuffling, some aspect of the data in your test set (which is what you most recently collected) is underrepresented in your training set. By shuffling the data, you allow those more recent samples to also be present in your training set, and thus your test set performance improves.


How should I tackle this real-life hypermarket problem?

Question

I feel that the question needs to be reformulated for clarity to:

  Given a fixed set of purchased products $S=\{s_i\}$ with associated prices $p_i$, find the greatest number of disjoint subsets $S_1, S_2, etc$ such that the number of reward points attained in each subset equals the number of points attained for the whole.

  A reward point is obtained for every whole number of 200 cents divisible into the total price of the set.

In other words you have a set of products that you will definitely buy then the maximal number of points you can acquire from those purchases is $floor(\frac{1}{200}\sum_i p_i)$. Since you cannot acquire more points than this any subdivisions you make, representing different trips to the hypermarket, need to reach the same amount of points.

Answer

The brute force way of considering every possible combination will be an exponential time algorithm. So you probably need to develop an algorithm which uses greedy approaches.

For example, one such approach would be to identify subsets. Say you had four products:
{milk, cheese, bread, oilves} with prices {189c, 200c, 211c, 205}, possible 4 reward points,
then a preliminary scan (over a single item) to detect prices with zero residuals will return two disjoint sets;
{cheese} and {milk, bread, olives}.
Now you can recursively treat the 3-set as the same problem, in this case no single product has a multiple of 200, but milk and bread does sum to 400 so that is a perfect split:
{cheese} and {milk. bread} and {olives}
And you have arrived at the result.

In general you cannot know how many products might be needed in a combination to make a perfect multiple of 200, but you can use some heuristics derived from your initial set. In this case $\sum_i p_i mod 200 = 805 mod 200 \equiv 5$, so you know that you cannot have the sum of residuals of subgroups totalling more than 5 otherwise your solution returns less reward points.

For example if you grouped {cheese, bread} you get 2 reward points with a residual of 11 cents. There is no combination of the remaining {milk, olives} that can collect 2 more reward points. In that case the residual is 194 cents, which actually suggests the two disjoint sets can be combined to produce an additional reward point since 11 + 194 > 200.

PS

I have no idea why you mention weekdays without giving any information about their significance to the problem. Unless you want me to assume that you only visit the hypermarket once per day in which case the total number of disjoint subsets you can have is 5.


How do I get my Neural network to ignore certain values?

CNNs work because of those zeros (the zeros create the boundary on which the change is values is the highest i.e. what networks learn). They are not the problem. Look at your regularizing your network (use dropout, reduce filters if overfitting or increase if underfitting).


Classifying ultrasound videos with a small dataset

One option in similar problems is to use transfer learning on raw images. For similar problems (few hundred images for training), solution that worked for me is transfer learning + image pre-processing to generate augmented images (E.g.: Stretched / Rotated versions of images).

High level design is :

Use a model that has been trained on large number of images (millions). This model has already learnt image processing as a domain (such as edge detection, phase / color changes and so on) and might remove need to feature engineering (like HOGS)
Add few more layers to be trained for Ultrasound images
Train these new layers with images (Existing layers are frozen since they are already trained)

Examples of this approach are :

https://www.researchgate.net/publication/311094437_Transfer_Learning_with_Convolutional_Neural_Networks_for_Classification_of_Abdominal_Ultrasound_Images

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5359213/

https://www.sciencedirect.com/science/article/pii/S2095809918301887


Data augmentation based on the class type in the CNN model

Try under/ or oversampling and aditionally use data augmentation for your input data.


RL Invertible Value Function approach - why it prevents rewards from exploding?

The answer of your first question can be answered by considering when you do not step forward and $n = 0$. So what happend in your case? $y = h(Q(S_t, a^*))$. However, $y= Q(S_t, a^*)$ could be more reasonable. Hence, we need that $h^{-1}$ to have a more explainable reward in the edge case.

One answer to the second question could back to the differentiation of the $h$ function. If the $\epsilon x$ is removed, the $\frac{dh(x)}{dx}$ goes to zero when $x$ grows. But, by the presence of $\epsilon$, we will have $\frac{dh(x)}{dx} \to \epsilon$ for big enough values of $x$. Hence, the reward function will not cap for the big values of $x$ and grows increasingly.


Try specifying mode so that PIL is aware of data format.

img = Image.fromarray(source_array, mode="CMYK")

If that does not work, what is the shape of source array ?


How to determine feature importance in a neural network?

Don't remove a feature to find out its importance, but instead randomize or shuffle it.

Run the training 10 times, randomize a different feature column each time and then compare the performance. There is no need to tune hyper-parameters when done this way.

Here's the theory behind my suggestion: feature importance


AUC and classification report in Logistic regression in python

In order to calculate the AUC, you need to have probabilities. Therefore you should use the following function:

roc=roc_auc_score(y_test, model.predict_proba(X_test)[:,1])

This will give you the probability for each sample in X_test having label 1.


Reinforcement learning: decreasing loss without increasing reward

How should I interpret this? If a lower loss means more accurate predictions of value, naively I would have expected the agent to take more high-reward actions.

A lower loss means more accurate predictions of value for the current policy (technically it is more complicated for Q-learning off-policy estimates, but the covergence will still be limited by experience reachable in the current policy). Unfortunately a loss metric in RL cannot capture how good that policy is.

So what it means is that your policy has settled into a pattern where values can be estimated well by the neural network that you are using for Q. For some reason it is not finding improvements to that policy - typically it should be doing that before the loss metric drops, as each improvement in value estimates should reveal better possible actions, and once those start being taken by a new policy, the value estimates become out of date, and the loss increases again.

  Could this be a sign of the agent not having explored enough, of being stuck in a local minimum?

Exploration could be an issue. The "local minimum" in that case is probably not an issue with the neural network, but that small variations in policy are all worse than the current policy. As you are learning off-policy, then increasing the exploration rate may help find the better states, at the expense of slower overall learning. Also, methods that explore more widely than randomly on each action could be better - e.g. action selection methods that consistently pick unexplored state/action pairs such as Upper Confidence Bound.

Also a possibility is that the structure of your network generalises well under the current policy, but is not able to cover better policies. In that case, whenever the exploration suggests a better policy, the network will also increase estimates of unrelated action choices - so it would try them, notice they are better, then back off as the new values also cause unwanted policy changes in other situations.

If you know a better policy than the one that is being found, then you could plot a learning curve with the policy fixed, see if the network can learn it. However, usually you will not know this, so you may be stuck with trying some variations of neural network architecture or other hyperparameters.

There are other methods than DQN (e.g. A3C, DDPG), as well as many add-ons and adjustments to DQN that you could try (e.g. eligibility traces, double learning).


How Adaboost calculates error for each weak learner in training?

When I started learning about ensemble methods, this youtube video help me greatly.
The idea is that you fit the model on the data, calculate the error and then work with the error. So yes you should use a weak learner to predict your data and then, use another weak learner.
There are some more videos in the same channel that give a great explanation to have an intuition.


How to improve the binary classification model for text (News Articles) of Recurrent Neural Net with word emmbeding?

I think you should be careful as to which algorithms you tend to use.

A machine learning algorithm should be structured as follows: feature extraction and then your model. These are two things that should be done separately.

Feature Extraction

This is the bag of words, n_grams and word2vec. These are all good choices for text examples. I think bag of words is a good choice in your case. However, if this generates a sparse matrix then maybe n_grams can be better. You can test all 3 methods. If you have a vocabulary size of 100,000 then you really need to use some extra feature extraction.

The Model

Theoretically, the more parameters in your model the more data you need to train it sufficiently otherwise you will retain a large amount of bias. This means a high error rate. Neural networks tend to have a very high number of parameters. Thus they require a lot of data to be trained.

But, you have 8000 instances!!! Yes. But, you also have 100,000 features for each instance. This is insufficient even for shallow machine learning models. For a neural network, i usually suggest to follow this very general rule of thumb,

$\#examples = 100 * \#features$.

So you will need a MASSIVE amount of data to properly train your neural network model. Moreover, if you have a skewed dataset then you should expect to be using even more training examples, to show sufficient examples such that the model is capable of distinguishing the two classes.

I would suggest a less intensive model. You should try to use: naive bayes, kernel-SVM or knn for text classification. These methods would do MUCH MUCH better than a neural network. Especially considering you have a skewed dataset!!

My Suggestion

I would start with bag of words and then use kernel-SVM. This should be a good starting point.

A recurrent neural network is 0% recommended for the amount of data you have.


Question about scaling in (R) Principal Components Regression

I agree that if scaling is used on the training data, it should also be used on the test data.
However, from what I see in the pcr documentation there is not scale option in the function. This seems to be confirmed by running the code from the example in the documentation:
pcr: Prinicpal component regression transfer function models
> library(analogue)
> data(ImbrieKipp)
> data(SumSST)
> pcr(ImbrieKipp, SumSST, tranFun = Hellinger, scale = TRUE)

    Principal Component Regression Model

Call:
pcr(x = ImbrieKipp, y = SumSST, tranFun = Hellinger, scale = TRUE)

No. of Components: 27

RMSE (Apparent):
     PC1      PC2      PC3      PC4      PC5      PC6      PC7      PC8      PC9     PC10     PC11
2.381215 1.707588 1.680896 1.679774 1.608903 1.535145 1.507995 1.496939 1.496908 1.432142 1.426444
    PC12     PC13     PC14     PC15     PC16     PC17     PC18     PC19     PC20     PC21     PC22
1.405155 1.391348 1.349172 1.349172 1.315284 1.313187 1.311801 1.291201 1.206484 1.188438 1.187503
    PC23     PC24     PC25     PC26     PC27
1.171215 1.170947 1.170380 1.162497 1.162355
> pcr(ImbrieKipp, SumSST, tranFun = Hellinger, scale = FALSE)

    Principal Component Regression Model

Call:
pcr(x = ImbrieKipp, y = SumSST, tranFun = Hellinger, scale = FALSE)

No. of Components: 27

RMSE (Apparent):
     PC1      PC2      PC3      PC4      PC5      PC6      PC7      PC8      PC9     PC10     PC11
2.381215 1.707588 1.680896 1.679774 1.608903 1.535145 1.507995 1.496939 1.496908 1.432142 1.426444
    PC12     PC13     PC14     PC15     PC16     PC17     PC18     PC19     PC20     PC21     PC22
1.405155 1.391348 1.349172 1.349172 1.315284 1.313187 1.311801 1.291201 1.206484 1.188438 1.187503
    PC23     PC24     PC25     PC26     PC27
1.171215 1.170947 1.170380 1.162497 1.162355

I did check the the data were not standardised to begin with !
So in your case I would simply use scale() on the data before running pcr

Edit to address the point raised in comments. In the pcr function there is no warning or error if you use a completely fictitious variable:
> pcr(ImbrieKipp, SumSST, tranFun = Hellinger, SomethingIjustMadeUp = TRUE)

    Principal Component Regression Model

Call:
pcr(x = ImbrieKipp, y = SumSST, tranFun = Hellinger, SomethingIjustMadeUp = TRUE)

No. of Components: 27

RMSE (Apparent):
     PC1      PC2      PC3      PC4      PC5      PC6      PC7      PC8      PC9     PC10     PC11
2.381215 1.707588 1.680896 1.679774 1.608903 1.535145 1.507995 1.496939 1.496908 1.432142 1.426444
    PC12     PC13     PC14     PC15     PC16     PC17     PC18     PC19     PC20     PC21     PC22
1.405155 1.391348 1.349172 1.349172 1.315284 1.313187 1.311801 1.291201 1.206484 1.188438 1.187503
    PC23     PC24     PC25     PC26     PC27
1.171215 1.170947 1.170380 1.162497 1.162355

No warning or error


Disadvantages of hyperparameter tuning on a random sample of dataset

One of the good practices is to create a split in the dataset for each tuning/ training step of your pipeline. Since you have large datasets, you should have enough data to split the original dataset into multiple subsets and still have a relevant number of rows for each step. As such, as an example, you can divide your dataset in 60% training, 20% hyperparameter tuning and 20% for the test.

It is important to avoid optimizing the hyperparameters with the same data you train on because this can lead to overfitting both tuning steps of your model to the same source of data.

Also, be careful on how you sample the original datasource. When dealing with highly skewed categorical features, random sampling can lead to categories in the test set which are not observed during training, which can cause some models to break. Also, numerical features should have a similar distribution between the training and the test set.


I see that Kobol offers a REST API. Thus, one could create WebDataConnector to fetch the data in Tableau. If that's an overkill, you could write a script in e.g. Python which extracts the data from the REST API and creates a csv file.

REST API
http://support.kobotoolbox.org/en/articles/592398-api-and-rest-services

REST API -> Tableau
https://www.youtube.com/watch?v=1y5LsqQd00s&list=WL&index=2&t=0s


Your reasoning is perfectly correct. Augmentation is just a process, which helps you cover your domain better. You should only pick operators that help you. Abusing augmentation can definitely mess up your model. It's always good idea to print data at those limits, to check yourself. Try also to think, how data will be acquired on production. Albumentations is a nice repo with a lot of available methods.


Evaluating machine learning model with missing features

If less than 20-25% of behavioral data is missing, maybe you could try to impute missing data  using one of the following solutions :

Impute missing behavioral data using some business rule or by training a machine learning model with demographics data as input and behavioral data as output variable.
Impute missing data with feature mean/median.
Impute missing data by picking up random value in the feature distribution (hot-deck).

In case you have more than 20-25% missing data, it will be really hard to impute values. I think in this situation you should consider creating a new model such as :

The new model doesn't use behavioral data anymore.
The new model is based on a different train-val-test split in order to have behavioral data in each dataset.

If you can't create a new model neither impute missing data, I guess hot-deck would be the best option you have to avoid bad performance on unseen data.


train_test_split : stratify can not be recognized?

You need to pass an array containing the class-labels (or whatever the criterion for stratifying is) as an argument to stratify.

In your case, the answer is probably loan['Loan_Status'].values.


You could look at string similarity measures and TFIDF (usually with cosine). If you want a measure which works at both levels of words and sentences, there are more advanced options such as SoftTFIDF.


According to graph theory, wouldn't this make Person B also very influential?

No. You need to think about how to set up your graph first and then go further. In this case your graph is not simple but both weighted and directed thus handling a request of B by A neither reduces A's influence nor improves B's. Because the weights of directed edges are determining the influence.

Is there a better way than just counting number of interactions?

Community Detection or what computer scientists call Graph Clustering. If you do not have to limit yourself to hierarchies, you can extract communities who have the most interaction and share suggestions inside the community.


Pearson correlation method using absolute values and relative values

In general, the correlation coefficient is "invariant to separate changes in location and scale in the two variables". In particular, you can mix relative with absolute values.

However, that only works if you scale the variables globally. You can't scale every individual data point (here on a city level). If this was a county wide election, you could scale the city values by the county population.

But it sounds like your crime rates are on a per city level. In this case you should scale the votes on a city level as well to make them comparable. This will change the correlation coefficient and give a different result than with absolute values. I think using percentages is more intuitive in your case.


How to interpret PR and ROC Curve for an unbalanced test set

For binary-class classification problems and data being highly unbalanced,

Go for AUC and f1 score as metrics.
Plot the confusion matrix.
Split the data into train:valid:test::60:20:20 or 80:10:10, and do Cross-validation and hyperparameter tuning on train and valid sets. Then go for test set.
You could also try bootstrap resampling.


Has Gradient decent any effects on the actual predictions or designed for practical convenience?

will they affect on the actual predictions?

Yes. The type of gradient descent that you use has an impact on how the model is trained. Therefore, it may lead to different predictions as the trained models may be different.

as we know they always attempt to reach a local minimum

Yes, each gradient descent technique has the same goal, which is to optimize a cost function by following its gradient. However, as you point out, gradient descent may end up in different local minima and some of them may be better than others. And each gradient descent technique is more or less prone to finding the global minimum, which may lead to a difference in predictions.

I would also recommend that you experiment with these different approaches and look at how the training progresses. This may help you gain intuition about the impact of each technique. Also, I find this blog to be quite useful to better understand the differences: https://towardsdatascience.com/gradient-descent-algorithm-and-its-variants-10f652806a3


TData science pipelines are designed to manage the end-to-end data lifecycle (e.g., clean data, fit model, and serve model).
CI/CD pipelines are more general purpose software engineering tools around the automation of common tasks (e.g., running a testing suite).
The advantage of using data science pipelines is that the pipelines have already primitives for common data science tasks. The disadvantages are that the tools are often immature and require a user to adhere to the tools established workflow.
In general, both systems do not optimize for compute or memory. They are just code runners. The code or platform would have to optimize for compute or memory.
These system can sometimes parallelize the parts of the code that is easy to parallelize. For example, serving models is often independent for different users.


How Box cox and other transformations convert data into Normal Distributions?

Box Cox and other transformations find the exponent needed to transform the data into a normal distribution.

Transformations happen by finding the estimate of $\lambda$ and some values in the neighborhood are chosen to transform the original data.

Once the data is transformed, we assume the data is normally distributed.

The original formula proposed by Box & Cox is:

$$
y(\lambda) = \left\{
        \begin{array}{ll}
           \frac{y^{\lambda} -1 }\lambda \quad if \lambda \neq 0; \\
            log(y), \quad  if \lambda = 0.
        \end{array}
    \right.
$$

Which basically turns into choosing $\lambda$ as a model parameter. This value is found by maximizing log-likelihood.


No, actually it doesn’t make sense to apply two scaling methods to your data simultaneously. Generally, I would suggest using standardization because it is more robust to outlier samples compared to normalization. Having removed outlier samples from your dataset, you can safely use normalization instead.


Multinomial Naive bayes giving wrong result

There is only one feature for xx.

For a MultinomialNB, the modelled probability of the feature per class $p(x_i|y)$ is always the same ($\frac{3}{3}=1$ for class $1$ and $\frac{7}{7}=1$ for class $0$). The prediction is therefore the class prior $p(y_i)$ which is also the same here ($p(0) = 0.5$ and $p(1)=0.5$) so by default class $0$. Having one feature does not really makes sense because it gives no information to the model on what gives discriminative power toward the classes.

For a GaussianNB this is different because the modelled probability is given by $P(x_i \mid y) = \frac{1}{\sqrt{2\pi\sigma^2_y}} \exp\left(-\frac{(x_i - \mu_y)^2}{2\sigma^2_y}\right)$ so the mean and variance per class of the single feature give sufficient information.


I don't think the second picture necessarily has more noise, it's just less linear than the first one. If you are doing regression (which is the case in both your problems) you can use the same loss function to optimize your family of functions that you are using to fit your data. The only difference is that with certain function families gradient descent might not get to a global optimum but will get stuck in a local one. The loss function only determines how to penalize certain errors (based on the residuals).


Missing outputs in multiple-output neural net

Since we are talking about multiple different types of targets (classes versus numerical for example) we already need a composite loss function. I will consider how to balance the different composite parts of the loss function outside of the scope of this answer but if you look into multi-task learning there are solutions to this. What you could do (both in training and evaluation) is to mask the parts of the loss function that are unknown.

You can do this fairly easily (but hacky) by passing weights as input during training or prediction for each output in your sample whether it was available or not as 1 or 0 and multiply that component with the weight. For analysis of the training and prediction you need to make sure to take the mean of only the available labels because otherwise your loss values are too optimistic.

I do think this approach is very valuable in a lot of cases, this can also help with semi-supervised learning where you do have labels available for a similar task.


Why do we use a softmax activation function in Convolutional Autoencoders?

Since its a multi-class classification problem, each class will have its own probability value between 0 and 1 and sum of all probabilities of classes will equal to 1.


What are the effects of clipping the reward in stability?

You could clip for several reasons.

If you clip the gradient, the stabilizing effect is to force the optimizer to do only small changes in the backward step. Of course, you could also decrease the learning rate but the effect is slightly different. When you decrease the learning rate you basically say "learn more slowly". Instead, with gradient clipping, you say "learn as usual but if you have to change your mind rapidly don't do it" (I'm not sure that this sentence is understandable, English isn't my first language).
If you clip the error, the effect is the same. Maybe it changes a bit in a mathematical point of view but the bigger result is equal to clipping the gradient.
Clipping the reward doesn't give you any direct stabilizing effect. It is only a particular case of the more general reward shaping. In Playing Atari with Deep Reinforcement Learning, it's stated that:

  Since the scale of scores varies greatly from game to game, we fixed all positive rewards to be 1 and all negative rewards to be −1, leaving 0 rewards unchanged. Clipping the rewards in this manner limits the scale of the error derivatives and makes it easier to use the same learning rate across multiple games. At the same time, it could affect the performance of our agent since it cannot differentiate between rewards of different magnitude.

Basically, they've done it to make the environments similar to each other in rewards terms when seen by an agent. If you think of it, it's not difficult for you to play to a game when the score is given as multiple of thousand and then play to another one where even one hundred could be a great score. This doesn't apply to RL agents and so they reshaped the reward.


Your Prediction Model works. What to do after?

You could do the following:

Find out the most important predictors used by your model; the methods used to identify importance depend on the model used.
For each of the major pedictors try to find out how does each predictor segment the customer population of will-churn vs. will-not-churn. That would be the starting point for framing hypothesis of why these set of customers are churning. Generate reports/info-graphics showing how these predictors impact the tendency to churn. Sometimes predictors are not causes but side-effects of the hidden cause showing up elsewhere. So be open to skepticism about the predictors.
Involve the business/users and have brainstorming sessions to go through each of these hypothesis and evolve the most practical and economic actions that could be taken to reduce churn. Before you get into a discussion, plan for a sufficient amount of time, come up with your best thought actions which you feel will address the situation and have all the details gleaned from your analysis available for sharing in the session.

This is an art more than a science and your ability to understand the domain as well as good communication will help you achieve this goal.


Dissecting performance issues with Random Forest

There are several things you can do, even though giving a detailed descripton of your datasets would help a lot:

The first thing is that you should is to thoroughly compare the two datasets:

Are the same features present in both datasets?
Is the data collected in the exact same way? As an example, if age is a feature make sure that it is not recorded differently. Let's say one dataset could use the categories (<25, 25-50, 50+) and the other the exact age.
How do they differ in size?
Plot the distribution of each feature in both dataset: do they differ significantly, and why? You could test the hypothesis that a feature is drawn from the same distribution in both datasets.
Look for correlated features: correlated with each other and with the target distribution

The amelioration of your algorithm will entirely depend on the answer to these questions. If you observe significant differences between the two datasets then your historical data will not be a good training set for this task.

Independently from that, your 100% accuracy is very suspicious and your algorithm is probably overfitting.


Including the validation file in the training process after tunning

It depends on the distribution of the train, valid and holdout/test set.

There are a couple of possibilities (basically permutations). In general any different distribution=covariate shift is bad and you should repair it. If this is the case, including valid is the least of your problems (but you should include it in this case to make corrections) and you should worry about covarite shift.

If distributions are the same between the sets, it wont make any negative difference and it could only help if you add the valid-hyperparam tuning dataset to the train.


Improve performance for known unlabelled test set

You are asking something different than standard machine learning. In standard machine learning, a test set is a dataset with labels that is evaluated a single time. The result of the test set evaluation is never used to change the model.
Another way to frame your question is - "I have high quality unlabeled data that I would combine with my labeled data to create a model."
The answer to that question is semi-supervised learning.


Dealing with optional data in an Artificial Neural Network

Several options, depending on your algorithms:

Default to 0
Default to 1
Default to Average


How does xgboost work with linear booster?

Refer to the authors' paper on xgboost here - XGBoost: A Scalable Tree Boosting System.

Equation 3 on page 2 mentions that in each step all the predictors are used to greedily fit the next additive tree.
However, since column sub-sampling is also employed to prevent over-fitting, not all features may actually be used for each step, see section 2.3 of the same paper.

Hence, the specific answer to your query is that the algorithm uses all features to fit the residual.


Mean normalization is a form of feature scaling, so these are not really two different approaches. Feature scaling is just a more general term. What kind of feature scaling, e.g. mean normalization,  you need to use ultimately depends on the data.


Assigning manually a score based on some statistics?

If you use a ML model to learn a score that you built based on your features the best thing that could happen is that your model learn exactly the rules you applied on your features. It would not be usefull as it would be equivalent to applying your rules in the first place. Supervised ML is only usefull to learn unknown rules.
Basically you know have two approaches :

Don't use ML and keep your rules as is. Even if it wouldn't be the most predictive model it might already be usefull to observe unusual behaviors.
If you want to go the ML way you need to build relevant target. Generally speaking that means having a solid, undiscutable definition of your target. In your case the target could be an action from the moderation. Then ML could help you find the behavior that leads to moderator intervention.


Does any cloud service provide free GPU access?

Nvidia has it's own online labs where they can help you learn GPU based data processing. Might be worth checking out for you if you want to learn and play around within the online lab environment.

https://developer.nvidia.com/dli/onlinelabs

All the best.


KMeans Clustering on Heterogeneous Dataset

K-means attempts to group distributions into $k$ similar categories based on some metric of nearness. If your dataset contains repeated values this is not a problem. It simply means that you have 2 distributions from which you are drawing instances, and the variance of these distributions is essentially zero.

For example, if you have a dataset from which you want to cluster two species of fish with 2 features: color and length. However, the fish of each species are all clones (ignore other biology stuff i know nothing about like environmental factors). Thus they will all have exactly the same color and the same length in either group. Each measurement of each individual will be exactly the same. This is fine. We simply have 0 variance in each distribution.

Clustering is very well suited for very tight distributions because they are easily separable. However, distributions with high variance will have significant overlap and will thus cause many instances to be wrongfully classified as being a part of the other cluster.

The ideal number of clusters

This value can be calculated by determining the average distance between each instance and its nearest cluster. You will see that once you surpass 2, the average distance will not decrease in any further in this case, which means a single distribution is being split into 2 groups. Thus 2 is the ideal number of clusters.


Classification Threshold Tuning with GridSearchCV

As far as I know you cannot add the model's threshold as hyperparameter but in order to find the optimal threshold you can do as follows:

make a the standard GridSearchCV but use the roc_auc as metric as per step 2
 model = DecisionTreeClassifier()
 params = [{'criterion':["gini","entropy"],"max_depth":[1,2,3,4,5,6,7,8,9,10],"class_weight":["balanced"]}]
 GSCV = GridSearchCV(model,params,scoring="roc_auc")
 GSCV.fit(X_train,y_train)
 GSCV.best_params_
 best_model = GSCV.best_estimator_

Once you have the best hyper parameters set you can obtain the threshold that maximizes the roc curve as follows:
 from sklearn.metrics import roc_curve
 preds = best_model.predict_proba(X_train)[:,1]

 fpr, tpr, thresholds = roc_curve(y_train, preds)
 optimal_idx = np.argmax(tpr - fpr)
 optimal_threshold = thresholds[optimal_idx]

This threshold will give you the lowest false positive rate and the highest true positive rate


Stateful LSTM : Using different training window

I don’t see anything wrong with that. In fact it sounds like a good form of data augmentation.

It does sound like you are training with a batch size of 1, which may be slow. You could think about creating batches in a similar way, where the first elements of the batches are the sequences 1-10, 11-20, ..., the second elements of the batches are 2-12, 13-23, ..., all the way to 9-19, 10-29, ...


Are there other large margin classifiers than SVMs?

Yes, one famous example are boosting techniques like Adaboost. It uses small classifiers to create a big one.

Here you can find more info about margin classifiers.


The topic you are interest in is called "PU learning" or "positive and unlabeled learning".
You can start by having a look into  survey literature.


Bidirectional GRU: validation loss stuck on plateau diverges from well performing training loss

Your training and test sets are different. If your test set has less errors than your training set then it means that the data contained in the test set is easier for the algorithm than the data in the train set. For example in the training set you have Shakespeare and in the test set you only have text with short sentences, clear structure and low vocabulary.

The effect that comes after the first phase of learning is called overfitting. The algorithm finds some logic in the data that does not really exist. For example, in the text above word "logic" is always preceded by word "some". But this rule does not really exist and you can still freely say "there is logic" without word "some". You can see this effect in how people think. Imagine you are in a vary dark forest. Your brain will see animals even if there is just a tree or the wind blew some leaves. It is better to see patterns when they do not exist than not see patterns when they exist (or the tiger will eat you). And you can see it in the training. Your model "sees" stuff that does not exist but at the same time still improves pattern recognition that really matters. This goes till epoch 150 in your data where the model starts "seeing" more stuff that does not exist and starts to lose the previously learned patterns. Non realistic patterns, that are only specific to the train set, start to overwhelm the good patterns. This is considered normal for neural networks. Almost all neural networks should stop learning before the training error becomes zero.


Does Feature Normalization affect Gradient Descent | Linear Regression

Standard normalization let gradient descent converge faster. That's why most cases normalization is applied (of course it also eliminate effect of different scale features values).

In this quora topic it is explained well.

Your learning rate is very small. I think there is not enough number of iterations to coverge local minimum. This graph explains well the effect of learning rate to gradient descent:

You may want to check my another answer for it.


Convert a binary neural network classifier to one verses all classifier

You may keep N neurons and Sigmoid and treat is like multi-label though your data will always be multi-class. So it should learn like multi-class

Create your own One-vs-One

Have 3 copy of the same model
Fit with A|B, B|C and C|A respectively
For new data, you need the average of two for each Class probability

Use scikit-learn Classifier

OneVsOneClassifier Link
OneVsRestClassifier Link


Generally with multiple classes you have to make a distinction between exclusive and inclusive groups. The simplest cases are "all classes are exclusive" (predict only one class), and "all classes are compatible" (predict list of classes that apply).

Either way, label the classes as you would want your trained model to predict them. If you expect your classifier to predict an example is in both garbage and footpath, then you should label such an example with both. If you want it to disambiguate between them, then label with a single correct class.

To train a classifier to predict multiple target classes at once, it is usually just a matter of picking the correct objective function and a classifier with architecture that can support it.

For example, with a neural network, you would avoid using a "softmax" output which is geared towards predicting a single class - instead you might use a regular "sigmoid" function and predict class membership on a simple threshold on each output.

You can get also more sophisticated perhaps with a pipeline model if your data can be split into several exclusive groups - predict the group in the first stage, and have multiple group-specific models predicting the combination of classes in each group in a second stage. This may be overkill for your problem, although it may still be handy if it keeps your individual models simple (e.g. they could all be logistic regression, and the first stage may gain some accuracy if the groups are easier to separate).


That is generally not true, to be more accurate we can say that L1 promotes sparsity. if a weight is larger than 1 then L2 cares more about it than L1 while if a weight is less than 1 then L1 cares more about it than L2.

For a quick example imagine two weights, $w_1 = 15$ and $w_2 = 0.02$, let's imagine that the model considers reducing both of those weights by a small number $\epsilon=0.001$ (for the sake of simplicity reducing either weight by $\epsilon$ increases the model-error part of the loss at the same rate). Now, let's calculate how much the loss changes for both weights under both regularization terms.

For $w_1$ and L1: $\Delta L=|15|-|14.999| = 0.001 $ (Loss decreased by 0.001)

For $w_1$ and L2: $\Delta L=|15|^2-|14.999|^2 = 0.029999 $ (Loss decreased by 0.029999)

We can see that for $w_1$ the loss decreases about 28 times more for L2 compared to L1. So L2 regularizes the weights more in this case (i.e. L2 is willing to sacrifice more model complexity to regularize this weight more).

For $w_2$ and L1: $\Delta L=|0.02|-|0.019| = 0.001 $ (Loss decreased by 0.001)

For $w_2$ and L2: $\Delta L=|0.02|^2-|0.019|^2 = 0.000039 $ (Loss decreased by 0.000039)

We can see that for $w_2$ the loss decreases about 24 times more for L1! So for smaller weights L1 is willing to sacrifice more model expressiveness just to reduce the weights while L2 barely pays any attention (compared to L1) to weights close to 0.

It's possible that in practice you perceive L1 regularizes weights more than L2 becauses when you perform L1 regularization you notice a lot of the weights tend to be exactly 0 while in L2 almost no weights are exactly 0 so you might tend to think that L1 is "stronger", but that just comes from the point we notices above where L2 barely pays any attention to weights close to 0 while L1 still sees a benefit of $\epsilon$ no matter how small the weight was before the $\epsilon$ change and thus it promotes sparsity.

There are a lot of practical and theoretical differences between L1 and L2 regularization, too many to list here. For example one practical difference is that L1 can be a form of feature elimination in linear regression. A theoretical difference is how L2 regularization comes from the MAP of a Normal Distributed prior while the L1 comes from a Laplacean prior.

EDIT: I just reread your post and yes, looking at the derivatives you should also get the same insight. For $w>1 \Rightarrow w\lambda > \lambda$ thus L2 regularizes large weights more while for $w<1 \Rightarrow w\lambda < \lambda$ thus L1 regularizes small weights more.


Regression model with variable number of parameters in dataset?

One simple idea, no imputation needed: build a model using the parameters have always existed, then each time a new set of parameters gets added, use them to model the residual of the previous model. Then you can sum the contributions of all the models that apply to the data you happen to have. (If effects tend to multiply rather than add, you could do this in log space.)


The difference between `Dense` and `TimeDistributedDense` of `Keras`

Let's say you have time-series data with $N$ rows and $700$ columns which you want to feed to a SimpleRNN(200, return_sequence=True) layer in Keras. Before you feed that to the RNN, you need to reshape the previous data to a 3D tensor. So it becomes a $N \times 700 \times 1$.

$ $

The image is taken from https://colah.github.io/posts/2015-08-Understanding-LSTMs

$ $

In RNN, your columns (the "700 columns") is the timesteps of RNN. Your data is processed from $t=1 \  to \ 700$. After feeding the data to the RNN, now it have 700 outputs which are $h_1$ to $h_{700}$, not $h_1$ to $h_{200}$. Remember that now the shape of your data is $N \times 700 \times 200$ which is samples (the rows) x timesteps (the columns) x channels.

And then, when you apply a TimeDistributedDense, you're applying a Dense layer on each timestep, which means you're applying a Dense layer on each $h_1$, $h_2$,...,$h_t$ respectively. Which means: actually you're  applying the fully-connected operation on each of its channels (the "200" one) respectively, from $h_1$ to $h_{700}$. The 1st "$1 \times 1 \times 200$" until the 700th "$1 \times 1 \times 200$".

Why are we doing this? Because you don't want to flatten the RNN output.

Why not flattening the RNN output? Because you want to keep each timestep values separate.

Why keep each timestep values separate? Because:

you're only want to interacting the values between its own timestep
you don't want to have a random interaction between different timesteps and channels.


Should you perform feature scaling/mean normalization when predicting with a model?

Yes, testing data should follow the same preprocessing as the training data. Otherwise, testing data will have nothing comparable with what the algorithm learned, leading to (very) bad performances.

note: In Sklearn, the Pipeline class helps you to respect the fundamentals of ML modeling like data leakage and applying the same transformations to train and test set.


Heavy regression loss for false non 0 prediction

One option would be Huber Loss which could be setup to increase the weight for some types of errors compared to other errors.


Random Forest - ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

You should imput missing data in your dataset, or delete thos rows if they are not many.

Alternatively, if the values are too big to be represented by float32 datatype, try to convert them to float64 (takes more RAM).


Also you have misunderstood the -1..
It's not the exponent rather the symbol of inverse of a Matrix

Also use .T for Transpose...(a bit Pythonic Convenience)

Just to help you out,
Search np.linalg.inv

For your second Query,

Refer here

Just to add a little, you stop when you see that your loss isn't improving any more as such or is improving at the 5th decimal place..

Hope this helps..


KNN custom transformer shows same accuracy for every k i set

You dont use self.k in your closest_n method (for starters).

U dont use it at all, you just initialise the property.

I would check sklearn implementation, just look under the hood here


Anomaly Detection in Time Series: How to label the data

Anomaly or Outlier detection are usually used when we have cases that happens rarely. So it is not easy to gather data for those cases. Then what we do is, we just train with our normal data. It is also the same for time series.

It is usually falls under Unsupervised learning, as there is no need to label your data. You feed all the data that you have to the model (For example one Class SVM) and it would try to understand or find clusters/groups in the data. Afterwards for test data, it would give you a number as a result of prediction/detection. This number would represent how close or far this test data is to normal group/clusters.

Of course you should have labeled data for the propose of testing the model. For this case you can either generate anomaly data manually just for the sake of testing or you should identify some anomalies by analysing the data. Then you can test the model with these labeled data and see if it can actually recognise them or not.

Of course you can tackle this issue as two-class classification but then this is not anomaly detection anymore, and it will be a normal classification model.

Hope this helps.

For further info you can take a look at these:

https://scikit-learn.org/stable/modules/outlier_detection.html

https://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html#sklearn-svm-oneclasssvm


Kaggle Titanic submission score is higher than local accuracy score

First question: on certain competitions on kaggle you can select your submission when you go to the submissions window. There you may not be able to on titanic one so you are stuck with 100 percent.

Regarding your second question and why do you have different results locally, there is a couple of explanations, first its randomization did you set all the seeds, second all of the modules/library versions you have locally sam as the ones on kaggle? Third different hardware could also be the reason


Why do the arrange of clusters change when change the unit

When you change the scale or range of your features the euclidean distance between pairs of observations in your dataset changes significantly and one feature with the larger range dominates over all the others.
So, in your case, k-means was tricked into believing that the depth in metres is more important than the other feature with the lower scale.
You can fix this by normalizing all features in your dataset to a similar scale so as to reduce the bias towards specific features.


The input to the predict method should be a 2d array of shape (n_samples, n_features), which in your case with one feature and one sample would be (1, 1). Therefore add a second dimension for the number of samples: reg.predict([['10000']]).


how to create multiple subplots( scatterplot) in for loop

Just get rid of the plt.figure(figsize=[5,5])
count=1
for i in df.columns:
    plt.subplot(n,1,count)
    sns.scatterplot(df["cnt"],df[i])
    count+=1

plt.show()

or use this to make it more beautier (Adjust the first two parameters of subplot function, i.e. 3, 2, to if you have more columns. First parameter specifies the number of rows for subplots in the plot, and second parameter specifies number of columns)
count=1
plt.subplots(figsize=(10, 8))
for i in df.columns:
    plt.subplot(3,2,count)
    sns.scatterplot(df["cnt"],df[i])
    count+=1

plt.show()


One way of stating what you are looking for is to find a simple mathematical model to explain your data.

One thing about neural networks is that (once they have more than 2 layers, and enough neurons total) they can in theory emulate any function, no matter how complex. This is useful for machine learning as often the function we want to predict is complex and cannot be expressed simply with a few operators. However, it is kind of the opposite of what you want - the neural network behaves like a "black box" and you don't get a simple function out, even if there is one driving the data.

You can try to fit a model (any model) to your data using very simple forms of regression, such as linear regression. So if you are reasonably sure that your system is a cubic equation $y= ax^3 + bx^2 +cx +d$ then you could create a table like this:

  bias   |   x  |  x*x  |  x*x*x  |     y
     1       0       0         0        0
     1       2       4         8        4
     1       3       9        27        9
     1       .         .        .       .
     1     100   1000000    10000   10000

and then use a linear regression optimiser (sci-kit learn's SGD optimiser linked). With the above data this should quickly tell you $b=1, a,c,d=0$. But what it won't tell you is whether your model is the best possible or somehow "correct". You can scan for more possible formulae by creating more columns - any function of any combination of inputs (if there is more than one) that could be feasible.

However, the more columns you add in this way, the more likely it is you will find an incorrect overfit solution that matches all your data using a clever combination of parameters, but which is not a good general predictor. To address this, you will need to add regularisation - a simple L1 or L2 regularisation of the parameters will do (in the link I gave to scikit-learn, the penalty argument can control this), which will penalise large parameters and help you home in on a simple formula if there is one.


Maths question on mean squared error being dervied to bias and variance

Because it's a costant, everything that is a costant value remains unchanged by the E, that's why you can "move" it outside.

For example:

If y is a costant or it is known, the E doesn't affect it, so you can "move" it outside the symbol, and write just y.

The only thing that is unknown is the estimator yhat, in fact you have  and not just .

That's what's happening here.

I found this on CrossValidated that might be more clear.


Is there an R package for Locally Interpretable Model Agnostic Explanations?

Yes, there is now a port to R, which is available here.
It purports to provide LIME explanations for any classifier that implements a predict() method accepting a type = 'prob' argument. I have not yet tested it myself.


Difference between MultiDiGraph and DiGraph

The first way (with DiGraphs) is correct, but the two paths [0, 5] and [0, 5, '*'] both have the same (weighted) length, and the method just returns one of them.  Which one depends on how your version of python orders dictionaries, but I think since vertex 5 appears first in the topological ordering it's rather likely to appear first.

You could hack it by making the edges into '*' have weight 1, then subtract that from the longest path length.

dag_longest_path doesn't appear to be built to work for MultiDiGraphs.  For a vertex's predecessors' .items(), a DiGraph has just the dictionary of attributes, but the MultiDigraph has a dictionary of edge-attributes pairs.  dag_longest_path doesn't try to take that into account, so never sees any weight attribute to use.


Classification training using probabilites and not raw classes (factors)

Beta Regression

You could use beta-regression. I have no practical experience with this type of regression. However, it might be the right method for your task. As far as I understand, the link function is chosen so to restrict $\hat{y} \in [0,1]$.

Here is an R implementation, where the docs say:

  Fit beta regression models for rates and proportions via maximum
  likelihood using a parametrization with mean (depending through a link
  function on the covariates) and precision parameter (called phi).

Example:

library("betareg")
data("GasolineYield", package = "betareg")
summary(GasolineYield$yield)

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.0280  0.1165  0.1780  0.1966  0.2705  0.4570

br = betareg(yield ~ batch + temp, data = GasolineYield)
preds = predict(br, newdata=GasolineYield)
summary(preds)

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.04571 0.10309 0.16364 0.19655 0.26429 0.50792

Regression Models For Ordinal Data

Ordinal Logistic Regression could be used for this problem since classes are ordered and multinomial classification does not take the order of classes into consideration. In practice, this algorithm doesn't scale to many classes or many observations because its computationally expensive.

Here is an example of fitting a cumulative link model (CLM) such as the proportional odds model to data using the ordinal package in R.

require("ordinal")
fm1 <- clm(rating ~ contact + temp, data=wine)
summary(fm1)

formula: rating ~ contact + temp
data:    wine

link  threshold nobs logLik AIC    niter max.grad cond.H
logit flexible  72   -86.49 184.98 6(0)  4.01e-12 2.7e+01

Coefficients:
       Estimate Std. Error z value Pr(>|z|)
contactyes   1.5278     0.4766   3.205  0.00135 **
tempwarm     2.5031     0.5287   4.735 2.19e-06 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Threshold coefficients:
    Estimate Std. Error z value
1|2  -1.3444     0.5171  -2.600
2|3   1.2508     0.4379   2.857
3|4   3.4669     0.5978   5.800
4|5   5.0064     0.7309   6.850

Regression with a Logistic Link Function

As suggested by Ben Reiniger in the comments of the question, another alternative is simply to use a Logistic Link function in a regression model.

An example would be using xgboost with reg:logistic as the objective function. However, many libraries may not support this behavior as they need the target be either one or zero.


What is the difference between normalization and re-scaling?

Re-scaling means that you multiply your variable by a factor, i.e. $x \to x/a$. A normalization is a specific kind of re-scaling, where the factor $a$ is such that the values of $x$ become of order one. Its form depends on the context and what you are trying to do. Examples are $a= \langle x \rangle$ or $a= \textrm{max}(x)$. Perhaps the most common one is when you do $x \to (x-\langle x \rangle)/\sigma$, because if $x$ follows a Gaussian distribution, the re-scaled variable follows a normal distribution (average 0 and std 1). Note, this last transformation actually involves a re-scaling, but also a translation.


What you have described is that you want to consider the distribution of observations in a way that models the population to serve a better purpose.

I would recommend reading into how different types of distributions can be transformed so that your model might achieve a higher score.

In terms of your plots during analysis of your dataset, I would look at resampling at different frequencies, this will change how your plots will look and allow you to identify issues with seasonality, such as breaks in the account activity.

If the observations are missing because the person simply wasn't active, then this is a good representation of reality, the question is maybe how would you increase the users engagement?


I've done this (using anaconda) with the following libraries.

from mlxtend.frequent_patterns import apriori
from mlxtend.frequent_patterns import association_rules

Have a look here.


The accuracy is different because there are k-classifiers made for each number of k-folds, and a new accuracy is found.

You don't select a fold yourself.  K-Fold cross-validation is used to test the general accuracy of your model based on how you setup the parameters and hyper-parameters of your model fitting function.

What you do select is the number of folds, so in your example of 5 folds, it will do the following:

split up your training set into 5 different subsets (folds)
create a classifier for each of the 5 folds by using k-1 folds for
fitting the model, and test the classifier accuracy using the fold
left out

After it's done you can see how your classifier fared over the average of these folds.

If you're trying to find the optimal parameters to configure your model for the best accuracy you should be using grid search.  Depending on your language the implementation will be different: python using sklearn.model_selection.GridSearchCV

, and R uses the 'carat' library and the train() function.  Once you've run grid search you can either use the resulting model if you're programming in R or in python, you can add the new hyperparameters to your model fitting function and re-fit your model.

Your train/test ratio will depend on the number of folds. If you have 100 rows in your training set and you have 5 folds, then you'll have an 80/20 split train/test.  If you have 10 folds it will be a 90/10 split.  The cross-validation function for k-folds uses k-1 folds for fitting the model, and the fold left out is used for testing.


lat is a series in your data, if you have even only one nan value in your series ,lat, then you will loose all of your series.
Another problem might be in the lat column values might not be suitable for converting to pd.to_numeric such as "1,2" (not "1.2"), "a", "nan" etc. Then you will loose your series again.
Therefore:

Check:
len(lat.dropna()) == len(lat)
if it is True then check:

len(df2[df2.str.contains(",")]) > 0

I hope it works!


Adding Features To Time Series Model LSTM

For RNNs (e.g., LSTMs and GRUs), the layer input is a list of timesteps, and each timestep is a feature tensor. That means that you could have a input tensor like this (in Pythonic notation):

# Input tensor to RNN
[
    # Timestep 1
    [ temperature_in_paris, value_of_nasdaq, unemployment_rate ],
    # Timestep 2
    [ temperature_in_paris, value_of_nasdaq, unemployment_rate ],
    # Timestep 3
    [ temperature_in_paris, value_of_nasdaq, unemployment_rate ],
    ...
]

So absolutely, you can have multiple features at each timestep. In my mind, weather is a time series feature: where I live, it happens to be a function of time. So it would be quite reasonable to encode weather information as one of your features in each timestep (with an appropriate encoding, like cloudy=0, sunny=1, etc.).

If you have non-time-series data, then it doesn't really make sense to pass it through the LSTM, though. Maybe the LSTM will work anyway, but even if it does, it will probably come at the cost of higher loss / lower accuracy per training time.

Alternatively, you can introduce this sort of "extra" information into your model outside of the LSTM by means of additional layers. You might have a data flow like this:

TIME_SERIES_INPUT ------> LSTM -------\
                                       *---> MERGE ---> [more processing]
AUXILIARY_INPUTS --> [do something] --/

So you would merge your auxiliary inputs into the LSTM outputs, and continue your network from there. Now your model is simply multi-input.

For example, let's say that in your particular application, you only keep the last output of the LSTM output sequence. Let's say that it is a vector of length 10. You auxiliary input might be your encoded weather (so a scalar). Your merge layer could simply append the auxiliary weather information onto the end of the LSTM output vector to produce a single vector of length 11. But you don't need to just keep the last LSTM output timestep: if the LSTM outputted 100 timesteps, each with a 10-vector of features, you could still tack on your auxiliary weather information, resulting in 100 timesteps, each consisting of a vector of 11 datapoints.

The Keras documentation on its functional API has a good overview of this.

In other cases, as @horaceT points out, you may want to condition the LSTM on non-temporal data. For example, predict the weather tomorrow, given location. In this case, here are three suggestions, each with positive/negatives:

Have the first timestep contain your conditioning data, since it will effectively "set" the internal/hidden state of your RNN. Frankly, I would not do this, for a bunch of reasons: your conditioning data needs to be the same shape as the rest of your features, makes it harder to create stateful RNNs (in terms of being really careful to track how you feed data into the network), the network may "forget" the conditioning data with enough time (e.g., long training sequences, or long prediction sequences), etc.
Include the data as part of the temporal data itself. So each feature vector at a particular timestep includes "mostly" time-series data, but then has the conditioning data appended to the end of each feature vector. Will the network learn to recognize this? Probably, but even then, you are creating a harder learning task by polluting the sequence data with non-sequential information. So I would also discourage this.
Probably the best approach would be to directly affect the hidden state of the RNN at time zero. This is the approach taken by Karpathy and Fei-Fei and by Vinyals et al. This is how it works:

For each training sample, take your condition variables $\vec{x}$.
Transform/reshape your condition variables with an affine transformation to get it into the right shape as the internal state of the RNN: $\vec{v} = \mathbf{W} \vec{x} + \vec{b}$ (these $\mathbf{W}$ and $\vec{b}$ are trainable weights). You can obtain it with a Dense layer in keras.
For the very first timestep, add $\vec{v}$ to the hidden state of the RNN when calculating its value.

This approach is the most "theoretically" correct, since it properly conditions the RNN on your non-temporal inputs, naturally solves the shape problem, and also avoids polluting your inputs timesteps with additional, non-temporal information. The downside is that this approach often requires graph-level control of your architecture, so if you are using a higher-level abstraction like Keras, you will find it hard to implement unless you add your own layer type.


Compare Neural Network generalization results

As Neil said in the comments, split out a test set (data you don't use in training either model) and see how each trained model performs on the test set.


Reformatting data- Giving each value in a list it's own row while retaining the list's ID

You can use split with iterrows:
import pandas as pd
df = pd.DataFrame([{'Site_ID': 1, 'Section_ID': 'a,b,c'},
               {'Site_ID': 2, 'Section_ID': 'd,e,f'}])

df

  Site_ID   Section_ID
0   1   a,b,c
1   2   d,e,f

pd.concat([pd.Series( row['Site_ID'], row['Section_ID'].split(',') ) for _, row in df.iterrows()])

a    1
b    1
c    1
d    2
e    2
f    2

iterrows goes through the series row by row and split will find the separate values in a string that is separated by a comma.
If you want the column names back, you can convert back to a dataframe:
import numpy as np
df1 = pd.DataFrame(np.array(pd.concat([pd.Series( row['Site_ID'], row['Section_ID'].split(',') ) for _, row in df.iterrows()]).reset_index()), columns=['Site_ID','Section_ID'])

df1

 Site_ID Section_ID
0   a   1
1   b   1
2   c   1
3   d   2
4   e   2
5   f   2


What is the best way of combining audio and visual data to make predictions?

You can try to fuse the image and audio data. You can fuse in different ways:

1) Early fusion - fusion at feature level

2) Late fusion - fusion at output level (similar to your idea)

Here is some papers for your reference:

Look, Listen and Learn
https://arxiv.org/pdf/1705.08168.pdf

Learn to Combine Modalities in Multimodal Deep
Learning
https://arxiv.org/pdf/1805.11730.pdf


LSTM - How many times should I look back to predict next six hours -Multivariate Time-Series

The second option sounds fun but better don't do it: errors will sum up and it's much harder to track where errors come from. Like you have some error for the 1st hour, feed it back in the net, predict the 2nd hour get an error there and can't really tell how much of that is caused by the error of the first hour.

Besides that, I haven't seen any application which works with this feedback mechanism.


Why we use Pickle instead of saving the weights in HDF5 format in Deep Learning

Avoid pickle, there are a couple of problems
Even in keras documentation they say
"It is not recommended to use pickle or cPickle to save a Keras model."
Also look at this solution.


You can start with rule-based methods, such as a combination of regular expression and if/then statements.


Stats degree from 25 years ago: What do I need to learn now to get into Data Science?

To my mind you have to start from basic understanding of what is Data Science, and what are basic concepts, before going deeper. You could almost be pretty good at data science not knowing a lot on statistic, since the global logic behind what you are doing is often way more important than the exact statistical detail.
I'd really suggest you to start by finding a global point of view, of what global concepts are, and what's the global process behind analysing data, and then you could go more specifically onto many things, like how models work, what are the precise issues you can encounter in your analysis, in your model building, what are different types of variables, how to handle them, what is "parameter tuning", ... but only in details in a second time.
I don't really have books to recommand you, since I just ended my studies, and my different teachers already provided me a lot of information to learn and work on, but to me the most important thing is to first understand global concepts and global logical behind data science, so you're not just applying technical things with making huge logical mistakes in the same time.


There can be other reasons related to the model but the most simple explanation is that the data contains contradicting patterns: if the same features correspond to different target values, there is no way for any model to achieve perfect performance.
Let me illustrate with a toy example:
x   y
0   4
0   4
1   3
1   4
1   3
1   3
2   8
2   7
2   7
2   6

The best a model can do with this data is to associate x=0 with y=4, x=1 with y=3 and x=2 with y=7. Applying this best model to the training set would lead to 3 errors and the MAE would be 0.3.


By googling: HDBSCAN is order of n squared whereas optics is order of n times log(n).


Very Deep Convolutional Networks for Text Classification: Clarifying skip connections

As you mentioned, the paper doesn't clarify. However, my guess is that this is not due to concatenating 2 previous layers (I don't really see an specific reason to do this here) but because of concatenating the ResNet shorcut.

Generally, all conv layers have a number of filters, thus determining the output size (num_filters, size) regardless the inputs. On the other hand, MaxPooling does keep the input num_filters (though in this case reducing the size). In the paper, note that the num_filters is doubled at the output of all convlayer except for the one that does not keep the ResNet shorcut (last 512 conv layer). So my guess is that they are concatenating the output of the conv layer and the shorcut which would explain the output size.

Hope this helps!


As the name implies, doc2vec generates vectors representing documents (sentences, paragraphs) but not single words. So training doc2vec won't give you word vectors but document vectors. This means you can't replace word2vec by doc2vec at all.

Here's how the authors of the underlying paper describe what doc2vec does:

  Our algorithm represents each document by a dense vector which is trained to predict words in the document.


Replacing missing value by class conditional mean

Your intuition about 'no effect' is true in some sense. But this replacement may be not the best use of the information you have.

The choice of missing value treatment depends on your initial problem statement. In all the cases I assume that you have already somehow estimated conditional means $\mu_0$ and $\mu_1$ and the common variance matrix $S$.

Problem 1. For your test observation, you know $y$, but know nothing about $x$, and want to estimate $x_n$. If this is the case, $\mu_{y, n}$ is indeed your best guess of $x_n$.

Problem 2. For your test observation, you know $x_{-n}$ (that is, $x$ except $x_n$) and $y$, and want to predict $x_n$ as best as possible. If it is the case, you should use expectation conditional not only on $y$, but also on $x_{-n}$:
$$
\hat{x_n} = \mathbb{E}(x_n|y, x_{-n}) = \mu_{(y,n)} + S_{(n,-n)}S_{(-n)}^{-1}(x_{-n}-\mu_{(y,-n)})
$$
where $S_{*}$, $\mu_{y,*}$ are corresponding parts of covariance matrix and conditional expectation vector.

Problem 3. You know $x_{-n}$ but don't know $y$, and you want to estimate $x_n$. In this case, conditional distribution of $x_n$ is a mixture of two normal distributions, $p(x_n|y=0, x_{-n})$ and $p(x_n|y=1, x_{-n})$, with weights, proportional to $p(y=0|x_{-n})$ and $p(y=1|x_{-n})$ respectively. The mean of this distribution is just the weighted average of two means conditional on $y$. But this distribution may be bimodal, and its mean may be not a very useful statistic.

Problem 4. You know only $x_{-n}$, and your final goal is to estimate probability of $y=1$. In this case, you should just forget about $x_n$ (you don't know anything of it anyway), and use the distribution of $x_{-n}$ conditional on $y$ for whatever algorithm you apply.


Which machine learning algorithm should I apply for differentiate question difficulty level with users' result

I suggest you look at item response theory and the Rasch model.  They construct a model that attempts to identify both (a) the proficiency/knowledge/ability of the user and (b) the level of difficulty of the question.


How to decode encoded labels in Decision tree classifier

First, setting feature_names will replace all the X[2] and such.  Using feature_names=data.columns is sensible, provided your feature names aren't too long for easy display.
Now, you'll still be comparing against the label encodings.  By saving the label encoder objects (your encoder dict), you can retrieve which levels correspond to the integer labels (with the classes_ attribute, or possibly the inverse_transform method).  If your features have many levels, then you probably don't want to display all the levels in each split, and just providing them off to the side somewhere might be best.  If your features have few levels, maybe hacking the resulting DOT code will work; see this related answer of mine, which concerns one-hot encoded variables.


Could the Input shape of the LSTM layer not be a constant?

That line doesn't work because the architecture of ANNs must be kept constant, in order to train them. If you vary the number of input cells, you change the architecture of the ANN itself.

If your input sequences are not of constant length, I suggest you to use zero padding. This is the most common and used padding technique. In other words, you "complete" the input series with zeros at the end, so that they all have the same length. A properly defined Neural Network would then learn to ignore the zeros and capture only the relvant information.

You can read more about padding here.


Plot a training/validation curve in Pytorch Training

You should use Tensorboard. It has been integrated with PyTorch. See this.


Does PCA decrease the feature on my Data set or just decrease the dimension?

From the documentation:

  coeff = pca(X) returns the principal component coefficients, also known as loadings, for the $n$-by-$p$ data matrix X. Rows of X correspond to observations and columns correspond to variables. The coefficient matrix is $p$-by-$p$. Each column of coeff contains coefficients for one principal component, and the columns are in descending order of component variance. By default, pca centers the data and uses the singular value decomposition (SVD) algorithm.

The values in coef represent the transformation from the original features (rows of coef) to the principal components (columns of coef).  You'll want to keep only the first $k$ columns, then multiply your data matrix by this matrix.

Your features are the dimensions that your data lives in, so number of features and dimension are the same.  Very roughly speaking, PCA rotates the the feature axes to align to the most significant directions rather than the original feature directions, then selects only the most significant directions to keep, thus reducing the dimensionality of your problem.  But it also means your new columns won't represent pure features anymore, but linear combinations of them.

If you keep all of the new columns, you won't reduce dimensionality; but your new features will be ordered in such a way that the first ones capture the most variance of your data.


Causes of vanishing and exploding gradient

Gradient vanishing and exploding depend mostly on the following: too much multiplication in combination with too small values (gradient vanishing) or too large values (gradient exploding).

Activation functions are just one step in that multiplication when doing the backpropagation. If you have a good activation function, it could help in reducing these bad effects. One such activation function is ReLu. See this tutorial.

Other sources:

A Gentle Introduction to Exploding Gradients in Neural Networks
Deep Learning book by Ian Goodfellow at el.: This book has a lot of examples with gradient problems (eg. Chapter 8.2.4: Cliffs and Exploding Gradients)


How to deal with different length entities in a Keras DataGenerator?

I faced a similar problem and the quick solution I came up with was iterate through the main data set, create an instance of TimeseriesGenerator for each of the subsets and feed it to the model. So in your case that would be: go through the collection of articles, instantiate TimeseriesGenerator for each article and feed the windowed samples to your model, ie.:
from keras.preprocessing.sequence import TimeseriesGenerator

# Set the training parameters
window_length = ...
ix0, ix1 = ...
batch_size = ...

for article in articles:
        # Insert your code to generate the article's features and targets
        x, y = ...

        model.fit_generator(TimeseriesGenerator(
                        x,
                        y,
                        window_length,
                        batch_size=batch_size,
                        start_index=ix0,
                        end_index=ix1
                ),
                steps_per_epoch=int(np.ceil((x.shape[0] - ix0 - ix1 - window_length)/float(batch_size))),
                epochs=50,
                verbose=1
        )

Now depending on your dataset size and composition you can either:

run a number of epochs per article
or train one article per epoch, but for that you would need another loop.


I believe that the algorithm that you want to use is something called a latent dirichlet allocation (LDA) model. This model is designed to uncover the topics in a corpus of documents.

Scikit learn has an implementation.

They even have a tutorial which teaches you how to extract topics. The tutorial also describes Non-negative Matrix Factorization (NNMF) as a method to extract the topics. I can't vouch for this algorithm, because I haven't used it personally (as opposed to LDA which I have used before), but from their tutorial NNMF does seem to give reasonable results.

Using cosine similarity will help you to group tweets that are most similar, but it wouldn't give you their topics. Which may be what you want? It really is hard to say, because only you would know how you should have the system behave. Unfortunately, that doesn't help you figure out what is trending, and you will need to do some heavy post-processing to make whatever algorithm you use spit out something that is useful to you.

Good luck!


Euclidean distance for more than two datapoints

You can use sklearn's euclidean_distances function.

http://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.euclidean_distances.html#sklearn.metrics.pairwise.euclidean_distances

In [1]: from sklearn.metrics.pairwise import euclidean_distances

In [2]: import numpy as np

In [3]: x = np.random.rand(10, 3)

In [4]: euclidean_distances(x)


Is the result of RL is a classic function to (re)apply at each time step?
In some manner yes, when using RL to find the best policy you end up with a policy that can be described as a function(classic or not) from possible states to possible actions.

As if it was a regression?
No, regression algorithms 'solves' the function between feature space and target space.
 In RL both of these spaces have no(very different) meanings.
In addition, RL algorithms take into consideration multi-step predictions(state transitions + rewards), which is not very straight-forward in regression problems.


Please review my sketch of the Machine Learning process

This process will result in data leaks. The split needs to happen earlier. Normalizing data before the split means that your training data contains information about your test data. I would put the split at 3. in your flow chart.

A common step I think you have missed is imputation of missing values. I would put that before feature engineering.

Overall I think this is a good rough outline for a beginner to follow. It is overly simplistic and leaves a lot out, but I think you know that and you have to start somewhere.


Is it valuable to normalize/rescale labels in neural network regression?

Yes, you should do this. Given the initialization schemes and normalized inputs, the expected values for the outputs are 0. This means that you will not be too far off from the start, which helps convergence. If your target is 1000, your mean squared error will be huge which means your gradients will also be huge which can lead to numerical instabiliy.


Browse the following sources to see if you could find them:

UCI ML Repository
OpenML webpage (login required to download)
List of datasets for ML research (Wiki)

Hope this helps.


Based on your comments, you are looking for agglomerative hierarchical clustering.

You start with one point as its own cluster. Then iterate over pairs of clusters, merging them according to some criterion.

Typically you need to select a "cut point" after which you stop combining clusters. This is not an easy problem in general, and for the most part involves eyeballing your data until it "looks right", much like choosing K in K-means. In your case, however, you can use the external criterion you have in mind. You will need to recompute its value at every step, and then simply stop when its value passes the desired threshold.


How to calculate key points:

Take your image $I(x,y)$ and convolve it using a Gaussian $G(x,y,k\sigma$) at different scales $k \sigma$ to obtain $L(x,y,k\sigma)$. This produces several versions (one for each scale $k\sigma$) of your image $I(x,y)$ with different degrees of blurring.
Separate your blurred images according to octaves (an octave is usually taken as $2\sigma$). Within a given octave, take the difference between adjacent blurred images $L(x,y,k_{i}\sigma)$ and  $L(x,y,k_{j}\sigma)$. This difference is called a Difference of Gaussians (DoG) $D(x,y,\sigma) = L(x,y,k_{i}\sigma) - L(x,y,k_{j}\sigma)$. At this point, an image will be helpful (source: opencv):

As you can see, you have the blurred/convolved/filtered images on the left and the differences between adjacent images on the right.

Then, take one pixel on a DoG and compare it to its 8 neighbors in the same DoG, as well as to the 9 equivalent pixels of the DoG located in the next scale  and to the other 9 pixels in the previous DoG. In the drawing, this corresponds to the DoG's located above and below. If this pixel is a local extrema, then you have a candidate keypoint. Keep in mind that candidates can be discarded based on other criteria.

Once you have these keypoints, you proceed to generate descriptors. For each keypoint, calculate an image gradient. As you should know, gradients tell you the direction of maximum rate of change. Therefore, you can construct a grid around each keypoint that is oriented according to the dominant gradient around that point. This grid has subregions (usually 16 subregions) and for each subregion, calculate an 8-bin histogram. Finally, concatenate every histogram you obtained from every subregion in your grid and that is the feature vector for that keypoint (actually, the full feature vector also includes the location and rotation angle.) A helpful illustration (from Solem's book Programming Computer Vision with Python):

At this point, you have a feature vector with 132 values for each keypoint. As for your second question, I'm not sure about the best approach to store an array in a database. Maybe others can expand on this point. Of course, there are several options:

Create an array datatype to store your data
Simply use a VARCHAR field
Store it as a binary file
Use a database specifically designed to handle arrays.


ValueError: could not convert string to float: '���'

Though not the best solution, I found some success by converting it into pandas dataframe and working along.

code snippet

# convert X into dataframe
X_pd = pd.DataFrame(data=X)
# replace all instances of URC with 0
X_replace = X_pd.replace('�',0, regex=True)
# convert it back to numpy array
X_np = X_replace.values
# set the object type as float
X_fa = X_np.astype(float)

input

array([['85', '0', '0', '1980', '0', '0'],
       ['233', '54', '27', '-1', '0', '0'],
       ['���', '�', '�����', '�', '��', '���']], dtype='<U5')

output

array([[ 8.50e+01,  0.00e+00,  0.00e+00,  1.98e+03,  0.00e+00,  0.00e+00],
       [ 2.33e+02,  5.40e+01,  2.70e+01, -1.00e+00,  0.00e+00,  0.00e+00],
       [ 0.00e+00,  0.00e+00,  0.00e+00,  0.00e+00,  0.00e+00,  0.00e+00]])


Predict_proba for Binary classifier in Tensorflow

Since a neural net that ends with a sigmoid activation outputs probabilities, you can take the output of the network as is.

If you're referring to scikit-learn's predict_proba, it is equivalent to taking the sigmoid-activated output of the model in tensorflow. In fact that's exactly what scikit-learn does.

E.g. if tour model outputs $0.8$ for class $1$, you would classify this as $1$ (since $0.8 > 0.5$), with a probability of $0.8$. S


K Nearest Neighbour with different distance matrix to each datapoint

As pointed out by @Pratik Deoghare you can create a custom metric on sklearn kNN and you can see how in the link he provided.

But you want a function to that is different for each $x_i$, that is not a metric in the mathematical sense, but I can see how that could benefit the algorithm either way.

the function you pass as a metric (see how in the link) could be defined as

def creatmydist(AllA):
    Alist = AllA
    def mydist(x, y):
        nonlocal Alist
        if x[-1] == 0: i = y[-1]
        else: i=x[-1]
        A = Alist[i]
        x = np.dot(A,x[0:len(x)-1])
        y = np.dot(A,y[0:len(y)-1])
        return np.dot(x-y,x-y)
    return mydist

Where AllA should be a list with all $A = (A^T)^{-1}Q$ and every $x_i$  should have as last element it's index $i$


Using categorical and continuous variables in Deep Learning

If I understand well your main issue is to transform some Categorical variables into Continuous, or some Continuous into Categorical.
You have many ways to do that:
Continuous to Categorical :

You can yourself set intervals, transforming your continuous value into a category. For example, for a variable Age, you define the following intervals : [0;10[, [10;20[, ... [90;100[. If someone is 36 yo, he'll be on the interval [30;40[. You'll then only have 10 categories. The complicated part is to find what are the best intervals to lose avoid losing to much information

Categorical to Continuous

You can apply some Sklearn Category Encoder, which, knowing your targer on training set, will transform each category to a value between 0 and 1. This value is calculated using statistics (different for each encoder), and illustrates how often your category ends up affecting your target.


How to manage missing data in meteorological time series?

It is a question related to the domain of your project. You should know the cause of the missingness.

If some values are missing, because there is no applicable measure then this values are a special case, therefore they're missing not at random. In such case you can impute missing values with the mean value of the non-missing data and add another feature, which indicates special cases.

On the other hand, if the value is missing, because some sensor temporarily stopped working, then it is missing at random, so the measured value is just not known. In this situation you can perform linear regression (or any other regression, but you should start with the linear one) between non-missing data.

In case your problem has mixed types of missingness you should also perform some regression.


How to do webscrapping in R on this webpage?

This should be possible with  rvest in R. Two things make is possible

URL pattern is predictable, https://www.ecb.europa.eu/press/pressconf/2012/html/index.en.html    (replace 2012 with other year values)
Html page applies predictable CSS for INTRODUCTORYSTATEMENT_CONTENT (E.g.: doc-title and doc-subtitle )

Following articles have examples :

https://towardsdatascience.com/web-scraping-tutorial-in-r-5e71fd107f32
https://www.datacamp.com/community/tutorials/r-web-scraping-rvest
https://www.analyticsvidhya.com/blog/2017/03/beginners-guide-on-web-scraping-in-r-using-rvest-with-hands-on-knowledge/


LinearRegression with multiple binary features sometimes performs poorly

You should always consider normalizing your output to some predefined range, otherwise there is a possibility of the gradients exploding as the loss will be of high magnitudes. It also becomes hard to output such a wide range. Try transforming your output using some StandardScaler, or a RobustScaler if there are significant outliers, and try again.


How to determine if a neural-network has a static computation graph?

I believe current implementations of the best known libraries will allow you to implement any network you come across. So it boils down to the aspects of taste and practicality.

As far as deciding on a framework based on the network you want to create; if the network has branches or separate control flows (data transfer or logical) that are used conditionally during training, PyTorch is likely more useful. Otherwise, the best starting point is likely to use Keras - a high-level API over Tensorflow.

The (very) general rule of thumb is:

  Use Pytorch for experimentation, and Tensorflow for production

Now the caveats to the above sweeping statement:

Both libraries are making massive efforts to cover "the other" library's strengths:

The latest release of Pytorch (v1.0) offers several methods of making production ready code - so ways of compiling it to improve performance as well as the use of 16-bit floating point operations (as opposed to the standard 32-bit). They have also started rolling out a C++ API. These are all steps to make full-scale products for deployment. Tensorflow has introduced its eager API, for eager execution, meaning you can essentially build dynamic graphs, just as in PyTorch. The enables faster and more interactive prototyping.

The experience you have while developing might be more important:

If you are a pure Python developer, PyTorch will feel more natural. If you have been a general software engineer in other statically types languages (such as C++ and Java), Tensorflow may feel more logical.

Summary

Overall, Tensorflow along with (the now built-in) Keras wrapper, is a more mature library with a larger community and longer history, meaning more support online. PyTorch has a higher level of excitemnet around it, as far as I can tell. It is used very actively in research and newer open-source projects/courses, which indicates it may become a dominant force.


Intuition behind using the inverse of a Hessian matrix for automatically estimating the learning rate (aggression parameter) in gradient descent.

It's not an intuition, it's mathematics. And more precisely, the quadratic expansion of a function.

Say you have $f(x)=f(x_n+\Delta x)=f(x_n)+f'(x_n)\Delta x + f''(x_n)\Delta x^2$ and you want to get to a point where $f'(x) = 0$, then you will use $\Delta x=-\frac{f'(x_n)}{f''(x_n)}$.

Generalizing to n dimension, $f''$ is the Hessian of the equation.

So of course this works perfectly for least squares, it will only "work" close to the solution for non quadratic functions.


Using TF-IDF with other features in scikit-learn

scikit-learn's FeatureUnion concatenates features from different vectorizers. An example of combining heterogeneous data, including text, can be found here.


Could I use logloss as classifier performance measures when "Test & Score"?

Hover on the scoring header and use right click. This should open a window where you can select logloss as well.


Using simulations to train ML algorithms

Is it possible to use data generated by a huge number of simulations to train a classification algorithm to perform this detection online?

Yes, it is always possible to train a classification algorithm when you have labeled i.i.d. training data, and there is no hard reason why you cannot use a simulator to generate that.

Whether or not such a trained model is fit for purpose is hard to say in advance of trying it.

Using a simulation as your data source has some benefits:

Generating more training and test data is straightforward.
You will automatically have high quality ground truth labels (assuming your goal is to match the simulation).
If you find a problem with certain parameter values, you can target them when collecting more training data.

Just as with data taken from real world measurements, you will need to test your results to get a sense of how accurate your model is.

  What are the considerations when using simulated data to train an algorithm that will then be used online with real data (expect from the obvious that the simulation needs to be very very accurate)?

Your model is a function approximator. At best it will match the output of the simulator. In practice it will usually fall short of it by some amount. You will have to measure this difference by testing the model, and decide whether the cost of occasional false negative or false positive is outweighed by the performance improvement.
Statistical machine learning models perform best when interpolating between data points, and often perform badly at extrapolating. So when you say that inputs can vary infinitely, hopefully that is within some constrained parameter space of real values, as opposed to getting inputs that are completely different from anything you have considered before - the simulation would cope with such inputs, but a statistics-based function approximator most likely would not.
If your simulation has areas where the class switches rapidly with small changes in parameter values, then you will need to sample densely in those areas.
If your simulation produces near chaotic behaviour in any region (class value varies a lot and is highly sensitive to small changes in value of one or more parameters), then this is something that is very hard to approximate.
If you have some natural scale factor, dimensionless number or other easy to compute summary of behaviour in your physical system, it may be worth using it as an engineered feature instead of getting the machine learning code to figure that out statistically. For instance, in fluid dynamics, the Reynolds number can characterise flow, and could be useful feature for neural network predicting vortex shedding.

  Any references to such examples?

The examples I have found here are about are in renderings of fluid simulations and other complex physical systems where a full simulation can be approximated and they all use neural networks to achieve a speed improvement over full simulation.

Accelerating Eulerian Fluid Simulation With Convolutional Networks - I saw a video about this on YouTube's Two Minute Papers channel.
Using neural networks in weather prediction ensembles to improve performance
Fast Neural Network Emulation of Dynamical Systems for Computer Animation

However, I don't think any of these are classifiers.


Error opening Iris.tab file in Orange load data widget

This was an error in Orange 3.3.11, and fixed in release 3.3.12. Update to the newest release to get rid of the error.


Anonymize continuous variable for masking purposes

I think you can use a much more complicated monotonic transformation, like
log(1.234578 + sqrt(x + 7.4142) ** 3)

which will be harder to invert than a simple log. But, as Nikos says, strictly monotonic functions are invertible, so all you can do is make it very hard to compute the inverse by composing many monotonic functions.


Implementing single variable Linear Regression in python

The Equation for Least Square method shall be as below-

theta(0)+theta(1).X , since you have 1 variable.

if theta(0) =0 and theta(1)=0 since you are adding it theta = np.zeros(shape=(2, 1)).

the value of Y shall be 0 hence error is 0.

To breakdown nicely you can add it like-

n = X.shape[1]
theta = np.zeros((1, n))

Now in the next run the values of theta shall change as per learning rates and in then the errors shall change too, that part you haven't reached yet, by looking at your code if I might add.

Here have a look-

https://towardsdatascience.com/linear-regression-using-python-b136c91bf0a2


How to choose dimensionality of the Dense layer in LSTM?

This is a very broad and difficult question, because these parameters depend completely on the complexity of your dataset. There is (currently) no "scientific method" to determine these parameters. The best solution is to apply a form of hyperparameter optimization.

See this similar post for more answers.