Why do we use different image processing at train and test time for an image classification tasl

The key difference is that for training the preprocessing serves the purpose of data augmentation: it increases variance in the training data and thereby helps to better generalize. That is why you will find random cropping, flipping, rotating and zooming here.

But during inference that is not required and these random preprocessing steps would not be helpful. Therefore, you only do the preprocessing steps required  and eliminate randomness (e.g. crop images to the correct input size with centercrop instead of randomcrop).


Anomaly detection - relation between thresholds and anomalies

To give a helpful answer more information is necessary. When you want to detect anomalies, I would say, from my feeling and understanding, that it doesn't make sense to train the model every day. You should define your normal system once and try to catch anomalies from there. Trends are hard to catch but this is always hard.

Identifying anomalies means also you have to know your system but here I have to refer to the first sentence: What is your system? Is it a technical system (then the task becomes quite complex) or is it "just" some pattern recognition or fraud detection and so on?

However, when I did that for a large machine with ~60 variables (and 12 months of data), I used the first 3 months to train and defined this state as normal. Then I set the thresholds such that an anomaly is identified when the RMSE reaches $ \sqrt{max (X_i)} $. This worked quite well with the exception of 1-2 variables.
But as I said, it depends on your situation. For example, when you have a steam machine and the normal temperature is at 98 °C and everything bursts at 101 °C, you probably have to use a custom threshold as e.g. a standard deviation wouldn't help here much.

To add some more information: Especially in case of anomaly detection ,it is often the case that one approach is not sufficient. I would implement some basic approaches like using standard deviations for the most trivial cases which definitely have to be detected and the more advanced methods (like LSTM) to track more complex patterns.


partial numerical array - pattern matching

According to your definition (consecutive, order matters, max +/-2 difference), it's not a fuzzy matching case. It's just a minor variant of searching a subsequence:

for i=0 to len(source)-len(test) {
  j=0
  while (j<len(test)) && (abs(source[i+j]-test[j]) <= 2) {
    j++
  if (j == len(test)) {
     // match found
  }
}

This is the simple version, in case efficiency is an issue there is a more efficient version using dynamic programming.


In the Rainbow approach, theoretical correctness of the off-policy return values is completely ignored, and it just uses:

$$G_{t:t+n} = \gamma^n\text{max}_{a'}[Q(S_{t+n},a')] + \sum_{k=0}^{n-1} \gamma^{k}R_{t+k+1}$$

It still works and improves results over using single-step returns.

They rely on a few things for this to work:

$n$ is not large, compared to amount of variability in the environment that occurs over those steps. So the return is usually correct even if there was an exploratory action taken. Many game events (and also real-world physics) have some fuzziness - e.g. as long as in Pong you hit the ball with a part of the paddle, it will be a reasonable result, and only some (literal) edge cases will cause a major difference in rewards.
Exploration is low ($\epsilon$ set to $0.01$ or $0.001$)
Rewards are sparse, so the n-step return very often only depends on state progression. As long at $S_{t+n}$ is as likely to be reached starting from $S_t$ under both the exploratory and target policies, then the simple returns are approximately correct.
The policy changes slowly, maybe taking a million frames to converge to optimal, which is far larger than the replay memory.

From a theoretical basis in Q learning, it would be better to use "up to n-step" returns that truncate at the time step when the action taken diverges from the  current learned policy. However, that is more complex, as you have to assess the learned policy at each of the n steps in order to decide when to truncate - I suspect given the researchers know how $Q(\lambda)$ works, that they may have tried this and just found the extra correctness was not worth the extra complexity and CPU cost. In other environments it may be worth the effort to implement this.


I have to quote Tukey, perhaps the grandfather of data science:

  The combination of some data and an aching desire for an answer does
  not ensure that a reasonable answer can be extracted from a given body
  of data.

I see nothing wrong with your Poisson model. In fact its a pretty good fit to the data. The data is noisy. There is nothing you can do about it. Perhaps the noise if due to whatever else is on TV at the time, or the weather, or the phase of the moon. Whatever it is, its not in your data.

If you reasonably think the weather might be affecting your data, get the weather data and add it. If it decreases the log-likelihood enough for each degree of freedom then it's doing a good job and you leave it in. This is regression modelling 101.

Of course there's a zillion other things you can do. Scale the data by any old transformation you want. Fit a quadratic. A quartic. A quintic. A spline.  You could include the date and possible temporal correlation effects. But always bear in mind what Tukey was saying - if your data is noisy, you won't get anything much out of it. So it goes.


Orange Geo Map does not show background map

The issue with the Geo Map widget has been solved in a new release of Orange3. The Geo Map widget now works correctly again in Orange3 version 3.23.1 and Geo Add-on version 0.26. Great!


Training a model, related to information extraction, in general, and named entity recognition/resolution (NER), in particular, is described in detail in Chapter 7 of the NLTK Book, available online at this URL: http://www.nltk.org/book/ch07.html.

Additionally, I think that you might find useful my related answer on Cross Validated site. It has a lot of references to relevant sources on NER and related topics as well as to various related software tools.


Regarding your second question:

  The users who did not have and tweet for the time range, is this okay to assign difference in mean zero, or I should exclude them from the samples?

You essentially have missing data in this case. How you can deal with this will depend on the model you are using, and if it is robust to missing data. If the model can ignore $\mu = 0, \sigma = 0$ values, then try it out. Otherwise, you might want to leave them out as you suggest, or perhaps even impute them with their previous known values.
If you are e.g. using something like an ARIMA model, then it keeps track of a moving average. In this case, using zero values will have an undesired impact (assuming zeros are not common in general).

I'm not sure I understand what you are asking in your first question. What have you tried already? Have you got some results?


You first need to specify if you are going to work on anomaly detection or novelty detection.

The difference between them is that in anomaly detection, your data contains anomalies, but in novelty detection, your data contains only valid data.

After that you need to choose the algorithms depending on what you have chosen before, there are some algorithms that work only with one type, and there are algorithms that work with both types.


Splitting a pdf containing batch of scanned documents

"Machine Learning for Digital Document Processing: from Layout Analysis to Metadata Extraction" by Esposito, Ferilli, Basile, and Mauro goes into detail about how to create a custom system for parsing digital documents, including pdfs. It proposes a generalized process to learn any structure within documents.


Remove Outliers - Market Basket Analysis

Since I cannot comment to ask for clarification, I am asking it here. What is your reason to think about removing the most frequent value in your dataset? If the second column actually represent frequency of usage, you probably should not remove it and I think it would be illogical to throw away that piece of information. Having said that, you may consider removing that tag if it is a "less meaningful" word (e.g. a, an etc).

Can you give a little bit more context on what you are trying to achieve?

In general, one way to find outlier is to look at points that lie beyond 1.5 times of inter-quartile range of the distribution i.e. for the frequency count in your data.

Just a quick thought, did you try clustering for finding similar tags? What are the ways you are considering to find similar tags?


why does performance of machine learning models plateau after certain amount of training data

In part feature learning has a lot to do with the fact that deep learning models can learn more data distributions. Although the graph that you are showing isn't completely true. For instance you can see the No Free Lunch Theorem for machine learning, that states that no machine learning model is universally any better than any other.

Either way, the goal of machine learning is not to seek a universal learning algorithm. Instead our goal is to understand what kind of dat distributions are relevant to the the real world that an AI agent experiences, and what kinds of machine learning algorithms performs well on data drawn from the kinds of data generating distributions we care about.

In conclusion, when facing a machine learning problem you should try to find the model that best learns the kind of data you are trying to work with and this graph is poorly informative and has no  foundations.


What do mean and variance mean for high dimensional data?

It's just the expansion of one dimensional mean and standard deviation. Suppose that you are trying to estimate the weight of a person and you have two inputs, salary and height. For finding the weight, you have two inputs which are of different scales, so you try to find the mean and variance of each feature, salary and height, separately using the data samples. Suppose you have two data samples in a tuple like (salary, height). They are as follows:

  (10000, 180)
  (5000, 175)

Although the first guy may seem so rich, the real point about the first and second data samples is that their scale is not the same. The approach is like this:

Find the mean of first feature among data samples, salary here. (10000 + 5000) / 2 = 7500
Find the mean of second feature among data samples, height here. (180 + 175) / 2 = 177.5
Find the standard deviation of first feature, among data samples, which is 2500 for salary.
Find the standard deviation of second feature, among data samples, which is 2.5 for height.
Reduce the amount of mean from corresponding feature for each data sample and divide each with the standard deviation of corresponding feature.
(10000 - 7500) / 2500 = 1
(5000 - 7500) / 2500 = -1

(180 - 177.5) / 2.5 = 1
(175 - 177.5) / 2.5 = -1

Data samples which have been normalized now are like as follows:

  (1, 1)
  (-1, -1)

Whenever you normalize your data, your cost function would be so easier to learn. The weights don't have to struggle to reach to high values in situations where your features are not in same range.


GridSearchCV vs RandomSearchCV and How it works?

Imagine the following scenario:

params = {
   epoch = [20, 30, 40, 50], #those numbers are only for example
   dense_layer_size = [20, 30],
   second_danse_layer = [30, 40]
}

In GridSearch you try all the combinations of your parameters, in this case:

(4 * 2 * 2) = 16 #Total of parameters

In RandomSearch as in documentation:

  Not all parameter values are tried out, but rather a fixed number of parameter settings is sampled from the specified distributions. The number of parameter settings that are tried is given by n_iter.

In this case, he select "n_iter" number of combinations and try then. Is good for optimizing less parameters, but, in cases when you don't have sure about multiples parameters, may let aside some combinations that would be better.

Another good approach, is Using an genetic algorithm to optimize your network.

Genetic algorithm generate a few combinations of "individuals" (combination of parameters) and at each step, select the best individuals(called parents) and crossover then, generating more individuals with the characteristic from both parents. This way, in some case you can optimize your network and add random elements when you don't have sure about what parameter would be increase your result.

If you want an easy to integrate genetic algorithm you can check this project.


Pattern Recognition (Bishop) - Maximum Likelihood

The partial derivative you have calculated is incorrect. You are probably confused by the index k in the equation.
Consider expanding the Langrangian:
$$ L = m_{1}ln\mu _{1}+ m_{2}ln\mu_{2} + ... + m_{K}ln\mu_{K} + \lambda(\mu_{1} + \mu_{2} + ... + \mu_{2} - 1) $$

Now, take the derivative with respect to $ \mu_{1} $

$$ \frac{\partial L}{\partial \mu _{1}} = \frac{m_{1}}{\mu _{1}} + \lambda $$

Then, in general,
$$ \frac{\partial L}{\partial \mu _{k}} = \frac{m_{k}}{\mu _{k}} + \lambda $$

Equate it to zero to get $ \mu _{k} = -m_{k} / \lambda $

Now substitute it in the constraint
$$ \sum_{k=1}^{K}\mu_{k} = \sum_{k=1}^{K}\frac{-m_{k}}{\lambda} = 1 $$

Then $ \lambda = \sum_{k=1}^{K}-m_{k} = -N $

Finally, you get your answer
$$ \mu_{k} = \frac{m_{k}}{N} $$


Essentially, each non-linear layer in a neural network is a map from $\mathbb{R}^n$ input to a $\mathbb{R}^m$ output. There is no requirement for these dimensions to be separate and uncorrelated, nor for the map to be reversible.

The best performance is usually found if each layer is roughly scaled to approximate a normal distribution with mean 0 and standard deviation 1. However, that is an independent issue to any kind of space-like mapping.

If you know a suitable mapping that relates to your problem and would simplify its expression, then you can apply a spacial transform between co-ordinate systems (typically by mapping the input features) and use it within the neural network. You could map co-ordinates to any curved space if you think it might help. Choosing between polar or cartesian co-ordinates for features is one choice that may crop up for example.

However, even without making this helpful mapping, you can analyse a neural network layer-by-layer, and see that it can effectively learn these kind of spacial mappings from the training data anyway. This is an intuition that has been called the "manifold hypothesis" - the link has some images showing how a neural network might learn to separate classes depending on how they are arranged in feature space, and this article from a DARPA researcher has a nice series of images showing successive transformations of a 2D spiral that allow the last layer to perform linear separation.


How to select the learned model using $k$-fold cross validation?

$k$-fold is just performed to obtain a measure of your accuracy, as using the training accuracy is generally a too optimistic measure of the accuracy. If you want to deploy a final model, what is recommended is to train a last model with all the data. In fact, when you compare two models, $f$ and $g$, what you do is obtain two error rates $r_f$ and $r_g$ by cross-validation, and you keep the model with the lowest error rate. After that, if the model with the lowest error rate is $f$, you retrain $f$ with all your data.

To sum up, $k$-fold cross-validation is a method to give measures of the performance of the model, if you want to obtain the best model just train it with all your data.


Prediction interval for general non-linear regression

Bootstrapping (which I think you are referring to as bagging, a specific algorithm that incorporates bootstrapping) is your friend here.

Simply train N models using N datasets, where these N datasets are created by sampling from your original dataset with replacement. Using these N different models, generate N predictions on an observation of interest.

You can now use any sort of method to derive the confidence interval of interest with these N predictions, whether that is using a normality assumption, empirical percentiles (simple bootstrapping), etc.

The one drawback I see with this method is that if you have many observations to predict, this will quickly become computationally expensive. That being said, bootstrapping is easily made parallel.


Well the names are pretty straight-forward and should give you a clear idea of vector representations.

The  Word2Vec Algorithm builds distributed semantic representation of words. There are two main approaches to training, Distributed Bag of Words and The skip gram model. One involves predicting the context words using a centre word, while the other involves predicting the word using the context words. You can read about it in much detail in Mikolov's  paper.

The same idea can be extended to sentences and complete documents where instead of learning feature representations for words, you learn it for sentences or documents. However, to get a general idea of a SentenceToVec, think of it as a mathematical average of the word vector representations of all the words in the sentence. You can get a very good approximation just by averaging and without training any SentenceToVec but of course, it has its limitations.

Doc2Vec extends the idea of SentenceToVec or rather Word2Vec because sentences can also be considered as documents. The idea of training remains similar. You can read Mikolov's Doc2Vec paper for more details.

Coming to the applications, it would depend on the task. A Word2Vec effectively captures semantic relations between words hence can be used to calculate word similarities or fed as features to various NLP tasks such as sentiment analysis etc. However words can only capture so much, there are times when you need relationships between sentences and documents and not just words. For example, if you are trying to figure out, whether two stack overflow questions are duplicates of each other.

A simple google search will lead you to a number of applications of these algorithms.


Example of binary classifier with numerical features using deep learning

I used Binary classification for sentiment analysis of texts. I converted sentences into vectors by taking appropriate vectorizer and classified using OneVsRest classifier.

On another approach, my words were converted into vectors and there, I used a CNN based approach to classify. Both when tested on my datasets were giving comparable results as of now.

If you have vectors, there are already really good approaches available for binary classification which you can try. On Binary Classification with Single–Layer
Convolutional Neural Networks is a good read for you for classification using CNNs for starters.

This is one of the first blogs I read to gain more knowledge about this and doesn't require much of pre-requisites to understand(I am assuming you know the basics about convolution and Neural Networks).


Similarity measure based on multiple classes from a hierarchical taxonomy?

While I don't have enough expertise to advise you on selection of the best similarity measure, I've seen a number of them in various papers. The following collection of research papers hopefully will be useful to you in determining the optimal measure for your research. Please note that I intentionally included papers, using both frequentist and Bayesian approaches to hierarchical classification, including class information, for the sake of more comprehensive coverage.

Frequentist approach:

Semantic similarity based on corpus statistics and lexical taxonomy
Can’t see the forest for the leaves: Similarity and distance measures for hierarchical taxonomies with a patent classification example (also see additional results and data)
Learning hierarchical similarity metrics
A new similarity measure for taxonomy based on edge counting
Hierarchical classification of real life documents
Hierarchical document classification using automatically generated hierarchy
Split-Order distance for clustering and classification hierarchies
A hierarchical k-NN classifier for textual data

Bayesian approach:

Improving classification when a class hierarchy is available using a hierarchy-based prior
Bayesian aggregation for hierarchical genre classification
Hierarchical classification for multiple, distributed web databases


What's the best graphic in R to study air pollution?

Well you can just start with simple line graph. Each line in graph will represent Specie. The X axis will be years from your data and Y axis will be the median. I am not sure why you have median and not actual data but this is how you can chart with visualization. Use this link to start.


I've tangled with modeling pricing systems over the last two years and one of my key learnings applies here:
Available sales data is often a bad basis for straight-forward prediction tasks and the reason for this is fairly simple:
If you classify all prices (or transactions of a given product at a price x) into "Accepted" and "Not accepted" by the customer you will realize that the data provided by your customer only contains instances of "Accepted" prices.
Therefore a straight-forward modeling of y ~ x with y = demand and x = price is impossible because your y does not vary!
There are several ways around this however. In my comment I mentioned discount being a valuable information!
You have already noticed that discounts are not depended on logical variables, they are seemingly random, this isn't true!
Discounts in most organizations are very, very flexible and often applied manually based on negotiations. This means they are a great indicator of our target y "Acceptance"/"Non Acceptance".
Consider this:
Discount = Demand x Undiscounted_Price
This means that large discounts indicate that the demand is low / negative for the undiscounted price and low discounts indicate demand is high.
To truly discover this relation you might need to model codependent factors and then remove them by training new models on the residuals.
Edit:
An important thing to add especially for the B2B domain. Demand for a certain product is almost set in stone for a customer. Unlike consumers companies do not buy surplus or refrain from purchase due to the prices.
What they do is switch suppliers! This means that there is a really important unknown variable "Customer Demand for product X". You do not want model this variable but you need it to model what you actually want to do:
Share of Wallet or the percentage of the fixed customer demand that was satisfied by your company with the goal being to identify the price that will optimize that percentage. This is helpful to keep in mind because it constrains the performance of any model as you never know whether the historic demand you have in your data is already 0% or 100% of the total demand and thus could not decrease/increase regardless of price.


A few comments:

The term "POS" (Part Of Speech) has the specific meaning of "grammatical category" so it's not really what you mean here, you should probably say "custom entities tags/categories/labels".
Your plan makes sense, but be aware that that you're addressing a really difficult problem so even with a really good dataset and method it's not going to work perfectly (probably far from it).
You're probably going to need a very large amount of annotated data for this to work, because there are many different ways to express a medical problem so the model will need many examples to extract the relevant patterns.
The standard way to address this kind of problem is a bit different when there are terminology  resources for the language: the medical terms can be annotated directly simply by pattern matching. If you have such resources and they are complete enough, this saves you the need to manually annotate and train a custom model.
Doing a recommender system on top of that is a second problem, and I think you need to think carefully about the goal:

if it's meant to be used by non-experts, it wouldn't make sense to recommend research articles.
The vast majority of the research literature is published in English, so there would be an additional problem to address here.
In case you're considering the whole medical literature, you might have a complexity issue because it's huge (PubMed contains around 30 millions abstracts, and that's without considering the full articles in PMC)
What would be the basis for recommendation? Is it based only on matching the terms in the query with the documents?


weight for each node?  $w_i$

weight for each model? $\alpha_i$

table weight column (more confusing)? The initial weight is set as $\frac{1}{M}$ where M is the number of data points. I assume the data2_1 is the update of data1_1 after being updated by a new weight for feature1 but it's a bit obscure due to lack of description about the title, etc. of the table.

weight for each node is being updated after a classifier is trained on the data and it's weight ($\alpha_i$) and the score is calculated.

$T(x_i)$ is the final classifier but seemingly is lacks a sign before the expression. which is the sum of multiplication of each classifier with it's weight $\alpha_i$


What does many low important feature indicate?

Adding low-value features might not help you surpass your current accuracy.
Getting good quality data and add more data to the dataset or training for more epochs if it doesn't converge might help you get more accuracy.


How to justify a predictor in influencing the outcome?

Cross Validation

You had a post where we discussed causality, but with ML models assumption is that data represents your problem entirely and has all the information in it. In other words every pattern that you pickup in your train data you can expect it to behave pretty similar in production, hence with this assumption what you want is to "evaluate the entire train" (thats what you can do with CV.) and if it score good on average over all folds you want to add this change (be it a new feature)


Is it possible to use grayscale images to existing model?

It depends on the model. You'll have to dig into the model definition and see.

It's not uncommon for models to grayscale images in pre-processing. In that case, you will be fine.

However, many recent deep convolutional models operate on RGB images as 3D tensors (4D with the batch dimension). In this case, you should consider modifying the model so that it operates on a single color channel before training it. Multi-channel convolution is discussed in the convent's chapter of Goodfellow's Deep Learning.


There isn't a particular approach or model that is definitively the best. You might want to try multiple types of models. A good starting point would be to visualize your data. If there are many non-linear relationships between your data and the the outputs you are predicting then you might want to start with a tree based model. It will be much less work to make it perform well. If the relationship between your variables and the outcomes you are predicting are fairly linear then a logistic regression model is a good approach.

If you have to have a silver bullet model to solve all your problems then I recommend trying xgboost. Use grid search and you will have a good model.

I actually worked on a similar project for my university. Aggressive feature engineering combined with logistic regression performed and xgboost were the two best models.


Who invented the concept of over-fitting?

Yes! It seems like first statistics reference to over fitting appears in The Quarterly Review of Biology  . It says
"Perhaps we are old
fashioned but to us a six-variate analysis
based on thirteen observations seems rather
like overfitting.
I am attaching the screenshot of that particular page for your reference.


How word embedding work for word similarity?

I think you have it mostly correct.

Word embeddings can be summed up by: A word is known by the company it keeps. You either predict the word given the context or vice versa. In either case similarity of word vectors is similarity in terms of replaceability. i.e. if two words are similar one could replace the other in the same context. Note that this means that "hot" and "cold" are (or might be) similar within this context.

If you want to use word embeddings for a similarity measure of tweets there are a couple approaches you can take. One is to compute paragraph vectors (AKA doc2vec) on the corpus, treating each tweet as a separate document. (There are good examples of running doc2vec on Gensim on the web.) An alternate approach is to AVERAGE the individual word vectors from within each tweet, thus representing each document as an average of its word2vec vectors. There are a number of other issues involved in optimizing similarity on tweet text (normalizing text, etc) but that is a different topic.


What is meant by Average Content Distance in Videos generated by GANs?

This article presents 2 ways to implement ACD metric. You talk about ACD-I (using the article's notation). As I understood,

you first average colors, i.e. sum over pixels in the image plane, for each color in all frames: $\mathrm{avg}_i = \frac{1}{MN}\sum_{x,y}\mathrm{Frame}_i $.
then, in the resulting 3D vectors for every two consecutive ones, you calculate L2 distance: $d_i = \sqrt{\sum_{l=1}^3(\mathrm{avg}_{i,l} - \mathrm{avg}_{i+1,l})^2}$. In general, you might want to use other distances, which is not prohibited.
sum it up (since we take into account all frames we have, not just one pair) and divide by the number of frames (because the metric should not depend on it [at least, I suppose so], otherwise longer videos will have larger metric values): $\mathrm{ACD} = \frac{1}{K-1}\sum_i d_i$ (if there are K frames).

ACD-C is obtained in the same way but instead frames you use feature vectors extracted from images (frames) with 'encoding-like' network. OpenFace probably is a good choice when dealing with facial expressions.

Your formula might work differently from the authors' intention. Imagine a white spot on a black screen. From frame to frame that spot is gradually moving from one side to the other. Your metric shows that the content in this video is changing. And the faster the spot moves, the greater changes happen (ok, in case of a spot and a black screen at a certain speed the saturation point will be reached and your metric will stop changing [when in one frame the spot moves a distance equal to its diameter], but that's another story). However, in fact, there are no changes in the content. You still have the spot and the black screen. This is why you need averaging (not summing 'changes' and making it independent from the image size - this is what your formula does)

Here you can see the implementation of ACD metric(s). I cannot vouch that is 100% correct. So, let me know if there are any kind of uncertainties.


Orange wont allow me to set target for corpus

Can't you do something like this? (or am I missing something?)


What are the possible values of a filter in a CNN?

Weights are learned with backpropogation, that's how model learns to identify different patterns with data. The point of a supervised model is to learn weights.

They can be floating value, they are usually represented as [[101][101][101]], because it identifies vertical line, and it's more intuitive to understand.

The filter usually do not contain info on depth, they are square matrix with depth equal to number of channels in input layer, with each filter layer spewing one output layer, so to get 4 output layers, you need to have 4 filters

Stride=2 means it will skip 2 every time, that is, it will go from 0,1 to 3,4


What is the best way to normalize histogram vectors to get distribution?

It's okay, what you are saying, except the

  each data point is subtracted from its means

part because it's in the opposite way, you have to subtract the means from their data points.

I guess your problem is coming from that you put your vectors into the matrix as row vectors, but np.mean(..., axis = 0) calculates the column means.

If you want to use the above formula for normalizing then you should transpose your matrix.

vectors = np.array([[0,4,0,0,1],[5,0,0,3,1],[1,0,0,0,0],[0,0,6,0,0]])
vectors.T
array([[0, 5, 1, 0],
       [4, 0, 0, 0],
       [0, 0, 0, 6],
       [0, 3, 0, 0],
       [1, 1, 0, 0]])

And now the np.mean(vectors.T, axis=0) gives the correct means for your vectors:

array([ 1. ,  1.8,  0.2,  1.2])

Substract the means from the transposed matrix like vectors.T - np.mean(vectors.T, axis=0)

array([[-1. ,  3.2,  0.8, -1.2],
       [ 3. , -1.8, -0.2, -1.2],
       [-1. , -1.8, -0.2,  4.8],
       [-1. ,  1.2, -0.2, -1.2],
       [ 0. , -0.8, -0.2, -1.2]])

And finally divide it by the standard deviation of the transposed matrix

t_vectors = (vectors.T-np.mean(vectors.T,axis=0)) / np.std(vectors.T,axis=0)
t_vectors
array([[-0.64549722,  1.65027399,  2.        , -0.5       ],
       [ 1.93649167, -0.92827912, -0.5       , -0.5       ],
       [-0.64549722, -0.92827912, -0.5       ,  2.        ],
       [-0.64549722,  0.61885275, -0.5       , -0.5       ],
       [ 0.        , -0.4125685 , -0.5       , -0.5       ]])

Sklearn gives the same sklearn.preprocessing.scale(vectors.T).


Test dataset with categorical variable value not present in train dataset & transformer

I usually replace unseen and NaN values with the global target mean.

There are also already implemented transformers for target encoding that you could use that gives you some options such as smoothing: scikit contrib - target encoder.


Correct interpretation of summary_plot shap graph

I think your interpretation is not entirely correct. Loosely rephrasing Lundberg et al. [arXiv:1802.03888], the SHAP value of feature $i$ is
$$ E[f(x) \mid S \cup \{i\}] - E[f(x) \mid S] $$
averaged over all possible subsets of features $S$, $i \notin S$. Here $f(x)$ is the prediction of the model for inputs $x$. Figure 2 from the preprint is a good illustration:

We are interested in the (signed) length of arrow for a particular feature, averaged over all permutations of the features. For a linear model $y(x) = w^Tx$ the SHAP value of feature $i$ would be simply $w_i x_i$.

Now to your items:

SHAP value of 0 for some feature means that in the current example the value of that feature is ignored by the model. In the linear model this feature's weight would be 0.
SHAP value of 4 means that the value of that feature in the current example increases the model's output by 4. Let me use your summary plot as an illustration. It was produced with Boston housing data, and we can check in the dataset that the maximal value of feature ‘RM’ (average number of rooms per house) is around 9. The plot shows that the brightest shade of red for this feature corresponds to SHAP values of around 3, 4, and 8. This means that having 9 rooms in a house tends to increase its price by 3, 4, or 8 thousand USD.
The summary is just a swarm plot of SHAP values for all examples. The example whose power plot you include below corresponds to the points with $\text{SHAP}_\text{LSTAT} = 4.98$, $\text{SHAP}_\text{RM} = 6.575$, and so on in the summary plot.


Interpreting the Root Mean Squared Error (RMSE)!

How can I interpret RMSE?

RMSE is exactly what's defined. $24.5 is the square root of the average of squared differences between your prediction and your actual observation. Taking squared differences is more common than absolute difference in statistics, as you might have learnt from the classical linear regression.

  It confuses me a little. And I could not find any reliable reference
  to also clearly state that one can safely interpret RSME as one does
  MAE. Is RMSE is simply a only mathematically more convenient for
  optimization etc., and we are better off with MAE for the
  interpretation?

I think this post should help you. I'll answer your questions directly:

RMSE is easier mathematically and also practically. Have you heard of derivative? The derivative for MAE is undefined when your prediction is equal to observation but well defined everywhere for RMSE. In machine learning, a well defined gradient function is generally better.
Both RMSE and MAE are useful, but they are two very different metrics. In regression, it's generally about choosing between linear regression and quantile regression. They are two very different models!
As stated in the link, if you don't want your residuals affect your model too much, MAE could be better. Otherwise, if your data set is well defined (not many residuals), RMSE could be better.

There is no right or wrong on which one is better. Think like two different algorithms for doing modelling.


Understanding dimensions of Keras LSTM target

If num_steps is set to 5, the data consumed as the input data for a given sample would be “The cat sat on the”. In this case, because we are predicted the very next word in the sequence via our model, for each time step, the matching output y or target data would be “cat sat on the mat”.

In this example, y is a sequence of words with length num_steps starting at the very next word. The input and output both have length num_steps.

Keras will return a sequence for the output when return_sequences is True. https://keras.io/layers/recurrent/#lstm

  return_sequences: Boolean. Whether to return the last output in the output sequence, or the full sequence.

In general, the input to an LSTM has dimensions (batch, time, features). The batch dimension should not be dependent on time. The output can either be a sequence or a single number, which in Keras is handled by return_sequences.


For a multi-class classification problem, how to transform the target variable to a form that is usable by sklearn algorithms?

Welcome to the site! Do a google search for "one hot encoding" and all will become clear.

In a nutshell, don't think of it as numbers "assigned" of 0-159. It's more about a "slot" for each category and using one hot encoding to let the algorithm know which slot a particular record belongs to.


Is boosting resistant to overfitting for both number of iterations and number of features?

I can't speak from a theoretical perspective, but I can say that in practice overfitting with a large number of features is not an issue with gradient boosting. I work at a huge financial institution that has been doing credit risk modeling for decades, and for the past decade or so they have almost exclusively been doing their variable selection using gradient boosting. Gradient boosting (along with any tree-based method) can be used to find relative feature importances (based on how much error is reduced after each split).

Statisticians where I work will often use gradient boosting to narrow down 2,000 or so features into a more manageable 10-15 by taking the top ranked features. These features are then fed into a logistic regression model, which is more interpretable than gradient boosting. Even on the hold-out partition, the gradient boosting algorithm built on the 2,000 features generally outperforms the logistic regression model. The gradient boosting model is thus used for variable selection and predictive performance benchmarking.


In Q-learning, how to tell the agent that action $a_7$  is unavailable from within state $s_{t}$?

Partially, it depends on what you mean by "unavailable". If the environment is such that it is possible to take the action, but that the consequences are very bad, then a negative reward due to the consequences is probably the best option. You might do this for a real-time system where the action can actually be taken, cannot be automatically blocked in any way, and has unwanted consequences.

Perhaps an action is unavailable, because it is not possible to even attempt it in the supplied environment. This could occur in a board game where certain moves are not allowed by the rules of the game. There you have a few options, depending on how you have constructed the agent.

A simple approach is to only present the agent with actions that it is allowed to take in the first place. The code for the environment should already know this list, so if you were considering penalising the choice, then you can also choose not to present it. This works well for value-based methods such as Q-learning, when you have implemented $Q(s,a)$. If the set of allowed actions is $\mathcal{A}(s)$, then the greedy action choice from state $s_t$ is $a_t = \text{argmax}_{a \in \mathcal{A}(s_t)} Q(s_t, a)$. All you need to do is implement $\mathcal{A}(s)$ in code and loop over different calls to Q(s,a) to find the best value.

If you have implemented the Q function in parallel over the whole action space (a good choice for efficiency with neural networks), then the network takes only the state $s$ as input, and outputs a vector e.g. $[Q(s, a_0), Q(s, a_1), Q(s, a_2), Q(s, a_4), Q(s, a_5) ... Q(s, a_N)]$. In that case you would use your implementation of $\mathcal{A}(s)$ to mask out unwanted actions before choosing the best action. It is a little bit wasteful to calculate the non-required actions, but if it was a rare enough exception, then probably still more efficient overall.

Beyond Q-learning, you could also use a similar action mask when you had a policy network that output softmax probabilities of actions. In that case you would also want to re-normalise the probabilities before generating the selected action.

Unless you have a specific goal or benefit in mind for the agent to learn about actions that are blocked by the environment, then it is simplest to query the environment to support the blocking, as described above. This is definitely the case for game-playing bots covering moves which are illegal according to the game rules. The justification is that the agent is learning to play the game, and the goal of it learning what the rules are for the game is a less interesting distraction.

Alternatively, it should work to allow the incorrect action selection, then give a negative reward (can be a small one, in same scale as other rewards in the system). Note this should also increment the time step and return the unaltered state. An agent will eventually learn not to take such a pointless, barred action. Maybe sometimes you are interested in whether the agent can learn the rules.


You definitely have interval data, that is, data which takes on discrete values, as opposed to continuous data, which takes on values along a continuum.

It may be of value to additionally determine if the data is ordinal, meaning that the order of the values is important, for example if [0, 1, 2] signifies [small, medium, large] or some analogous system.

In the case of ordinal data, it may be best to keep the data as exposed to the SVM training process in integer form, as the integer representation encodes some information about the relationship between the categories.

This approach would also be more reasonable if the values that the variable could take on in a production setting could expand beyond the values you've already observed in the training set- a categorical approach would be less able to handle new values in that context.

If there are no ordinal relationships and you suspect all of the possible values are enumerated in the training set,treating the variable as categorical would be approriate.


Determining first name and surname from a name pair?

You can apply a simpler strategy here that will likely lead to better results than a neural network. Luckily we have census data for most countries. For example you can use US census data.

We can find the commonality of the two names in the array using their search function. Then we can say that the name with the higher rank is the first name. For example we can see that for the name ['John', 'Smith'], John has a rank of 27 for the most recent year whereas Smith is not listed. So we can assume that John is the first name and Smith is the last name.

This technique is a very old school means of prediction, we already have the distribution of the data through polls. So we can use it directly, we do not need to train a model to learn the underlining distribution (this would be very difficult with something like names which is has a lot of noise).

If you have additional data for each person like a date of birth or a gender then you will get better results through this prior information. This is similar to what you would achieve when applying Naive Bayes algorithm, however in this instance once again the distribution has already been determined.


Papers presenting results that are worse than random chance

Not all probabilities are 50/50.

I'm assuming that the paper you are looking at is eYOLOv3-Lite: A Lightweight Crack Detection Network.

Under the evaluation metrics section, I see

  For each image, the intersection over union ($I_{oU}$) between the bounding box of the detected crack
  and ground truth can be calculated as: $I_{oU}$ = $\frac{A_o}{A_u}$
  , where $I_{oU}$ is the intersection over union, $A_o$ is the
  area of overlap, and $A_u$ is the area of union.
  When the $I_{oU}$ of the predicted bounding box and ground truth is greater than a certain threshold
  value (e.g., 0.5), it is considered to be a true positive; otherwise, it is a false positive.

It looks like the problem is to assign a bounding box to each image such that the bounding box contains the crack (or the ground truth box associated with the crack). It's not an easy probability to calculate exactly, but it's quite clear that assigning a bounding box completely at random will yield a success much lower than 50%.


Can I load my own "*.h5" file?

Yes, you can load your own .h5 file. But to load weights into a model, you need to have a model architecture defined for it. If the weight dimensions or layer sizes does not match, it will throw an error, of course.

But instead of saving weights, if you save the model by using model.save_model, it does have architecture of the model saved allowing you to re-create the model, weights of the model, training configuration (loss, optimizer), state of the optimizer, allowing you to resume training. But this does not load hyperparameters that are present for that epoch like decay in learning rate etc. You have to write a custom callback for this.

  Can I load my last epoch?

Yes, if specified. Keras does not automatically save the model for every epoch or anything until you ask it to. So if you had interrupted something, you cant resume it from there. You have to write a custom callback to make it happen. For details on what are callbacks. For details on how to save the model after every epoch, check out this answer.


Regression in Predicting Tenancy Lengths

You may have done this already, but if your target values are positive integers, it could be worth transforming your output layer in such a way that constrains it to take an appropriate range of values.

Tenancy length values presumably have some recognisable distribution that might fruitfully be modelled by a known statistical process (Poisson process, survival analysis, etc.), and so instead of using your NN to predict the tenancy length directly, you could use the NN to parametrise a relevant probability distribution, and make a point estimate of that distribution in your output layer.


Related concepts, but not the same.

ROC-receiver operating curve

AUC area under the curve

Thank this post for explanation :

Abbreviations

AUC = Area Under the Curve.
AUROC = Area Under the Receiver Operating Characteristic curve.

AUC is used most of the time to mean AUROC, which is a bad practice since as Marc Claesen pointed out AUC is ambiguous (could be any curve) while AUROC is not.

Interpreting the AUROC

The AUROC has several equivalent interpretations:

The expectation that a uniformly drawn random positive is ranked before a uniformly drawn random negative.
The expected proportion of positives ranked before a uniformly drawn random negative.
The expected true positive rate if the ranking is split just before a uniformly drawn random negative.
The expected proportion of negatives ranked after a uniformly drawn random positive.
The expected false positive rate if the ranking is split just after a uniformly drawn random positive.

Going further: How to derive the probabilistic interpretation of the AUROC?

Computing the AUROC

Assume we have a probabilistic, binary classifier such as logistic regression.

Before presenting the ROC curve (= Receiver Operating Characteristic curve), the concept of confusion matrix must be understood. When we make a binary prediction, there can be 4 types of outcomes:

We predict 0 while the true class is actually 0: this is called a True Negative, i.e. we correctly predict that the class is negative (0). For example, an antivirus did not detect a harmless file as a virus .
We predict 0 while the true class is actually 1: this is called a False Negative, i.e. we incorrectly predict that the class is negative (0). For example, an antivirus failed to detect a virus.
We predict 1 while the true class is actually 0: this is called a False Positive, i.e. we incorrectly predict that the class is positive (1). For example, an antivirus considered a harmless file to be a virus.
We predict 1 while the true class is actually 1: this is called a True Positive, i.e. we correctly predict that the class is positive (1). For example, an antivirus rightfully detected a virus.

To get the confusion matrix, we go over all the predictions made by the model, and count how many times each of those 4 types of outcomes occur:

In this example of a confusion matrix, among the 50 data points that are classified, 45 are correctly classified and the 5 are misclassified.

Since to compare two different models it is often more convenient to have a single metric rather than several ones, we compute two metrics from the confusion matrix, which we will later combine into one:

True positive rate (TPR), aka. sensitivity, hit rate, and recall, which is defined as $ \frac{TP}{TP+FN}$. Intuitively this metric corresponds to the proportion of positive data points that are correctly considered as positive, with respect to all positive data points. In other words, the higher TPR, the fewer positive data points we will miss.
False positive rate (FPR), aka. fall-out, which is defined as $ \frac{FP}{FP+TN}$. Intuitively this metric corresponds to the proportion of negative data points that are mistakenly considered as positive, with respect to all negative data points.  In other words, the higher FPR, the more negative data points will be missclassified.

To combine the FPR and the TPR into one single metric, we first compute the two former metrics with many different threshold (for example $0.00; 0.01, 0.02, \dots, 1.00$) for the logistic regression, then plot them on a single graph, with the FPR values on the abscissa and the TPR values on the ordinate. The resulting curve is called ROC curve, and the metric we consider is the AUC of this curve, which we call AUROC.

The following figure shows the AUROC graphically:

In this figure, the blue area corresponds to the Area Under the curve of the Receiver Operating Characteristic (AUROC). The dashed line in the diagonal we present the ROC curve of a random predictor: it has an AUROC of 0.5. The random predictor is commonly used as a baseline to see whether the model is useful.

If you want to get some first-hand experience:

Python: http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html
MATLAB: http://www.mathworks.com/help/stats/perfcurve.html


Understanding math notation in infoGAN paper

The notation $D_\mathrm{KL}(P \| Q)$ is pretty much standard for the Kullback-Leibler divergence. There is an interesting discussion on Mathematics SE on the reasons for the fairly unusual notation used for divergences.

In general, $x \sim G(z, c)$ means "$x$ is a random variable distributed according to $G(z, c)$". In the subscript the author means to take the expectation with respect to the random variable $x$, supposing that $x$ has that distribution.

If you have a variable, such as $a$, it is not unusual to call a related variable $a'$ (usually pronounced "a prime"). In this case $c'$ is just a random variable distributed according to $P(c\mid x)$, with the name $c'$ probably meant to remind you that the variable is in some way connected to $c$. In cases where derivatives are obviously not the right thing, this is probably what a prime symbol means.


Using local outlier factor score to detect outliers at run time

You should find an appropriate threshold of the LOF score, above which the point will be considered as an outlier. Typically, a point with score below 1 is considered as normal (non-outlier), but the threshold above 1 at which to consider a point as outlier depends on the dataset.

If you have a dataset with outlier points labeled, then you can use that to fine tune the value of the threshold, such that you get an acceptable tradeoff between false-positive rate (out of points labeled as outliers, fraction of normal points), and recall (out of all actual outliers, fraction which are labeled as outliers).

At run time, if new points are getting added to your dataset, then the local densities and LOF scores across points could change, and these should be recomputed. However, the threshold of LOF score that you determined as optimal should continue to apply, and requires revisiting only over a larger time period.


Optimal input setup for character-level text classification RNN

For character-level NLP, one-dimensional convolutions are often used to shrink the number of hidden states to a reasonable number.
For instance, ELMo first tokenizes the input into words and then uses a CNN with max-pooling to obtain character-based word embeddings that are used as an input to an LSTM.
In machine translation, there is an approach that shrinks the character-level input into pseudo-word hidden states. This is a one-dimensional CNN over the character input (here, without segmenting the input into words), the CNN is followed by max-pooling with window step 5 (which is the average word length) and a bunch of highway-layers. These pseudo-word states are then used again as an input to an LSTM.


What's wrong with RF/SVM with word embedding (GloVe)?

Usually these are highly dimensional matrices, and all of the information can not be modeled by less-complex models like RF and SVM.


How can we convert time series data to supervised learning problem?

Since you always have a fixed number of days, I think your approach is good. In order to help the learning algorithm you might consider adding some statistics as features for every instance, for example:

mean of the last N days
difference dayN-day(N-1) (evolution)
...

Of course this can work only if there is actually a dependency between the features and the predicted speed.


Prediction from data series with varying features

If you need to impute each missing value you could consider multiple imputation or interpolation methods for Time Series.  e.g. https://stackoverflow.com/questions/49308530/missing-values-in-time-series-in-python

If the goal is to simply predict the entry time to Stage 10 and not worry about when they hit the intermediate stages, you can treat is a regression problem i.e. just predict time to Stage 10.  The simplest way might be this - just treat it like a normal supervised learning problem with some missing data. The only extra step is you need to create some missingness in your training/test data to match the data you will score.

Your train/test data will consist of all records with the full data up to stage 10.

Use cross-validation or a train/test split within this data to create a predictive model to predict time to Stg10. Within this data randomly blank out data between Stg1 and Stg9 to simulate the scoring dataset but keep Stg10 as you need a label to predict.  Use a tree based method such as Random Forests as they are OK with the dependence between the variables and will deal with missing data.

Use this model to score the unlabelled data (i.e. where there is no Stg10 info).  The remaining problem is the level of completeness within the unlabelled dataset.  If they are filled up to Stg9 you have no problem.  If they all go to Stg1 you have a problem! So you will have to see if you have enough completeness in your data to support this method.

But it's a valid place to start and may turn out to be sufficient for your purpose..


np.loadtxt function showing error, could not convert string to float: 'ï»¿"Date"'

The problem might arise because of the meta-text in the .csv or .txt file that is not really written there but is copied when its content is loaded somewhere.

I think it is better to first import your text in an array or a string and then split it and save into the dataframe specifically when your data is not too large.

import csv
arrays = []
path = "C:\\Users\\Souro\\Downloads\\AXISBANK.csv"
with open(path, 'r') as f:
   reader = csv.reader(f)
   for row in reader:
       row = str(row).replace('\\', '') #deleting backslash
       arrays.append(row)

Then take a look at arrays[:10] to find where the meta data ends and delete the unwanted data (meta data)
and then converting the 'arrays' array into the dataframe.
for instance:

arrays = arrays[9:]
df = pd.DataFrame(arrays[1:], columns=arrays[0]) #arrays[0] is the columns names

about your comments:

if you look at the text in each row (print each row), you would find out that a backslash is at the end of each row, so by replace('\',' ') we are substituting each backslash with nothing(''). why two \? It is the way that we declare backslash, otherwise, it won't be recognized.

row=str(row).replace('\\',' ')

and about

open('text.txt','r')

It opens the file 'text.txt' in reading mode (r).


When should I use Gini Impurity as opposed to Information Gain (Entropy)?

Gini impurity and Information Gain Entropy are pretty much the same. And people do use the values interchangeably. Below are the formulae of both:

$\textit{Gini}: \mathit{Gini}(E) = 1 - \sum_{j=1}^{c}p_j^2$
$\textit{Entropy}: H(E) = -\sum_{j=1}^{c}p_j\log p_j$

Given a choice, I would use the Gini impurity, as it doesn't require me to compute logarithmic functions, which are computationally intensive. The closed form of it's solution can also be found.

  Which metric is better to use in different scenarios while using
  decision trees ?

The Gini impurity, for reasons stated above.

So, they are pretty much same when it comes to CART analytics.

Helpful reference for computational comparison of the two methods


How the vector of weights is assigned to a Neural Network

Yes, I am asking about Backpropagation algorithm. I want to understand how the weights get assigned to the neural network. If we want to assign a weight matrix to the neural network, how will we assign it?


Graphically Speaking - how weight vector is perpendicular to hyperplane

I think if you play with the scales of the axes and the viewing angle, you may find it is perpendicular.

Especially, try to rotate the view that you look sideways to the plane. You will see the perpendicular vector (regardless of scale of the other axes).

Any random view angle perpendicular to the weight vector should work.


Machine learning - Algorithm suggestion for my problem using NLP

There is a really good video about this topic from a PyCon in 2016. There is a pretty in-depth description on how to vectorize your sentences as well as make predictions based on those vectors.

I think that would greatly help you out. That is what I used when I was learning about how to perform sentiment analysis.


How can I encode a 'Name' so that similar names are represented by vectors close in n-dimensional plane?

You can do that. I propose the simplest one conditioned on the fact that number of data is not very large. In case you need more ideas please drop a comment.
In this case, you can use the idea of similarity encoding based on Fuzzy String Matching and get the spectral embedding. The amount of data is crucial here as you need to do order of $n^2$ comparisons to get affinity matrix for spectral embedding. Follow the code bellow (and get a wonderful idea from this paper)
data = ['sarah connor', 'sara jones', 'jack blabla', 'jackie jones', ' jakob blabla', 'sara conor']
n = len(data)

aff_mat = np.zeros((n,n)) # This is the S matrix in the paper
D = np.zeros((n,n))

for ii in range(n):
    for jj in range(n):
        name1 = data[ii]
        name2 = data[jj]

        surname1 = name1.split()[0]
        lastname1 = name1.split()[1]

        surname2 = name2.split()[0]
        lastname2 = name2.split()[1]

        aff_name1_name2 = fuzz.ratio(surname1,surname2) + fuzz.ratio(lastname1,lastname2)
        # Fuzz ratios are betweein 0 and 100 and we add 2 of them
        # so we normalize the whole score to 0 and 1 by dividing by 200
        aff_mat[ii,jj] = aff_name1_name2/200

for ii in range(n):
    D[ii,ii] = np.sum(aff_mat[:,ii])

L = D - aff_mat # This is Laplacian matrix

Having the Laplacian matrix, you simply calculate the eigenvectors, directly from the code inside the paper. Here I choose second and third eigenvector as the forst eigenvector is trivial. please not that there are tones of ways to calculate Laplacian matrix and what we did here is different that the one in the paper. So despite the paper which chooses first $k$ eigenvectors, we drop the first one. For more details on this you may refer to the literature.
# compute eigenvectors / eigenvalues of L
evals, evcts = eig(L)
# extract "smallest" 2 eigenvectors (ignoring first one)
sortedevals = argsort(evals)
U = evcts[:,sortedevals[1:3]]

Now U is your embedding in 2 dimensions. Just plot and see the result:
for (x,y), label in zip(U, data):
    plt.text(x, y, label, ha='center', size=10)
plt.xlim((-1,1))
plt.ylim((-1,1))
plt.show()

and this is the result:

Now it's up to you how you would like to query similar names. The main job is done.
PS: As obvious above, I assumed you are interested in similarity of both first names and last names. In case you want the same code only for first name just simply take the $lastname$ variable out.
Hope it helped. Good Luck!


Reinforcement Learning in NLP for chatbots

That's because using NN for chatbots have proven super challenging. Basically once you type a query the NN (whether its NN or RL) has to tag it to a particular intent based on which u give a templatized response ( generating a human like response goes into even more complex territory of natural language generation)

The issue with NN like LSTM , GRU etc is that they don't understand "turns" in a conversation. So if u type in a query one after the other the bot will treat them as separate sentences and try and tag them individually. But a human would form a context based on the whole conversation and can handle turns. There's some labelled data for a few verticals that shows turns in the conversation but it's just not enough

Coming to RL , it seems like a good fit for handling conversation since the current state (sentence) totally depends on the prior state. But what kind of reward function do you have in mind ? How, based on the action of the agent, will the network learn a policy to identify a "conversation"? When will u stop the episode ? When a conversation ends ? Or a fixed number?  Me thinks these are still hard problems to solve since convergence is a challenge in such use cases

If someone has cracked it, they most likely will not publish anything since a true researcher will only publish stuff that is reproducible or they are just being secretive :)


How to decrease the gap between training and validation error?

As Learning is a mess points out, more information would help make useful suggestions. What is the size of the model (in terms of the trainable parameters)? Also, how much training data do you have?
However, you can try these:
a. Dropouts - this a regularization method, it is a kind of proxy for ensemble learning. Not sure if you are using this.
b. Smaller Batches: 8/16/32 - Empirically it has been found that smaller batches find better minimums in the sense of generalization on the test/validation sets. See https://arxiv.org/pdf/1609.04836.pdf.
c. Holding the number of parameters the same, deeper models (more layers) have been found to generalize better. See https://arxiv.org/abs/1706.10239.


python linear search code, what’s a better example if a linear search Algorithm?

A few issues with the code above:

Misspelling of variable name in line 1 - linerar_search instead of linear_search
None is the reserved keyword in python, not none. Trying to return none would throw a NameError: name 'none' is not defined
The indent for function verify is off but it shouldn't cause errors.

The code below runs for me:

def linear_search(list, target):
    """ Returns the index position of the target if found, else returns None"""
    for i in range(0, len(list)):
        if list[i] == target:
            return i
        return None

def verify(index):
    if index is not None:
        print("Target found at index: ", index)
    else:
        print("Target not found in list")

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

result = linear_search(numbers, 12)
verify(result)

Hope this helps! :)

P.S: This seems more so like a python problem then data science. Perhaps you could have posted this question in Stackoverflow instead.


Answer:
Since the difference can't be seen by the human eye, you may need to try teaching an algorithm the difference.
You might try two approaches:

Train 1 model per cable defining the initial state implicitly

Train a model with a triplet loss to define an anchor in the initial state.

Observation:
I strongly recommend to keep the images as raw files or use compression without loss for this kind of image treatment as that will avoid adding artifacts that will reduce your precision. If your data acquisition system cannot sustain the load of transferring raw image files, I would recommend using PNG or jpeg2000(used by NASA to compress images, way more efficient and the transformation used as base is used as inspiration for multiple object detection algorithms)


How does imblearn apply the transformations during prediction?

You generally shouldn't apply resampling to the test set (although there are some differing opinions on whether to do so on various levels of validation data).  imblearn has its own version of the pipeline to accomplish this; in particular, the pipeline docs say:

  The samplers are only applied during fit.


As you have phrased it the answer is no.

Models typically have a fixed input and output size (except RNNs, which are not really used for images).
This means that you can not input smaller images to the network.
If it appears that you are doing so it is either because the network is automatically resizing or padding your small image up to the model input size.
In these cases, the time performance of the model will be fairly constant.

If you were to train the same model with a smaller image input size, you would be correct in saying that inference (and train) time would be faster than the larger image input size model.
You could also load the pre-trained weights from the larger model into the smaller model, in most cases.

Note that what you are considering will change the input resolution and as such the features will change, decreasing the loss-performance of the model.


How to measure deviance resulting from different random seeds in machine learning?

Unfortunately it actually can be substantial. This nice article goes into depth about it and this question shows some clear impact as well.
So depending on the model performance, the model / algorithm used and especially the distribution of your data set you can expect the random seed to influence your results almost as much as optimizing any other parameter (~2-4% points in both examples).
You can minimize this by reducing imbalance in your data sets (e.g. resampling train and test data would have similar effects) or simply fixing your random seed and never touching it.
Now that you are where you are there is not a lot you can do. Maybe pointing at the two sources I have will let you get away with a fixed "interval" or maybe you simply use your last result.


Best way to model a problem where multiple rows contribute to the result

You could try the following.

Filter the results the way you describe; for all the valid options of X in the row assign equal probabilities
Apply the law of total probability. I guess in this case it's just summing all the probabilities for all possible X and divide by number of rows.

I.e. in you case

for 30 it's either 15, 10, 6 or 5 with probability 0.25 each
for both 10 it's 10, 5 or 2 with prob 0.33
for 15 it's 15, 5, 3 with prob 0.33
for 5 it's 5 or 1 with prob 0.5

That's mean in total Y is 5 with probability ~0.35 or 10 with prob ~0.18
If the X is integer, then some level of noise in Y should not be that big of a deal. Should be solved by small allowance of discrepancy on filter stage.
Slowly changing is trickier, but you probably could just make window smoothing, like say the probability of 10 is the sum of probs for 9, 10, 11.
I would not bother with estimation by number of rows - you could work out something based on CLT, but in fact the implicit sampling on step 2 already takes this into account.


training model on random samples from a large dataset

Depends entirely on your data, if your variables are mostly numerical then you can get by with small samples. If however you have a lot of categorical variables you need to make sure that each category of every variable is well represented in the subsample. If they are all numerical I would go for 1000 data points repeated 1000 times.


Question about BERT embeddings with high cosine similarity

It's really hard to say if the BERT assigns similar embeddings to the same words in the similar context. Most likely it will not. This is because, the embeddings, is not just function of context (other words in the sentence), but the position as well. Hence, even if the words occurs in similar context, with different positions, they can have different embeddings.


PCA: projection of positive data on negative side of plane

Yes, it is, there is no reason why a component cannot have negative values.

Let's take this matrix:

[[1, 1],[0.5, 1]]

The eigenvectors have negative values:

[[ 0.81649658, -0.81649658],
    [ 0.57735027,  0.57735027]]


unimportant features impact on model's performance

Yes, unimportant features can hurt the model's performance. This happens in my experience in a few ways:

Efficiency - they make the fitting process slower. Particularly if you're one-hot encoding categorical features, and end up with a large and useless sparse matrix. If these reductions in efficiency force you into taking steps that might impact on actually informative features, like a PCA transformation, you're then potentially also directly impacting on predictive ability.
A minimum number of samples is needed per feature for models to effectively learn. By including redundant features, this sample:feature ratio is diminished, which makes it more difficult for many models to determine whether or not a feature is a useful predictor.
You make it more likely to overfit to the "noisy" features.

You can recursively reject features that the model's feature_importances_ routine has decided are unimportant using sklearns recursive feature elimination. Or, in the exploratory phase of building your model, you can assess predictive power using visualisation or hypothesis testing.

This nicely highlights the improvement in performance by ditching redundant features:

https://scikit-learn.org/stable/auto_examples/feature_selection/plot_rfe_with_cross_validation.html#sphx-glr-auto-examples-feature-selection-plot-rfe-with-cross-validation-py


What is the interpretation for quadratic functions?

$Displacement$ component gives us a "line" to fit over the data points. To get more freedom add $Displacement^{2}$ a "curve" element. This adds to the flexibility - with same feature/ variable - to map the data points.
Please refer page 90, 91 on Introduction to Statistical Learning in R - Hastie, Tibshirani


In essence you want to shift your outcome variable by 1, that is place december's outcome to november. In doing so you are saying that the values in the current month (november) caused the next month's (december) outcome. As to the question of processing your data that depends entirely on your environment / software used for the analysis, however they seem good as they are now.


How to use text classification where the training source are txt files in categorized folders?

Assuming that your folders are your classes, you can match any document with the correspondent tag.
Then, for every document:
1.- Normalize the text, i.e. remove stop words (unless they make sense), stem and / or lemmatize it (unless it doesn't makes sense).
2.- Vectorize the documents, you can choose TFIDF, BOW, word embeddings etc
3.- Depending on your documents train with an MLP (in case of BOW) or an LSTM in case of words embeddings.

When you have a new document, you need to repeat the procedure using the vocabulary you created for the training set.

I had a similar use case and was enough to use BOW with a multi layer perceptron, the accuracy was above 95%, but, documents were different for each category and I removed most frequent words because there where to common.

Another solution is to perform a topic modeling on documents binding those topics to the category and then training a simple classifier (an MLP or SVM will work)


How to design two different neural nets for actor and critic RL?

There are many papers out there that deal with neural networks and RL. This blog will give a very good insight on a Policy Gradient Network: Deep RL with PG

Now for your question. You really need to be familiar on how we train a neural network. A simple one for classification. If you check the derivations and how the weights are getting updated things will be very clear to you on how you can implement the above.

I will describe it to you as simple as possible so you get the link. A Neural Network in a very broad sense consists of nested functions. The function that contains all the others is the one at your output layer. In case of Stochastic Policy Gradients this is your Boltzmann function. So your output layer will take all the previous layer's outputs and will pass them through the Boltzmann function. In the case of NN the parametrization comes from all previous layers.

The blog link I sent you describes a very nice and simple example of a vanilla Policy Gradient with NN (REINFORCE algorithm). By using the code (plus your understanding on feedforward networks) you will see that the gradients are multiplied by the reward. It is a very good exercise!

For Actor-Critic, you need in general a network performing PG (stochastic or deterministic) -- you Actor -- and a network that will give you the reward signal (like the simple case in the blog. However, for various reasons, instead of the actual reward we use another network that estimates the reward by performing Q-learning as in Deep-Q learning (minimizing square error between estimated reward and true reward).

Hope this helps!


model.cuda() by default will send your model to the "current device", which can be set with torch.cuda.set_device(device).

An alternative way to send the model to a specific device is model.to(torch.device('cuda:0')).

This, of course, is subject to the device visibility specified in the environment variable CUDA_VISIBLE_DEVICES.

You can check GPU usage with nvidia-smi. Also, nvtop is very nice for this.

The standard way in PyTorch to train a model in multiple GPUs is to use nn.DataParallel which copies the model to the GPUs and during training splits the batch among them and combines the individual outputs.


Problem trying to build my own sklean transformer

The definition of dic should happen inside the fit method; you're wanting the rates to be based on the dataframe passed to fit.  So you'll want to set dic as another class attribute, to be referenced in the transform method.

(This kind of replacement is called target encoding, and there's an existing package to do it in sklearn.)


Are Undergraduate Statistics Concepts Used in Practice?

t-test and ANOVA are used pretty often, more in statistical data analysis which is a "must know" for a data scientist but not necessarily their everyday work. More you go toward medical/bio statistics or social sciences, you see them more used.

In everyday life of a data scientist, the problem of feature selection, for example, is one of situations where ANOVA helps. For instance imagine numerical features and discrete classes in a classification problem. A way to select good features is to compare the distributions of features values for each class and see if those distributions differ significantly.

I also have used Wilcoxon test once in a low-sample size project where I was trying to recognize patterns in Complex Networks, created from time-series of two control groups. The story was to see which features of those networks (centrality measures, clustering coefficient, average path length, etc.) makes the significant difference (so can be used for recognition i.e. classification problem).

This difference was the difference between distribution of feature values among those two groups where those distributions were not necessarily Gaussian. Wilcoxon helped a lot there, as some bold difference between two groups for certain features were actually statistically not significant and without such test the analysis could go the wrong way.

Hope it helped!


Connecting Authors with Published Papers

It's not an easy question to answer even for groups with a lot of leverage such as Research Gate (RG). RG has it's own (I assume proprietary) author matching algorithm which has caused problems in the past.

They use the name of the author (in different combinations) to suggest authorship to RG users (so has you said, it does causes a lot of problems). Every once in a while users that are not the author accept the suggestions and, from the portal point of view, gain the equivalent reputation. It's a serious business that requires quite a bit of R&D before making a decision.

That being said I can't answer with certainties only with reasonable possibilities. A few questions I would make (and hope to answer with a bit of data analysis):

What is the probability that an author with a confirmed publication in an identified journal will publish in that journal again? - Journal Name
What is the probability that an author that has partnered with other authors will repeat the same co-authoring combination? Co-author Names
What is the expected time story for publishing for each author? Authors rarely publish articles 20 years apart. Typically they publish more and more, or less and less. Time frame
How frequently do authors change the institution they belong to? Institution Name
What are the preferred keywords for a given author? Key Names
What are the preferred citations made by a given author? Bibliography

All of the questions above require quite a bit of Text Mining and String Matching, as well as solid dataset to start your mining.

Some publishers have their own API although I can't say much about permissiveness (never tried it myself). RG has been promising one for years but, as far as I know, it still does not exist.

An unlikely thing I remember now is the inspiring story of Aaron Swartz. This activist, along with other persons, successfully managed to create large open archives for books and articles. Should that information still exist it might be worth your time to take a look there.

Also if you have a list of the journals you are considering (you've mentioned only "doctors" which is a bit vague) you can try and see with the publisher if they have any way of accessing their database.


AttributeError: 'Functional' object has no attribute 'uses_learning_phase'

Could you please share some snippets of your code where the error arises? Generally attribute errors are caused when documentations are updated and that specific attribute is no longer present for the version! Importing new libraries (which you have tried already) or lowering the library version can help but for a definitive answer the code snippet is required!
P.S Sorry for posting this as an answer, I can't comment yet haha.


Amazon Cloud Image istance most suited for R data mining

It very much depends on the calculations you are doing as well as the tools you are using to implement them and the size of data you are working with.

Some small rules of thumb:

R processes (generally) tend towards being RAM bound, so you want may want to get a RAM optimised
R Studio IDE has a profiling tool you can use to check how your code executes and where the time is spent
It may be easier and cheaper to optimise code before scaling with tools like data.table and RCPP

ML models are likely not going to be effected by this, more your data prep, check the profiler

If you are REALLY keen on getting the model running fast, look into the GPU/TPU instances, however, in the case of R using caret, I don't know if it will utilise these assets. Do your research first as these are the most expensive flavour of EC2. TPU units are (from my limited research) specifically optimised to tensorflow ML.

A plan B/alternative would be to work in aws Sagemaker notebooks to start. Then you can abstract away all the faff of managing the EC2 and just focus on building the ML.

For extra credit on your EC2, use one of the R community AMI's put together by this lovely person: http://www.louisaslett.com/RStudio_AMI/


Relationship between log-odds and weighted sums in Logistic Regression

Ordinary regression, Logistic regression, Poisson regression are all examples of Generalized Linear Models (GLM).

A GLM can be broken down into 3 components:

A random component: Target Variable $Y_i$ which is assumed to follow a distribution with $E(Y_i)$ and $Var(Y_i)$
A systematic component: A linear combination of the features variables known as the linear predictor; $\gamma_i= \beta_{0} + \beta_1{x_1} \dots + \beta_n{x_n}$
Link Component: The link function, $g(\cdot)$, connects the random component to the systematic component. If $E(Y_i) = u_{i}$, then the link function connects $\mu_{i}$ to $\gamma_{i}$ through some function $g(\cdot)$, and we then have that
$g(\mu_i)= \beta_{0} + \beta_1{x_1} \dots + \beta_n{x_n}$. Moreover, $E(Y_i) = \mu_i = g^{-1}(\beta_{0} + \beta_1{x_1} \dots + \beta_n{x_n})$

The link function for linear regression is the identify link: $g(\mu_i) = \mu_i$

The link function for logistic regression is the logit link: $g(\mu) = log(\frac{\mu}{1-\mu})$. The logit link is used when $\mu$ takes values between 0 and 1 since the function maps to -infinity to infinity.

There are a lot details which I have skimmed over on purpose, GLM is a rich subject. I would recommend reading more about it, here are some references below:

References:

Categorical Data Analysis by Alan Agresti

An Introduction to Categorical Data Analysis

Hope this helps!


Pretrained models for Propositional logic

As far as I know, automated reasoning with propositional logic can be done with a solver like Prolog (and it's not new). I don't know if it has been done but I don't think it would make sense to train a ML model for propositional logic, since it's entirely symbolic (as opposed to statistical): the right answer can be found deterministically.
The question of logic reasoning from text like in the proposed example is a bit different, because it involves a step of representing the text. I think it could make sense to train a model which converts a text into a formal logic proposition (and back). The logic reasoning should still be done with a tool meant specifically for that imho. Note that question answering doesn't involve any logical reasoning, even if it might look this way to a user. As far as I know a QA system learns patterns about matching a type of question to its corresponding answer, the system is completely oblivious to the meaning.


Which Classification Metrics Are Appropriate For Each Class Distribution Scenario?

First of all don't change the distribution of your data. Your classifier won't perform good at test time if the real data is not balanced.

If you have approximately equal amount of data for each class and they have same importance, accuracy is a good metric which can help you understand how much your classifier has performed well without regarding to the number of labels, binary or multi-class classificaitons.

Binary classification
There is no difference between 0 or 1 because you yourself choose to assign each class to a label but it is common to set the rare and maybe, in some occasions, the desired output to 1.

In cases which data is distributed badly and it is so much skewed it recommended to use F1 score. For last, imagine a 75/25 ratio for either I personally prefer to use F1 because it is unbalanced.

Also in cases where one class is much important than the other class and maybe the data is skewed, it is better to use recall, although you have to change the cost function to emphasize the importance of one class over the other. As an example, suppose that you want to know whether a patient has a bad illness or not. In such cases you try to predict correctly all of those who has that illness because if you tell that someone has an illness but you're wrong, it will have less harm than saying a patient is healthy. So in these cases recall is so much better than precision.

Multi-Class classification

  Majority of samples are dominated by one class (very common in my experience)

In this case first of all, if the data is so much skewed, I suggest you using anomaly detection. But if you have more than few data, you can again use F1 score.

  Classes have roughly equally the same amount

In this case you can use accuracy if the classes have the same importance. If they don't, again use accuracy but change the cost function to emphasize on the important class.

  Many class labels with few samples in each (imagine 1000 samples with 100 classes ranging from 2 - 20 samples in each)

In this case I don't think you will have a good learning. Anyway, you have to use confusion matrix in this for tracking all classes. Although I guess your model will be over-fitted because of the amount of provided data.

Consider that in all cases you can use confusion matrix.

Finally I suggest you taking a look at here and here.


Why does averaging over vector values cause errors?

Change this:

import bumpy as np
a = np.array([1,2,3])
b = np.array([4,5,6])
vec_line = [a,b]

print(np.mean(vec_line, axis=0, keepdims=True)[:,None])
>>[[[2.5 3.5 4.5]]]
np.mean(vec_line, axis=0, keepdims=True)[:,None].shape
>>(1,1,3)

To this:

print(np.mean(vec_line, axis=0))
>>[2.5 3.5 4.5]
np.mean(vec_line, axis=0).shape
>>(3,)


pandas: How to impute the categorical column by the nearest neighbors?

The following script will give the value of the most frequent item to the nan value. It is a list of 7 items, since it checks the three samples before the nan, the nan itself and the three after the nan samples.

iput = ['s','s','s','s','s','s',np.nan,'s',np.nan,'s','s','s','s','r',np.nan,np.nan,'r','r','r','r','s','s','s','s','s',np.nan,np.nan,'s','s','s','l','l','l','l','l',np.nan,'l','l','l']
for i in range(len(iput)):
    if type(iput[i]) is float:
        iput[i]=max(iput[i-3:i+3],key=iput[i-3:i+3].count)


A simple and direct answer is that skewness and kurtosis are both defined in terms of the $Z-$values, $Z = (X-\mu)/\sigma$ : Skewness = $E(Z^3)$ and Kurtosis = $E(Z^4)$. When talking about a data set, you can just replace the expectation operator "$E$" with the ordinary average.

Since raising a number to either the third or fourth power amplifies values that are greater than 1 (in absolute value) and diminishes values that are less than 1 (in absolute value), both skewness and kurtosis are primarily measures of the heaviness of the tails of the distribution of $X$.

Note that the $Z$-values than are less than one in absolute value correspond to $X$-values that are less than one standard deviation from the mean; greater than one correspond to values that are more than one standard deviation from the mean. Hence, the averages of the $Z^3$ and $Z^4$ values are highly influenced by the $X$-values that are many standard deviations from the mean; i.e., by the tails or outliers.

Consider skewness first.  By definition of skewness, and by definition of center of gravity, the graph of the probability distribution of $Z^3$ balances (i.e., has a center of gravity) at the skewness of $X$. Hence, if the tails are heavier on the left side of the distribution than on the right, then skewness is negative. Conversely,  if the tails are heavier on the right side of the distribution than on the left, then skewness is positive.

Now consider kurtosis. The graph of the probability distribution of $Z^4$ balances (i.e., has a center of gravity) at the kurtosis of $X$. Note that the kurtosis of the normal distribution is exactly 3.0. Hence, if the tails are heavier on either or both sides of the distribution than the tails of the normal distribution, then the kurtosis is greater than 3.0, and the probability distribution of $Z^4$ falls to the right when a fulcrum is placed at 3.0. Conversely, if the tails are lighter on either or both sides of the distribution than the tails of the normal distribution, then the kurtosis is less than 3.0, and the probability distribution of $Z^4$ falls to the left when a fulcrum is placed at 3.0

Hence, skewness and kurtosis really do not measure things that differently - they are both measures of tail heaviness. The biggest difference is that skewness considers relative heaviness in one tail versus the other. The Wikipedia entry on kurtosis displays both lower and upper bounds for kurtosis in terms of skewness.

There is a mistaken but remarkably persistent meme out there that kurtosis measures "peakedness or flatness" that might be a source of confusion.  Your work shows (yet again) that this characterization is false. See  here for a clear explanation of why the "peakedness/flatness" characterization is wrong.


How to make a Python code execute on GPU and not CPU

You may apply Wolfram Language to your project. There is a free Wolfram Engine for developers and with the Wolfram Client Library for Python you can use these functions in Python.

Either the CUDALink Overview and CUDALink Guide or the OpenCLLink Overview and OpenCLLink Guide will enable you to run code on your GPU.  However, the CUDALink guide image process functions built-in so perhaps best to start there for your project.

from wolframclient.evaluation import WolframLanguageSession
from wolframclient.language import wl, wlexpr

Start the Wolfram session.

wolfSession = WolframLanguageSession()

The CUDA package is included with the Wolfram Engine but has to be explicitly loaded.

wolfSession.evaluate(wl.Needs('CUDALink`'))

Then you can check if the engine has located one or more compatible GPUs with CUDAQ.

print(wolfSession.evaluate(wl.CUDALink.CUDAQ()))

True

There is a one-time download of CUDA binaries when CUDAQ is evaluated. You can make your first call directly in the kernel to track the one-time download instead of first calling through Python. Run woframscript in the terminal and evaluate the Needs and CUDAQ functions at the input prompt.

Detailed information can be returned with CUDAInformation.

print(wolfSession.evaluate(wl.CUDALink.CUDAInformation(1,'Core Count')))

160

Grab an image for some CUDA image function examples.

imgWolfram=wolfSession.evaluate(
    wl.Rasterize(
        wl.Import('https://i2.wp.com/oemscorp.com/wp-content/uploads/2015/07/wolfram-language-logo.png'),
        'Image'
    )
);

The image can be Exported in one of the supported Raster Image Formats or Vector Graphics Formats.

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        imgWolfram
        )
    )
)

A few example CUDA image functions include

CUDAImageConvolve

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        wl.CUDALink.CUDAImageConvolve(imgWolfram,  [[-1,0,1],[-2,0,2],[-1,0,1]])
    )
)

CUDAErosion

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        wl.CUDALink.CUDAErosion(imgWolfram,6)
    )
)

CUDAOpening

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>',
        wl.CUDALink.CUDAOpening(imgWolfram,6)
    )
)

Terminate the Wolfram session.

wolfSession.terminate()

Since you have processing many image files you would get slightly better performance by processing directly in wolfram script with the Wolfram Engine rather than by calling the engine from Python.

Hope this helps.


what is the complexity of a bidirectional recurrent neural network?

The computational complexity of simple single-layer recurrent networks, either vanilla RNNs, LSTMs or GRUs is linear with the length of the input sequence, both at training time and inference time, so $O(n)$, where $n$ is the length of the input sequence. This is because in order to get the last time step output, you need to compute all the previous ones.
This is assuming that there is a single output. If there are multiple output time steps, then it is linear on the sum of both input and output lengths.
Take into account that, inside LSTMs and GRUS there are internal steps that account for a multiplication by a constant in the complexity.
You can complicate the network architecture in many different ways (more layers, skip connections, etc), and this can affect its computational complexity. Here you can find an in-depth study of the computational complexity of different architectural variations.


Publicly available social network datasets/APIs

A couple of words about social networks APIs. About a year ago I wrote a review of popular social networks’ APIs for researchers. Unfortunately, it is in Russian. Here is a summary:

Twitter (https://dev.twitter.com/docs/api/1.1)

almost all data about tweets/texts and users is available;
lack of sociodemographic data;
great streaming API: useful for real time text processing;
a lot of wrappers for programing languages;
getting network structure (connections) is possible, but time-expensive (1 request per 1 minute).

Facebook (https://developers.facebook.com/docs/reference/api/)

rate limits: about 1 request per second;
well documented, sandbox present;
FQL (SQL-like) and «regular Rest» Graph API;
friendship data and sociodemographic features present;
a lot of data is beyond event horizon: only friends' and friends' of friends data is more or less complete, almost nothing could be investigated about random user;
some strange API bugs, and looks like nobody cares about it (e.g., some features available through FQL, but not through Graph API synonym).

Instagram (http://instagram.com/developer/)

rate limits: 5000 requests per hour;
real-time API (like Streaming API for Twitter, but with photos) - connection to it is a little bit tricky: callbacks are used;
lack of sociodemographic data;
photos, filters data available;
unexpected imperfections (e.g., it’s possible to collect only 150 comments to post/photo).

Foursquare (https://developer.foursquare.com/overview/)

rate limits: 5000 requests per hour;
kingdom of geosocial data :)
quite closed from researches because of privacy issues. To collect checkins data one need to build composite parser working with 4sq, bit.ly, and twitter APIs at once;
again: lack of sociodemographic data.

Google+ (https://developers.google.com/+/api/latest/)

about 5 requests per second (try to verify);
main methods: activities and people;
like on Facebook, a lot of personal data for random user is hidden;
lack of user connections data.

And out-of-competition: I reviewed social networks for Russian readers, and #1 network here is vk.com. It’s translated to many languages, but popular only in Russia and other CIS countries. API docs link: http://vk.com/dev/. And from my point of view, it’s the best choice for homebrew social media research. At least, in Russia. That’s why:

rate limits: 3 requests per second;
public text and media data available;
sociodemographic data available: for random user availability level is about 60-70%;
connections between users are also available: almost all friendships data for random user is available;
some special methods: e.g., there is a method to get online/offline status for exact user in realtime, and one could build schedule for his audience.


Multi-Output Regression with neural network in Keras

I found some mistakes:

input data must be numpy objects, not pandas
this Network has 6 output nodes, not 2
the number of layers is completely exagerated IMHO
the Flatten() layer at the beginning is not correct
the way you called ReLU's is not correct

This should be enough:

from tf.keras.models import Sequential
from tf.keras.layers import Dense
from tf.keras.activations import relu

model = Sequential([
    tf.keras.layers.Dense(128, activation = relu),

    tf.keras.layers.Dense(128, activation = relu),

    tf.keras.layers.Dense(2, activation = None)
])

Check if the loss works at this point. Alternatively, you need to write your own custom loss function using Keras backend functions.


Commonly you divide the domain up into different accuracy,stability regimes and apply a different approximation within each.
For $|x|<1$ you can use a polynomial approximation, like Chebyshev. For $1<|x|\leq E$, $E$ depending on implementation of the formula, you can use a related formula to the one you described.
For $|x|>E$ you can use $\tanh(x) \approx sign(x).$
More details here from a documented implementation.


XGBOOST (sklearn interface) REGRESSION error

Upgrade your xgboost version. reg:squarederror was added in 0.83 release

(In version 0.82 or below, use reg:linear)

In general, XGBoost support information is at https://discuss.xgboost.ai/